Follow-up to the v2 trajectory post and the noise characterization experiment. v3 adds image-based visual servoing.

The problem with v1

v1 tracked horizontally by steering the car body. Ackermann steering has a minimum turning radius — if the tag moves outside a cone in front of the car, the only recovery is a multi-point turn. The camera was locked forward and the car had to point at what it wanted to see.

IBVS decouples the camera from the chassis. The pan/tilt gimbal tracks the tag in pixel space regardless of where the car is pointing, and the car centering logic works from the gimbal's pan angle rather than raw image error. The camera can follow a target that the car physically can't yet reach.

What v3 adds

IBVS core — pan and tilt are driven by pixel error feedback: eu = tag_x − cx, ev = tag_y − cy. Error is smoothed with an EWMA (alpha=0.8), a 10px deadband prevents hunting at center, and corrections are capped at 2° per frame. The result is a gimbal that follows the tag continuously rather than only when the car is pointed at it.

Four operational modes — ibvs_test (gimbal only, no drive), manual_ibvs (gimbal + WASD), rat_chase (gimbal + autonomous centering + forward drive), and world_ibvs (full v2 dual-tag world frame behavior). The modes let each layer be validated in isolation before combining them.

rat_chase — the autonomous mode shown in the video. Car centering derives a steering angle from the gimbal's current pan angle via a feed-forward gain, then drives forward at a configurable speed and stops at a distance threshold. The car is on a test rig in this clip so the wheels are suspended — this is a hardware-in-the-loop simulation of the drive logic before putting it on the floor.

ibvs_anchor_mode world frame — tag0 alone is now enough to anchor the world frame. First detection seeds T_world_anchor; every subsequent frame derives world → car_base from that anchor and the current tag0 pose. No second tag required. The full URDF renders in RViz2 and the car trajectory publishes live.

Trajectory visualizer — trajviz.py reads the PLY files output by tf_bridge and produces an interactive Plotly HTML with a color gradient over time, cubic spline overlay, and sliders for spline order and smoothing weight.

What the video shows

The car is suspended on a test rig — wheels off the floor — running in rat_chase mode. Pan/tilt hunts briefly at the start while the EWMA settles, then locks onto the tag and tracks it. The gimbal motion looks smooth on the car itself; some snappiness is visible in the camera output feed, which is the per-frame correction still present at the edges of the lazy band. Drive and steering commands are being issued but the wheels aren't in contact with anything.

Next step is putting it on the floor and running it for real.

Bugs worth mentioning

TF never broadcast in ibvs_test/manual_ibvs. tf_pub.on_frame() was only called inside _do_chasing(), which the ibvs_test and manual_ibvs branches never reach — they return early. Pi was detecting the tag correctly but zero messages reached tf_bridge. Fix: added the on_frame() call directly in the early-return branch.

Z filter rejecting all valid frames — twice. The first version checked car_base_pos[2] against a floor threshold; car base is at Z≈0 so every frame failed. Fixed to check camera height instead. Second version: valid camera height readings clustered just below the threshold (0.000–0.054m vs 0.055m cutoff) and still got rejected universally. Root cause was that I was physically lifting the car during testing so Z filtering is inappropriate in that context. Made the filter an opt-in ROS2 param, defaulted off.

Velocity gate blocking hand-carried movement. The jump gate inherited from v2 was set at 10cm — fine for autonomous driving, but every footstep when carrying the car exceeds that. Result: 62 skipped frames in 11 seconds, 2 trajectory points recorded. Added a separate ibvs_max_jump_m param (default 1.0m) for the ibvs_anchor path.

What's next

Put rat_chase on the floor with the wheels down. The steering and drive logic is implemented and confirmed sending commands — it just hasn't chased anything yet under its own power in v3. That's the next session.

References

Post history

• v2 noise characterization experiment
• v2 trajectory post
• v1 tag chaser
• PiCar-X introduction

Hardware / code

• PiCar-X on Amazon
• Git repo

Follow-up to my v2 trajectory post. The RViz trajectory had visible zig-zag jitter even when the robot was stationary. Before deciding what filter to apply, I wanted to actually measure the noise and understand what's driving it.

The problem

The v2 system uses a physically fixed AprilTag (tag1) as a world frame anchor. The Pi detects it each frame, inverts the camera→tag transform to get world→camera, and publishes that as a TF. The zig-zags in the trajectory come from frame-to-frame instability in that pose estimate.

The root cause is AprilTag PnP pose ambiguity — the solver has two valid geometric solutions for a planar tag and flips between them. The flip shows up as a large swing on one axis, typically ±15cm, even with the camera stationary. On top of that, small angular errors get amplified into position noise through the matrix inversion: at ~74cm tag distance, a 5° rotation error becomes ~6.5cm of position noise in world frame.

The question I wanted to answer before touching the filter: how much does tag size actually move the needle?

Method

Added a single_tag_world_mode flag to config so ManualTracker can run with just the world anchor tag in frame — no chase target needed. Camera held stationary, pointed directly at the tag, for ~2–3 minutes per condition. Raw camera-frame poses recorded automatically to JSON. Four conditions: 5cm and 20cm printed tags, each with room lights on and off.

All four plots below share identical axis scales so the distributions are directly comparable.

Results

Condition	σ X	σ Y	σ Z (depth)
5cm — lights off	3.4 cm	0.5 cm	4.5 cm
5cm — lights on	5.1 cm	1.7 cm	3.7 cm
20cm — lights on	2.7 cm	0.4 cm	1.4 cm
20cm — lights off	2.1 cm	1.0 cm	1.7 cm

(Images: 5cm lights off → 5cm lights on → 20cm lights on → 20cm lights off)

What the plots show

Tag size dominates. Going from 5cm to 20cm cuts depth noise by roughly 3x. The distributions tighten and become more unimodal — the PnP flip signature (broad or bimodal histogram on X and Z) is clearly visible in the 5cm sessions and largely absent in the 20cm sessions.

Lighting is secondary. For the 5cm tag, lights-on is actually worse on X (σ 5.1 vs 3.4cm), likely because uncontrolled ambient light causes glare that degrades corner localization on a small tag. For the 20cm tag the lighting effect is small enough that it's not the thing to optimize.

Best condition across all three axes simultaneously: 20cm + lights on (σX=2.7cm, σY=0.4cm, σZ=1.4cm).

What's next

This experiment was groundwork, not a fix. The noise is reduced but still present — 2cm+ std dev on X and Z with a stationary camera is not acceptable for a usable world frame. The next step is a filter, but the right choice (EWMA, velocity gate, Kalman, or some combination) depends on understanding the noise characteristics, which is what this data was for.

Still deciding. Open to suggestions from anyone who's dealt with PnP jitter on planar markers before.

References

Post history

• v2 trajectory post
• v1 tag chaser
• PiCar-X introduction

Hardware / code

• PiCar-X on Amazon
• Git repo

v1 post here if you want the background.

Where v2 is now

The big addition in v2 is a world-frame coordinate system and live trajectory visualization in RViz2. The robot now runs two AprilTags simultaneously: tag0 is the chase target, tag1 is physically fixed to the wall and acts as the world anchor.

A ROS2 node on Ubuntu (tf_bridge) connects to the Pi over WebSocket, ingests raw camera-frame poses, and computes a floor-anchored world frame on first tag1 detection. World origin is the floor point directly below the camera. From there it publishes /trajectory/car and /trajectory/tag0 as LINE_STRIP markers to RViz2, and writes per-cycle PLY point clouds for inspection in MeshLab.

The URDF visualization is also live — a picar_description ROS2 package provides a tracking URDF anchored to the camera TF frame, so the robot mesh follows its world-frame position in RViz2 in real time.

Manual Track mode lets me drive with WASD while tag detection and TF publishing run in the background, which is what the video shows.

What the video shows

Split view: dashboard on the right, RViz2 on the left. I'm driving forward with keyboard controls. You can see the trajectory building in RViz2 as the robot moves — and you can also clearly see the problem: the path is a zig-zag even when the motion is roughly straight. That's not wheel slip or steering noise. That's measurement noise from the AprilTag pose solver, visualized honestly.

The jitter problem

The zig-zags are real and understood. Root cause is AprilTag PnP pose ambiguity — the solver has two valid solutions for a planar tag and flips between them frame to frame. One axis swings ±15cm per frame while the robot is stationary. On top of that, a small angular error in the tag1 pose gets amplified into position noise in world frame: at ~74cm tag distance, a 5° rotation error becomes ~6.5cm of position error. Every raw frame goes straight to TF with no filtering, so one bad frame is a spike on the trajectory.

What's next

Two things need fixing before the trajectory is useful:

Fix the world frame geometry. The current floor-anchoring logic and world frame initialization are approximate. Tag1 needs to be treated more carefully — its pose relative to the world origin needs to be stable across the session, not just initialized once and held.

Add a noise filter. An EWMA filter with a velocity gate in _process_frame would reject the frame-to-frame pose flips without introducing lag on real motion. This was prototyped and tested during the v2 session but pulled out to keep the debrief clean — it's the next thing going in.

Once those two are solid the trajectory should be smooth enough to actually reason about where the robot has been.

Stack: Raspberry Pi 4B · PiCar-X v2.0 · Picamera2 · pupil-apriltags · FastAPI · ROS2 Humble · Python 3.13

References

Post history

• v1 tag chaser
• PiCar-X introduction

Hardware / code

• PiCar-X on Amazon
• Git repo

Been working on a PiCar-X build on a Raspberry Pi 4B. v1 goal: detect an AprilTag (36h11 family, ID 0), steer toward it with a PID controller, drive forward, and stop at a configured distance threshold. Toggle it on from a browser dashboard, 3-second countdown, and it goes.

I built this entirely with Claude Code. It’s been a massive productivity boost while balancing a full-time job, and the process of building agentically has been a great learning experience.

WebSocket concurrent send corruption

The broadcast coroutine and the sensor push loop were both calling send_json() concurrently. At await boundaries they interleaved, Starlette threw, and the client was silently dropped from the send set — meaning the toggle-off confirmation never arrived and the button stayed stuck in active state even after the car stopped.

Fixed by replacing the shared client set with a per-connection asyncio.Queue and a single drain task per connection.

Camera color inversion that didn't respond to the obvious fixes

BGR888 didn't fix it. RGB888 + cvtColor didn't fix it either.

Root cause: capture_array() on this Pi hardware returns RGB regardless of the format name, and this platform's libjpeg encodes from RGB input correctly without any conversion. One-line fix once the actual data layout was confirmed via a frame diagnostic log.

Had to fully remove Vilib

It uses a Picamera2 internal API (allocator) removed in 0.3.36 — crashes on any camera restart after a chase session. Server now owns Picamera2 directly for the full session lifetime.

What's next

v2 candidates on the list: distance-proportional speed, latching stop behavior, camera tilt tracking, and operator override during chase.

Stack: Raspberry Pi 4B · PiCar-X v2.0 · Picamera2 · pupil-apriltags · FastAPI · Python 3.13

I've recently completed the assembly of a SunFounder PiCar-X and am currently running it on a legacy Raspberry Pi B. I have the base movement and motor control working and am currently prepping to get it chasing ArUco/AprilTags this coming week.

I'm looking to evolve this platform into something capable of SLAM and eventually Structure from Motion (SfM). I'd love to get some community advice on the best way to handle these upgrades:

Traction

The stock wheels are quite slippery. Has anyone found direct-fit replacement tires or wheels that offer better grip on smooth indoor surfaces?

Odometry

Since the stock motors lack encoders, my dead reckoning is non-existent. Should I attempt to mount external encoders to these motors, or is it better to swap out the motor/gearbox assembly entirely for something with integrated feedback?

IMU for SLAM

I'm planning to add an accelerometer/gyroscope. Any specific sensors (such as the BNO055 vs. MPU6050) that are currently considered the "gold standard" for stability and ease of integration on a Raspberry Pi?

Computer Vision

The current camera resolution is limiting for SfM. Any recommendations for a higher-resolution CSI or USB camera that fits well within the PiCar's chassis?

ROS 2 / Distributed Computing

A specific question on the software side:

I'm planning to move this platform to ROS 2. Given that I'm working with a legacy Raspberry Pi B, is this a lost cause, or should I keep the Pi as a low-level hardware node and offload the heavy ROS 2 processing, SLAM, and visualization tasks to a more powerful machine on my network?

If a distributed setup is the preferred approach, what does the typical workflow look like? For example:

• Pi handles motor control, sensors, and camera acquisition
• ROS 2 nodes run on a desktop/laptop workstation
• Visualization and mapping performed via RViz on the workstation
• Communication over Wi-Fi using DDS

Is this the recommended architecture, or are there better approaches for a platform like the PiCar-X?

General Advice

Any feedback on the hardware upgrade path, software architecture, or general "gotchas" with this kit would be greatly appreciated.

Thanks in advance!