A closed-loop tracking system split across two machines. One host captures video at 240 Hz and encodes regions of interest; a remote GPU runs inference and sends corrections back over a custom pipelined TCP protocol; a Raspberry Pi turns those corrections into motion. About 20 milliseconds end to end, camera to actuation.
The hard problem is that by the time a frame has been analyzed, the world has moved. The controller keeps a delay line of commands that are already in flight and subtracts them from the measured error, the same idea as a Smith predictor. That cut overshoot from roughly 100 percent to 18.
After sending a command it also has to wait out its own blind window, and it ends that window on evidence rather than a timer: it waits for a frame timestamped after the command left. That way the two machines never need synchronized clocks.
I used YOLO as the teacher model and distilled a smaller, faster student from it with similar accuracy. The teacher labels footage I collected myself, and the student trains on those labels, plain supervised learning against a stronger model’s output. The student is small enough to run in the loop and scores mAP50 0.92, mAP50-95 0.81 on that dataset. Between detections, Lucas-Kanade optical flow carries the track through fast, blurred pans that would otherwise drop it.
I tested the system with video games as the input detection data, a cheap source of fast, unpredictable motion. The pipeline transfers directly to anything in real life that needs low-latency object detection and tracking on minimal hardware.