A latency budget for robot inference

Model inference is usually less than half of end-to-end latency. Teams spend months quantizing a network and never measure the 25 ms their camera driver is holding frames.

Ask a team what their policy's latency is and you will usually get the forward pass time. Ask what the time is from photon to torque and you will usually get a pause.

That second number is the one that determines stability. A control loop closed around a 40 ms delay behaves very differently from one closed around 120 ms, and the difference shows up as the arm hunting around its target — which then gets diagnosed as a controller gain problem and fixed by making the robot slower.

The full chain#

Every stage below is real and measurable. The numbers are representative of an embedded-GPU manipulation stack; yours will differ, but the shape usually does not.

That is 81 ms before the first action of the chunk is even commanded, and the last action of a 16-step chunk at 50 Hz is executed 300 ms after the frame it was computed from. If you have not measured the first three stages, you do not know your latency.

Sensor to host#

The largest and most commonly ignored term. Contributors, in rough order:

• Exposure time. A 20 ms exposure means the image is a 20 ms average of the world, and its effective timestamp is the middle of that window. Under low light, auto-exposure will quietly lengthen it and add latency you did not ask for. Lock exposure for deployment.
• Driver buffering. Most capture APIs queue frames. If you are reading the oldest queued frame rather than the newest, you are adding a full buffer's worth of delay. Set the queue depth to one and drop frames rather than accumulating them.
• Rolling shutter. Rows are captured at different times. For a moving camera or a fast-moving object, the top and bottom of the image disagree about when "now" is by several milliseconds.

# measure it, don't assume it
import cv2, time
cap = cv2.VideoCapture(dev)
cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)         # do not queue
cap.set(cv2.CAP_PROP_AUTO_EXPOSURE, 0.25)   # manual
cap.set(cv2.CAP_PROP_EXPOSURE, exposure_us)

# a monitor showing a millisecond counter, filmed by the camera, gives you
# the true photon-to-numpy latency in one shot — cheap and unambiguous

Preprocessing#

Usually small and usually avoidable. The pattern that costs the most is doing resize and normalization on the CPU in Python, in the same process as the control loop, allocating a new array each frame. Move the resize to the GPU, keep a pinned host buffer and reuse it, and overlap the host-to-device copy with the previous forward pass on a separate CUDA stream.

The model#

This is where the optimization literature lives, and the ordering of what to try is fairly stable:

| Step | Typical speedup | Accuracy cost | Effort | | --- | --- | --- | --- | | Compile (TensorRT / torch.compile) | 1.5–2.5× | None | Low | | FP16 / BF16 | 1.5–2× | Negligible | Low | | Static shapes, no dynamic control flow | 1.1–1.4× | None | Low | | KV-cache reuse across chunk decode | 1.3–2× | None | Medium | | INT8 post-training quantization | 1.5–2× | Small, measurable | Medium | | Structured pruning | 1.2–1.6× | Task-dependent | High | | Distillation to a smaller student | 2–5× | Task-dependent | High |

The first three are close to free and should be done before anything else is discussed. INT8 is where judgement starts: quantize the vision encoder aggressively and the action head conservatively. Action outputs are the thing you are actually shipping, and quantization error there lands directly on the robot's end-effector.

Transport and control#

Serialization, IPC, and the controller's own loop. Small individually, and they add up. Two rules keep them small: never cross a process boundary with a Python object where a shared memory ring buffer will do, and never let the control loop allocate.

Jitter is worse than latency#

A constant 100 ms delay can be compensated — you predict forward, you tune the controller for it, the system is stable. A delay that varies between 60 ms and 200 ms cannot be compensated, because the compensation itself is now wrong most of the time.

Sources of jitter, in the order we usually find them:

1. Python garbage collection. A collection pause in the control process at the wrong moment is a 10–50 ms stall. Disable automatic GC in the control loop and collect explicitly between episodes.
2. CPU frequency scaling and scheduling. Pin the control thread to an isolated core, set a real-time scheduling policy, and disable the governor.
3. Dynamic shapes. A model whose input dimensions change triggers recompilation or kernel reselection. Pad to fixed shapes.
4. Logging in the hot path. Writing to disk or, worse, to a network endpoint from the control thread. Buffer and flush from a separate thread.

Report p50 and p99, never the mean. The mean hides exactly the events that break a control loop, and the p99 is the number your safety margin has to cover.

Instrumenting it properly#

One timestamp propagated through the whole chain is worth more than a dozen local timers, because it lets you attribute delay rather than merely observe it.

@dataclass
class Frame:
    image: np.ndarray
    t_capture: float      # from the driver, at exposure midpoint
    t_host: float = 0.0
    t_preprocessed: float = 0.0
    t_inferred: float = 0.0
    t_commanded: float = 0.0

def record(frame: Frame) -> None:
    stages = {
        "capture_to_host":  frame.t_host - frame.t_capture,
        "preprocess":       frame.t_preprocessed - frame.t_host,
        "inference":        frame.t_inferred - frame.t_preprocessed,
        "transport":        frame.t_commanded - frame.t_inferred,
        "end_to_end":       frame.t_commanded - frame.t_capture,
    }
    for name, dt in stages.items():
        histogram(f"latency.{name}").observe(dt * 1e3)

Put the end-to-end p99 on a dashboard that the whole team sees, and treat a regression in it as seriously as a regression in success rate. It is the number that silently degrades every time someone adds a feature to the perception stack.

Where the budget usually goes wrong#

Three patterns account for most of the surprises we have found:

The camera was never configured. Default buffer depth, auto-exposure on, sometimes an MJPEG pipeline doing a decode nobody knew about. Fixing this is often a bigger win than any model optimization, and it takes an afternoon.

Inference and control share a process. The GIL, the allocator, and the GC are all shared, so a slow batch in one stalls the other. Separate processes with a shared-memory ring buffer is the correct architecture and is not much more code.

Nobody owns the number. Latency is everyone's problem and therefore no one's. Assign it, chart it, and gate releases on it, or it will drift upward one reasonable-seeming feature at a time.

The one-page version#

Measure photon-to-torque, not forward-pass. Lock exposure and set buffer depth to one. Compile and use half precision before considering anything exotic. Quantize the encoder harder than the head, and validate on task success. Separate the control process from the inference process. Track p99. Then, if you still need the milliseconds, start distilling.