Action chunking and the latency wall

Chunking decouples the control rate from the inference rate. Everything difficult about it follows from the fact that the last action in a chunk was computed from an observation that is now stale.

Here is the arithmetic that shapes every modern manipulation policy.

A dexterous task — bimanual folding, insertion, anything with contact — wants commands at 30 to 50 Hz. That is a budget of 20 to 33 milliseconds per action. A 3-billion-parameter vision-language-action model, on a good embedded GPU, with a compiled graph and quantized weights, takes somewhere between 60 and 150 milliseconds for a forward pass. You are short by a factor of five, and the gap does not close by buying a better card, because the card is on the robot.

Action chunking is the resolution: predict H future actions in one forward pass, execute them at the control rate while the next forward pass runs. The control loop is now decoupled from the model. This is not a trick; it is the structural reason chunked policies took over.

What a chunk actually costs you#

The moment you commit to H actions, action H is executed against an observation that is H control periods old. At 50 Hz with a chunk of 16, the final action in the chunk is acting on a picture of the world from 320 milliseconds ago. If something moved, you do not know.

A chunk is a bet that the world will not surprise you for H control periods. The right chunk length is the length of time that bet is safe for your task.

This gives a much better design heuristic than "tune H on the validation set": estimate the timescale on which your task's contact events happen, and keep the chunk shorter than that.

Adaptive horizon — long chunks in free space, short ones near contact — is attractive and mostly unnecessary. A single conservative horizon plus temporal ensembling gets you most of the benefit with none of the mode-switching bugs.

Temporal ensembling#

Naive chunking has a visible artefact: the robot executes chunk k, then jumps to the first action of chunk k+1, which was computed from fresher data and disagrees. You get a discontinuity every H steps — a stutter you can hear.

The fix, from the ACT line of work, is to overlap. Run inference every S steps where S < H, so at any moment several chunks have an opinion about what the robot should do now, and blend them.

from collections import deque
import numpy as np

class TemporalEnsembler:
    """Blend overlapping action chunks into one smooth command stream.

    Each chunk votes on the timesteps it covers. Older chunks are downweighted
    exponentially: they saw a staler observation, so they get less say.
    """

    def __init__(self, horizon: int, m: float = 0.08):
        self.horizon = horizon
        self.m = m                       # decay rate; larger = trust fresh more
        self.chunks: deque = deque()     # (issue_step, actions[H, D])

    def submit(self, step: int, actions: np.ndarray) -> None:
        self.chunks.append((step, actions))
        while self.chunks and step - self.chunks[0][0] >= self.horizon:
            self.chunks.popleft()        # fully consumed

    def command(self, step: int) -> np.ndarray:
        votes, weights = [], []
        for issued, actions in self.chunks:
            offset = step - issued
            if 0 <= offset < self.horizon:
                votes.append(actions[offset])
                weights.append(np.exp(-self.m * offset))
        if not votes:
            raise RuntimeError("no chunk covers this step — inference fell behind")
        w = np.asarray(weights)
        return (np.stack(votes) * w[:, None]).sum(0) / w.sum()

Two details decide whether this works.

The decay rate is a smoothness/latency dial. m near zero averages all chunks equally: maximally smooth, maximally laggy — the robot responds to changes about H/2 steps late. Large m collapses to "use the newest chunk": responsive and jumpy. We generally start around m = 0.05 for free-space work and push it up toward 0.2 for anything reactive, then tune by watching the commanded-velocity trace rather than task success, because the artefact is visible there long before it shows up in a success rate.

Averaging is only valid in a space where averaging means something. Blending end-effector positions is fine. Blending quaternions component-wise is not — you get an unnormalized quaternion that is not the rotation between the two. Use spherical interpolation, or represent rotation as a 6D continuous vector and re-orthogonalize after blending.

The runtime that keeps the loop fed#

The chunking arithmetic only holds if inference actually returns before the current chunk runs out. That is a scheduling problem, and it is the part that tends to be under-engineered.

Run the policy in its own process at real-time priority, publishing chunks into a lock-free ring buffer. The control loop reads from the buffer and never blocks on the model. Then handle the two failure modes explicitly:

1. Inference overruns. The buffer is about to empty. Do not freeze and do not repeat the last command — both are unsafe with a moving arm. Extrapolate the final chunk with a decaying velocity toward zero, and if the buffer empties entirely, execute a controlled stop. Log it loudly; a run with overruns is not a valid evaluation run.
2. Inference stalls entirely. A watchdog with a hard deadline that trips the safety controller. A policy that has stopped publishing is indistinguishable from a policy that has decided to do nothing, and only one of those is safe to assume.

# control thread — never allocates, never blocks
def control_step(t, buffer, ensembler, robot):
    chunk = buffer.try_pop()             # non-blocking
    if chunk is not None:
        ensembler.submit(chunk.step, chunk.actions)
    try:
        cmd = ensembler.command(t)
    except RuntimeError:
        cmd = decay_to_stop(robot.last_command)   # graceful, not frozen
        metrics.incr("control.starved")
    robot.send(cmd)

Measuring it honestly#

Task success rate hides latency problems completely — a policy that stutters can still succeed. Three instruments actually tell you what is happening:

• Chunk age at execution. Histogram of how stale each executed action was. If the tail extends past your intended horizon, inference is falling behind.
• Commanded jerk. The third derivative of the commanded trajectory. Chunk seams show up here as spikes at exactly the inference period, which makes them trivially attributable.
• Disagreement between overlapping chunks. When consecutive chunks disagree strongly about the same timestep, the policy is uncertain — and that signal arrives before the failure. It is one of the few cheap, honest uncertainty estimates you get from a deterministic policy, and it makes a good trigger for slowing down or handing back to a human.

That last one is worth more than it looks. Most policies have no calibrated uncertainty at all. Chunk disagreement is free, requires no architectural change, and correlates with the situations where you would want a human watching.

The short version#

Chunking is not an optimization you add later. It is the assumption the rest of the stack is built on: your data loader emits chunks, your loss is over chunks, your runtime schedules chunks, your safety layer bounds chunks. Retrofitting it into a single-step policy means rewriting all four.

Pick the horizon from the physics of your task, ensemble the overlaps, weight recent chunks harder, never average a gripper, and instrument chunk age from day one.