A field guide to vision-language-action models

Almost every meaningful difference between VLAs reduces to three choices: how actions are represented, how much of the vision-language prior you are willing to damage, and how many actions you commit to per forward pass.

A vision-language-action model is a single network that takes camera frames and a natural-language instruction and emits robot actions. That sentence is short enough to be misleading. The interesting part is not that it is one network — it is that the network starts life as a vision-language model trained on web data, and that inheritance is the whole point. The robot data alone is far too small to teach a policy what a colander is.

This is a working map of the space: the lineage, the axes that actually matter, and the things we would want a team to know before they start fine-tuning something on their own hardware.

The lineage in five steps#

The line from "transformer for robot control" to "generalist policy" is unusually legible.

RT-1 (2022) established the template: a transformer over a short history of frames plus a language instruction, emitting end-effector deltas. Its important move was discretizing actions — each dimension binned into 256 buckets and predicted as a classification problem. Regression heads on multimodal action distributions collapse to the mean; classification does not.

RT-2 (2023) made the leap that named the category. Instead of training a control transformer from scratch, take a pretrained vision-language model, encode actions as text tokens, and co-fine-tune on a mixture of robot trajectories and the original web data. The result inherited semantic generalization: it could act on objects and concepts that never appeared in a single robot episode, because the backbone already knew them.

Open X-Embodiment (2023) attacked the data problem from the other side — pooling demonstrations across dozens of robot platforms into one corpus, and showing that policies trained on the pool outperformed policies trained on any single embodiment's data. Cross-embodiment transfer is real, if noisy.

OpenVLA (2024) made the recipe reproducible in the open: a 7B backbone with fused vision encoders, trained on roughly a million episodes from that pool, still using discretized action bins. Crucially, it came with fine-tuning recipes that worked on a single node, which is what turned VLAs from a lab result into something a small team could actually use.

π0 and the flow-matching generation (2024–) changed the output head rather than the backbone: keep the VLM, bolt on an action expert that generates continuous action chunks by flow matching, and run the whole thing at control rates that are useful for dexterous manipulation rather than for slow pick-and-place.

The backbone stopped being the research frontier around 2024. The action head and the data pipeline are where the remaining performance lives.

Axis one: how actions are represented#

This is the single most consequential choice, and it is mostly a question about the shape of the action distribution you are trying to model.

| Representation | Mechanism | Strength | Cost | | --- | --- | --- | --- | | Discrete bins | Per-dimension classification over ~256 buckets | Models multimodality; reuses the LM head unchanged | Quantization floor; autoregressive decoding is slow | | Direct regression | MLP head, L1 or L2 loss | Trivial, fast, continuous | Averages across modes — the classic "robot freezes between two valid grasps" failure | | Diffusion | Iterative denoising of an action sequence | Excellent multimodal fit; strong on contact-rich tasks | N denoising steps per decision | | Flow matching | Learned velocity field, integrated in few steps | Diffusion-class quality at a fraction of the steps | More delicate to train; newer tooling |

The discrete-bin approach deserves its popularity for an unobvious reason: it requires no architectural change at all. You are still doing next-token prediction, so every optimization, every serving stack, every LoRA recipe built for language models applies unchanged.

class ActionTokenizer:
    """Bin each action dimension into the vocabulary's least-used token ids.

    Percentile clipping (not min/max) is the detail that matters: a single
    teleop jerk at the tail of the dataset will otherwise consume most of
    your bins and quantize everything useful into two buckets.
    """

    def __init__(self, stats, n_bins=256, vocab_offset=31744):
        # stats: per-dimension 1st and 99th percentile from the training set
        self.low, self.high = stats["q01"], stats["q99"]
        self.n_bins, self.offset = n_bins, vocab_offset

    def encode(self, action):                      # (D,) float -> (D,) int
        x = np.clip(action, self.low, self.high)
        u = (x - self.low) / (self.high - self.low + 1e-8)
        return self.offset + np.round(u * (self.n_bins - 1)).astype(int)

    def decode(self, tokens):
        u = (tokens - self.offset) / (self.n_bins - 1)
        return u * (self.high - self.low) + self.low

Axis two: how much of the prior you damage#

Fine-tuning a VLM on robot data is a controlled act of forgetting. You want the model to keep its grip on "the blue mug on the left" and lose nothing else — but gradient descent on a narrow, highly-correlated corpus of teleoperation episodes is a fast way to destroy general visual grounding.

Three mitigations, in ascending order of cost:

The empirical signal we look for is simple: hold out a small language generalization probe — object-naming, spatial-relation questions on your own scene images — and score it at every checkpoint alongside task success. If probe accuracy is falling while success rises, you are buying task performance with generalization, and you will pay it back the first time the scene changes.

Axis three: how many actions per forward pass#

Predicting one action per inference step is the intuitive design and the wrong one. It produces jittery, indecisive behaviour, and it makes your control rate a hostage to your model's latency.

Chunking — predicting H future actions in one pass and executing them open-loop — fixes both. It also fixes a subtler problem: human teleoperation data is full of pauses and hesitations, and a single-step policy learns to imitate the pauses, producing the notorious idle-forever failure. A chunk forces the model to commit to a trajectory.

def chunked_bc_loss(model, batch, horizon=16):
    """Behaviour cloning over an action chunk, with a validity mask so
    episode boundaries don't teach the model to drive into the next task."""
    pred = model(batch["images"], batch["instruction"], batch["state"])
    # pred: (B, H, D)   target: (B, H, D)   mask: (B, H)
    per_step = (pred - batch["actions"]).abs().mean(-1)        # (B, H)
    # weight early steps higher: they are executed with the least stale
    # observation, and errors there compound through the rest of the chunk
    w = torch.linspace(1.0, 0.5, horizon, device=pred.device)
    return ((per_step * w) * batch["mask"]).sum() / batch["mask"].sum()

Horizon is a real tradeoff, not a hyperparameter to grid-search blindly. Short chunks stay reactive and drift less; long chunks are smoother, cheaper per action, and blind for longer. For table-top manipulation at 30–50 Hz, chunks of 8–20 steps (roughly 0.2–0.5 s of open-loop commitment) are the usual landing zone. Anything that involves contact events you might need to react to — sliding, slipping, insertion — wants the short end.

We have written separately about the latency arithmetic that makes chunking not merely helpful but structurally necessary.

The data problem is the actual problem#

Every team we have talked to underestimates this. A VLA fine-tune is approximately as good as the demonstrations underneath it, and demonstration quality is not a single number.

• Coverage beats volume. Two hundred episodes across varied initial states, lighting, and distractor objects beat two thousand from the same starting pose. The policy will learn whatever is constant across your dataset, including the things you did not mean it to learn.
• Operator identity is a latent variable. Different teleoperators have different speeds and recovery habits, and the model will learn a blend of them. If you cannot avoid multiple operators, record who did what — it makes the eventual "why does it hesitate here" investigation tractable.
• Failure data is not automatically useful. Naively including failed episodes teaches failure. Including failures followed by recovery teaches recovery. The difference is entirely in whether the episode was relabelled.
• Proprioception is a shortcut waiting to be exploited. If the state vector is sufficient to predict the action on your training set, the model will learn to ignore the cameras, and the first novel scene will expose it. Dropping the state input with some probability during training is a cheap insurance policy.

Where they still break#

An honest list, because the demo reels do not include it:

1. Out-of-distribution lighting and backgrounds remain the most common cause of silent degradation. The policy does not fail loudly; it gets slightly worse at everything.
2. Long horizons are unsolved by imitation alone. Beyond roughly a minute of task, error accumulation dominates and you need either a higher-level planner or explicit subgoal conditioning.
3. Force-sensitive tasks — anything where the right action depends on felt resistance — are poorly served by vision-and-proprioception-only policies. Tactile and force-torque inputs help materially and are still comparatively rare in the public datasets.
4. Evaluation is expensive and noisy. Twenty rollouts on hardware gives you a confidence interval wide enough to hide most of the improvements you care about. Budget for this or you will be tuning on noise.

What we would tell a team starting today#

Start from an open VLA checkpoint with a published fine-tuning recipe rather than training from scratch — the compute saving is roughly two orders of magnitude and the generalization is not something you can reproduce with your own data.

Spend the first two weeks on the data pipeline, not the model: synchronized timestamps, per-dimension normalization statistics, episode-boundary masks, a deterministic replay of any episode by id. Every debugging session for the next six months routes through that pipeline.

Fix your evaluation protocol before you look at a single result — the initial state distribution, the number of rollouts, what counts as success — and then do not change it. Moving the goalposts mid-project is the most common way teams end up believing something works when it does not.

Then, and only then, argue about action heads.