Grasping, and the quiet difficulty of 6-DoF pose

Object pose is the wrong abstraction for most manipulation. Predicting grasps directly from geometry sidesteps the hardest part of the perception problem — and works on objects you have never seen.

The textbook pipeline for picking something up is: detect the object, estimate its 6-DoF pose, look up a grasp in a database keyed by object identity, transform it into the world frame, execute. It is clean, it is modular, and every stage fails in a way that the next stage cannot recover from.

The modern alternative is to skip the middle: predict grasps directly from observed geometry, without ever deciding what the object is. That change turns "works on the twelve objects in the CAD database" into "works on a thing it has never seen".

Why pose is harder than it looks#

Symmetry makes it ill-posed. A cylinder has infinitely many valid poses about its axis. A regression model trained on a single ground-truth angle per image learns to predict the average of a symmetry group, which is not a pose. You either encode symmetry explicitly in the loss, or you predict a distribution rather than a point.

Occlusion is the normal case. In any real bin, most objects are partially hidden. Pose estimators degrade gracefully in benchmarks, where occlusion is mild and synthetic, and much less gracefully in a cluttered tote.

Depth sensors lie in specific ways. Structured-light and time-of-flight sensors produce holes on dark, transparent, or specular surfaces, and flying pixels at depth discontinuities. Since exactly those materials are common in industrial and domestic settings, depth quality is a persistent tax rather than an occasional problem.

Pose is not what you needed. Even a perfect pose does not tell you how to grasp: that depends on what is reachable given the current clutter, on the gripper geometry, and on where the object's centre of mass is. Pose is an intermediate variable that discards the information the planner actually wants.

Estimating pose to look up a grasp is solving a harder problem than the one you have. Predict the grasp.

Predicting grasps from geometry#

The category-agnostic formulation: take a point cloud, output a set of 6-DoF gripper poses with confidence scores. No object identity, no pose, no CAD model.

A grasp is parameterized by an approach direction, a wrist rotation about it, a grasp centre, and a gripper width — plus a score. Networks in the Contact-GraspNet family predict these densely, one candidate per point, which turns grasp generation into a per-point regression problem with a natural spatial prior.

def select_grasp(cloud, gripper, obstacles, target_mask=None):
    """Score, filter, and rank candidates. The network proposes; kinematics
    and collision checking dispose."""
    grasps = grasp_net(cloud)                       # (N, 4x4 pose), (N,) width, (N,) score

    keep = grasps.score > 0.4
    if target_mask is not None:                     # segment-then-grasp, if asked
        keep &= grasps.contact_in(target_mask)

    ranked = []
    for g in grasps[keep].sorted_by_score():
        if not gripper.fits(g.width):               # mechanical limits
            continue
        if collides(gripper.mesh_at(g.pose), obstacles, margin=0.005):
            continue
        ik = arm.solve(g.pose, seed=arm.current_q)  # reachable, in joint limits
        if ik is None:
            continue
        if not clear(arm.path_to(ik), obstacles):   # approach path, not just the pose
            continue
        ranked.append((score_with_priors(g, arm, ik), g, ik))

    return max(ranked, default=None)

The filtering stages routinely reject 90% or more of proposals, and that is correct behaviour. A grasp is only useful if the gripper fits, the arm can reach it, and the approach path is clear — none of which the network knows about. The useful mental model is that the network is a proposal distribution and the classical stack is the acceptance test.

score_with_priors is where task knowledge enters: prefer top-down approaches when the scene is cluttered, prefer grasps near the object's centroid for stability, penalize configurations near joint limits or singularities, prefer wrist angles from which the next motion is easy. These priors are hand-written, tunable, and account for a large share of real-world reliability.

Where the remaining difficulty lives#

Transparent and specular objects. The depth sensor returns nothing useful. The practical answers are stereo networks that infer depth from RGB rather than projected light, or a completion model trained specifically on transparent objects. Neither is as good as depth on a matte surface, and this remains a real limitation.

Deformables. A towel has no rigid pose and no stable grasp database. The approaches that work operate on affordances — grasp here on this edge — rather than on object-level reasoning, and they need task-specific data.

Dense clutter. When objects are touching, segmentation and grasping become mutually dependent: you cannot segment reliably without knowing what is one object, and grasp candidates that span two objects look geometrically fine. Interaction helps here — a nudge to separate before grasping — and it is underused because it requires the system to plan a non-prehensile action first.

Post-grasp verification. Whether the grasp succeeded is a separate question from whether it looked good. Gripper width after closing, a force reading, and a quick visual check together catch most failures. Without verification, a failed grasp propagates into a place operation on nothing, which is worse than either failure alone.

When pose estimation is still the right answer#

The argument above is against pose as a general abstraction, not against it everywhere. It is the right tool when:

• The object set is closed and known, with CAD models available — most industrial assembly.
• The task requires a specific relationship to object features, not just a stable hold. Inserting a connector needs the connector's axis, not any grasp of it.
• Downstream steps need the pose anyway — placing into a fixture, mating parts, verifying assembly.

For those cases, render-and-compare methods that iteratively refine a hypothesis by minimizing the difference between a rendered model and the observation remain the accuracy leader, and they compose well with a coarse learned initialization.

The pragmatic architecture#

For a system that has to handle unknown objects:

1. Segment the scene into object instances with a class-agnostic model.
2. Propose dense 6-DoF grasps from the point cloud.
3. Filter by gripper geometry, collision, reachability, and approach clearance.
4. Rank with task priors.
5. Execute with force monitoring and a safety envelope that stops on unexpected contact.
6. Verify, and treat a failed verification as a first-class outcome with its own recovery, not as an exception.

Steps 3 through 6 are ordinary robotics engineering and are where the reliability comes from. The learned component proposes; the classical stack is what makes it trustworthy. That division of labour is, we think, the correct shape for most perception-driven manipulation today.