CoRL 2026 Workshop · Do Robots Need World Models?

Same Goal, Different Words

Spatial Instruction Sensitivity in World-Action Models

Ali-Adeeb Abbas^1,2, Esmaeil Seraj¹, Behrad Toghi¹, Taskin Padir²

¹General Motors Robotics · ²RIVeR Lab, Northeastern University

Same model, same start, same goal. Only the wording changes. Naming the reference before the object to move lowers stable placement by 12.2 points across three world-action models and 864 rollouts.

Introduction Evaluation code

TF Target-first Loading

“Move the mustard bottle on the table so that the mustard bottle is to the right of the raisin box.”

Kept Lifts the mustard at 9 s and sets it down to the right of the raisin box at 23 s.

RF Reference-first Loading

“Move the mustard bottle on the table so that the raisin box is to the left of the mustard bottle.”

Never placed Pushes the mustard without lifting it. From 25 s the gripper is on the raisin box instead.

0.0 / 0.0 s

Cosmos3 Edge, start state 2 of 8, played at 2×. Unedited simulator recordings from an exterior camera, cropped and rotated upright as in the paper. The camera faces the robot, so image left is the robot’s right. The lanes are scored from simulator state after the fact.

01 · Introduction

Equivalent instructions should lead to the same manipulation

World-action models (WAMs) generate robot actions together with predictions of future observations. Their usefulness as general-purpose policies depends on turning a user’s description into the intended manipulation. For a spatial placement, that means identifying both the object to move and its destination, which is fixed relative to a reference object.

Wrist camera: the raisin box and the mustard bottle on the table, gripper below — wrist

One goal, three ways to say it

A placement goal names two objects: the target to move and the reference that fixes where it goes. The same arrangement can be stated from either side, because the mustard is to the right of the raisin box exactly when the box is to the left of the mustard. We therefore write every goal three ways and run each from the same start.

Initial scene seen from across the table: the robot at the back, a raisin box, and the mustard bottle on the robot’s left of the box — The paper’s Table 1. Direct (DIR) is a concise command. Target-first (TF) and reference-first (RF) differ in the subject of the spatial clause: TF describes the mustard relative to the box, RF the box relative to the mustard. Close view of the exterior camera, rotated 90° as in the paper; the camera faces the robot, so image left is the robot’s right.

The question

Does the policy respond consistently to equivalent language instructions?

Prior work asks neighbouring questions. We focus on a specific consistency requirement: preserving the intended manipulation when the same spatial goal is described from another object’s perspective.

Evaluation
Language in look-alike scenes

Whether policies respond to language when scenes are visually similar.
- TRI LBM Team, 2025
Evaluation
Changed goals

Whether policies follow a goal that differs from the one the scene suggests.
- Fang et al., 2026
Robustness
Paraphrases and irrelevant context

Whether policies withstand reworded instructions and distracting additions.
- LIBERO-Plus, Fei et al., 2026
- LIBERO-Para, Kim et al., 2026
- Moskalenko et al., 2026
Methods
Better instruction following

Counterfactual supervision and semantic guidance to make policies follow instructions.
- CAST, Glossop et al., 2025
- SG-WAM, He et al., 2026
This study
The same spatial goal, described from another object’s perspective

Target-first and reference-first describe one arrangement from opposite objects. A consistent policy should move the same object to the same place under both.

What we did

We compare three released WAMs under alternative instructions from identical initial scenes. Success under one description does not imply consistency across equivalent descriptions, and opposite effects across spatial relations can cancel in an overall score. The rollouts also show a limit of spatial scoring: the requested relation can become true because the wrong object moved, or disappear under continued control. To separate language understanding from action generation, we also test each model’s vision–language backbone on its own.

02 · The finding

Put the reference first, and placements drop

Target-first and reference-first start with the same command and ask the robot to move the same object. They differ only after “so that”: reference-first makes the reference the subject and states the relation from its side. Across the three models, that lowers the rate of stable placement by 12.2 percentage points, and the gap appears in every model.

Cube·bowl

Butter·raisin box

Mustard·raisin box

Two bowls

Try , the largest gap, and , where it reverses.

Move the mustard bottle on the table so that the mustard bottleisto the right ofthe raisin box.

The object to move is the subject of the clause.

DIR Direct command, for comparison: “Place the mustard bottle on the table to the right of the raisin box.”

One of the eight starts, seen from above as the camera sees it: robot at the top, so its left is your right. Counts when the mustard bottle’s origin (+) lies within 45° of straight right of the raisin box’s origin, in the robot’s frame. Origins are the assets’ own reference points and can sit just outside the object. Dashed: one example placement.

DIR 19/24

TF 20/24

RF 2/24

Same model, same start: 18 placed under TF only, 0 under RF only.

Placed, kept
Placed, then lost
Never placed
TF only
RF only

TF and RF change both the order of the objects and the relation word, so this comparison cannot say which of the two drives the effect. Per-goal and per-model numbers are descriptive.

points

Higher rate of stable placement under target-first than reference-first, any time in the episode.

Nano +14.6Edge +8.1FLUX +13.8

95% interval 8.6–15.4 · adjusted p < 0.001

How the effect varies

Paired runs where only TF placed versus only RF. The other 229 of 288 pairs agreed.

Same start, same model

of 383

Stable placements undone before the episode ended. The overall rate halves from any time to the final second.

Any time 43.2% At the end 21.8%

Still placed in the final second

Undone before the end

Rates weight the four scenes equally.

Where runs fail

of 10

RF instructions that FLUX’s text encoder, tested alone, read with the relation reversed. TF: 0 of 10 errors.

A test of the language model, not a causal explanation of rollouts.

Language tests

03 · Setup

Same start, same goal, same model. Only the words change.

Each model attempts each goal from the same recorded starts, once per wording. The robot and objects are reset before every instruction and the random seed is fixed, so the three wordings begin from the same saved simulator state. Every placement rate on this site comes from these 864 paired episodes.

Start 2 of the mustard scene from the exterior camera: the robot at the top, the mustard bottle and the raisin box on the table. — Every dot is one episode. Rows are goals, grouped by scene; columns are the 8 starts. Each cell holds the DIR, TF and RF runs of one model, goal and start, so its three dots differ only in wording. ✓ marks the three goals that already hold at the start. **Kept**: the placement conditions hold throughout the final second. **Lost**: a stable placement at some point, but not at the end. **Never**: no stable placement.

Four scenes, eight starts each

The scenes come from RoboLab, a simulation benchmark for generalist manipulation policies. Seven goals are RoboLab’s own; five more reuse the same objects. Each start is a small change in the object poses that a scripted controller can solve. Every goal, wording and model in a scene shares the same eight starts. A scene with two bins had no qualifying start and was dropped before any outcomes were collected.

Cube, bowl and banana

30 s per episode

Robot Start 1

Objects

Rubik's cube
banana, in no goal
bowl

Goals

RoboLab Cube left of bowl
Added Cube right of bowl ✓
RoboLab Cube in front of bowl
RoboLab Cube behind bowl

Butter box and raisin box

40 s per episode

Robot Start 1

Objects

butter box
raisin box

Goals

RoboLab Butter box on raisin box
Added Butter box left of raisin box
Added Butter box right of raisin box ✓

Mustard bottle and raisin box

40 s per episode

Robot Start 1

Objects

mustard bottle
raisin box

Goals

RoboLab Mustard bottle on raisin box
Added Mustard bottle left of raisin box ✓
Added Mustard bottle right of raisin box

Two bowls

20 s per episode

Robot Start 1

Objects

right bowl
left bowl

Goals

RoboLab Left bowl onto right bowl
RoboLab Right bowl onto left bowl

Top views at the first step, rotated so that the robot sits at the top edge. The robot’s left is therefore the image’s right. “RoboLab” marks the benchmark’s own goals; “Added” goals reuse the same objects. ✓ marks a goal that already holds at the start. The logged object poses at the first step are identical for every goal, wording and model that shares a start; the rendered frames differ only by small rendering noise.

All 36 instructions

The direct command (DIR) keeps RoboLab’s original wording where it exists, spelling included. Target-first (TF) and reference-first (RF) start the same way and ask the robot to move the same object. After “so that”, TF describes the target relative to the reference. RF describes the reference relative to the target, with the relation inverted.

target reference relation

Each row is one goal; the three columns are its three wordings.

The 36 instructions: 12 goals, each in three wordings
Goal	DIR Direct	TF Target-first	RF Reference-first
Cube, bowl and banana
Cube left of bowl RoboLab	Put the rubiks cube to the left of the bowl	Place the Rubik's cube so that the Rubik's cube is to the left of the bowl.	Place the Rubik's cube so that the bowl is to the right of the Rubik's cube.
Cube right of bowl ✓ Added	Put the rubiks cube to the right of the bowl	Place the Rubik's cube so that the Rubik's cube is to the right of the bowl.	Place the Rubik's cube so that the bowl is to the left of the Rubik's cube.
Cube in front of bowl RoboLab	Put the rubiks cube in front of the bowl	Place the Rubik's cube so that the Rubik's cube is in front of the bowl.	Place the Rubik's cube so that the bowl is behind the Rubik's cube.
Cube behind bowl RoboLab	Put the rubiks cube behind the bowl	Place the Rubik's cube so that the Rubik's cube is behind the bowl.	Place the Rubik's cube so that the bowl is in front of the Rubik's cube.
Butter box and raisin box
Butter box on raisin box RoboLab	Pick up the butter box and place it on top of the raisin box	Place the butter box so that the butter box is on top of and supported by the raisin box.	Place the butter box so that the raisin box is underneath and supporting the butter box.
Butter box left of raisin box Added	Place the butter box on the table to the left of the raisin box.	Move the butter box on the table so that the butter box is to the left of the raisin box.	Move the butter box on the table so that the raisin box is to the right of the butter box.
Butter box right of raisin box ✓ Added	Place the butter box on the table to the right of the raisin box.	Move the butter box on the table so that the butter box is to the right of the raisin box.	Move the butter box on the table so that the raisin box is to the left of the butter box.
Mustard bottle and raisin box
Mustard bottle on raisin box RoboLab	Place the mustard on the raisin box.	Place the mustard bottle so that the mustard bottle is on top of and supported by the raisin box.	Place the mustard bottle so that the raisin box is underneath and supporting the mustard bottle.
Mustard bottle left of raisin box ✓ Added	Place the mustard bottle on the table to the left of the raisin box.	Move the mustard bottle on the table so that the mustard bottle is to the left of the raisin box.	Move the mustard bottle on the table so that the raisin box is to the right of the mustard bottle.
Mustard bottle right of raisin box Added	Place the mustard bottle on the table to the right of the raisin box.	Move the mustard bottle on the table so that the mustard bottle is to the right of the raisin box.	Move the mustard bottle on the table so that the raisin box is to the left of the mustard bottle.
Two bowls
Left bowl onto right bowl RoboLab	Stack the left bowl on the right bowl	Move the left bowl so that the left bowl is stacked on the right bowl.	Move the left bowl so that the right bowl is underneath and supporting the left bowl.
Right bowl onto left bowl RoboLab	Stack the right bowl on the left bowl	Move the right bowl so that the right bowl is stacked on the left bowl.	Move the right bowl so that the left bowl is underneath and supporting the right bowl.

✓ The goal already holds at the start, so the robot only has to leave it in place. Three goals are like this; the other nine must be achieved.
RoboLab goals come with the benchmark. Added goals (five) reuse the same objects. Direct commands keep RoboLab’s text, spelling and capitalization included.
For the bowls, reference-first says the reference is “underneath and supporting” the target. That is less specific than the nesting the scorer checks.

Three models, one protocol

We evaluate the DROID versions of Cosmos3 Nano, Cosmos3 Edge and FLUX 3 Action. Each model keeps its released implementation and image preprocessing. The cameras, the action rate and the seed are the same for all three.

Nano

Cosmos3 Nano

Selected for strong RoboLab performance.

Language backbone Qwen3-VL-8B-Instruct

Edge

Cosmos3 Edge

Provides a smaller Cosmos comparison.

Language backbone Cosmos3-Edge reasoner

FLUX

FLUX 3 Action

Selected for strong RoboLab performance.

Language backbone Qwen3-VL-4B-Instruct

Inputs: The wrist camera and two exterior cameras, plus the robot’s proprioceptive state. The scorer’s output is never an input.
Actions: 32 actions per request at 15 Hz, so each request covers about 2.1 s of motion.
Seed: 6100 for every episode.
Length: 20–40 s by scene. Control continues after a placement until the time runs out.
Reset: Robot and objects return to the saved start before every instruction.

Inference settings (Appendix B)
Setting	Cosmos3 Nano	Cosmos3 Edge	FLUX 3 Action
Arithmetic	bfloat16	bfloat16	bfloat16
Sampling steps	4	4	4
Sampler	UniPC	UniPC	Cosmos UniPC
Noise shift	5.0	5.0	5.0
Visual guidance	3.0	3.0	4.0
Action guidance	—	—	1.0
Observation history	1	1	1
Actions per request	32	32	32
Control frequency	15 Hz	15 Hz	15 Hz
Prompt JSON formatting	No	Yes	No
Input/canvas setting	540×640	540×640	544×736

The Cosmos models use joint-position actions and a resolution setting of 480; Edge's guidance interval is [960, 1001].
FLUX uses its DROID camera layout and absolute actions with a scale of 2.0.
Different input dimensions reflect image preprocessing; the simulated cameras are unchanged.
The exact full-policy revision for FLUX was not recorded; its text encoder was verified separately.

What counts as a placement

A scorer checks the requested arrangement from the simulator’s object poses and contacts. A stable placement needs the relation, support, a target released from the gripper, and a speed below 2 cm/s, all held for one continuous second. Directions use the robot’s frame. Try it:

Goal

Gripper

Target

mustard bottle (target) raisin box (reference) where the relation holds

Mustard bottle right of the raisin box: noits origin within 45° of the robot’s right
Supported by the table: yesnow: resting on the table
Clear of the gripper: yesnow: released
Slower than 2 cm/s: yesnow: 0.0 cm/s

Held one continuous second Waiting for all four checks

Drag the mustard bottle, or focus it and use the arrow keys (Shift moves 5 cm). Moving it counts as speed, so each step restarts the timer. The scorer measures direction between the objects’ origins (+) in a 90° cone, and “on top” from the target’s centre (•). Speed is the change in the logged position between steps at 15 Hz.

TF Edge · Target-first Loading

“Move the mustard bottle on the table so that the mustard bottle is to the right of the raisin box.”

0.0 / 0.0 s

The scorer at 0.0 s

Loading the simulator log…

All four, held one second Not yet

The same checks, read from the logged simulator state of one rollout. Drag along the lanes, or focus them and use the arrow keys, to scrub.

Any time, or at the end

The scorer gives every run two outcomes. Both use the same stable-placement conditions: the requested relation, support, a target released from the gripper and moving slower than 2 cm/s.

Main outcome

Stable placement at any time

The conditions hold for one continuous second at some point during the episode.

43.2%

383 of 864 runs

Stricter

Still holds at the end

The same conditions hold throughout the final second of the episode.

21.8%

191 of 864 runs

Percentages weight the 4 scenes equally, as in the paper; counts are raw. 192 of the 383 runs credited at some point are no longer in place at the end. Where those placements go →

Must be achieved the mustard bottle to the right of the raisin box

Already true at the start the mustard bottle to the left of the raisin box

Start

End · 40 s

Placed and kept Edge, target-first (TF), start 2: the mustard bottle to the right of the raisin box.

How the wordings are compared

Every comparison is paired: two wordings, same model, same start, same goal. Choose a comparison and watch how the paper turns 288 pairs into an estimate, an interval and a test.

Wordings

Outcome

Model

Reported in the paper +12.2 points 95% interval 8.6 to 15.4 Holm-adjusted p < 0.001

1
Pair. For each model, start and goal, subtract the RF outcome from the TF outcome: +1 if only TF reached the placement, −1 if only RF did, 0 if they agree. 45 pairs +1 14 pairs −1 229 pairs 0
2
Average per start. Average over the goals in the start’s scene, then over the models. That leaves one value per physical start, eight per scene. Their mean weights the four scenes equally.

Each column is one start; values in points Observed mean +12.2

Bootstrap the starts. Within each scene, draw eight of its starts with replacement, keeping every goal, model and wording of a start together, and recompute the mean. Repeat 10,000 times; the middle 95% of the means is the interval.

Draws 0 of 10,000 This illustration —

Seeded draws illustrate the procedure, so their interval and p-value can differ slightly from the paper’s. The values reported in the paper are shown at the top of this panel. Units are percentage points of the placement rate.

The scorer reads simulator state after each run; the policies never see it. Its two outcomes describe placements, not complete instruction following: neither identifies which object moved. Across all 4 scenes the set-up is identical for the three wordings, so the differences in the next sections come from the words.

04 · Results

Where the gap holds, and where it flips

The 12.2-point target-first advantage is an average. Each of the 32 physical starts gives its own paired difference: 21 favour TF, 4 favour RF and 7 tie. By goal the effect ranges from −25.0 to +75.0 points, reverses for one goal, and cancels out completely in the cube scene. This section shows every number behind the headline, including the appendix tables.

Same start, same model: every paired difference

Each dot is one physical start: the difference in placement rate between two wordings from that start, averaged over its goals (and, in the top row, over the three models). Pairing this way cancels out how hard a start is. Switch the contrast to bring in the direct instruction, or the outcome to require the placement at episode end.

Contrast

Outcome

Highlight a scene

All models +12.2points 95% CI [8.6, 15.4] Holm-adjusted p < 0.001 21 starts favour TF, 4 RF, 7 tie

Each model separately · descriptive

Nano +14.6points 95% CI [8.1, 21.1] Descriptive 15 starts favour TF, 3 RF, 14 tie

Edge +8.1points 95% CI [3.6, 12.8] Descriptive 8 starts favour TF, 0 RF, 24 tie

FLUX +13.8points 95% CI [7.6, 20.1] Descriptive 11 starts favour TF, 3 RF, 18 tie

Paired difference in stable-placement rate, percentage points

TF and RF change both object order and relation vocabulary, so this comparison does not identify which aspect drives the effect.

Each dot is one of the 32 physical starts (eight per scene). Its position is the paired TF − RF difference in stable placement for that start, averaged over the scene’s goals and, in the top row, over the three models. Tied dots stack vertically; a full stack continues one dot-width to the side. Diamonds and bars show the mean and its 95% interval from 10,000 bootstrap draws of starts within scenes. Adjusted p-values appear only for the two original pooled any-time tests (TF − RF and DIR − TF, Holm correction); all other rows are descriptive.

Paired runs that disagree

Same model, start and goal; at any time. The remaining pairs agree: both wordings placed, or neither did.

Only RF placed Only TF placed Agree Nano · 66 of 96 agree 8 22 66 of 96 Edge · 88 of 96 agree 0 8 88 of 96 FLUX · 75 of 96 agree 6 15 75 of 96 All · 229 of 288 agree 14 45 229 of 288

Raw counts of paired runs in which exactly one wording reached a stable placement: 96 pairs per model (12 goals × 8 starts). They include the three goals already true at the start, which never disagree at any time. Unlike the dots above, these counts are not averaged within scenes. Computed from the released episode outcomes; the paper does not report them.

The average hides a reversal

Pooled over models, the TF − RF gap ranges from −25.0 points for cube-behind to +75.0 for mustard-right. The heatmap gives each model’s placements out of eight for every goal and wording; the forest gives the pooled difference with its interval. Each goal rests on eight physical starts, so these per-goal estimates are descriptive: they show where the effect is, not a rule for every task.

Heatmap outcome

Jump to

Cube · bowl

Butter box · raisin box

Mustard · raisin box

Two bowls

Cube · bowl

Butter box · raisin box

Mustard · raisin box

Two bowls

Mustard bottle right of raisin box

Largest TF − RF gap Added goal

DIR Place the mustard bottle on the table to the right of the raisin box.
TF Move the mustard bottle on the table so that the mustard bottle is to the right of the raisin box.
RF Move the mustard bottle on the table so that the raisin box is to the left of the mustard bottle.

TF − RF +75.0 points

95% CI [58.3, 87.5] · descriptive, from eight physical starts

Out of 8 starts per model; 24 for all models.
Any time / end	DIR	TF	RF
Nano	7/3	8/6	0/0
Edge	4/3	4/4	0/0
FLUX	8/8	8/3	2/0
All models	19/14	20/13	2/0

Watch all 72 runs TF and RF only

Left: runs out of 8 that reached a stable placement, for each model and wording. Right: the paired TF − RF difference for each goal at any time, pooled over the three models, with a 95% interval from that scene’s eight physical starts. Counts on the right are any-time placements out of 24 runs. Per-goal estimates rest on only eight starts each and are descriptive. Goals marked † are already true in all eight starting states: they pass at any time under every wording and contribute zero to the difference.

Opposite effects can cancel

In the cube scene, effects in opposite directions cancel: RF does better for cube-behind even though the scene’s average gap is zero. An aggregate score can therefore hide sensitivity to individual relations.

Scene

In the cube scene, cube-left (+16.7) and cube-front (+8.3) favour TF, while cube-behind (−25.0) favours RF. The opposite effects cancel exactly: the scene average is ±0.0 points, although its goals respond to the wording in opposite directions. Across the scene’s 96 runs per wording, TF placed 35 times and RF 35 times.

Pooled TF − RF, scenes weighted equally

Sum ÷ 4 = +12.2 points

Each bar adds one goal’s paired TF − RF difference at any time, pooled over the three models; the counts beside each goal are its TF and RF placements out of 24. Bars to the right favour TF, bars to the left favour RF. Goals marked † are already true at every start and add zero. The pooled estimate averages the goals within each scene and then weights the four scenes equally, so a scene whose goals cancel contributes nothing to it.

Placement rates by model and wording

The rates behind the paired differences, balanced across scenes as in the paper. RF is the lowest of the three wordings for every model, under both outcomes and with or without the goals already true at reset. Fewer than half of the any-time placements also pass the episode-end check.

Outcome

Goals

Scene-balanced placement rates in percent, at any time, all goals
Model	DIR	TF	RF	All
Nano	50.3	50.8	36.2	45.7
Edge	36.5	31.0	22.9	30.1
FLUX	63.3	56.0	42.2	53.8
All models	50.0	45.9	33.8	43.2

Overall rate, all models and wordings: 43.2% at any time.
RF is the lowest of the three wordings for every model in this view.
The all-models line falls 12.2 points from TF to RF: exactly the paired TF − RF estimate at any time, because both average goals within scenes and weight scenes equally.
At any time the three goals already true at reset always count as placed, which sets a floor of 22.9%. Edge’s RF rate sits exactly on it: no such run placed a goal that was not already true.

Rates average the goals within each scene and weight the four scenes equally, as in the paper; per-model rates are descriptive. Any-time and end scores assess placement only; neither shows that the requested object was the one moved.

Placed, then failed the end check: raw counts

Across the 864 episodes, 383 reach a stable placement and 191 also pass the episode-end check, so 192 of the 383 any-time passes (50.1%) fail at the end. Among the 648 episodes whose goals are not true at reset, 167 reach a placement and 94 pass the final check; the other 73 are 43.7% of that group’s any-time passes. For goals already true at reset, 97 of 216 episodes pass the final check.

Raw counts pooled over models and wordings, so they need not equal the scene-balanced rates above. An episode can lose and regain a placement, so passing at the end does not show that the placement was held continuously after it was first reached.

Checks on the headline

The bowl scene needs separate reading: its RF prompt describes the lower bowl as underneath and supporting the target, which does not explicitly require the nesting the benchmark checks. Without that scene the estimate is 10.65 points, so the direction does not depend on the bowl-language ambiguity; that check is descriptive and does not establish that the bowl prompts are equivalent. In an exploratory comparison, the short direct instruction also outperforms reference-first wording on average, by 16.2 points. That comparison was added after the original analyses, and because the forms also differ in length and sentence construction it cannot isolate the effect of changing spatial perspective.

Sensitivity of the main estimate

Diamonds are pooled TF − RF estimates in percentage points; bars are 95% intervals from resampling the physical starts within each scene. Only the any-time estimate belongs to the original test family. The end-of-episode and bowl-free summaries are descriptive, and their intervals come from the released study data rather than the paper’s text.

Direct versus reference-first

Exploratory DIR − RF gaps in percentage points with 95% intervals
Model	At any time	At episode end
All models	16.2 [11.7, 20.4]	14.4 [10.0, 18.9]
Nano	14.1 [7.0, 20.8]	16.1 [4.2, 27.6]
Edge	13.5 [7.8, 19.5]	8.6 [0.0, 17.7]
FLUX	21.1 [10.4, 30.7]	18.5 [9.1, 27.6]

Exploratory DIR − RF gaps in percentage points, with 95% intervals from resampling paired physical starts. Positive values favour the direct instruction. No new multiplicity-adjusted significance claims are made.

Every count

The appendix count tables in one place, with totals. Every rate and difference in this section can be rebuilt from them, and they download as CSV.

Every count: 12 goals × 3 wordings × 3 models 108 cells, each any time / at episode end, out of 8

Each entry gives any-time and episode-end stable-placement counts, both out of eight starts. A dagger marks a goal already satisfied in all eight starting states. Zeros are observed failures. These scores assess the resulting arrangement and do not establish that the requested object moved.

Stable-placement counts by goal, model and wording
Goal	Nano			Edge			FLUX
Goal	DIR	TF	RF	DIR	TF	RF	DIR	TF	RF
Cube, bowl and banana
Cube left of bowl	8/2	6/3	0/0	3/1	2/1	0/0	1/0	0/0	4/3
Cube right of bowl†	8/3	8/2	8/0	8/4	8/3	8/0	8/2	8/2	8/8
Cube in front of bowl	1/1	1/1	0/0	0/0	1/0	0/0	0/0	0/0	0/0
Cube behind bowl	0/0	0/0	7/6	1/0	0/0	0/0	2/0	1/0	0/0
Butter box and raisin box
Butter box on raisin box	6/5	6/5	2/1	3/2	0/0	0/0	2/2	0/0	0/0
Butter box left of raisin box	0/0	0/0	0/0	0/0	0/0	0/0	0/0	0/0	0/0
Butter box right of raisin box†	8/5	8/4	8/6	8/5	8/3	8/7	8/4	8/6	8/3
Mustard bottle and raisin box
Mustard bottle on raisin box	5/5	6/5	4/2	0/0	0/0	0/0	7/2	5/2	6/2
Mustard bottle left of raisin box†	8/5	8/5	8/0	8/3	8/4	8/3	8/6	8/4	8/0
Mustard bottle right of raisin box	7/3	8/6	0/0	4/3	4/4	0/0	8/8	8/3	2/0
Two bowls
Left bowl onto right bowl	0/0	0/0	0/0	0/0	0/0	0/0	7/6	8/5	1/1
Right bowl onto left bowl	1/1	1/1	1/0	2/1	1/1	0/0	6/0	4/0	4/0
All twelve goals, of 96	52/30	52/32	38/15	37/19	32/16	24/10	57/30	50/22	41/17

Entries are any-time / end stable-placement counts, each out of eight; † marks a goal already true at reset. Bold entries placed at any time from all eight starts on a goal that was not already true. Goal names open their episodes in the explorer.

05 · Failures

When a placement fails, is lost, or is reached the wrong way

The scorer checks spatial outcomes only. A placement counts once the target holds the requested relation for 1 continuous second while it rests on the table, moves slower than 2 cm/s and is clear of the gripper. Neither check identifies which object moved.

A placement reached by moving the wrong object

Both Nano instructions ask the robot to move the Rubik’s cube behind the bowl. Play them side by side and watch what the gripper holds.

0.0 / 0.0 s

TF Target-first Loading

“Place the Rubik's cube so that the Rubik's cube is behind the bowl.”

Never placed Lifted, never reached the goal.

Rubik’s cube: ended 26.9 cm from its start · bowl: at most 1.3 cm from its start

RF Reference-first Loading

“Place the Rubik's cube so that the bowl is in front of the Rubik's cube.”

Placed, then lost Stable placement completed at 25.4 s; the final arrangement fails the end check.

Rubik’s cube: ended 1.9 cm from its start · bowl: at most 28.3 cm from its start

What the scorer reads at 0.0 s
Check	TFTarget-first	RFReference-first
Cube behind bowlwithin 45°	…	…
Target supported by the table	…	…
Target clear of the gripper	…	…
Goal metall three	…	…
Goal met and still for 1 sunder 2 cm/s	…	…
Gripper onnot scored	…	…

Every row is read from simulator state at the current moment, as the lanes are. Jump to 25.4 s stops both rollouts where the paper’s snapshot was taken.

Both Nano instructions ask the robot to move the cube behind the bowl. In the reference-first rollout, however, the robot holds the bowl while the cube remains on the tabletop. Moving the bowl changes the relative positions enough for the cube-behind check to pass. The target-first rollout does not pass that check. Thus, the reference-first score advantage in this example is not evidence of better instruction following: the predicate credits a relation that became true through manipulation of the wrong object.

Appendix D.1

Selected from the recorded outcome differences before their videos were inspected. The example illustrates possible behaviour and is not used to estimate how often it happens. Labels describe recorded spatial outcomes, not complete instruction success.

Camera context

The close views are a fixed crop of one exterior camera. Step through both example pairs at the paper’s snapshot times, or scrub them, in the close view or in a second camera.

Close view. A fixed crop of the exterior camera, rotated 90° for readability, as in the paper’s close views. The camera faces the robot, so image left is the robot’s right. The crop is identical across instruction forms, and no image content is retouched.

Edge Mustard bottle right of raisin box start state 2

End of the 40 s episode

Initial sceneboth rollouts start here

0.0 s

TFTarget-first

40.0 s

Loading Placed and kept

RFReference-first

40.0 s

Loading Never placed

Time 40.0 s

Nano Cube behind bowl start state 1

The paper’s 25.4 s snapshot

Initial sceneboth rollouts start here

0.0 s

TFTarget-first

25.4 s

Loading Never placed

RFReference-first

25.4 s

Loading Placed, then lost

Time 25.4 s

Within each pair, both rollouts begin from the same physical state and are shown at the same elapsed time. These are executed observations, not model-generated predictions; the labels under each frame are scored from simulator state at that moment and over the whole episode.

Placed, then lost: any time versus the end

The main score credits a stable placement reached at any time. A stricter check asks whether the placement conditions still hold throughout the final 1 s of the episode. Filter by model or wording to see how many placements survive to the end.

All episodes864

Placed at some point383 44.3% of 864

Still placed at the end191 22.1% of 864

Placed and still placed at the end 191 Placed, then lost 192 (50.1% of placements)

Goal must be achieved not true at reset648 episodes

kept 94 lost 73 (43.7% of its placements) never placed 481

Goal already true at reset only has to be kept216 episodes

kept 97 lost 119 (55.1% of its placements) never placed 0

Across the 864 episodes, 383 reach a stable placement and 191 also satisfy the episode-end criterion. Thus, 192 of the 383 any-time passes (50.1%) fail the episode-end check. Among the 648 episodes whose goals are not satisfied initially, 167 reach a stable placement and 94 pass the final check; the remaining 73 account for 43.7% of that group’s any-time passes. For initially-satisfied goals, 97 of 216 episodes pass the final check.

An episode can lose and regain a placement, so endpoint success does not establish continuous retention after first attainment.

Scene-balanced rates, as in the paper’s figure

All goals: 43.2% any time 21.8% at the end
Goals not true at reset: 26.8% any time 15.3% at the end

These average goals within each scene and weight scenes equally, so they need not equal the raw fractions above (44.3% and 22.1%).

Where runs stop

Each episode is assigned the furthest stage it reached: touching the target, lifting it, meeting the goal, and holding a stable placement.

Descriptive no test of significance

1 Never touched the target
2 Touched, never lifted
3 Lifted, never reached the goal
4 Reached the goal, never stable
5 Stable placement
Stable because the goal was already true at reset

Each bar is 96 episodes of one model and wording (288 for all models), split by the furthest stage reached; the number at the right is the share that reached a stable placement at some point. Hatched: episodes whose goal was already true at reset, which start in the goal and only have to keep it.

Reference-first minus target-first, all models, episodes per stage (of 288 each)

Under reference-first wording, more runs stop before lifting the target and fewer reach a stable placement. This describes the recorded distribution only; the paper does not test these stage differences, and the stage breakdown was computed for this website from the same simulator logs.

Episodes by furthest stage reached, per model and wording
Model	Wording	Never touched the target	Touched, never lifted	Lifted, never reached the goal	Reached the goal, never stable	Stable placement	Episodes
Nano	Direct	6	9	26	3	52	96
Nano	Target-first	3	10	25	6	52	96
Nano	Reference-first	9	18	27	4	38	96
Edge	Direct	12	10	33	4	37	96
Edge	Target-first	11	17	35	1	32	96
Edge	Reference-first	13	20	39	0	24	96
FLUX	Direct	0	1	35	3	57	96
FLUX	Target-first	0	1	40	5	50	96
FLUX	Reference-first	15	8	27	5	41	96
All models	Direct	18	20	94	10	146	288
All models	Target-first	14	28	100	12	134	288
All models	Reference-first	37	46	93	9	103	288
All models	Reference-first minus target-first	+23	+18	−7	−3	−31

One placement that was lost

A single rollout that is credited at any time but fails the end check: watch where the goal stops holding.

0.0 / 0.0 s

DIR Edge · Direct Loading

“Place the mustard bottle on the table to the right of the raisin box.”

Placed, then lost

What the scorer reads at 0.0 s
Check	DIRDirect
Mustard bottle right of raisin boxwithin 45°	…
Target supported by the table	…
Target clear of the gripper	…
Goal metall three	…
Goal met and still for 1 sunder 2 cm/s	…
Gripper onnot scored	…

The episode is credited at any time once the goal has held still for 1 s. Jump to the loss stops where the goal stops holding.

Chosen for this website to illustrate; not one of the paper’s examples, and not a prevalence estimate. Rule: among the 10 Nano and Edge episodes in the mustard scene that reach a stable placement at some point, fail the end check and whose goal was not already true at reset, take the one whose goal does not hold at the last frame and whose last goal-holding interval is longest. The loss was then confirmed in the video.

06 · Interpreting the goal

Some backbones misread the goal before the robot moves

To separate understanding from manipulation, we gave each policy's vision-language backbone the same 30 instructions and asked it to name the object to move, the reference and the requested relation. FLUX's text encoder reverses reference-first relations, Edge's reasoner errs even on direct commands, and Nano reads every sentence correctly from text, until camera images are added.

Same 30 instructions The ten cube, butter and mustard goals, each in direct, target-first and reference-first wording. Bowl stacking is excluded.
Four prompt conditions Text alone, text with camera descriptions, text with the three camera images of each of eight starts, and both.
One answer, all three parts Greedy decoding into a fixed JSON vocabulary. Object to move, reference and relation must all be right.

Camera images

Camera descriptions

FLUX's text encoder reverses a reference-first instruction

Tested on its own with text only, the encoder names the mustard and the raisin box correctly but puts the mustard to the left of the box: the requested relation, reversed.

Same goal, target-first: correct

Name the mustard first and the answer is right. FLUX reads all ten direct and all ten target-first instructions correctly, but only four of ten reference-first ones, and each of its six errors reverses the relation.

Nano's language model reads all 30 from text

From text alone, Nano answers every instruction correctly, including this reference-first one. Camera descriptions without images change none of its answers.

Adding the camera images reverses the same sentence

With images and camera descriptions, Nano reverses this butter instruction at five of eight starts and cube-behind at three. Reference-first accuracy falls to 89.9%, 10.1 points below the same prompt without images (95% CI 5.2–14.6); direct and target-first stay at 100%.

Edge misreads even direct commands

Edge's reasoner answers “left of” for “Put the rubiks cube in front of the bowl.” From text it gets 6/10 direct, 8/10 target-first and 4/10 reference-first instructions right, so its difficulty is not restricted to reference-first wording.

FLUX · Text only · Mustard bottle right of raisin box · Reference-first

Pick any cell for its exact prompt and decoded answer.

DIRTFRF cube and bowl left right front behind butter and raisin box on top left right mustard and raisin box on top left right

DIR 10/10TF 10/10RF 4/10

Correct 24
Relation reversed 6

Prompt 215 tokens 215 text

Camera views of the current scene, captured at the same moment:

empty line: no image

Object vocabulary: mustard_bottle, raisin_box

Relation vocabulary: left_of, right_of, in_front_of, behind, on_top_supported, stacked_on, unknown

Instruction:

Move the mustard bottle on the table so that the raisin box is to the left of the mustard bottle.

Reference-first Move the mustard bottle on the table so that the raisin box is to the left of the mustard bottle.

Output 25/25 tokens

{"target": "mustard_bottle", "reference": "raisin_box", "relation": "left_of"}

target mustard_bottleraisin_box

reference mustard_bottleraisin_box

relation left_ofright_ofin_front_ofbehindon_top_supportedstacked_onunknown

greedy · temperature 0 · JSON constrained to these values · thinking off

Requested

mustard right of raisin box

Model's reading

mustard left of raisin box

Relation reversed Right objects, opposite relation (left for right, front for behind).

A · Text only

Reference-first wording is misread from text alone

Nano 10 10 10

Edge 6 8 4

FLUX 10 10 4

Correct
Relation reversed
Other wrong relation

FLUX's text encoder reads every direct and target-first instruction correctly but only four of ten reference-first ones. All six errors name the right objects and reverse the relation. Edge also reads fewer reference-first than target-first instructions correctly, but it errs on direct and target-first ones too, so its difficulty is not limited to reference-first wording. Nano reads all 30.

B · Nano with camera images

Adding images reverses correct readings

Change in reference-first accuracy, points (95% CI)

Add images to the text −6.9

Add images to the text with camera descriptions −10.1

Add camera descriptions to the images −3.1

Reference-first answers at the eight starts

Butter box right of raisin box

Cube behind bowl

Butter box on raisin box

Correct
Relation reversed

Reference-first accuracy 89.9% · direct 100% · target-first 100%

Nano reads all 30 instructions correctly from text, with or without camera descriptions. With images and descriptions it makes eight relation reversals, five for butter-right and three for cube-behind: reference-first accuracy falls 10.1 points. Images alone cause four butter-right reversals and one butter-on-top error that swaps in a stacking label. The descriptions' extra effect remains unresolved.

C · Placement controls

Moving the reference phrase changes Edge's answers

Reference early Relative to the bowl, place the Rubik's cube to the left.

Reference late Place the Rubik's cube to the left, relative to the bowl.

Edge, eight goals per placement
Reference	Text only	Images + desc.
Early	5/8	41.7%
Late	8/8	59.4%

Sixteen controls move only the reference phrase and keep the direction. Nano and FLUX answer all of them correctly. Edge answers 5 of 8 reference-early and 8 of 8 reference-late sentences from text. With images and descriptions its reference-early accuracy is 17.7 points lower (95% CI 10.4–25.0).

Relative to the bowl, place the Rubik's cube in front. Edge: cube left of bowl
Relative to the bowl, place the Rubik's cube behind. Edge: cube left of bowl
Relative to the raisin box, place the mustard bottle to the right. Edge: mustard left of raisin box

These controls use different sentences from target-first and reference-first, so they do not settle general noun-order effects.

Every condition, every model

Accuracy for each backbone, wording and prompt condition. Select a cell to replay its first wrong answer in the console; perfect cells replay a correct one.

Text only correct of 10 · Sec. 4
Model	DIR	TF	RF
Nano
Edge
FLUX

Text + camera descriptions correct of 10 · App. F
Model	DIR	TF	RF
Nano
Edge
FLUX

Images scene-balanced % · App. F
Model	DIR	TF	RF
Nano
Edge
FLUX(auxiliary)

Images + camera descriptions scene-balanced % · Sec. 4
Model	DIR	TF	RF
Nano
Edge
FLUX(auxiliary)

Correct
Relation reversed
Roles swapped, relation flipped
Roles swapped
Support/stacking label swap
Other wrong relation

Bars under each score split that cell's answers by class: 10 instructions without images, 80 image prompts with them. Select a cell to load its first error in the console. ^†FLUX's policy uses this encoder for text only and processes images separately, so its image results are auxiliary.

Judging the visible arrangement was inconclusive

We also asked whether the current camera views already show the requested relation, allowing an insufficient-evidence answer. Labels come from saved simulator geometry and contact, and each item is asked under both assignments of the yes and no answer codes.

How often each backbone answers “yes”

Answerable items, both code orders. Outlined bars give how often the views truly show the relation.

Initial images · 234 items

Truth 28.2%

Nano 0.9%

Edge 1.9%

FLUX 0.0%

Saved-rollout images · 360 items

Truth 25.0%

Nano 0.6%

Edge 4.4%

FLUX 0.0%

Balanced accuracy, 95% CI

Weights both answers equally, so always answering “no” scores exactly 50%.

Nano Initial 49.4

Rollout 49.8

Edge Initial 48.5

Rollout 50.1

FLUX Initial 50.0

Rollout 50.0

Every backbone almost always answers “no”: at most 4.4% “yes”, although 25–28% of the items do show the relation. None of the 4,320 image answers chose insufficient evidence. Balanced accuracy ranges from 48.5% to 50.1%. Only Edge ever answers “yes” when that is true: its recall on matching rollout frames is 5.2% (CI 1.4–9.9).

Same start, same goal, the relation changes

Each image set is asked whether the views show the relation in the instruction, twice, with the yes and no answer codes swapped. A dot marks a pair answered correctly for both frames.

Cube Butter Mustard

Cube behind bowl · start 4

Place the Rubik's cube so that the bowl is in front of the Rubik's cube.

Left exterior, cube behind bowl start 4, relation shown — Shows the relation 15 s into a Nano target-first rollout · Watch

Wrist camera, cube behind bowl start 4, relation not shown — Does not show it Initial image

Left exterior, cube behind bowl start 4, relation not shown — Does not show it Initial image

Both frames right under both code orders, RF: Nano 0/11 · Edge 1/11 · FLUX 0/11

What these tests can and cannot show

Exploratory. A preliminary prespecified test, which asked whether an image matched the instruction, was inconclusive. This protocol was fixed after seeing it, so it is not an independent confirmatory test.
Matched components. Nano's language component has the same weights as Qwen3-VL-8B-Instruct and Edge's matches the released Cosmos3-Edge reasoner. FLUX's frozen encoder matches Qwen3-VL-4B-Instruct, which its policy uses only for text, so its image results (†) are auxiliary. These comparisons do not show degradation of component weights during robot adaptation.
Scoring. Constrained decoding fixes the JSON format and vocabulary but allows wrong choices, and all three fields must be right. Scores are not comparable to the preliminary unrestricted answers.
Uncertainty. Image accuracy averages starts within goals, goals within scenes and the three scenes equally. Intervals come from 10,000 bootstrap resamples of starts within scenes, conditional on the text answers, scenes and wording templates, not on new instructions.
Scope. Bowl stacking is excluded. The images are 24 initial and 48 saved-rollout sets of three views from 24 physical starts, using existing checkpoints and no new rollouts.
Not a cause. These question-answering tests do not reveal the policy's internal goal representation or establish that a wrong answer causes a rollout failure.

07 · Forecasts

Do the models’ imagined futures warn of their mistakes?

World-action models generate robot actions together with predictions of future observations, so each of the three models here also shows what it expects to happen next. Whether those generated videos flag an impending error was the study’s original question. With vision–language models standing in for human annotators, the answer is null.

The original question

Before wording became the focus, this was the study’s primary question: does a policy’s generated video help identify an impending manipulation error, such as handling the wrong object or releasing the target into a relation it was not asked for, beyond what is already apparent from the current scene and proposed actions?

The behavioural results arrived first and made wording the paper’s focus. At that point the planned human video annotations were replaced with two vision–language models, before their answers were inspected. The analysis found no measurable benefit from their labels, but the labels are too unreliable to say whether the information is in the videos themselves.

of 864 chunks

contained a detected error in the actions that accompanied the selected forecast: 14 wrong-object and 33 contradictory release.

Detected from simulator state; not all visually verified.

change in held-out Brier error when the video labels join a classifier of those errors. Negative means they made it slightly worse.

95% interval [−0.00177, +0.00016] includes zero.

of forecasts on which the two annotators agreed on every field. On the final spatial relation alone they agreed in 20.1%.

Agreement, not accuracy: neither annotator was checked against people.

See what the model imagined

Each time a model is queried it returns a chunk of actions together with a short generated video of the future it predicts. Pick any of the 864 episodes to see the forecast this analysis selected, beside what the cameras then recorded. Wipe and Overlay put the two on top of each other.

Examples

Detected errors

– of 47

Model

Goal

Wording

Start

View

Generated prediction Executed

0.0 / 0.0 s

The generated prediction is the video the model produced with its chunk of up to 32 actions; the executed view is what the same three cameras recorded while those actions ran, wrist camera above two exterior cameras. Frames are paired by index from the request at frame 0; nominal frame times do not establish the pace of the generated motion. Nano and Edge decode a few rows short of this layout, so their predictions end in a thin black strip. The annotators saw eight generated frames and a numbered episode-start image, never the executed half. Examples follow a fixed rule: the cube-behind episode from start 1 with direct wording for each model, and the first wrong-object and contradictory-release chunks by episode identifier.

Whole-episode forecasts

For a few episodes the whole sequence of predictions was rendered. Each new request restarts the imagined future from the current scene, so the generated side jumps every time the model replans.

Model

Episode

Generated prediction Executed

0.0 / 0.0 s

Loading timeline…

Each model’s generated video beside the executed episode, at the same moment. Every new request re-anchors the prediction on the current scene, so the generated half jumps at each boundary on the top strip; within one prediction, frames are paired by index and do not establish the pace of generated motion. A black generated frame would mean no decodable future was available for that moment, which is missing evidence rather than a zero or a failure; none of these 17 episodes has one. The thin black strip under Nano and Edge predictions is padding. Lanes are scored from simulator state. These 17 are the episodes with whole-episode media; none has a detected error in its selected chunk.

How forecasts were labelled

One forecast per episode was chosen by a rule fixed without viewing any forecast or outcome. Two vision–language models then labelled it without seeing the instruction, the model or what actually happened.

864 rollouts

3 models × 12 goals × 3 wordings × 8 starts. Each request returned a chunk of up to 32 actions at 15 Hz together with a generated video of the future.
One forecast per rollout

Chosen without viewing the forecast or the outcome: the first prediction made while the gripper was within 10 cm of a movable task object. Two Nano episodes never met that condition and use their first prediction. Only frames assigned to actions the policy then performed, before its next prediction, are kept.
3
What the annotators saw

Eight forecast frames in a common three-camera layout, and an episode-start image with numbered objects and direction arrows. Ambiguous observations could be labelled unknown.
- Instruction
- Requested goal
- Policy
- Wording
- Rollout
- Outcome
12
Two vision–language models, independently

Qwen3-VL-235B-A22B-Instruct-FP8 GLM-4.5V-FP8

They replaced the planned human video annotation before anyone inspected their answers.
5
7 fields each
- Moving object
- Motion direction
- Final spatial relation
- Comparison object
- Possible release
- Missing object
- Hallucinated object
The comparison object is not necessarily the instruction’s reference.
Disagreements resolved by one of the two

Qwen chose between the two answers and an unknown option, in random order. Because Qwen also supplied one of the original answers, this step is not independent validation.
7
Classified against the task

A requested or alternative relation counted only when the labels named the intended target as moving and matched that goal’s comparison object and relation.
- Requested relation
- Another registered relation
- Neither
- Unknown
share of 864 forecasts

Agreement, not accuracy

With no human labels there is no ground truth to score against, only whether the two annotators agree. They agreed on the moving object in 59.1% of forecasts, on the final relation in only 20.1%, and on every field at once in 5.9%.

Agreement between the two video annotators on 864 selected forecasts
Field	Agreement (%)	Cohen’s κ
Moving object	59.1	0.398
Motion direction	48.6	0.108
Final spatial relation	20.1	0.077
Comparison object	33.3	0.111
Possible release	93.4	0.026
Missing object	97.7	0.000
Hallucinated object	98.5	0.231
All 7 fields at once	5.9	not reported

On the 862 additional episode-start forecasts, every field’s κ stayed below 0.115.

Agreement between the two independent annotators on the 864 selected forecasts, before disagreements were resolved. High agreement on a rare answer mostly reflects its rarity: the annotators agreed on possible release in 93.4% of forecasts, yet κ was 0.026, and only 9 of 864 final labels said release was possible. Neither annotation set has been validated against human judgments, so these numbers measure agreement, not accuracy.

What the forecasts show

After disagreements were resolved, the labels were classified against each episode’s request: only 4.5% were credited with the requested relation. Separately, 77.9% indicated upward-only motion.

All models · n = 864

Final relation, classified against the goal

4.5% Requested relation 39 forecasts
11.5% Another registered relation 99 forecasts
70.4% Neither 608 forecasts
13.7% Unknown 118 forecasts

Motion direction

Up only 77.9% 673
Goal direction 1.9% 16
Lateral 5.7% 49
Other direction 6.7% 58
None 7.4% 64
Unknown 0.5% 4

Moving object

Target 62.5% 540
Reference 28.6% 247
None 6.3% 54
Other object 1.7% 15
Several objects 0.8% 7
Unknown 0.1% 1

Possible release

No 98.8% 854
Yes 1.0% 9
Unknown 0.1% 1

Hallucinated object

No 99.3% 858
Yes 0.7% 6

Missing object was labelled no in 864 of 864 forecasts.

Final labels after disagreements were resolved; counts are forecasts. A requested or alternative relation was credited only when the labels identified the intended target as moving and matched that goal’s comparison object and relation, so “Neither” includes forecasts that named a different moving object. Direction categories keep the annotation’s own names. These model-generated labels are unverified: short grasping or lifting clips may reveal little about the eventual placement, but annotation errors are another explanation.

Can forecasts predict errors?

The test asks whether the labels add anything to what the scene and the proposed actions already reveal. For each selected forecast, the scorer checked the chunk of actions that accompanied it for two kinds of error.

Wrong-object manipulation

At least 0.2 s of continuous gripper contact with a non-target object, which moves at least 2 cm from its position at the start of the chunk.

Contradictory release

The target detaches after contact, then rests supported and detached for at least 0.2 s in exactly one registered alternative relation while the requested relation is false, all within the chunk. Alternatives were the other tested cube, butter and mustard relations; bowls had none.

Both checks read simulator state for the chunk of actions that accompanied each selected forecast. They differ from the 1-second stable-placement endpoint used for task success, and they have not all been visually verified: contact followed by detachment can count as a release without substantive prior manipulation.

Wrong-object 14 Contradictory release 33 No detected error 817

Cosmos3 Nano 15 of 288 chunks 3 wrong-object · 12 release

Cube13/96

Butter2/72

Mustard0/72

Bowls0/48

Cosmos3 Edge 8 of 288 chunks 1 wrong-object · 7 release

Cube6/96

Butter1/72

Mustard0/72

Bowls1/48

FLUX 3 Action 24 of 288 chunks 10 wrong-object · 14 release

Cube17/96

Butter4/72

Mustard2/72

Bowls1/48

Detected errors per model and scene, as wrong-object plus contradictory-release chunks out of scored chunks
Model	Cube	Butter	Mustard	Bowls	Total
Cosmos3 Nano	2 wrong-object, 11 release, of 96	1 wrong-object, 1 release, of 72	0 wrong-object, 0 release, of 72	0 wrong-object, 0 release, of 48	15 of 288
Cosmos3 Edge	0 wrong-object, 6 release, of 96	0 wrong-object, 1 release, of 72	0 wrong-object, 0 release, of 72	1 wrong-object, 0 release, of 48	8 of 288
FLUX 3 Action	7 wrong-object, 10 release, of 96	2 wrong-object, 2 release, of 72	0 wrong-object, 2 release, of 72	1 wrong-object, 0 release, of 48	24 of 288

Every cell is the chunk of actions that accompanied one of the 864 selected forecasts. Within each scene, columns group the three wordings of each goal and rows are the eight starts. The scorer marked 47 chunks: 14 wrong-object and 33 contradictory-release detections. To watch the forecasts behind them, step through the detected errors in the viewer above.

Two classifiers then predicted which chunks contained an error, one with the information available at the moment of the request and one with the forecast’s labels added.

Baseline classifier

Model, scene and goal
Current object and gripper states
Distance to the goal, and whether it already held
Object proximity along the proposed actions’ forward kinematics
Prior contact and lift indicators

With video labels

Everything in the baseline
The selected forecast’s video annotations, including unknown categories

Both: L2-regularized logistic regression, C = 1, 8 folds each holding out the same initial-state slot across all conditions, preprocessing fitted within the training fold. Wording, future states and final task success were excluded. The comparison uses privileged simulator information and is not a deployable failure detector.

Change in held-out Brier error from adding the video labels, ΔB = B_baseline − B_with video

Observed Weights models equally, scenes equally within models and goals equally within scenes.

All models, pooled −0.0007395% interval [−0.00177, +0.00016]

Per model, descriptive Classifiers refitted on each policy’s episodes; these rows do not decompose the pooled result.

Cosmos3 Nano −0.00139

Cosmos3 Edge −0.00040

FLUX 3 Action +0.00106

Controls The no-motion control did not pass repeated current images through the video annotators.

Missing-annotation indicators only +0.000011

Synthetic no-motion labels with the current-goal flag +0.000009

Permutation null A permutation distribution, not a confidence interval for the observed effect.

Forecasts shuffled across starts, 100 seeds mean −0.000532.5th–97.5th percentile [−0.00177, +0.00049]

Positive ΔB would favour the video labels. Adding them slightly worsened the pooled score; the 95% interval includes zero, and the small positive FLUX estimate does not establish a model-specific benefit. The interval resamples starts within scenes with fitted predictions held fixed, so it excludes annotation uncertainty and variation from refitting.

Model	Events / forecasts	Baseline	With video	ΔB
All models	47 / 864	0.04012	0.04085	−0.00073
Cosmos3 Nano	15 / 288	0.04012	0.04151	−0.00139
Cosmos3 Edge	8 / 288	0.02491	0.02530	−0.00040
FLUX 3 Action	24 / 288	0.06210	0.06104	+0.00106

Held-out Brier error for predicting detected manipulation errors; lower is better. Model rows are descriptive.

One exploratory pattern

One comparison, made after seeing the results, did show a difference: detected errors were more common when the labels named the reference object as moving (23 of 247, 9.3%) than when they named the requested object (19 of 540, 3.5%).

Exploratory Comparison made after seeing the results

Labels named the reference object as moving

9.3% detected errors in 23 of 247 chunks

10 wrong-object 13 contradictory release

Labels named the requested object as moving

3.5% detected errors in 19 of 540 chunks

2 wrong-object 17 contradictory release

Each cell is one selected forecast, grouped by the moving object its final labels named; filled cells are chunks with a detected error of either kind. The remaining 77 forecasts, whose labels named no moving object, another object, several objects or could not tell, accompanied 5 of the 47 detected errors among all 864. This comparison does not adjust for scene, goal or model, so it does not establish added predictive value.

Caveats

The labels are unvalidated. Neither annotation set has been checked against human judgments; the agreement figures describe agreement, not accuracy.
Vision–language models, not people. They replaced the planned human annotation. Disagreements were resolved by Qwen, which had also supplied one of the two original answers, so that step is not independent validation.
A null result. The analysis cannot distinguish an uninformative short forecast from a failure to extract useful information from it. Human validation of object identity, motion, spatial relations and detected errors is needed before attributing it to the generated videos.
Imperfect inputs. Two inspected examples showed overlapping object numbers and direction arrows, occasional object-name confusions and image stretching in the common layout, and the numbered scene image could precede the forecast by several actions. Those two predictions matched their actions and camera, but the check was not extended to every forecast, and it was not established that generated motion follows the rollout’s timing.
Privileged information. Event detection and the classifier use simulator state, so this is not a deployable failure detector, and the detected events have not all been visually verified.
Independent of the main results. The behavioural findings elsewhere on this page do not depend on these video annotations.

08 · Takeaways

Success under one wording is not steerability

Every result on this page comes from the same 864 rollouts, the same starts and the same scorer. Together they argue for evaluating world-action models on consistency across equivalent instructions, not on task success alone.

02 · The finding12.2pointsSaying the reference first lowers stable placementSame goal, same start, same model. Only the wording changes, and the target-first advantage appears in all three models (95% CI 8.6–15.4).
04 · Results−25.0 to +75.0points by goalAverages hide reversalsThe gap flips sign across relations, from cube behind bowl to mustard bottle right of raisin box. Opposite effects can cancel to zero in a scene score.
05 · Failures43.2% → 21.8%any time → endPlacements do not last, and can come from the wrong objectOnly 191 of the 383 episodes that reach a stable placement still hold it at the end, and a spatial check can pass because the robot moved the reference instead of the target.
06 · Interpreting the goal4 / 10reference-first read correctlyThe language backbones already misread the goalTested alone, FLUX's text encoder reverses the relation in every reference-first error. Nano reads all ten correctly from text, yet drops to 89.9% once images and camera descriptions are added.
07 · Forecasts−0.0007Brier change from videoGenerated futures added no measurable warningVLM labels of each policy's predicted video did not improve held-out prediction of manipulation errors (95% CI −0.0018 to +0.0002). Unreliable annotations keep this from being a verdict on the videos themselves.

What a steerability evaluation should check

Pair equivalent wordings from identical initial states, and report every relation, not only the average.
Verify that the intended object moved, not just that the relation became true.
Check that the requested arrangement still holds when the episode ends.
Test the language backbone on the same goals, with and without images.

Explore all 864 rollouts →Code

Cite

@inproceedings{same_goal_different_words_2026,
  title     = {Same Goal, Different Words: Spatial Instruction Sensitivity in World-Action Models},
  author    = {Abbas, Ali-Adeeb and Seraj, Esmaeil and Toghi, Behrad and Padir, Taskin},
  booktitle = {CoRL 2026 Workshop: Do Robots Need World Models?},
  year      = {2026},
  note      = {Under review}
}

One goal, three ways to say it

The question

Language in look-alike scenes

Changed goals

Paraphrases and irrelevant context

Better instruction following

The same spatial goal, described from another object’s perspective

What we did

Four scenes, eight starts each

All 36 instructions

Three models, one protocol

Cosmos3 Nano

Cosmos3 Edge

FLUX 3 Action

What counts as a placement

Any time, or at the end

Stable placement at any time

Still holds at the end

How the wordings are compared

Same start, same model: every paired difference

Paired runs that disagree

The average hides a reversal

Mustard bottle right of raisin box

Opposite effects can cancel

Placement rates by model and wording

Placed, then failed the end check: raw counts

Checks on the headline

Sensitivity of the main estimate

Direct versus reference-first

Every count

A placement reached by moving the wrong object

Camera context

Placed, then lost: any time versus the end

Where runs stop

One placement that was lost

Reference-first wording is misread from text alone

Adding images reverses correct readings

Moving the reference phrase changes Edge's answers

Every condition, every model

Judging the visible arrangement was inconclusive

How often each backbone answers “yes”

Balanced accuracy, 95% CI

Same start, same goal, the relation changes

The original question

See what the model imagined

Whole-episode forecasts

How forecasts were labelled

864 rollouts

One forecast per rollout

What the annotators saw

Two vision–language models, independently

7 fields each

Disagreements resolved by one of the two

Classified against the task

Agreement, not accuracy

What the forecasts show

Final relation, classified against the goal

Motion direction

Moving object

Possible release

Hallucinated object

Can forecasts predict errors?

Wrong-object manipulation

Contradictory release

Baseline classifier

With video labels

One exploratory pattern

Caveats

What a steerability evaluation should check

Cite