Same Goal, Different Words
Spatial Instruction Sensitivity in World-Action Models
Same model, same start, same goal. Only the word order changes.
1General Motors Robotics · 2RIVeR Lab, Northeastern University
“Move the mustard bottle on the table so that …
TF … the mustard bottle is to the right of the raisin box.”
RF … the raisin box is to the left of the mustard bottle.”
Cosmos3 Edge from one saved start, once per wording, at 2× speed. The camera faces the robot, so image left is the robot’s right.
If a robot understands the goal, the wording shouldn’t matter.
World-action models turn an instruction and camera views into robot actions, while predicting what the cameras will see next. To place an object, the model has to work out which object to move and where it belongs relative to another. The same placement can be described from either object’s side. We asked three released models to make the same placement, from the same start, under three equivalent wordings.
- DIRDirect
Place the mustard bottle on the table to the right of the raisin box.
- TFTarget-first
Move the mustard bottle on the table so that the mustard bottle is to the right of the raisin box.
- RFReference-first
Move the mustard bottle on the table so that the raisin box is to the left of the mustard bottle.
mustard bottle is the object to move, raisin box is the reference. All three ask for the same arrangement, the mustard to the right of the box.
Four simulated tabletop scenes. A rollout is scored a success when the requested arrangement holds steadily for the first time; simulator state is used only for scoring, never shown to the robot.
Name the reference first, and the robot succeeds less often.
Across three world-action models, the same goals worded reference-first were placed 12.2 points less often than when worded target-first.
Stable placement at any point in the episode, 96 rollouts per dot, with the four scenes weighted equally. 95% CI 8.6–15.4 points. Among pairs with the same model, goal and start, 45 succeeded only with target-first wording and 14 only with reference-first. All results →
Twenty placements against two.
Goal: the mustard bottle to the right of the raisin box. Each wording got the same 24 tries: three models from the same eight starts.
Each tile is one rollout, sampled at twelve moments from start to end; rows are models, columns the eight starts. Click a tile to watch it. Every goal →
The right arrangement, with the wrong object.
One goal favoured reference-first wording: put the cube behind the bowl. Every one of its reference-first successes came from moving the bowl.
“Place the Rubik's cube so that …
TF … the Rubik's cube is behind the bowl.”
RF … the bowl is in front of the Rubik's cube.”
reference-first successes on this goal moved the bowl. The cube was never lifted.
The only one of twelve goals where reference-first wording came out ahead, and every one of those wins moved the wrong object.
Cosmos3 Nano from one saved start, at 2× speed. A relation check alone counts the bowl move as a success; checking which object moved does not. Failure analysis →
Half the successes came undone before the end.
Of all 864 rollouts, 383 satisfied the goal at some point. Only 191 still did when the episode ended.
383 rollouts satisfied the goal at some point; 191 still did at the end.
216 of the 383 started with the arrangement already true, and the robot undid it in 119. Of the rollouts that had to create it, 167 did and 94 kept it. Where rollouts fail →
The language backbone already misreads the goal.
We gave each model’s own language backbone just the instruction text, and asked for the object to move, the reference and the final relation. On reference-first wording, two of the three misread six of the ten goals.
Move the mustard bottle on the table so that the raisin box is to the left of the mustard bottle.
The model answered target mustard bottle, reference raisin box, relation left_of.
Reversed. The bottle should end up right_of the raisin box.
Ten goals per wording; bowl stacking is excluded. Adding camera images and their descriptions does not fix it: on reference-first goals Edge answers 23.3% correctly and FLUX 41.7%, and even Nano drops to 89.9%. A misreading alone does not explain a failed rollout. Language probe →
The imagined future added no measurable warning.
Each model also generates a video of what it expects to happen next. Labels of those videos did not help predict manipulation errors, such as handling the wrong object.
- Change in error prediction (Brier) from adding the video labels
- −0.000795% CI −0.0018 to +0.0002
Labels came from two vision–language models standing in for human annotators. They agreed on every label for only 5.9% of the videos, so this is not a verdict on the videos themselves. Forecast analysis →
Success under one wording is not steerability.
Every number on this page comes from the same 864 rollouts, the same starts and the same scorer. To know whether a policy follows instructions, an evaluation should:
- Pair the wordingsRun equivalent instructions from identical starts, and report every relation, not only the average.
- Check the objectVerify that the intended object moved, not just that the relation became true.
- Check the endingConfirm the requested arrangement still holds when the episode ends.
- Test the language modelProbe the backbone on the same goals, with and without images.
Cite
@inproceedings{same_goal_different_words_2026,
title = {Same Goal, Different Words: Spatial Instruction Sensitivity in World-Action Models},
author = {Abbas, Ali-Adeeb and Seraj, Esmaeil and Toghi, Behrad and Padir, Taskin},
booktitle = {CoRL 2026 Workshop: Do Robots Need World Models?},
year = {2026},
note = {Under review}
}