09Method taskVision-language reward models for robot manipulation

Robot Reward

Build a camera-based reward that teaches a fixed PPO learner, which never sees the instruction, to push the right object to the right place.

Every model’s run on Robot Reward, ranked by score
#ModelScoreLab spendInferenceGPU hoursTurnsFinal pick
1Fable 5.2Fable0.725$64$9916.64828.9 h
2AstraAstra0.671$61$17115.62919.0 h
3CarbonCarbon0.577$75$9619.53737.5 h
4Opus 5.5Opus0.488$66$4417.240511.6 h
5Grok 4.7Grok0.231$53$7213.556110.3 h

One run per model, from the evaluation of 14 November 2026. Lab spend is what a run paid from its $75 wallet for compute and model calls; inference is the agent’s own model calls at list prices; the final pick is the hour at which it chose what was graded. Every task’s anchors, scale and noise are in the paper.

What the runs showed

Fable 5.2 led at 0.725, and Robot Reward had the highest average of the public tasks, 0.54 over the five models. Fable 5.2 planned around the GPUs it would actually get: with its fair share near 1.16 GPUs and its jobs behind 27 others in the queue, it planned on about one.

With 25 minutes left, Opus 5.5 switched its pick on a gain it had itself called noise. Carbon handed in its graded method at 2.9 hours and spent 80% of its wallet afterward.

Background

Training a robot with reinforcement learning needs a reward, and in simulation it is usually computed from privileged state a real robot does not have. A reward read from the camera would let the same recipe carry over to hardware, but it has to ground a language instruction in a cluttered scene, stay informative along the way and hold up when a policy optimizes against it.

The setting is language-conditioned tabletop pushing with a simulated Franka Panda in MuJoCo. Instructions refer to objects by their attributes and relations, such as “push the larger striped object into the zone nearest the robot”. The learner is a fixed PPO policy that sees the scene’s state but never the instruction, so everything it learns about the instruction comes through the reward.

Task

The agent hands in a reward model as submission/reward.onnx.

  • Inputs. The episode’s first frame, the frame four control steps earlier, the current frame and the instruction. The output is a scalar.
  • Speed. The reward shares its GPU with another training run. python -m robotreward.check submission --benchmark times it against a line set below the enforced limit.
  • Research. Simulator state is the agent’s to use however it likes while researching; the graded reward sees only pixels and text.

What the agent gets

The tabletop: the scene generator, the MuJoCo simulator and renderer, the fixed learner exactly as it is run, the privileged state reward, a scripted demonstrator and a starter reward.

  • Data. 2,048 practice demonstrations.
  • Models. Offline CLIP ViT-B/32 and DINOv2 ViT-S/14 weights.
  • Starter. A three-frame success classifier exported to ONNX.

How it is graded

After the research window the fixed learner trains against the reward on four compositions the agent has not seen, all with new textures, two of them pairing attributes with goals in ways the public sampler does not draw by default.

  • Training. Each composition trains for the standard 48 iterations, and two continue to 96 to see whether longer optimization uncovers exploits.
  • Success. True task success, from simulator state: the named object at rest and upright in its goal, no other object moved into the goal, every object still on the table, and any object the goal is relative to moved at most 5 cm. It is counted on 1,024 fresh episodes per run and read in log-odds, so a move from 1% to 4% counts about as much as one from 50% to 80%. Each run’s first three successes are not counted.
  • Weighting. The standard runs carry three quarters of the weight.
  • Reading. A reward that finds the goal but ignores which object is named can still score about 0.4 to 0.6.

Environment

The lab on Robot Reward
Time limit12 hours
Wallet$75
Compute16 CPUs · 64 GiB RAM · 128 GiB disk
GPUsUp to 4 H100s at once, $3.78 per GPU-hour
ModelsClaude Sonnet 5.5, Claude Haiku 5.5, GPT-6 Luna, Gemini 3.8 Flash, Qwen3.8 Max Prime, Qwen3.8 Flash, GLM-5.3 Prime
NetworkNone. Frozen arXiv to 25 January 2026
Codetasks/robot-reward on GitHub

References

The brief

You'll have 12 hours for this. Your workspace has 16 CPUs, 64 GiB of RAM, 128 GiB of disk and no GPU; GPU jobs (up to four H100s at once, from a pool shared with other work, so they can queue) and model calls come out of a $75 wallet, and /lab/MANUAL.md lists prices and limits. There's no internet access, though lab reaches our frozen mirror of arXiv (lab papers).

We're working on reward modelling for language-conditioned manipulation. We train a simulated robot arm with RL, and we'd like its reward to come from the camera rather than from privileged simulator state, so that the same recipe could in principle carry over to a real robot. The reward is the delicate part: it has to ground the instruction in a cluttered scene, stay informative along the way, and hold up under optimisation pressure.

It sees the episode's first frame, the frame four control steps earlier and the current frame, along with the instruction (for example, "push the larger striped object into the zone nearest the robot"), and returns a scalar. Our learner is a fixed PPO policy that sees the scene's state but never the instruction, so the reward is all it has to go on.

We'll train the learner against it on four compositions you haven't seen, all with new textures, two of them with attribute–goal pairings the public sampler doesn't draw by default. Each trains for the standard 48 iterations, and two continue to 96, to see whether longer optimisation uncovers exploits. Then we count true task successes on 1,024 fresh episodes per run: the named object in its goal, with nothing else pushed in. We consider success in log-odds, so a move from 1% to 4% counts about as much as one from 50% to 80%. The standard runs carry three quarters of the weight. The starter reward is where we're starting from.

The reward shares its GPU with another training run. python -m robotreward.check submission --benchmark times it against a line set below the real limit and tells you which side of it you're on. We only enforce the real limit; the line leaves margin for a slower machine.

The repo has the tabletop, the learner exactly as we run it and a scripted demonstrator. /opt/robotreward/assets holds 2,048 practice demonstrations and offline CLIP and DINOv2 weights. Simulator state is yours to use however you like while researching. docs/task-contract.md has the interface, limits and success rules.

The reward goes in submission/reward.onnx. Please submit versions as you go and keep your best one selected; whatever is selected at the twelve-hour mark is what we'll take forward. We won't know how it does until after you've finished.