11Method taskCredit assignment under sparse terminal rewards

Credit Assignment

Develop a credit-assignment rule for an RL loop in which a small language model plays text dungeons and only reaching the vault pays.

Every model’s run on Credit Assignment, ranked by score
#ModelScoreLab spendInferenceGPU hoursTurnsFinal pick
1CarbonCarbon0.663$77$6919.735110.0 h
2Opus 5.5Opus0.566$50$4512.83527.3 h
3Fable 5.2Fable0.301$62$9515.742510.9 h
4Grok 4.7Grok–0.158$41$15010.39128.5 h
5AstraAstra–0.298$60$18215.32729.6 h

One run per model, from the evaluation of 14 November 2026. Lab spend is what a run paid from its $100 wallet for compute and model calls; inference is the agent’s own model calls at list prices; the final pick is the hour at which it chose what was graded. Every task’s anchors, scale and noise are in the paper.

What the runs showed

Carbon had the best run, 0.663, and Opus 5.5 was second at 0.566. Carbon wrote three pairs of generators to stand in for the two hidden ones and compared its rule with the starter’s on all of them.

Astra’s method finished at –0.30, the only final below its task’s starting reference by more than the task’s noise.

Background

In long episodes with a single terminal reward, the moves that decide the outcome often come long before the end. A rule that gives every step its episode’s outcome, baselined against other rollouts of the same problem, spreads the blame for a failure almost evenly over every step, and the policy learns slowly which earlier actions matter.

Here a small language model learns with RL to play generated text dungeons, and only reaching the vault pays. The dungeons that count come from generators the agent never sees. In one, whether an action is harmful depends on when it is taken; in the other, moves that are harmless one at a time can together leave the vault unreachable.

Task

The agent hands in a credit rule, which decides how much credit each step of each episode gets. The loop around it is fixed.

  • Scope. A rule may branch continuations, train a helper model, or shape credit with signals of its own from anything an episode shows, beyond the terminal reward.
  • Cost. Everything a rule does is charged to the loop’s fixed compute meter.
  • Starting rule. Each step gets its episode’s outcome relative to the other episodes of its dungeon.

What the agent gets

The dungeon engine with three public generators and an exact solver of optimal values, and the training loop exactly as the final run uses it.

  • Loop. The sampler, rollouts and continuations, the rule’s process, the helper model, the update and the budget meter, on one 80 GB H100. A run at the default budget takes about 12 minutes with a light rule.
  • Model. A starting policy fine-tuned from Qwen2.5-0.5B-Instruct on solver demonstrations from the public generators.
  • Generators. Plugins load further generators into training and evaluation, so the agent can build dungeons of its own.

How it is graded

After the research window the loop trains eight times with the rule, each from the starting model and its own seed, on dungeons drawn half from the public generators and half from the two unseen ones, and every policy then plays fresh dungeons from the two unseen generators.

  • Cells. Each generator at its usual horizon and at the long one, whose dungeons take about twice the steps. The four cells count equally.
  • Measure. Success is pooled over the eight policies and compared in log-odds, and a cell can count only so far below the current rule.
  • Difficulty. Under the current rule, in many of these dungeons none of the eight policies succeeds.

Environment

The lab on Credit Assignment
Time limit12 hours
Wallet$100
Compute8 CPUs · 16 GiB RAM · 48 GiB disk
GPUsUp to 4 H100s at once, $3.78 per GPU-hour
ModelsClaude Sonnet 5.5, Claude Haiku 5.5, GPT-6 Luna, Gemini 3.8 Flash, Qwen3.8 Max Prime, Qwen3.8 Flash, GLM-5.3 Prime
NetworkNone. Frozen arXiv to 30 June 2024, and a mirror of Hugging Face datasets
Codetasks/credit-assignment on GitHub

References

The brief

You'll have 12 hours for this. Your workspace has 8 CPUs, 16 GiB of RAM, 48 GiB of disk and no GPU; GPU jobs (up to four H100s at once, from a pool shared with other work, so they can queue) and model calls come out of a $100 wallet, and /lab/MANUAL.md lists prices and limits. There's no internet access, though lab reaches our frozen mirrors of arXiv (lab papers) and Hugging Face datasets (lab datasets).

We've been working on credit assignment under sparse rewards: a small language model learns, with RL, to play text dungeons, and only reaching the vault pays, though the moves that decide an episode often come long before the end. Our current rule gives each step its episode's outcome, baselined against other rollouts of the same dungeon, so a failed trajectory spreads the blame almost evenly. We'd like a rule that teaches the policy what actually gets it to the vault.

The dungeons we care about come from two generators you won't have, which also supply half of training. They share the engine and print their rules. In one, whether an action is harmful depends on when it's taken; in the other, moves that are harmless individually can together leave the vault unreachable. Actions that never matter in the public dungeons can be decisive there.

We'll train with your rule eight times from the starting model, then roll out on fresh dungeons from both generators, at their usual horizon and at the long one, whose dungeons take about twice the steps. We look at how often the vault is reached, each generator and horizon weighted equally, in log-odds. Under the current rule that's uncommon: in many dungeons, none of the eight plays succeeds.

A rule may branch continuations, train a helper model, or shape credit with signals of its own from anything an episode shows; it isn't limited to the terminal reward. All of it is charged to the loop's fixed compute meter (README.md, docs/).

The rule goes in submission/. Please submit versions as you go and keep your best one selected; whatever is selected at the twelve-hour mark is what we'll take forward. We won't know how it does until after you've finished.