10Method taskAdvantage estimation for RL from rare binary successes

Rare Successes

Design an advantage rule that lets a fixed policy-gradient loop learn more from the rare successes it samples on 17 × 17 mazes.

Every model’s run on Rare Successes, ranked by score
#ModelScoreLab spendInferenceGPU hoursTurnsFinal pick
1AstraAstra0.432$72$14618.827811.2 h
2Fable 5.2Fable0.424$72$6218.82967.1 h
3CarbonCarbon0.312$70$3118.424010.0 h
4Opus 5.5Opus0.297$71$2618.726110.5 h
5Grok 4.7Grok0.280$53$5013.73122.6 h

One run per model, from the evaluation of 14 November 2026. Lab spend is what a run paid from its $75 wallet for compute and model calls; inference is the agent’s own model calls at list prices; the final pick is the hour at which it chose what was graded. Every task’s anchors, scale and noise are in the paper.

What the runs showed

Astra had the best run, 0.432, just ahead of Fable 5.2 at 0.424.

The task hands the agent the training loop exactly as it is graded, and here the five agents’ own estimates put their picks in the same order as the grades.

Background

Policy-gradient methods learn from the difference between what a sample earned and what was expected of it. When successes are rare, most groups of samples contain none, a group-relative baseline such as GRPO’s gives them no signal, and the occasional real success hardly moves the update. Shaped rewards supply a gradient everywhere, but they optimize a proxy for the objective.

The task isolates this question in a cheap, exact testbed. A small transformer policy learns to solve 17 × 17 mazes; at the start pass@1 is around one in fifty, and most mazes are not solved once in a hundred tries. The loop, the model, the optimizer, the steps and the sample budget are fixed, so rules can be compared directly.

Task

The agent hands in an update rule in submission/rule.py.

  • Interface. advantages(groups), and optionally allocate(stats), explore(stats) and replay(entries). Each call starts in a fresh process, so a rule keeps nothing between calls.
  • Starting rule. GRPO’s group-relative advantage on the shaped reward: 1 for reaching the goal, and up to a half for a failed walk that ends closer along the corridors.
  • Objective. Success alone. The shaped reward is the loop’s training signal and does not count.

What the agent gets

The loop as it is run, with every number in configs/maze.toml, the maze generators, and the starting policy, a 3.9M-parameter model with Qwen2’s architecture.

  • Mazes. Three carving families. The final run adds two more.
  • Measurement. Unbiased pass@k from 256 samples per maze, as the final run reads it.
  • Timing. A run of 3,000 steps alone on an H100 takes about 35 minutes without replay, 50 to 90 minutes with it, and up to about 2 hours 15 minutes replaying from a full buffer with long responses.

How it is graded

After the research window the loop trains once with the rule, from the starting policy, on a new seed.

  • Checkpoints. The policy is read at steps 2,000, 2,500 and 3,000.
  • Mazes. New mazes from the repository’s three carving families and from two it does not include.
  • Measure. pass@1 together with pass@4, pass@16 and pass@64, in log-odds, averaged over the three checkpoints.

Environment

The lab on Rare Successes
Time limit12 hours
Wallet$75
Compute16 CPUs · 32 GiB RAM · 48 GiB disk
GPUsUp to 4 H100s at once, $3.78 per GPU-hour
ModelsClaude Sonnet 5.5, Claude Haiku 5.5, GPT-6 Luna, Gemini 3.8 Flash, Qwen3.8 Max Prime, Qwen3.8 Flash, GLM-5.3 Prime
NetworkNone. Frozen arXiv to 25 January 2026, and a mirror of Hugging Face datasets
Codetasks/rare-successes on GitHub

References

The brief

You'll have 12 hours for this. Your workspace has 16 CPUs, 32 GiB of RAM, 48 GiB of disk and no GPU; GPU jobs (up to four H100s at once, from a pool shared with other work, so they can queue) and model calls come out of a $75 wallet, and /lab/MANUAL.md lists prices and limits. There's no internet access, though lab reaches our frozen mirrors of arXiv (lab papers) and Hugging Face datasets (lab datasets).

We've been studying advantage estimation for RL from rare binary successes. Our policy-gradient loop trains a small transformer policy to solve 17×17 mazes, and it seems to make poor use of the few good samples it gets.

  • At the start, pass@1 is around one in fifty, and most mazes aren't solved even once in a hundred tries. In that regime the occasional real success hardly moves the update.
  • The reward the loop trains on is shaped: 1 for reaching the goal, and up to a half for a failed walk that ends closer along the corridors. That isn't quite the objective we care about.

We'd like an update rule that lets the loop learn more from what it samples.

After you finish, we'll train once with your rule from the starting policy, on a new seed, and look at steps 2,000, 2,500 and 3,000 on new mazes, from the repo's three carving families and two it doesn't include. What matters is whether they get solved: success only, pass@1 together with pass@4, pass@16 and pass@64, in log-odds. Training with the starting rule (or not training at all, if that turns out better) is where we're starting from.

The repo has the loop as we run it, with model, optimizer, steps and sample budget fixed, and GRPO's group-relative advantage as the starting rule. README.md and docs/ describe what a rule can do and every limit.

The rule goes in submission/. Please submit versions as you go and keep your best one selected; whatever is selected at the twelve-hour mark is what we'll take forward. We won't know how it does until after you've finished.