12Method taskLearning from expert solutions when pass@32 = 0

pass@32 = 0

Develop a training method after which Qwen3-4B-Base solves contest problems on which it has no correct sample to learn from.

Every model’s run on pass@32 = 0, ranked by score
#ModelScoreLab spendInferenceGPU hoursTurnsFinal pick
1Fable 5.2Fable0.294$70$6218.036611.3 h
2Opus 5.5Opus0.252$70$3418.133810.9 h
3Grok 4.7Grok0.094$49$7812.55409.6 h
4CarbonCarbon0.071$76$4220.12438.7 h
5AstraAstra0.000$66$14017.12697.6 h

One run per model, from the evaluation of 14 November 2026. Lab spend is what a run paid from its $90 wallet for compute and model calls; inference is the agent’s own model calls at list prices; the final pick is the hour at which it chose what was graded. Every task’s anchors, scale and noise are in the paper.

What the runs showed

The hardest public task: averaged over the five models it scored 0.14, and the best run, Fable 5.2’s 0.294, closed 29% of the gap.

Grok 4.7 noted that a decision rested on one hard-tail draw and repeated the draw with new seeds before keeping the change.

Background

Reinforcement learning from verifiable rewards learns only from problems a model sometimes solves. On a contest problem with pass@128 = 0 there is no correct rollout and so no positive reward, and the problem contributes nothing. Worked solutions exist for many such problems, but imitating a terse published solution is not obviously the same as learning to find one.

The task asks what does transfer. The method runs from Qwen3-4B-Base on problems the base model never solves, and the trained model is tested on newer problems that the base model also never solves.

Task

The agent hands in a method package: code and configuration only, with no data or weights. The method takes the base model, training problems and a compute budget, and returns a LoRA adapter or full weights.

  • Inputs. 300 fresh math contest problems from before 2025 that the base model fails in 128 samples. Each has its final answer, and 200 have the contest’s published solution.
  • Budget. One hour on two H100s, offline.

What the agent gets

The exact prompts and decoding used in testing, sampling with vLLM, the math answer checker, the checks for a returned model and a method package, and a harness that runs a method end to end as the final run does.

  • Data. Earlier HMMT problems with their published solutions, and the base model’s correct samples out of 128 on HMMT and AIME problems.
  • Starter. A method that returns no model, with SFT and GRPO recipes.
  • Filter. A tool that builds never-solved problem sets the way the held-out problems were built.

How it is graded

After the research window the method runs once from the base model, and the returned model is tested on 199 math problems from 2025 and 2026 contests on which none of 1,024 base-model samples was correct.

  • Measure. pass@32 from 512 samples of at most 4,096 tokens, corrected for answer guessing against decoy answers.
  • Floor. What a model reaches by answering with no reasoning at all does not count as progress.

Environment

The lab on pass@32 = 0
Time limit12 hours
Wallet$90
Compute16 CPUs · 32 GiB RAM · 80 GiB disk
GPUsUp to 4 H100s at once, $3.78 per GPU-hour
ModelsClaude Sonnet 5.5, Claude Haiku 5.5, GPT-6 Luna, Gemini 3.8 Flash, Qwen3.8 Max Prime, Qwen3.8 Flash, GLM-5.3 Prime
NetworkNone. Frozen arXiv to 25 January 2026, and a mirror of Hugging Face datasets
Codetasks/zero-pass on GitHub

References

The brief

You'll have 12 hours for this. Your workspace has 16 CPUs, 32 GiB of RAM, 80 GiB of disk and no GPU; GPU jobs (up to four H100s at once, from a pool shared with other work, so they can queue) and model calls come out of a $90 wallet, and /lab/MANUAL.md lists prices and limits. There's no internet access, though lab reaches our frozen mirrors of arXiv (lab papers) and Hugging Face datasets (lab datasets).

We've been looking at what a model can learn from problems where RL gives it no signal at all. Our small model regularly meets contest problems with pass@128 = 0: with no correct rollout there's no positive reward, so those problems contribute nothing. We do have worked solutions for many of them, but imitating a terse published solution isn't obviously the same as learning to find one, and we're not sure what does transfer.

We'd like a training method after which Qwen3-4B-Base solves problems like these.

We'll run your method once, offline, from the base model, on two H100s for one hour, with 300 fresh pre-2025 contest problems the base model fails in 128 samples. Each has its final answer, and 200 have the contest's published solution. The method returns a LoRA adapter or full weights.

We then test that model on problems from 2025 and 2026 contests where none of 1,024 base-model samples, at our prompts and temperatures, was correct: pass@32 from 512 samples of at most 4,096 tokens, corrected for answer guessing. The base model is where we're starting from, and what a model can reach by answering with no reasoning at all doesn't count as progress.

The repo has the sampler and answer checker we use, a starter that returns no model, with SFT and GRPO recipes, earlier HMMT problems with solutions, the base model's correct samples out of 128 on HMMT and AIME problems, and docs/ for the method format and the final run.

The method package goes in submission/: code and configuration only, with no data or weights. Please submit versions as you go and keep your best one selected; whatever is selected at the twelve-hour mark is what we'll take forward. We won't know how it does until after you've finished.