12Method taskLearning from expert solutions when pass@32 = 0
pass@32 = 0
Develop a training method after which Qwen3-4B-Base solves contest problems on which it has no correct sample to learn from.
Results
Watch the best run| # | Model | Score | Lab spend | Inference | GPU hours | Turns | Final pick |
|---|---|---|---|---|---|---|---|
| 1 | 0.294 | $70 | $62 | 18.0 | 366 | 11.3 h | |
| 2 | 0.252 | $70 | $34 | 18.1 | 338 | 10.9 h | |
| 3 | 0.094 | $49 | $78 | 12.5 | 540 | 9.6 h | |
| 4 | 0.071 | $76 | $42 | 20.1 | 243 | 8.7 h | |
| 5 | 0.000 | $66 | $140 | 17.1 | 269 | 7.6 h |
One run per model, from the evaluation of 14 November 2026. Lab spend is what a run paid from its $90 wallet for compute and model calls; inference is the agent’s own model calls at list prices; the final pick is the hour at which it chose what was graded. Every task’s anchors, scale and noise are in the paper.
What the runs showed
The hardest public task: averaged over the five models it scored 0.14, and the best run, Fable 5.2’s 0.294, closed 29% of the gap.
Grok 4.7 noted that a decision rested on one hard-tail draw and repeated the draw with new seeds before keeping the change.
Background
Reinforcement learning from verifiable rewards learns only from problems a model sometimes solves. On a contest problem with pass@128 = 0 there is no correct rollout and so no positive reward, and the problem contributes nothing. Worked solutions exist for many such problems, but imitating a terse published solution is not obviously the same as learning to find one.
The task asks what does transfer. The method runs from Qwen3-4B-Base on problems the base model never solves, and the trained model is tested on newer problems that the base model also never solves.
Task
The agent hands in a method package: code and configuration only, with no data or weights. The method takes the base model, training problems and a compute budget, and returns a LoRA adapter or full weights.
- Inputs. 300 fresh math contest problems from before 2025 that the base model fails in 128 samples. Each has its final answer, and 200 have the contest’s published solution.
- Budget. One hour on two H100s, offline.
What the agent gets
The exact prompts and decoding used in testing, sampling with vLLM, the math answer checker, the checks for a returned model and a method package, and a harness that runs a method end to end as the final run does.
- Data. Earlier HMMT problems with their published solutions, and the base model’s correct samples out of 128 on HMMT and AIME problems.
- Starter. A method that returns no model, with SFT and GRPO recipes.
- Filter. A tool that builds never-solved problem sets the way the held-out problems were built.
How it is graded
After the research window the method runs once from the base model, and the returned model is tested on 199 math problems from 2025 and 2026 contests on which none of 1,024 base-model samples was correct.
- Measure. pass@32 from 512 samples of at most 4,096 tokens, corrected for answer guessing against decoy answers.
- Floor. What a model reaches by answering with no reasoning at all does not count as progress.
Environment
| Time limit | 12 hours |
|---|---|
| Wallet | $90 |
| Compute | 16 CPUs · 32 GiB RAM · 80 GiB disk |
| GPUs | Up to 4 H100s at once, $3.78 per GPU-hour |
| Models | Claude Sonnet 5.5, Claude Haiku 5.5, GPT-6 Luna, Gemini 3.8 Flash, Qwen3.8 Max Prime, Qwen3.8 Flash, GLM-5.3 Prime |
| Network | None. Frozen arXiv to 25 January 2026, and a mirror of Hugging Face datasets |
| Code | tasks/zero-pass on GitHub |
References
- Qwen3 technical report (Qwen Team, 2025): Qwen3-4B-Base, the model the method trains.
- Omni-MATH: a universal olympiad level mathematic benchmark for large language models (Gao et al., 2024): the training and practice problems with their solutions.
- Beyond benchmarks: MathArena as an evaluation platform for mathematics with LLMs (Dekoninck et al., 2026): 109 of the 199 test problems; the other 90 come from the contests’ own papers.
- Math-Verify (Hugging Face): the answer checker.
The brief
You'll have 12 hours for this. Your workspace has 16 CPUs, 32 GiB of RAM, 80 GiB of disk and no GPU; GPU jobs (up to four H100s at once, from a pool shared with other work, so they can queue) and model calls come out of a $90 wallet, and /lab/MANUAL.md lists prices and limits. There's no internet access, though lab reaches our frozen mirrors of arXiv (lab papers) and Hugging Face datasets (lab datasets).
We've been looking at what a model can learn from problems where RL gives it no signal at all. Our small model regularly meets contest problems with pass@128 = 0: with no correct rollout there's no positive reward, so those problems contribute nothing. We do have worked solutions for many of them, but imitating a terse published solution isn't obviously the same as learning to find one, and we're not sure what does transfer.
We'd like a training method after which Qwen3-4B-Base solves problems like these.
We'll run your method once, offline, from the base model, on two H100s for one hour, with 300 fresh pre-2025 contest problems the base model fails in 128 samples. Each has its final answer, and 200 have the contest's published solution. The method returns a LoRA adapter or full weights.
We then test that model on problems from 2025 and 2026 contests where none of 1,024 base-model samples, at our prompts and temperatures, was correct: pass@32 from 512 samples of at most 4,096 tokens, corrected for answer guessing. The base model is where we're starting from, and what a model can reach by answering with no reasoning at all doesn't count as progress.
The repo has the sampler and answer checker we use, a starter that returns no model, with SFT and GRPO recipes, earlier HMMT problems with solutions, the base model's correct samples out of 128 on HMMT and AIME problems, and docs/ for the method format and the final run.
The method package goes in submission/: code and configuration only, with no data or weights. Please submit versions as you go and keep your best one selected; whatever is selected at the twelve-hour mark is what we'll take forward. We won't know how it does until after you've finished.