14Judgment taskPretraining language models without backpropagation
Forward
Pretrain a fixed 37.7M-parameter transformer from scratch with forward passes alone, bringing held-out loss as close to backprop’s as it goes.
Results
Watch the best run| # | Model | Score | Lab spend | Inference | GPU hours | Turns | Final pick |
|---|---|---|---|---|---|---|---|
| 1 | 0.624 | $67 | $54 | 17.2 | 348 | 9.4 h | |
| 2 | 0.621 | $74 | $77 | 18.1 | 365 | 8.7 h | |
| 3 | 0.545 | $75 | $60 | 19.4 | 265 | 9.8 h | |
| 4 | 0.503 | $72 | $200 | 18.7 | 319 | 8.3 h | |
| 5 | 0.125 | $75 | $81 | 18.9 | 633 | 11.5 h |
One run per model, from the evaluation of 14 November 2026. Lab spend is what a run paid from its $75 wallet for compute and model calls; inference is the agent’s own model calls at list prices; the final pick is the hour at which it chose what was graded. Every task’s anchors, scale and noise are in the paper.
What the runs showed
Opus 5.5 had the best run, 0.624, just ahead of Fable 5.2 at 0.621; Carbon and Astra followed at 0.545 and 0.503. Opus 5.5’s and Carbon’s full-scale development losses predicted their grades to within 0.01.
In the paper’s restart ablation, three fresh agents each given the best learner all found that it gives no credit through attention keys and values; that was the only rise beyond grader noise in the ablation.
Background
Learning only from forward evaluations of the loss needs no stored activations or backward pass and parallelizes naturally. But a learning signal estimated from loss measurements is noisy, and the noise tends to grow with the number of parameters, so backprop-free methods have usually trailed backprop by a wide margin at any real size.
The task measures how far that gap can be closed. A small fixed transformer of 37,749,776 parameters and eight layers is pretrained from scratch on about 10M tokens of public FineWeb, using only forward passes, with every learning signal originating in a measured loss.
Task
The agent hands in a learner that pretrains the fixed model and exports its complete weights.
- Rules. No autodiff or backward pass in the learner, and at most one local nonlinear estimation step per measured signal, as
docs/rules.mddraws the line. Backprop is allowed while researching. - Review. Two automated reviewers, from OpenAI and Anthropic, read the learner’s source with comments stripped, and a third, from Z.ai, decides if they disagree. A learner that crosses the line cannot be used.
- Data. The learner consumes only its current batch, without replaying earlier ones, and never sees the held-out text.
What the agent gets
The pinned model and tokenizer with no pretrained weights, the runner and its worker processes, and the held-out evaluator the final run uses.
- Data. 4,880 rows of 2,049 tokens of training text and 544 rows for development, frozen FineWeb under the pinned tokenizer.
- Starter. Plain weight perturbation, as an example of the interface.
- Comparison. An untuned backprop script for development, distinct from the tuned reference the score is read against.
- Scale. Research jobs use at most four H100s at once, while the final run uses eight, so what holds up at eight is part of the work.
How it is graded
After the research window the learner trains the model from a fresh seed on about 10M new tokens, single epoch, on eight H100s for at most an hour.
- Measure. Next-token cross-entropy on about 4M held-out tokens, read against well-tuned backprop as how many tokens backprop would need to match it, on a log scale.
- Seeds. A second run follows when the first charges at most two H100-hours, and the two losses are averaged; it adds a second seed only when the learner uses the run seed it is given.
- Scope. The rules bound how credit may travel, not how closely a legal learner can approximate backpropagation, so a high score measures sample efficiency under the rules.
Environment
| Time limit | 12 hours |
|---|---|
| Wallet | $75 |
| Compute | 16 CPUs · 32 GiB RAM · 48 GiB disk |
| GPUs | Up to 4 H100s at once, $3.78 per GPU-hour |
| Models | Claude Haiku 5.5, GPT-6 Luna, Gemini 3.8 Flash, Qwen3.8 Max Prime, Qwen3.8 Flash |
| Network | None. Frozen arXiv to 25 January 2026, and a mirror of Hugging Face datasets |
| Code | tasks/forward on GitHub |
References
- The FineWeb datasets (Penedo et al., 2024, ODC-By 1.0): the training, development and held-out text.
- qlabs-eng/noise (MIT): the fixed model and tokenizer.
- Fine-tuning language models with just forward passes (Malladi et al., NeurIPS 2023): related work (MeZO).
- The forward-forward algorithm: some preliminary investigations (Hinton, 2022): related work.
The brief
You'll have 12 hours for this. Your workspace has 16 CPUs, 32 GiB of RAM, 48 GiB of disk and no GPU; GPU jobs (up to four H100s at once, from a pool shared with other work, so they can queue) and model calls come out of a $75 wallet, and /lab/MANUAL.md lists prices and limits. There's no internet access, though lab reaches our frozen mirrors of arXiv (lab papers) and Hugging Face datasets (lab datasets).
We're curious how far pretraining can get without backpropagation. Learning only from forward evaluations of the loss needs no stored activations or backward pass and parallelises naturally, but a signal estimated from loss measurements is noisy, and the noise tends to grow with the number of parameters. Backprop-free methods have usually trailed backprop by a wide margin at any real size. We don't think anyone knows how close the gap can be brought, and that's what we'd like to find out.
Here's a small fixed transformer (37,749,776 parameters, eight layers) and about 10M tokens of public FineWeb. We'd like a learner that pretrains it from scratch using only forward passes, every learning signal originating in a measured loss, and brings held-out loss as low as it can. The boundary matters, and docs/rules.md draws it precisely: no autodiff or backward pass in the learner, and at most one local nonlinear estimation step per measured signal. Backprop is fine while researching. Two reviewers will read the learner against the rules, with a third deciding if they disagree; one that crosses the line can't be used.
When you're done, we'll train it from a fresh seed on about 10M new tokens, single epoch, on eight H100s for at most an hour, and read next-token cross-entropy on about 4M held-out tokens against well-tuned backprop, as how many tokens backprop would need to match it, on a log scale. The repo's backprop script is only a starting point, not that baseline. Your own jobs can use at most four GPUs, so what holds up at eight is part of the work.
The learner goes in submission/; docs/ has the interface and every limit. Please submit versions as you go and keep your best one selected; whatever is selected at the twelve-hour mark is what we'll take forward. We won't know how it does until after you've finished.