08Method taskContinual learning without catastrophic forgetting
Rule Worlds
Build a learner that makes a small language model learn new rule-rewriting games by practice and keep what it learned from earlier ones.
Results
Watch the best run| # | Model | Score | Lab spend | Inference | GPU hours | Turns | Final pick |
|---|---|---|---|---|---|---|---|
| 1 | 0.574 | $66 | $71 | 17.2 | 369 | 10.6 h | |
| 2 | 0.526 | $52 | $51 | 13.5 | 429 | 8.9 h | |
| 3 | 0.473 | $73 | $255 | 17.2 | 417 | 9.7 h | |
| 4 | 0.312 | $65 | $91 | 17.1 | 331 | 11.6 h | |
| 5 | 0.158 | $25 | $78 | 6.2 | 575 | 9.3 h |
One run per model, from the evaluation of 14 November 2026. Lab spend is what a run paid from its $75 wallet for compute and model calls; inference is the agent’s own model calls at list prices; the final pick is the hour at which it chose what was graded. Every task’s anchors, scale and noise are in the paper.
What the runs showed
Fable 5.2 led at 0.574, with Opus 5.5 at 0.526 and Astra at 0.473; Carbon and Grok 4.7 finished at 0.312 and 0.158.
Astra had GPT-6 Luna write rationales for solver-verified solutions to use as training targets.
Background
People learn an unfamiliar game by practicing until its logic makes sense, then carry what they learned into the next one. Language models adapt poorly in this way: fine-tuning on a new task tends to overwrite what came before, and what is learned in context is lost with the context.
Rule worlds make the problem concrete. A level is a grid of objects whose rules are sentences of word tiles on the board, such as WALL IS STOP, so pushing tiles rewrites the rules in the middle of a level, in the style of Baba Is You. The task asks for a learner that makes a small language model good at learning new rule worlds by practice, and at keeping what it learned.
Task
The agent hands in a learner package: a program that runs practice, and optionally a starting model with Qwen3.5-4B’s architecture and a notes file.
- Package.
learner.jsonnames the command to start, the starting model (a LoRA adapter for Qwen3.5-4B or full weights) and the starting notes, in at most 20 GB. - Practice. The program plays and trains however it chooses, on one H100 with no network, and exports the model and notes it wants to carry forward.
- Carry-over. Only the model’s weights and up to 256 KiB of notes reach the exam, which the model plays alone.
What the agent gets
The rule-world engine and its public games, the exam harness and a starter learner, with Qwen3.5-4B staged offline.
- Software. Full and LoRA fine-tuning with PEFT or Miles, and vLLM and SGLang for sampling, all offline.
- Rehearsal. Local tools that run a learner’s practice and exam the way the final run does, and the checks applied to every package.
How it is graded
After the research window the learner runs unattended on one H100 with no network. It practices two games it has never seen, for an hour each, in an order it is not told, and its final export then takes one exam.
- Held-out games. Every word in them is made up except IS, each has one or two properties no public game has, and no public game uses their puzzle designs.
- Exam. 160 fresh levels of each game, played by the model alone and scored by solve rate. A starting model prepared in advance counts the same as what practice teaches it.
- Weighting. The two games count equally, and so do three kinds of level within each: levels built around a new property, short levels of 4 to 8 moves, and longer ones.
- Reported alongside. What practice added and what the first game kept, neither of which changes the measure.
Environment
| Time limit | 12 hours |
|---|---|
| Wallet | $75 |
| Compute | 8 CPUs · 32 GiB RAM · 128 GiB disk |
| GPUs | Up to 4 H100s at once, $3.78 per GPU-hour |
| Hosted training | Tinker |
| Models | Claude Sonnet 5.5, Claude Haiku 5.5, GPT-6 Luna, Gemini 3.8 Flash, Qwen3.8 Max Prime, Qwen3.8 Flash, GLM-5.3 Prime |
| Network | None. Frozen arXiv to 25 January 2026, without Baba Is AI, DiG-bench and the papers citing them, and a mirror of Hugging Face datasets |
| Code | tasks/rule-worlds on GitHub |
References
- Baba Is You (Arvi Teikari (Hempuli), 2019): the game the rule worlds follow; they are original games with their own levels, words and properties.
- Qwen3.5-4B (the Qwen team): the architecture every starting model shares.
- Overcoming catastrophic forgetting in neural networks (Kirkpatrick et al., PNAS 2017): related work.
The brief
You'll have 12 hours for this. Your workspace has 8 CPUs, 32 GiB of RAM, 128 GiB of disk and no GPU; GPU jobs (up to four H100s at once, from a pool shared with other work, so they can queue), Tinker and model calls come out of a $75 wallet, and /lab/MANUAL.md lists prices and limits. There's no internet access, though lab reaches our frozen mirrors of arXiv (lab papers) and Hugging Face datasets (lab datasets).
We're interested in continual learning without catastrophic forgetting, in a form that resembles how people pick up games: practise something unfamiliar until it makes sense, then carry what was learned into the next thing. Language models don't seem to do this well yet.
The games here are rule worlds: grids of objects whose rules are sentences of word tiles on the board, like WALL IS STOP, so that pushing tiles rewrites the rules mid-level. We'd like a learner that makes a small language model good at learning them by practice. The starter learner is where we're starting from.
After you're done, we'll run your learner unattended on one H100, with no network. It practises two games it has never seen, an hour each, in an order it isn't told. Every word in them is made up except IS, each has one or two properties no public game has, and no public game uses their puzzle designs. Afterwards, only the model's weights and up to 256 KiB of notes carry over to one exam: 160 fresh levels of each game, played by the model alone.
What we look at is the solve rate. The games count equally, and so do three kinds of level within each: those built around a new property, short ones of 4 to 8 moves, and longer ones. Only that exam counts, so a starting model you prepare in advance counts the same as what practice teaches it. We report separately what practice added and what game 1 kept, but they don't change the measure.
The README and docs/ describe the interface, formats and limits.
The learner package goes in submission/: a program and, if you want them, a starting model with Qwen3.5-4B's architecture and notes. Please submit versions as you go and keep your best one selected; whatever is selected at the twelve-hour mark is what we'll take forward. We won't know how it does until after you've finished.