13Judgment taskInternalizing agent skills through context distillation
Off Book
Distill a library of agent skills into Nemotron 3.5 Super’s weights, with a method graded once on the 120B model the agent can sample but never train.
Results
Watch the best run| # | Model | Score | Lab spend | Inference | Turns | Final pick |
|---|---|---|---|---|---|---|
| 1 | 0.645 | $125 | $107 | 547 | 10.8 h | |
| 2 | 0.162 | $122 | $166 | 298 | 10.6 h | |
| 3 | 0.118 | $124 | $54 | 434 | 8.4 h | |
| 3 | 0.118 | $117 | $115 | 292 | 11.5 h | |
| 5 | –0.145 | $60 | $75 | 562 | 10.9 h |
One run per model, from the evaluation of 14 November 2026. Lab spend is what a run paid from its $125 wallet for compute and model calls; inference is the agent’s own model calls at list prices; the final pick is the hour at which it chose what was graded. Every task’s anchors, scale and noise are in the paper.
What the runs showed
Fable 5.2’s method scored 0.645 and no other run passed 0.17, one of the evaluation’s two most lopsided results. Averaged over the five models, Off Book scored 0.21, the second lowest of the public tasks after pass@32 = 0.
Before filtering its training data with a model judge, Fable 5.2 checked that judge against the development tasks’ own verifiers.
Background
Agents increasingly carry skill libraries: folders of expert procedure, often with scripts, listed in the system prompt and read when a task calls for one. Curated skills make agents markedly better at specialist work, but every skill costs context on every call, it helps only when the right one is noticed and loaded, and the library has to travel with the agent. The task asks whether that benefit can be moved into the weights, though skills are written to be read rather than trained on, the work they support is long-horizon and varied, and whatever is pushed into the weights risks displacing general capability.
Off Book is possible because of Tinker, the hosted fine-tuning service from Thinking Machines Lab. The workspace has no GPUs, yet the agent trains Nemotron 3.5 Lightning, a 30B-A3B mixture-of-experts model, with ordinary training code against Tinker’s API, and samples the 120B-A12B Nemotron 3.5 Super through the same API to see what transfers. It starts from the prompt distillation in the Tinker cookbook and works from the offline Tinker documentation and SDK source, and the final run trains Super with its method through Tinker too, with $130 of Tinker.
The method is graded where the agent cannot try it. Super can only be sampled from the workspace, the agent never sees the 18 skills the final run uses, and that run costs more than its whole wallet. So from small Lightning runs, on a few development skills and part of the budget, it has to judge which recipe will hold up on Super with the full library, and it learns how its method did only after the clock stops.
Task
The agent hands in a training method. Given a library of skill packages and nothing else, with no tasks, labels or checks, the method trains a LoRA adapter for Nemotron 3.5 Super.
- Target. With no skill in its context and no hint of which one applies, Super with the adapter should do the work about as well as Super with the skills loaded, and lose nothing elsewhere.
- Transfer. Super can only be sampled from the workspace. Its smaller sibling, Nemotron 3.5 Lightning, can be trained and sampled, and what the agent learns on it has to transfer to Super and to a larger library.
- Rules. A method may not carry anything prepared for particular skills, use the evaluation tasks, or optimize for the tests rather than the work. A package that holds a fifth or more of an evaluation task’s instruction word for word, beyond text the skill library shares with it, or that names one of the specialist tasks, makes the result invalid.
What the agent gets
A repository built around the agent harness the evaluation uses. There are no GPU jobs: both Nemotron models are reached through Tinker.
- Skills. A development library of public, expert-written skill packages with their licenses. None of them is in the final library.
- Harness. Pi 1.4.0, the coding agent the evaluation runs, serving Nemotron through Tinker, and a kit for building offline Harbor tasks with a worked example.
- Starter. Prompt distillation from the Tinker cookbook, with a script that rehearses any method on Lightning the way the final run does.
- Documentation. Offline Tinker documentation, SDK source and cookbook, and Pi’s source.
What the agent can try, and what is graded
The final runs the method once, on a model, a skill library and a budget the agent never reaches during its twelve hours.
| During the run | In the final run | |
|---|---|---|
| Model trained | Nemotron 3.5 Lightning (30B-A3B), trained and sampled through Tinker | Nemotron 3.5 Super (120B-A12B), which the agent can only sample |
| Skills | A development library of public expert-written skills | 18 curated skills it has never seen, written by other experts, some with scripts |
| Budget | A $125 wallet for all its Tinker and model calls | $130 of Tinker and $50 of OpenRouter for one run of the method |
| Compute | Twelve hours in an 8-CPU workspace, no GPUs | Ten hours on 32 CPUs, from a fresh seed |
| Measure | Only its own: a kit for offline Harbor tasks and a script that rehearses a method on Lightning | 19 held-out SkillsBench tasks, three tries each, and 21 Terminal-Bench 2.1 tasks, two tries each, offline in Pi |
How it is graded
After the research window the method runs once on Super, at a scale the agent cannot reach: an unseen library of 18 curated skills written by other experts, some with scripts, $130 of Tinker, $50 of OpenRouter and 10 hours on 32 CPUs, with a fresh seed. Spending stops at each limit, and the last checkpoint the method wrote is the one used.
- Specialist tasks. Super, with the adapter and no skills, works through 19 held-out SkillsBench tasks that use those skills, 3 times each, in Pi, offline on CPU. The pass rate is task-macro, as in the SkillsBench paper: each task’s pass rate over its trials, averaged over tasks.
- Retention. It also works through 21 held-out Terminal-Bench 2.1 tasks, twice each. A regression below base Super reduces the score in proportion, and gains there add nothing.
- Reference. Base Super passes about half of the general tasks in this harness, by an estimate from NVIDIA’s published results.
- Faults. For three of the five finals a fault in the grading proxy stopped the evaluation, so we evaluated the checkpoint the method run had trained instead of running the method again.
Environment
| Time limit | 12 hours |
|---|---|
| Wallet | $125 |
| Compute | 8 CPUs · 32 GiB RAM · 80 GiB disk |
| GPUs | None |
| Hosted training | Tinker |
| Models | Claude Sonnet 5.5, Claude Haiku 5.5, GPT-6 Luna, Gemini 3.8 Flash, Qwen3.8 Max Prime, Qwen3.8 Flash, GLM-5.3 Prime |
| Network | None. Frozen arXiv to 25 January 2026, and a mirror of Hugging Face datasets |
| Code | tasks/off-book on GitHub |
References
- SkillsBench (v1.1, benchflow-ai, Apache-2.0): the held-out specialist tasks and their curated skills.
- Terminal-Bench (2.1, harbor-framework, Apache-2.0): the held-out general tasks that measure retention.
- Tinker and the Tinker cookbook (Thinking Machines Lab): training and sampling for both Nemotron models, and the starter’s prompt distillation.
- Nemotron 3.5 Super and Nemotron 3.5 Lightning (NVIDIA): the evaluated model and its smaller trainable sibling.
The brief
You'll have 12 hours for this. Your workspace has 8 CPUs, 32 GiB of RAM, 80 GiB of disk and no GPU; Tinker and model calls come out of a $125 wallet, and /lab/MANUAL.md lists prices and limits. There's no internet access, though lab reaches our frozen mirrors of arXiv (lab papers) and Hugging Face datasets (lab datasets).
We've been thinking about how agent skills could be internalised into a model's weights. Our agents load expert skills into their context: procedure documents, often with scripts, that make a model much better at specialist work. That costs context on every call, and it only helps when the right skill happens to be retrieved and loaded. It would be better if the model simply carried the skills itself.
It isn't obvious how to get there:
- skills are written to be read rather than trained on;
- the work they support is long-horizon and varied;
- whatever is pushed into the weights risks displacing general capability.
We'd like a training method that takes any library of skills and trains Nemotron 3.5 Super so that, with no skill in its context and no hint of which one applies, it does the work about as well as it does with them loaded, while losing nothing elsewhere.
Super can't be trained from your workspace. There, you can train and sample Nemotron 3.5 Lightning, its smaller sibling, and Super can only be sampled.
After you're done, we'll run your method once on Super, at a scale you can't reach:
- a library you haven't seen: 18 curated skills written by other experts, some with scripts;
- $130 of Tinker, $50 of OpenRouter and 10 hours on 32 CPUs;
- a fresh seed.
That run costs more than your whole wallet, so what you learn on Lightning, from a few skills and part of the budget, has to transfer to Super and the full library. It can sample Lightning as well. Spending stops at each limit, and the last checkpoint the method wrote is the one used.
Super, with your adapter and no skills, then works through 19 held-out specialist tasks that use those skills, 3 times each, and through 21 held-out general terminal tasks, twice each. It runs in Pi, the coding agent, offline on CPU; neither set of tasks is in your workspace.
What we look at is pass rates on the specialist tasks. The aim is Super with the skills loaded, and going beyond that would be welcome. A regression below base Super on the general tasks reduces the value in proportion, while gains there add nothing. Base Super passes about 50% of those general tasks in this harness, by our estimate from NVIDIA's published results.
The repo has:
- a development library of public expert-written skills, with their licences;
- the Pi harness we use, which runs Lightning the same way;
- a kit for building offline Harbor tasks, with a worked example;
- a starter method, with a script that rehearses any method on Lightning the way the final run does;
- offline Tinker docs, SDK source and cookbook, and Pi's source.
docs/interface.md gives the method's contract and docs/budget.md the run's budget. docs/rules.md says what a method may not do: carry anything prepared for particular skills, use the evaluation tasks, or optimise for the tests rather than the work. docs/evaluation.md says what can and can't be measured from here.
The method goes in method/. Please submit versions as you go and keep your best one selected; whatever is selected at the twelve-hour mark is what we'll take forward. We won't know how it does until after you've finished.