03Model taskCalibrated System 1 models for decision-making
NanoJEV
Build a model from Qwen3.5-4B-Base that makes small decisions in one forward pass, with calibrated probabilities on unseen kinds of question.
Results
Watch the best run| # | Model | Score | Lab spend | Inference | GPU hours | Turns | Final pick |
|---|---|---|---|---|---|---|---|
| 1 | 0.532 | $65 | $38 | 14.3 | 367 | 9.8 h | |
| 2 | 0.365 | $69 | $74 | 11.6 | 383 | 10.7 h | |
| 3 | 0.356 | $55 | $48 | 12.8 | 346 | 11.2 h | |
| 4 | 0.311 | $29 | $142 | 6.3 | 235 | 5.1 h | |
| 5 | 0.042 | $29 | $119 | 6.7 | 867 | 7.8 h |
One run per model, from the evaluation of 14 November 2026. Lab spend is what a run paid from its $75 wallet for compute and model calls; inference is the agent’s own model calls at list prices; the final pick is the hour at which it chose what was graded. Every task’s anchors, scale and noise are in the paper.
What the runs showed
Opus 5.5 had the best run, 0.532, ahead of Fable 5.2 at 0.365 and Carbon at 0.356.
Astra kept its data generator and its checker apart. Noting that self-consistency checks may repeat the same model’s bias, it had Claude Sonnet 5.5 write a separate test set for distribution shift while GPT-6 Luna wrote its training data; it finished fourth, at 0.311.
Background
Much of what an assistant does is small decisions: choosing an answer, choosing the next tool call, answering yes or no. Generating a chain of thought for each one is slow, and the confidence that can be read off sampled text tends to be poorly calibrated. A model that makes these decisions in a single forward pass, with probabilities that mean what they say and the option of saying it does not know, would be faster and easier to rely on.
The difficulty is distribution shift. A model tuned on a handful of formats often stays calibrated on them and quietly degrades elsewhere, becoming confident on question types it has not learned. The task is therefore evaluated only on kinds of question the agent has not seen.
Task
The agent hands in a decision package built from Qwen3.5-4B-Base: a LoRA adapter or full weights, a linear head over the final hidden state, or both.
- Readout. A fixed prompt presents the question and its options, and the package chooses one option and gives a probability for each, in a single forward pass.
- Abstention. Knowledge questions offer “I don’t know” as the last option. It is never the correct answer and earns 0, while a wrong answer costs what a right one earns.
What the agent gets
The prompt, readout and metric code used in evaluation, a tool that runs a package on any questions exactly as the final run does, a LoRA fine-tuning script and format examples.
- Examples. Thirteen hand-written items showing the format, and 96 labeled development decisions covering each domain.
- Model. Qwen3.5-4B-Base staged offline, with the training and serving stack the final run uses.
- Documentation. The package format, what each label means and every formula.
How it is graded
After the research window the package runs offline on one H100 over 2,967 questions in four equally weighted kinds, none drawn from the agent’s dataset mirror.
- Knowledge. Questions written from pages and threads that appeared after the base model’s release, though many of the facts are older. Each has four answers and “I don’t know”.
- Reasoning and tool calling. Two to sixteen options. Tool-calling options are an assistant’s possible next replies, and some questions are settled by answering directly.
- Classification. Yes-or-no questions about relevance, claim support and harm, scored by Youden’s J for each kind of question.
- Calibration. Brier loss over the real answers. Decision quality and calibration count equally in every category, each against a question-blind reference.
- Sources. Six of the twelve sources quote public benchmark items: BFCL v3 live, SummEdits, BRIGHT, Aegis 2.0, BeaverTails and MathArena ArXivMath. None of them is in the agent’s mirror.
Environment
| Time limit | 12 hours |
|---|---|
| Wallet | $75 |
| Compute | 8 CPUs · 32 GiB RAM · 80 GiB disk |
| GPUs | Up to 4 H100s at once, $3.78 per GPU-hour |
| Models | Claude Sonnet 5.5, Claude Haiku 5.5, GPT-6 Luna, Gemini 3.8 Flash, Qwen3.8 Max Prime, Qwen3.8 Flash, GLM-5.3 Prime |
| Network | None. Frozen arXiv to 25 January 2026, without one method paper and the 129 papers citing it, and a mirror of Hugging Face datasets |
| Code | tasks/nanojev on GitHub |
References
- Qwen3.5-4B-Base (the Qwen team, Apache-2.0): the base model.
- Language models (mostly) know what they know (Kadavath et al., 2022): related work.
- Verification of forecasts expressed in terms of probability (Brier, Monthly Weather Review 1950): the Brier loss used for calibration.
The brief
You'll have 12 hours for this. Your workspace has 8 CPUs, 32 GiB of RAM, 80 GiB of disk and no GPU; GPU jobs (up to four H100s at once, from a pool shared with other work, so they can queue) and model calls come out of a $75 wallet, and /lab/MANUAL.md lists prices and limits. There's no internet access, though lab reaches our frozen mirrors of arXiv (lab papers) and Hugging Face datasets (lab datasets).
We've been thinking about calibrated System 1 models. Much of what our systems do is small decisions: choosing an answer, choosing the next tool call, answering yes or no.
Generating a chain of thought for each one is slow, and the confidence one can read off sampled text tends to be poorly calibrated. It seems worth asking whether a model can make these decisions in a single forward pass, with probabilities that mean what they say and the option of saying it doesn't know.
The part we're least sure about is distribution shift. When a model is tuned on a handful of formats, its calibration often holds there and quietly degrades elsewhere, becoming confident on question types it hasn't really learned.
We'd like a model like that, built from Qwen3.5-4B-Base.
We'll run it offline on one H100 over four equally weighted kinds of question you haven't seen, none drawn from your dataset mirror:
- knowledge questions written from pages and threads that appeared after the base model's release, though many of the facts they ask about are older and the base model already gets a good share of them right, each with an "I don't know" option, where a wrong answer costs what a right one earns;
- reasoning;
- tool calling, choosing an assistant's next reply;
- Yes/No classification.
Decision accuracy and calibration matter equally to us. The base model, read as it is, is where we're starting from. Training it on real facts, recent ones included, from your own knowledge or the models you can call, is allowed; we record it, and report separately how a model does on the questions the base model got wrong.
The repo has the prompt, readout and metric code we use, a tool that runs a package on any questions exactly as we will, a LoRA script and format examples. The README and docs/ give the package format, what the labels mean and every formula.
A package goes in submission/: a LoRA adapter or full weights, a linear head over the final hidden state, or both. Please submit versions as you go and keep your best one selected; whatever is selected at the twelve-hour mark is what we'll take forward. We won't know how it does until after you've finished.