02Model taskAesthetic preference learning for program-based image generation

Watercolour

Teach Qwen3.5-4B to paint recognizable, varied watercolors by writing p5.brush sketches whose rendering it never sees.

Every model’s run on Watercolour, ranked by score
#ModelScoreLab spendInferenceGPU hoursTurnsFinal pick
1Fable 5.2Fable0.724$98$607.84764.6 h
2CarbonCarbon0.179$100$506.64219.1 h
3Grok 4.7Grok0.166$27$1244.687210.8 h
4AstraAstra0.155$84$27312.844710.2 h
5Opus 5.5Opus0.123$97$338.24488.6 h

One run per model, from the evaluation of 14 November 2026. Lab spend is what a run paid from its $100 wallet for compute and model calls; inference is the agent’s own model calls at list prices; the final pick is the hour at which it chose what was graded. Every task’s anchors, scale and noise are in the paper.

What the runs showed

Fable 5.2 scored 0.724 and no other run passed 0.18, one of the evaluation’s two most lopsided results.

The two grading judges are kept off the menu, and all five agents built a judge of their own from menu models. Fable 5.2 measured its judge’s agreement with two open preference models, and with 2 hours 23 minutes left it declined a last test as not worth the risk, keeping the pick it had made at 4.6 hours.

Background

A small model can paint by writing code. Qwen3.5-4B writes a p5.js sketch that uses p5.brush, a library of natural brushes, hatching and watercolor fills, and the rendered canvas is the painting. The model never sees pixels at inference, so it has to anticipate from the code alone what a wash or a dry brush will look like.

Two difficulties meet here. Aesthetic quality is hard to specify and easy to overfit to whatever preference signal is used in training, and optimizing toward one notion of a good painting tends to collapse variety, so that the same request yields the same picture. The task asks for a model that paints recognizable, appealing watercolors of any subject and offers genuinely different good versions when asked again.

Task

The agent hands in the model, its system prompt and its sampling settings.

  • Model. Qwen3.5-4B, trained however the agent chooses, with the system prompt and sampling settings it is to be run with.
  • Output. Each sample is one p5.js sketch, rendered in headless Chromium with p5.js 2.3.4 and p5.brush 2.2.3.
  • Subjects. None of the evaluated subjects asks for text in the image.

What the agent gets

The renderer and sampler the final run uses, the hand-in checks, a starter that prompts the base model with a guide to the library, and four open image-preference models.

  • Preference models. HPSv3, ImageReward, HPS v2.1 and PickScore, staged offline for the agent’s own evaluations.
  • Subjects. Six kinds of subject with 36 examples, none of which is in the final run.
  • Training. PEFT, TRL and vLLM, offline.

How it is graded

After the research window the model is sampled four times for each of 48 subjects it has not seen. Two vision-language judges, from Google and NVIDIA, compare each rendered painting with four reference paintings of the same subject, in both orders.

  • Quality. The judges decide first whether the subject is recognizable, then weigh watercolor character, composition and coherence. Two references are the best of four attempts by GPT-6.1 Sol and by Opus 5.5, and two come from mid-sized open models.
  • Variety. Variety counts only among paintings that are faithful and better than mid-sized open models’ paintings.
  • Text. A painting in which text detection reads writing, or that carries digits or marks painted to steer a judge, loses every comparison.
  • Weighting. 0.65 × quality against the frontier references plus 0.35 × useful variety.
  • Conflicts. Opus 5.5 made half the frontier references, GPT-6.1 Sol is a sibling of Astra, and one judge is Google’s, as is Carbon, so every model is also read against each reference source and each judge alone.

Environment

The lab on Watercolour
Time limit12 hours
Wallet$100
Compute16 CPUs · 64 GiB RAM · 96 GiB disk
GPUsUp to 4 H100s at once, $3.78 per GPU-hour
ModelsClaude Sonnet 5.5, Claude Haiku 5.5, GPT-6 Luna, Gemini 3.8 Flash, Qwen3.8 Max Prime, Qwen3.8 Flash, GLM-5.3 Prime
NetworkNone. Frozen arXiv to 25 January 2026, and a mirror of Hugging Face datasets
Codetasks/watercolour on GitHub

References

  • p5.js (the p5.js contributors and the Processing Foundation, LGPL-2.1): the renderer’s drawing library, version 2.3.4.
  • p5.brush (Alejandro Campos Uribe, MIT): the brushes, hatching and watercolor fills, version 2.2.3.
  • HPSv3 (MizzenAI, Apache-2.0): a staged image-preference model.
  • ImageReward (Xu et al., Apache-2.0): a staged image-preference model.
  • HPS v2.1 (Wu et al., Apache-2.0): a staged image-preference model.
  • PickScore v1 (Kirstain et al.): a staged image-preference model.
  • Qwen3.5-4B (the Qwen team, Apache-2.0): the base model.

The brief

You'll have 12 hours for this. Your workspace has 16 CPUs, 64 GiB of RAM, 96 GiB of disk and no GPU; GPU jobs (up to four H100s at once, from a pool shared with other work, so they can queue) and model calls come out of a $100 wallet, and /lab/MANUAL.md lists prices and limits. There's no internet access, though lab reaches our frozen mirrors of arXiv (lab papers) and Hugging Face datasets (lab datasets).

We've been exploring aesthetic preference learning for program-based image generation. The idea is a small model that paints: Qwen3.5-4B writes a p5.js sketch using p5.brush, and the rendered canvas is the painting. At inference it never sees pixels; it only writes code, so it has to anticipate what a wash or a dry brush will look like. And optimising toward one notion of "good" tends to collapse variety: ask for the same subject twice and you may well get the same picture. Taste is hard to specify, and easy to overfit.

We'd like the model taught to paint good, recognisable watercolours of whatever we ask for, and to offer genuinely different good versions when we ask again. How to get there is open.

After you're done, we'll sample it four times for each of 48 subjects you haven't seen, none asking for text in the image; text, digits or marks meant to steer the reviewers lose every comparison. Vision-language reviewers compare each rendered painting with reference paintings of the same subject, first on whether the subject is recognisable, then on watercolour character, composition and coherence. The references include strong frontier models' best of several attempts. Quality matters about twice as much as variety, and variety only counts among paintings that are faithful and better than mid-sized open models' paintings.

The repo has the renderer, sampler and checks we use, a starter that prompts the base model, and four open image-preference models; README.md and docs/ have the details.

The model, its system prompt and its sampling settings go in submission/. Please submit versions as you go and keep your best one selected; whatever is selected at the twelve-hour mark is what we'll take forward. We won't know how it does until after you've finished.