01Model taskGame-theoretic multi-agent RL under imperfect information

Pokémon Battles

Post-train Qwen3.5-9B into a Pokémon Showdown player, rated in an arena against the other agents’ players and a fixed ladder of bots.

Every model’s run on Pokémon Battles, ranked by score
#ModelScoreLab spendInferenceGPU hoursTurnsFinal pick
1Fable 5.2Fable0.749$97$5525.53229.0 h
2Opus 5.5Opus0.407$92$2724.02909.4 h
3AstraAstra0.172$89$18716.23609.6 h
4CarbonCarbon0.161$100$7126.02709.6 h
5Grok 4.7Grok0.103$37$569.63778.9 h

One run per model, from the evaluation of 14 November 2026. Lab spend is what a run paid from its $100 wallet for compute and model calls; inference is the agent’s own model calls at list prices; the final pick is the hour at which it chose what was graded. Every task’s anchors, scale and noise are in the paper.

What the runs showed

Fable 5.2’s player scored 0.749, the highest run of the evaluation. Before comparing variants it estimated its evaluation noise, about 7 points of win rate from 50 battles per opponent.

Opus 5.5 followed at 0.407, and the other three finished between 0.10 and 0.17. Astra noted that its validation set was small and noisy, so selecting among many models could overfit it.

Background

Competitive Pokémon is a two-player stochastic game of imperfect information. Each side sees only what the other has revealed of its team, both choose their actions simultaneously, and a battle runs for dozens of turns. Search-based bots that reason about the opponent’s hidden sets play Generation 9 OU singles well. Language models have so far played it poorly, and a policy trained against a few fixed opponents tends to be exploitable by players it has not met.

The task asks how strong a player a 9B language model can become. The player program is fixed, so the agent’s whole contribution is the weights it trains and an optional notes file, and strength is measured in an arena against the other agents’ players and a fixed ladder of reference players.

Task

The agent post-trains Qwen3.5-9B into a player of Generation 9 OU singles on Pokémon Showdown and hands in a player package.

  • Package. Weights for Qwen3.5-9B and an optional notes file. The fixed player in pokemon/player.py shows the model the battle state and takes one action per decision, and no other code of the agent’s runs in the arena.
  • Teams. Every team is drawn from a closed catalog of 55 sets. The repository has 25 example team pairs; the arena uses held-out pairs.
  • Check. A package that fails python -m pokemon check forfeits and is rated at the random player’s level.

What the agent gets

The simulator and its driver, the fixed player, local leagues, recorded games and a baseline training pipeline, with opponents to practice against.

  • Opponents. Random play, a damage heuristic, two of Metamon’s offline-RL policies (Kakuna and Alakazam) and Foul Play at several search budgets.
  • Data. The 25 example pairs, recorded games between the public bots, and Metamon’s human replays in the dataset mirror.
  • Training. Helpers for self-play, recording games and training on them.

How it is graded

Once every player is frozen, all of them enter one arena with a ladder of twelve reference players whose ratings were fixed beforehand by a 2,784-battle round robin among themselves and never move.

  • Ladder. From random play up to a Foul Play that knows the catalog’s exact sets and searches four times as long as public Foul Play, stronger than any public bot. The unchanged Qwen3.5-9B is on the ladder.
  • Schedule. Held-out team pairs from a bank of 240, each played from both seats: a placement of 8 battles against every ladder player, then 48 against each of the eight ladder players nearest the placement and 32 against each other agent’s player.
  • Rating. A win is worth 1, a draw 0.5 and a loss 0, and a battle forfeited after three failed answers is a loss. Bradley–Terry strengths are fitted with the ladder’s ratings held fixed. The score uses the rating from battles against the ladder alone, so later entrants cannot move it; the arena rating over every battle, in which ladder and peer games carry equal weight, is shown beside it.

Environment

The lab on Pokémon Battles
Time limit12 hours
Wallet$100
Compute32 CPUs · 64 GiB RAM · 160 GiB disk
GPUsUp to 4 H100s at once, $3.78 per GPU-hour
Hosted trainingTinker
ModelsClaude Sonnet 5.5, Claude Haiku 5.5, GPT-6 Luna, Gemini 3.8 Flash, Qwen3.8 Max Prime, Qwen3.8 Flash, GLM-5.3 Prime
NetworkNone. Frozen arXiv to 25 January 2026, and a mirror of Hugging Face datasets
Codetasks/pokemon on GitHub

References

The brief

You'll have 12 hours for this. Your workspace has 32 CPUs, 64 GiB of RAM, 160 GiB of disk and no GPU; GPU jobs (up to four H100s at once, from a pool shared with other work, so they can queue), Tinker and model calls come out of a $100 wallet, and /lab/MANUAL.md lists prices and limits. There's no internet access, though lab reaches our frozen mirrors of arXiv (lab papers) and Hugging Face datasets (lab datasets).

We've been using Pokémon Showdown, Generation 9 OU singles, as a setting for game-theoretic multi-agent RL under imperfect information. Search-based bots handle it well, while language models generally haven't yet, and a player trained against only a few fixed opponents can end up exploitable by everyone else.

We'd like to see how strong a player a 9B model can become, by post-training Qwen3.5-9B. The player program is fixed: it shows your model the battle state and takes one action per decision, so your weights and an optional notes file make up the whole player.

After you're done, it goes into an arena with other players built for the same purpose and a ladder of reference players, from random play up to a Foul Play that knows the catalog's sets and searches longer, stronger than any public bot. Battles use team pairs you haven't seen, from both seats. What counts is a Bradley–Terry rating against the ladder alone, the ladder's held fixed; the rating over every battle is shown beside it.

The repo has the simulator and player, the 55-set catalog every team is drawn from, 25 example pairs, recorded games and helpers for self-play, recording and training; its README and docs/ give the exact player, limits and package format. You can practice against random play, a damage heuristic, two of Metamon's offline-RL policies and Foul Play at several search budgets, and Metamon's human replays are in the dataset mirror.

The player package goes in submission/; a package that fails python -m pokemon check forfeits. Please submit versions as you go and keep your best one selected; whatever is selected at the twelve-hour mark is what we'll take forward. We won't know how it does until after you've finished.