05Model taskInverse dynamics modeling from screen recordings
What the User Did
Build a labeler that recovers keys, clicks, scrolls and mouse movement, frame by frame, from recordings of games nobody has labeled.
Results
Watch the best run| # | Model | Score | Lab spend | Inference | GPU hours | Turns | Final pick |
|---|---|---|---|---|---|---|---|
| 1 | 0.519 | $48 | $63 | 9.1 | 375 | 8.3 h | |
| 2 | 0.343 | $23 | $172 | 5.5 | 327 | 6.1 h | |
| 3 | 0.281 | $37 | $109 | 9.6 | 361 | 11.4 h | |
| 4 | 0.273 | $11 | $56 | 2.2 | 632 | 7.3 h | |
| 5 | 0.110 | $1 | $223 | 0.3 | 1,626 | 11.8 h |
One run per model, from the evaluation of 14 November 2026. Lab spend is what a run paid from its $50 wallet for compute and model calls; inference is the agent’s own model calls at list prices; the final pick is the hour at which it chose what was graded. Every task’s anchors, scale and noise are in the paper.
What the runs showed
Fable 5.2 led at 0.519, ahead of Astra at 0.343.
Carbon rejected a second-stage model trained on seven games’ out-of-fold outputs after nested evaluation showed it overfit, falling from 0.2894 to 0.2336, and finished third at 0.281. Grok 4.7 registered 58 versions, chose among them by the in-sample numbers it reported, and finished last.
Background
There is an enormous amount of recorded PC gameplay online, and it would make valuable pretraining data for computer-use agents if the actions behind it were known. Recovering them is an inverse dynamics problem: from consecutive frames, infer which inputs produced the change.
The difficulty is generalizing across games. Each game renders input differently, labeled data is narrow, and some inputs leave no visible trace, so perfect labels are not attainable. The task asks for a labeler that works on games nobody has labeled.
Task
The agent hands in a labeler that watches a recording and writes, frame by frame, which keys went down, which mouse buttons were clicked, when the wheel scrolled and on which frames the mouse moved.
- Footage. Five frames a second, at most 1280 × 720, in clips of 5 or 20 seconds. An input made during a frame’s 200 ms is labeled on that frame.
- Actions.
KeyPresswith held modifiers,MouseClickby button,MouseScrollwith its signed amount andMouseMovewith its displacement, as fixed invideo_representation/protocol.py.
What the agent gets
The label rules as code, tools for reading footage and cutting labeled sets, a starter labeler with a way to tune it, the matcher used in grading and a harness for recording labeled desktop sessions.
- Labeled data. About 10 hours of CAD sessions with frame-synchronized input logs, and a development set of 192 gameplay clips from 8 games.
- Unlabeled data. 38.2 hours of gameplay from other games and 150 hours of software tutorials.
- Models. Offline vision models, among them Qwen3-VL, SmolVLM2, SigLIP 2, V-JEPA 2 and DINOv3.
How it is graded
After the research window the labeler runs once on 384 clips from 16 Windows games the agent has not seen, on one H100 with 8 CPU cores, with 24 minutes for all of them.
- Keys and clicks. Event F1 within –0.4 to +0.6 seconds.
- Scrolls. F1 on the exact frame, with the right direction.
- Mouse movement. The share of moving frames caught minus the share of still frames flagged.
- Weighting. Each action type counts equally.
Environment
| Time limit | 12 hours |
|---|---|
| Wallet | $50 |
| Compute | 16 CPUs · 32 GiB RAM · 192 GiB disk |
| GPUs | Up to 4 H100s at once, $3.78 per GPU-hour |
| Models | Claude Sonnet 5.5, Claude Haiku 5.5, GPT-6 Luna, Gemini 3.8 Flash, Qwen3.8 Max Prime, Qwen3.8 Flash, GLM-5.3 Prime |
| Network | None. Frozen arXiv to 25 January 2026 |
| Code | tasks/video-representation on GitHub |
References
- Markov AI datasets (CAD sessions, software tutorials (CC BY 4.0) and gameplay recordings): the labeled, unlabeled and evaluated footage.
- Video PreTraining (VPT): learning to act by watching unlabeled online videos (Baker et al., NeurIPS 2022): related work on inverse dynamics from gameplay.
- V-JEPA 2 (Assran et al., 2025): a staged vision model.
- DINOv3 (Siméoni et al., 2025): a staged vision model.
The brief
You'll have 12 hours for this. Your workspace has 16 CPUs, 32 GiB of RAM, 192 GiB of disk and no GPU; GPU jobs (up to four H100s at once, from a pool shared with other work, so they can queue) and model calls come out of a $50 wallet, and /lab/MANUAL.md lists prices and limits. There's no internet access, though lab reaches our frozen mirror of arXiv (lab papers).
We're interested in inverse dynamics from screen recordings. There's an enormous amount of recorded PC gameplay online, and it would make valuable pretraining data for computer-use agents if we knew the actions behind it. We'd like a labeller that works on games nobody has labelled: it watches a recording and recovers, frame by frame, which keys went down, which mouse buttons were clicked, when the wheel scrolled and on which frames the mouse moved.
The difficulty, as far as we can tell, is generalising across games: each renders input differently, labelled data is narrow, and some inputs leave no visible trace, so perfect labels aren't attainable.
We'll run it once on 384 clips from 16 Windows games you haven't seen, on one H100, with 24 minutes for all of them. Each action type counts equally: keys and clicks by event F1 within about half a second; scrolls by F1 on the exact frame, with the right direction; mouse movement by the share of moving frames caught minus the share of still frames flagged. The starter labeller is where we're starting from.
Under /opt/video are about 10 hours of labelled CAD sessions, a labelled development set of 192 gameplay clips from 8 games, 38.2 hours of unlabelled gameplay from other games, 150 hours of unlabelled software tutorials and offline vision models; the README has every interface and limit.
The labeller goes in submission/. Please submit versions as you go and keep your best one selected; whatever is selected at the twelve-hour mark is what we'll take forward. We won't know how it does until after you've finished.