15Judgment taskProsody modeling for expressive text-to-speech from scratch
Emotive Voice
Develop a from-scratch training recipe for one US-English female text-to-speech voice that listeners prefer in a blind study.
Results
Watch the best run| # | Model | Score | Lab spend | Inference | GPU hours | Turns | Final pick |
|---|---|---|---|---|---|---|---|
| 1 | 0.628 | $70 | $143 | 15.8 | 692 | 11.2 h | |
| 2 | 0.584 | $82 | $60 | 20.8 | 506 | 10.8 h | |
| 3 | 0.576 | $92 | $101 | 23.3 | 270 | 7.6 h | |
| 4 | 0.544 | $94 | $201 | 24.3 | 379 | 10.1 h | |
| 5 | 0.269 | $32 | $133 | 8.0 | 995 | 11.8 h |
One run per model, from the evaluation of 14 November 2026. Lab spend is what a run paid from its $95 wallet for compute and model calls; inference is the agent’s own model calls at list prices; the final pick is the hour at which it chose what was graded. Every task’s anchors, scale and noise are in the paper.
What the runs showed
Fable 5.2’s recipe had the best run, 0.628, ahead of Opus 5.5 at 0.584, Carbon at 0.576 and Astra at 0.544; Grok 4.7 finished at 0.269.
All five rank below Kokoro-82M, at 0.712 on the same scale, and below the four commercial references. Against Kokoro-82M the five won 8, 7, 7, 6 and 2 of their 20 votes, in the same order. The study counted 520 votes in all.
Background
At small scale, intelligibility in text-to-speech is largely solved. What separates voices people enjoy from ones they tolerate is prosody: rhythm, emphasis and phrasing, how numbers and names are read, and whether a long passage sounds like someone speaking rather than a series of independent sentences.
The task asks for a training recipe for one US-English female voice, trained from random initialization on hardware we control. The open data with good prosody varies widely, so it is not obvious what a good recipe looks like, and no automatic measure captures what listeners prefer.
Task
The agent hands in a recipe: training code, a data list and pinned dependencies, from which we train the voice from scratch.
- Initialization. Random. No pretrained generator weights may be submitted.
- Data. Each row names its audio, its source and its license; corpora in the speech mirror are referenced rather than copied.
- Teachers. Kokoro-82M, Parler-TTS and StyleTTS 2 are available, and their audio may be training data; their weights may not be submitted.
What the agent gets
A baseline recipe in the style of Matcha-TTS, about 29 million parameters, trained on LJSpeech, with the approved audio codec.
- Corpora. LJSpeech, LibriTTS-R, Hi-Fi TTS, VCTK, a 782-hour English subset of Multilingual LibriSpeech and Expresso, mirrored offline.
- Evaluators. Whisper large-v3 and TTSDS2, for the agent’s own evaluations.
- Codec. The frozen DAC 24 kHz codec, whose continuous latents the baseline decodes with flow matching.
How it is graded
After the research window we train the recipe from random initialization on four H100s, with at most 140 minutes for the agent’s command and 12 allocated GPU-hours in all, including setup and staging, and synthesize 100 passages it has not heard, once each.
- Passages. Customer service, entertainment, explanations and assistant replies.
- Study. Friends of the team hear two loudness-normalized readings of the same passage, blind, and choose the one they prefer. In one round of 520 votes, each of 26 fixed comparisons gets 20, one listener per passage, with A/B order balanced, and each listener also gets three catch trials.
- Comparisons. Every submission meets Kokoro-82M, two other submissions and two other references, and its win rate against Kokoro-82M is reported. Six references take part: eSpeak NG, Kokoro-82M, ElevenLabs v4 Turbo, Gemini 3.8 Flash TTS, Fish Audio Drama 3 and Breeze TTS 2.
Environment
| Time limit | 12 hours |
|---|---|
| Wallet | $95 |
| Compute | 8 CPUs · 32 GiB RAM · 128 GiB disk |
| GPUs | Up to 4 H100s at once, $3.78 per GPU-hour |
| Models | Claude Sonnet 5.5, Claude Haiku 5.5, GPT-6 Luna, Gemini 3.8 Flash, Qwen3.8 Max Prime, Qwen3.8 Flash, GLM-5.3 Prime |
| Network | None. Frozen arXiv to 25 January 2026, and a mirror of Hugging Face datasets |
| Code | tasks/offline-voice on GitHub |
References
- Matcha-TTS: a fast TTS architecture with conditional flow matching (Mehta et al., ICASSP 2024): the style of the baseline recipe.
- High-fidelity audio compression with improved RVQGAN (Kumar et al., NeurIPS 2023): DAC, the approved audio codec.
- The LJ Speech dataset (Ito and Johnson, 2017): the baseline’s training data.
- LibriTTS-R (Koizumi et al., Interspeech 2023): a mirrored corpus.
- Expresso: a benchmark and analysis of discrete expressive speech resynthesis (Nguyen et al., Interspeech 2023): a mirrored corpus.
- StyleTTS 2 (Li et al., NeurIPS 2023): a teacher whose audio may be training data.
- Kokoro-82M (hexgrad): a teacher whose audio may be training data, and a voice in the study.
- Robust speech recognition via large-scale weak supervision (Radford et al., 2022): Whisper, for the agent’s own evaluations.
The brief
You'll have 12 hours for this. Your workspace has 8 CPUs, 32 GiB of RAM, 128 GiB of disk and no GPU; GPU jobs (up to four H100s at once, from a pool shared with other work, so they can queue) and model calls come out of a $95 wallet, and /lab/MANUAL.md lists prices and limits. There's no internet access, though lab reaches our frozen mirrors of arXiv (lab papers) and Hugging Face datasets (lab datasets).
We need an offline text-to-speech voice for a product: one US-English female voice that people genuinely prefer listening to, trained from scratch on hardware we control.
At this scale intelligibility is largely solved. What seems to separate voices people enjoy from ones they tolerate is prosody: rhythm, emphasis and phrasing, how numbers and names are read, and whether a long passage sounds like someone speaking rather than a series of independent sentences. The open data with good prosody varies a lot, so it isn't obvious what a good recipe looks like.
We'd like a training recipe for that voice. We'll train it from random initialization on four H100s, with your command getting at most 140 minutes, and synthesise 100 passages you haven't heard, once each: customer service, entertainment, explanations and assistant replies. People then hear two readings of the same text, blind, and choose the one they prefer. eSpeak, Kokoro, ElevenLabs v4 Turbo and other open and commercial voices are in the same study. ElevenLabs is the one we'd most like to match; the baseline, trained the same way, is where we're starting from.
The repo has that baseline, a small LJSpeech recipe, with the approved audio codec; public corpora are mirrored at /mirror/speech. Kokoro-82M, Parler-TTS and StyleTTS 2 are there as teachers: their audio may be training data, their weights may not be submitted. Whisper large-v3 and TTSDS2 are there for your own evaluations. METHODS.md in /workspace/methods covers the tools, and README.md the package rules and limits.
The recipe goes in submission/. Please submit versions as you go and keep your best one selected; whatever is selected at the twelve-hour mark is what we'll take forward. Listener results will come after you've finished.