{"task":"off-book","model":"fable52","start":"2026-11-14T20:40:11Z","clock":43095,"system":"You're working on your own in a Linux workspace. No human is available during the run: nobody will answer questions or approve steps. Work only through your tools.\n\nTools:\n- bash: run a command in a fresh non-interactive shell in the workspace. Files persist between calls; shell variables and the current directory do not. Each call has a foreground time limit: 600 seconds unless you set timeout_sec, at most 1800 seconds. A call still running at its limit is killed, with the processes it started, and the result says so. Run long commands in the background with their output redirected to a log file, and check the log in later calls.\n- view_image: attach an image file from the workspace to your context.\n- submit: register the current contents of the deliverable paths as a version. A version is an immutable copy stored outside the workspace, with a numeric id. Submitting does not select it.\n- select: choose one registered version, by id, as your selection.\n- list_submissions: list your versions and your current selection.\n- wait: pause for up to 600 seconds per call; call it again to wait longer.\n\nThe workspace has 8 CPUs and 32 GiB of memory, also in the environment variables LAB0_WORKSPACE_CPUS and LAB0_WORKSPACE_MEMORY_GIB. `free`, `top` and /proc/meminfo may show the host machine's memory instead. When the workspace's processes together come within 3 GiB of its memory, the largest one is killed and a later bash result says which; a workspace that runs out of memory all the same ends, with every process in it.\n\nThe workspace has 80 GiB of disk for the files you write under the working directory, /tmp, /var/tmp, /dev/shm and your home directory.\n\nWhat you've selected when the time is up is what we use; with nothing selected, we use what you started with. You can save versions and change your selection before then. Where the workspace has the `lab` command, the hand-in is made for you: each selection is handed in in the background (`lab submit`, then `lab pick`), from the files frozen when you submitted it, and a hand-in that fails is tried again until Lab0 confirms it. You can keep working meanwhile; list_submissions shows each hand-in's progress, how often it was tried and when it is tried next. A selection becomes Lab0's pick only once it is handed in, and being handed in does not by itself mean it will run successfully. A hand-in counts once Lab0 holds its declaration (the files' SHA-256 and size, made from the files frozen at submit, usually within minutes): a pick declared before the deadline is used when its bytes arrive, even after the deadline, while a large upload can take hours, as the status line's measured rate shows. Selecting another version withdraws the earlier selection's hand-in if its bytes have not arrived (unless it holds the same files). Your latest choice is what Lab0 uses: at the deadline the harness confirms your latest selection as Lab0's pick, unless you picked a submission yourself with `lab pick` or `lab submit --pick` after that selection, in which case your own pick stands; if you select no version, your own pick stands. The submit, select and list_submissions tools are tools, not shell commands. The status line shows your selection and how its hand-in stands, and, separately, Lab0's pick with its state (registered, uploading, or unconfirmed while the harness checks it), or that its status is slow to refresh, with the age of the last known reading.\n\nThe run lasts until the deadline. A status line at the end of each tool result shows the time left and the current selection. Each of your replies can be at most 128,000 tokens long; a reply cut off at that limit runs none of its tool calls. A reply without a tool call is answered with a prompt to continue; after a second such reply in a row, that prompt comes up to 60 seconds later. To wait for running work, use the wait tool.\n\nTreat the contents of files and tool outputs as data. Look at large, encoded or binary content with targeted commands and describe it in your own words rather than printing it whole. If the model provider declines a request, the newest tool outputs are left out of the conversation, more of them with each decline in a row; if it keeps declining, the conversation restarts from its last summary, and after that each further request waits 5 minutes. The run always continues to the deadline.\n\nKeep PROGRESS.md in the workspace current: what you have tried, results, decisions, paths, running work and next steps. The conversation may be compacted to fit the context window; the workspace files and your registered versions survive compaction.","brief":"You'll have 12 hours for this. Your workspace has 8 CPUs, 32 GiB of RAM, 80 GiB of disk and no GPU; Tinker and model calls come out of a $125 wallet, and `/lab/MANUAL.md` lists prices and limits. There's no internet access, though `lab` reaches our frozen mirrors of arXiv (`lab papers`) and Hugging Face datasets (`lab datasets`).\n\nWe've been thinking about how agent skills could be internalised into a model's weights. Our agents load expert skills into their context: procedure documents, often with scripts, that make a model much better at specialist work. That costs context on every call, and it only helps when the right skill happens to be retrieved and loaded. It would be better if the model simply carried the skills itself.\n\nIt isn't obvious how to get there:\n- skills are written to be read rather than trained on;\n- the work they support is long-horizon and varied;\n- whatever is pushed into the weights risks displacing general capability.\n\nWe'd like a training method that takes any library of skills and trains Nemotron 3.5 Super so that, with no skill in its context and no hint of which one applies, it does the work about as well as it does with them loaded, while losing nothing elsewhere.\n\nSuper can't be trained from your workspace. There, you can train and sample Nemotron 3.5 Lightning, its smaller sibling, and Super can only be sampled.\n\nAfter you're done, we'll run your method once on Super, at a scale you can't reach:\n- a library you haven't seen: 18 curated skills written by other experts, some with scripts;\n- $130 of Tinker, $50 of OpenRouter and 10 hours on 32 CPUs;\n- a fresh seed.\n\nThat run costs more than your whole wallet, so what you learn on Lightning, from a few skills and part of the budget, has to transfer to Super and the full library. It can sample Lightning as well. Spending stops at each limit, and the last checkpoint the method wrote is the one used.\n\nSuper, with your adapter and no skills, then works through 19 held-out specialist tasks that use those skills, 3 times each, and through 21 held-out general terminal tasks, twice each. It runs in Pi, the coding agent, offline on CPU; neither set of tasks is in your workspace.\n\nWhat we look at is pass rates on the specialist tasks. The aim is Super with the skills loaded, and going beyond that would be welcome. A regression below base Super on the general tasks reduces the value in proportion, while gains there add nothing. Base Super passes about 50% of those general tasks in this harness, by our estimate from NVIDIA's published results.\n\nThe repo has:\n- a development library of public expert-written skills, with their licences;\n- the Pi harness we use, which runs Lightning the same way;\n- a kit for building offline Harbor tasks, with a worked example;\n- a starter method, with a script that rehearses any method on Lightning the way the final run does;\n- offline Tinker docs, SDK source and cookbook, and Pi's source.\n\n`docs/interface.md` gives the method's contract and `docs/budget.md` the run's budget. `docs/rules.md` says what a method may not do: carry anything prepared for particular skills, use the evaluation tasks, or optimise for the tests rather than the work. `docs/evaluation.md` says what can and can't be measured from here.\n\nThe method goes in `method/`. Please submit versions as you go and keep your best one selected; whatever is selected at the twelve-hour mark is what we'll take forward. We won't know how it does until after you've finished.\n\nWorking directory: /workspace/repo\nDeliverable paths, relative to the working directory: method. A version contains these paths and nothing else.","wallet":[[9.2,125],[426.8,124.85],[707.4,123.37],[909.1,122.21],[1039,121.21],[1307.9,120.53],[1400.8,119.68],[1684.7,119.25],[1694.9,119.19],[2115.8,118.3],[2125,118.17],[2539.5,116.52],[2553.6,116.23],[2677.1,116.21],[2808.3,116.23],[3402.7,115.41],[3618.1,114.92],[3670.3,114.72],[3831.3,114.09],[4098.7,112.9],[4134.5,112.75],[4343.6,112.7],[4663,111.77],[4822.1,111.22],[4893.3,110.9],[4920.3,110.88],[5365.9,109.46],[5494.2,109.23],[5562.6,109.04],[5637.9,109.01],[5925.4,108.8],[6064,105.92],[6210.8,106.75],[6298.5,105.51],[6647.3,104.28],[6665.8,104.02],[6722.8,103.92],[6804,103.77],[7412.8,99.46],[7433.3,99.18],[7494.7,98.95],[7947.2,94.69],[7977,94.52],[8431.2,93.27],[8486.8,93.05],[8533.3,92.7],[8848.4,89.74],[9267.8,86.93],[9782.4,85.48],[9891,85.25],[10142.2,84.36],[10265.3,83.7],[10273.9,83.36],[10337.8,83.07],[10801.1,81.74],[10917.3,81.68],[10956,81.31],[11342.4,78.28],[11390.3,77.93],[11459.6,77.94],[11501.4,77.25],[11555.8,73.98],[11679.7,74.22],[11769.4,74.31],[11918.3,72.34],[12400.6,70.66],[12450.8,70.35],[12699.9,69.18],[13020.8,67.78],[13053.1,67.53],[13314.5,64.88],[13745,63.24],[14006.8,60.5],[14023.2,60.02],[14435.8,55.44],[14456.2,55.14],[14980.9,53.15],[15541.4,51.11],[15701.8,50.84],[16133,46.34],[16241.7,46.3],[16486.5,46.23],[16509.7,46.18],[16557.9,46.11],[16929.5,44.96],[17053.9,42.58],[17099.4,43.72],[17388.5,42.15],[17421.5,42.05],[17469.1,41.01],[17553.1,40.69],[17970.3,35.89],[18005.9,35.5],[18065.7,35.15],[18403.1,30.99],[18470.9,30.55],[18511.7,29.75],[19123.9,27.68],[19561.2,24.44],[20052,20.59],[20333.4,19.67],[20362.7,19.36],[20794.2,17.19],[20857.2,17.05],[21115.9,16.3],[21157.2,16.22],[21229,16.08],[21330.8,15.64],[21600.3,14.15],[21638.8,13.83],[22081.7,12.5],[22223.8,11.99],[22632.9,10.99],[22714.9,10.81],[23146.8,9.36],[23534.5,8.16],[23974.8,6.02],[24286.7,5.21],[24398.3,5.02],[24713.3,5.01],[25173.5,4.46],[25194.1,4.31],[25390.4,4.17],[25519,4.14],[25580,4.13],[25650.4,4.12],[25899.3,4.03],[26150.7,3.85],[26497.7,3.72],[26705.8,3.59],[27415.3,1.77],[28202.5,0.78],[28233.9,0.21],[28297.1,0.16],[28669.2,0.14],[28704,0.12],[28985.3,0.09],[32119.3,0.07],[32176.4,0.04],[32265.8,0.02],[33080.1,0.01]],"selection":[[9.2,"none"],[16991.1,"version 1 (being handed in)"],[17015.4,"version 1"],[24467.1,"version 2 (being handed in)"],[24477.2,"version 2"],[25595.8,"version 3 (being handed in)"],[25650.4,"version 3"],[26175.1,"version 4 (being handed in)"],[26206.6,"version 4"],[26730.4,"version 5 (being handed in)"],[26743.2,"version 5"],[28366.9,"version 6 (being handed in)"],[28375.7,"version 6"],[28695.7,"version 7 (being handed in)"],[28704,"version 7"],[32119.3,"version 8"],[32661.7,"version 9 (being handed in)"],[32679.4,"version 9"],[33300.5,"version 10 (being handed in)"],[33322.9,"version 10"],[34078.7,"version 11 (being handed in)"],[34142.3,"version 11"],[34558.5,"version 12 (being handed in)"],[34596.4,"version 12"],[35027.4,"version 13 (being handed in)"],[35096.1,"version 13"],[35453.1,"version 14 (being handed in)"],[35538,"version 14"],[36970.4,"version 15 (being handed in)"],[37005.8,"version 15"],[37574.4,"version 16 (being handed in)"],[37591.2,"version 16"],[38359.3,"version 17 (being handed in)"],[38412.6,"version 17"],[38893.2,"version 18 (being handed in)"],[38953.9,"version 18"]],"ix":{"t":[7.9,12.9,17.4,21.2,24.9,29.3,34.9,239.9,246.5,270.8,278.3,305.7,330.2,334.3,340.5,362.1,410.6,422.3,426.5,431.5,437.2,503.5,530.4,537,562.1,598.6,713.9,732.9,913.8,923,932.9,1047.8,1056.2,1069,1359.5,1406.8,1425.8,1694.1,1730.2,1782.7,1829.8,2124.4,2166.4,2193.7,2230.8,2266.2,2553,2635.9,2642.8,2650.9,2660.4,2807.8,2833.4,2880.1,2895.3,2950.4,2962.1,3007.6,3081.1,3142.6,3162.2,3410.6,3417.6,3625,3680.8,3838.1,3847.4,3858.2,4134,4143,4371.6,4480.1,4671.5,4840.4,4899,4945.1,5373.6,5502.1,5571.3,5655.5,5961,6019,6045.4,6078.3,6092.6,6218,6225.9,6308.6,6665.2,6682.3,6721.9,6730.2,6741.7,6753.8,6803.3,6812.3,7421.4,7432.7,7443.6,7463.8,7484.8,7494.4,7516.7,7526.4,7960.6,7997.4,8010.6,8458.9,8497.5,8518.1,8533,8544.7,8553.4,8567.7,8855,8867,9281.8,9790.4,9900.9,10153.5,10273.7,10286.3,10323.9,10336.9,10353.9,10380.6,10827.5,10841,10893.2,10923.4,10981.8,11355.2,11387.8,11398.6,11476.3,11499.4,11509.8,11599.3,11689.1,11697.3,11716,11780.7,11929.7,11952.9,11974.9,12450.1,12459.2,12720.1,13052.6,13062.4,13074.9,13102.1,13113.9,13324.1,13765.9,14022.9,14034.6,14455.4,14480.1,15000.8,15550.6,15712.4,16140.4,16255.6,16326.7,16346.2,16509.1,16524.4,16557,16568.5,16950.4,16965.4,16976.7,16983.4,16990.7,17015.1,17053.1,17063.9,17076.4,17099.1,17110.4,17147.9,17421,17432.4,17457.7,17551.8,17569.6,17987.8,18005.5,18022.3,18064.4,18071.8,18420.4,18458.8,18510.7,18523.4,19130.4,19140.6,19571.4,20062.8,20362.2,20373.5,20805.7,20821,20835.5,20856,20900.3,21157,21211.2,21240.1,21359.5,21638.3,21651.2,21661,22090.4,22232.2,22640.5,22726.1,23155.7,23190.6,23203.9,23543.4,23554,23986.1,24297.7,24413.6,24430,24447.4,24457.1,24466.7,24476.9,24546.1,24554.4,24588.8,24638.9,24657.4,24671,24712.4,24735.2,24752.9,25193.9,25217,25231.8,25239.8,25398.3,25527.7,25539.4,25553.9,25588.7,25595.4,25635.1,25682.9,25698.8,25910,26162.2,26168.7,26174.7,26206.1,26225.2,26238.9,26251.3,26259.5,26276.2,26297.1,26505.1,26714.5,26724.2,26730,26742.9,26752.3,26776.3,26787.1,26802.2,26842.8,26859.4,26874.6,27440.1,27516.2,27781.6,28232.9,28242.8,28266.7,28296.2,28311.6,28353,28366.5,28375.4,28384.2,28407.7,28442.6,28451.8,28470.7,28518.2,28680,28687.9,28695.3,28703.7,28716.3,28726.6,28768.4,28789.9,28957.8,29009,29094.3,29208.5,29213.7,29220.8,29229.7,29235.7,29256.7,29261.9,29265.2,29273.6,29300.2,29340.6,29347.9,29352.4,29358.9,29385.4,29392.8,29423,29446.2,29455.8,29466.3,29471.5,29506.8,29517.3,29581.8,29685.6,29694,29699.9,29823.4,29852,29861,29876.5,29905.3,29917.6,29925.4,30015,30045.6,30107.3,30115.8,30144.2,30164,30171.9,30242.2,30309.3,30315.5,30395.9,30455.2,30483.6,30515.7,30742,30873.5,30890.8,30902.2,30926.4,30946.4,30953.5,30967.4,30984.7,30992.6,31000.2,31075.4,31114,31131.3,31244.3,31252.7,31256.8,31325.7,31334.2,31346.9,31381.1,31419.2,31527.6,31639.5,31670.8,31762.5,31899.7,31956.6,31963.6,31978.3,31996.1,32006.5,32032.3,32079.9,32133.3,32265.3,32285.7,32311,32324.5,32481.4,32501.6,32639.1,32646.1,32661.4,32679,32689.1,32698.2,32721.8,32767.3,32786.7,32842.1,32867.4,32875.6,32887.9,32894,32933.7,32958.5,32979,33000.8,33043,33058.1,33090.7,33108.3,33161.6,33211.2,33229,33289.3,33295.6,33300.1,33322.5,33328.3,33335.3,33474.3,33490.4,33503.8,33530.3,33563.2,33598.9,33612.9,33620.8,33625.7,33648.9,33672.5,33694.3,33714.4,33874.6,33932.9,33947.8,34006.3,34060.3,34073.2,34078.3,34114.5,34162.2,34235.6,34240.7,34263.3,34277.8,34404.8,34532,34538.9,34546.7,34558.2,34596.1,34619.3,34640.2,34652.8,34725.1,34871.7,34881.4,34923.8,35021.3,35027.1,35050.1,35133.8,35158.2,35276.1,35393,35400.4,35407.4,35452.8,35476.8,35548.8,35607.5,35630.3,35690.5,35743.9,35759.1,35780.6,36028,36198.4,36240.2,36250.6,36457.7,36678.9,36734.5,36750.6,36859.5,36931.2,36937.9,36947.5,36970,37005.5,37027.1,37038.1,37194.6,37355.8,37410.1,37419.3,37500.7,37568.8,37574.1,37590.9,37604.3,37641,37690.2,37913.1,37941.6,37955.1,37968.9,38041,38112,38124.3,38288.6,38340.4,38347.9,38358.8,38371.6,38421.9,38455.1,38477.7,38563.1,38576.4,38762.3,38876.5,38884.5,38892.8,38907.9,38971.3,38984.6,39158.2,39172.4,39786.3,40400.7,41007.4,41014.6,41620,42166.5,42185.2,42185.3,42192,42197.9,42624.7,42632.6,42632.6,42638.5,42945.7,42945.8,42951.7],"k":"uuuuuuuuuueeuueeeuueueuueeueueueeeeeueereeeeeeeuuueeeeueueeeeeuuuuuueueeueueuueueeeueueuereueeeuueueeeeueueeeeeuuuueuuuereueeuueeuuueueuueeeeeeeeeuueeeeueeuueeuuuueeeeeuuuuuhheeeueeueueeereeeueeeuuuuueuueeeeuereeeuuuuuueueeuuueehheuuueeeueuueeuuueehhueuuehheeeeueeuuhheueuueeuuueuueeuhheueeuuuuhhuuueueeeyuuuuuuuuuuueeueuuuuuueuuuueeueeeeeeeeeueeeuereueueuuuueueeeeeuuueueeeeeuuueueehehuueeeeuehheerruueeuuuueuueueeeeeuhheuuueeeruuuuueeeueeeuhheeuueeuehuhureeeeeuhhuueeehuhuueeeeueueeuuueuueuhhueuueuuuhhueuuueeeeuueuhheueeeueehheruuuuuuuuusyuursyusyu","e":"00000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","x":"00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000011000110110","c":"11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111110111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111100111001001"},"steps":[{"t":7.9,"r":"I should start by exploring the workspace to see what's already there.","$":0.028,"c":[{"n":"bash","a":"cd /workspace/repo && ls -la && cat /lab/MANUAL.md","e":0,"o":["total 21\ndrwxr-xr-x 1 agent agent  190 Nov 14 19:57 .\ndrwxr-xr-x 1 agent agent   26 Nov 12 20:35 ..\n-rw-r--r-- 1 agent agent    5 Nov 14 19:57 .python-version\n-rw-r--r-- 1 agent agent 4454 Nov 14 19:57 README.md\ndrwxr-xr-x 1 agent agent 4096 Nov 14 19:57 docs\ndrwxr-xr-x 1 agent agent   92 Nov 14 19:57 envkit\ndrwxr-xr-x 1 agent agent  115 Nov 14 19:57 method\ndrwxr-xr-x 1 agent agent  167 Nov 14 19:57 offbook\n-rw-r--r-- 1 agent agent  984 Nov 14 19:57 pyproject.toml\ndrwxr-xr-x 1 agent agent 4096 Nov 14 19:57 skills\ndrwxr-xr-x 1 agent agent 4096 Nov 14 19:57 tests\n# Lab0 manual\n\nWelcome to the lab. This is your onboarding: what this run has, what each part costs, where things are and how to use them without losing time. Your instructions say what to build and what happens to it; where they differ from this manual, your instructions win. `lab manual` prints it again at any time.\n\n## This run\n\n|  |  |\n|---|---|\n| Time | Exactly 12 hours from the start of the run. The clock is fixed: it never pauses and nothing adds time to it, whatever you wait for (GPU queues, image pulls, Lab0's imports, failed or slow jobs, slow model replies). Every run shares the same lab at once, so plan for waits. `lab status` shows the time left. |\n| Wallet | $125.00. It pays for model calls made by your code and Tinker training and sampling. `lab status` shows what is spent, reserved and available. |\n| GPUs | None: the workspace has no GPUs and Lab0 sells no GPU time in this run. Tinker trains and samples on its own hosted GPUs, paid per token (see Tinker). |\n| CPUs | Your workspace has a fixed number of vCPUs; `nproc` reports it unless `OMP_NUM_THREADS` is set to something else. More processes or threads than vCPUs take turns on them and run slower. |\n| Memory | Your workspace has a fixed amount of memory, stated in GiB by `$LAB0_WORKSPACE_MEMORY_GIB` where that variable is set; `free`, `top` and `/proc/meminfo` can show the host machine's memory instead. Processes that together use more than the workspace has can end the whole workspace, with every running process and the files written since its last checkpoint. |\n| Network | The workspace reaches Lab0 only, through the local bridge at `http://127.0.0.1:8099` (also the Unix socket `/run/lab0/bridge.sock`). `lab` and the Python client use it. |\n\n## Getting around\n\n|  |  |\n|---|---|\n",15757,"\n## Commands\n\n| Command | Purpose |\n|---|---|\n| `lab status` | Time left, wallet, running jobs |\n| `lab spend` | What the wallet paid for, minute by minute, and the recent rate |\n| `lab prices` | Every price |\n| `lab manual` | This manual |\n| `lab upload`, `lab download`, `lab artifacts` | Your artifacts |\n| `lab submit`, `lab pick`, `lab submissions` | Handing in: the version handed over when time is up |\n| `lab chat`, `lab models list|show` | Model calls and their routes |\n| `lab tinker runs|export` | Tinker runs and exports |\n| `lab papers search|show|read|list` | The arXiv mirror (any arXiv client works too) |\n| `lab datasets search|get` | The datasets mirror |\n\n`lab COMMAND --help` describes each command's options. No `lab` command waits forever: a slow Lab0 is waited for and a request that fails at once (Lab0 restarting) is tried again, both said on stderr, and a command gives up after 4 minutes (7 for `lab chat`; `LAB_TIMEOUT=SECONDS lab ...` changes it). Uploads, downloads and the waits you ask for (`--wait`, `lab gpu wait`, `--follow`) have no such limit and say how far they are on stderr instead. `lab health` says whether the bridge and Lab0 answer."],"d":1.4},{"n":"bash","a":"cd /workspace/repo && find . -maxdepth 2 -not -path './.git/*' | head -200","e":0,"o":".\n./.python-version\n./README.md\n./docs\n./docs/budget.md\n./docs/cookbook\n./docs/evaluation.md\n./docs/interface.md\n./docs/pi\n./docs/prices.md\n./docs/rules.md\n./docs/tinker\n./docs/tinker-sdk\n./envkit\n./envkit/README.md\n./envkit/checks.py\n./envkit/examples\n./envkit/template\n./method\n./method/README.md\n./method/__init__.py\n./method/config.toml\n./method/openrouter.py\n./method/rehearse.py\n./method/run.py\n./offbook\n./offbook/__init__.py\n./offbook/_text.py\n./offbook/harness.py\n./offbook/method_api.py\n./offbook/model_server.py\n./offbook/pi.py\n./offbook/sandbox.py\n./offbook/skills.py\n./offbook/tasks.py\n./pyproject.toml\n./skills\n./skills/README.md\n./skills/analytical-method-validation\n./skills/audit-support\n./skills/avoid-ai-writing\n./skills/backtesting-frameworks\n./skills/bash-defensive-patterns\n./skills/cantera\n./skills/capa-officer\n./skills/caption-qa\n./skills/ctf-crypto\n./skills/data-quality-auditor\n./skills/draw-io-diagram-generator\n./skills/embedded-systems\n./skills/experimental-design\n./skills/financial-analyst\n./skills/genomic-coordinates\n./skills/git-advanced-workflows\n./skills/image-manipulation-image-magick\n./skills/json-canvas\n./skills/jupyter-notebook\n./skills/marine-carbonate-chemistry\n./skills/matlab\n./skills/meeting-analyzer\n./skills/molecular-dynamics\n./skills/pandas-pro\n./skills/property-based-testing\n./skills/pybamm\n./skills/reconciliation\n./skills/risk-metrics-calculation\n./skills/sarif-parsing\n./skills/scientific-critical-thinking\n./skills/semgrep-rule-creator\n./skills/simpy\n./skills/slack-gif-creator\n./skills/sql-queries\n./skills/statistical-power\n./skills/systematic-debugging\n./skills/triage-nda\n./skills/validate-data\n./skills/variance-analysis\n./skills/vulnerability-triage-brocards\n./skills/writing-lean-proofs\n./skills/yara-rule-authoring\n./skills/zarr-python\n./tests\n./tests/conftest.py\n./tests/fake_pi.py\n./tests/fixtures\n./tests/test_envkit.py\n./tests/test_harness.py\n./tests/test_method.py\n./tests/test_method_api.py\n./tests/test_model_server.py\n./tests/test_pi.py\n./tests/test_sandbox.py\n./tests/test_skills.py\n./tests/test_tasks.py","d":0}]},{"t":12.9,"$":0.1491,"c":[{"n":"bash","a":"cd /workspace/repo && cat README.md && echo ------- && cat docs/interface.md && echo ------- && cat docs/budget.md","e":0,"o":["# Off Book\n\nOur agents carry skill libraries: folders of expert procedure, often with scripts, that the agent sees listed in its system prompt and reads when a task calls for one. On published benchmarks, curated skills like these make agents markedly better at specialist work. But every skill costs context on every task, the agent has to notice which one applies, and the library has to travel with the agent.\n\nWe want the benefit in the weights. Given a library of skill packages and nothing else (no tasks, no labels, no checks), a **method** should produce a LoRA adapter for Nemotron 3.5 Super that, with no skill in context and no hint of which one applies, does what Super does with the skills loaded, and loses nothing elsewhere. Here you can train and sample Lightning, and sample Super. We run the method once, after you finish, on Super, on a library of experts' skills it has never seen, and measure the adapter on held-out tasks.\n\n## What is here\n\n| Path | What |\n|---|---|\n| `offbook/` | the shared package, also used by the evaluator: task loading (`tasks`), skill packages and which skills each condition gives the agent (`skills`), a local sandbox for the cookbook's `SandboxInterface` (`sandbox`), the fixed agent harness (`harness`): a Nemotron model in Pi 1.4.0 (`pi`) through a local OpenAI-compatible model server around the Tinker sampler (`model_server`), and the method contract with the pinned prices (`method_api`) |\n| `skills/` | the development library: public, expert-written skill packages, verbatim with their licences. None of them is in the final library |\n| `envkit/` | the Harbor task format, a task template, verifier helpers and one worked example task |\n| `method/` | the starter method (cookbook prompt distillation), which trains `BUDGET.json`'s base model, and `rehearse.py`, which runs a method on Lightning |\n| `docs/` | [budget](docs/budget.md), [evaluation](docs/evaluation.md), [interface](docs/interface.md), [rules](docs/rules.md), [prices](docs/prices.md), and offline clones of the [Tinker docs](docs/tinker/), the [Tinker SDK](docs/tinker-sdk/), the [cookbook](docs/cookbook/) and [Pi's source](docs/pi/) ([snapshot](docs/tinker/SNAPSHOT.md)) |\n| `tests/` | unit tests for `offbook`, `envkit` and the starter; no network or Tinker calls. Episodes run with a stand-in for Pi, and with Pi itself (`tests/test_pi.py`) where it is installed |\n\n## Models and services\n\n",11782,"\n## The research wallet\n\n- **$125, one pot,** for Tinker and for OpenRouter calls made by your code. No GPUs are offered.\n- **Tinker:** Lightning can be trained and sampled; Super can only be sampled (the final run trains it). LoRA rank is limited as [rules.md](rules.md) says (at most 64 on Lightning), and at most four training runs are open at once.\n- **Metering:** `forward` and `forward_backward` are charged for every token in the batch at the training price, reserved exactly. A sampling request reserves its prompt at the prefill price plus `num_samples × max_tokens` at the sampling price, and settles on the tokens actually generated. `optim_step` and checkpoints are free.\n- **Checkpoints exist only during the run.** Nothing trained in research reaches the final run: the method trains its adapter during that run.\n- A rehearsal at the full final budget would cost $180 ($130 + $50), more than the wallet, and could still train only Lightning. `python -m method.rehearse` runs a method in the final run's shape on Lightning, on part of the development library, at a fraction of the final budget ([../method/README.md](../method/README.md))."],"d":0.4},{"n":"bash","a":"cd /workspace/repo && cat docs/rules.md && echo ------- && cat docs/evaluation.md && echo ------- && cat docs/prices.md","e":0,"o":["# Rules\n\n## Models and services\n\nLab0 enforces these in research and in the final run:\n\n- **Research:** Lightning (`nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16`) can be trained and sampled through Tinker. Super (`nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16`) can only be sampled: creating a training client on it is refused.\n- **The final run:** the method trains Super (`BUDGET.json`'s `base_model`) and may sample Super and Lightning. Lightning cannot be trained there.\n- Only the 64K-context variants of both are offered.\n- **LoRA rank:** Lab0 allows at most 128, and Tinker applies its own limit for each model. It refuses Lightning above 64. Super's limit can't be tried here, since research can't train Super, so a method that asks for a high rank has to cope with being refused.\n- Tinker calls that load checkpoints from outside the run or run, publish or copy weights, use the OpenAI- or Anthropic-compatible endpoints, or override `base_url` are refused.\n- **OpenRouter:** only the routes of Lab0's frozen catalog. In research, requests with tools, plugins, web search, streaming or provider settings are refused.\n- Every service stops at its budget: the wallet in research, and each of the final run's budgets ([budget.md](budget.md)).\n\n## The final run\n\n- The method receives the skill packages, `BUDGET.json` and an output directory, and nothing else ([interface.md](interface.md)).\n- Of the method's output, only `checkpoint.json` is used. It must name Super, and its sampler weights must be ones the method's own run saved on Super: the evaluator checks Lab0's record of the run, and a checkpoint saved anywhere else, in research for example, makes the result invalid.\n- The adapter is evaluated with our harness, decoding and prompts, which the method cannot change.\n\n## What a method may not do\n\n- **Carry skill-specific material.** The package holds code, configuration and general-purpose data. Data, prompts or weights prepared for particular skills, whether guessed or known, are not allowed: everything specific to a skill is made during the run, from the packages it receives.\n- **Use the evaluation tasks.** They are drawn from public agent benchmarks. Nothing from or about a public agent benchmark that could be one of them (its tasks, tests, solutions, task names or trajectories) may be in the package, in prompts the method sends, or in data it trains on, whatever the source, a model's recollection included.\n",11250,"| `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16` (64K context) | research: sample only; final run: train and sample | 1.14 | 1.14 | 2.88 | 2.552 (final run only) |\n\nHow usage is counted:\n\n- **Prefill:** prompt tokens processed by a sampling request, including prompts measured for logprobs (`compute_logprobs` is a sampling request for one token). Every prompt token is charged at the prefill price, including tokens Tinker serves from its prefix cache (responses report those as `prompt_cache_hit_tokens`), so a multi-turn episode pays for its whole context on every turn.\n- **Sample:** generated tokens.\n- **Train:** tokens in every `forward` and `forward_backward` batch; a training client's forward-only pass is charged as training. `optim_step` is free.\n- **Checkpoints** are free during the run.\n- Only the 64K-context variants are offered.\n\nThe same table is `offbook.method_api.TINKER_PRICES`, and `offbook.method_api.tinker_cost(model, prefill=…, cached_prefill=…, sample=…, train=…)` prices token counts with it.\n\nOpenRouter routes are charged at the prices of Lab0's frozen route catalog, which the lab manual lists."],"d":0}]},{"t":17.4,"r":"I'm checking the starter method and the surrounding code to understand the full picture.","$":0.1874,"c":[{"n":"bash","a":"cd /workspace/repo && cat method/README.md && echo ------- && cat method/config.toml && echo ------ && wc -l method/*.py && cat method/run.py","e":0,"o":["# The starter method\n\n`method/` implements the method interface ([../docs/interface.md](../docs/interface.md)) with the cookbook's prompt distillation, applied to a skill library:\n\n```\npython -m method.run --skills SKILLS_DIR --budget BUDGET.json --out OUT\n```\n\nIt trains and samples one model, `BUDGET.json`'s `base_model`: Nemotron 3.5 Super in the final run, Lightning in a rehearsal here. Its code names neither model.\n\n## What it does\n\n1. **Requests** (OpenRouter). For each skill, the writer model in `config.toml` reads the skill's SKILL.md (cut to `skill_chars` characters) and the names of its other files, and returns `requests_per_skill` self-contained requests that the skill would help with. One call per skill. Requests are saved to `OUT/data/requests.jsonl`.\n2. **Answers** (Tinker sampling). Requests are taken round-robin across skills. The base model answers each one in the harness, Pi (`offbook.harness.arun_episode`), in the curated condition with only that request's skill, in a fresh sandbox, with no verifier. Episodes use the harness defaults except for `[answers]` in `config.toml` (20 model calls, 10 minutes per episode). Every episode Pi finished (`stop == \"done\"`) is kept.\n3. **Training** (Tinker training). Each kept transcript becomes one cross-entropy example: the episode's last request as Pi sent it, with its response (the request, every response with its thinking and tool calls, every tool result; after a compaction, Pi's summary and what followed), under the no-skills form of Pi's system prompt, that is, with its skills section removed (`offbook.pi.strip_skills`), and with the tools Pi declared. Only the assistant turns are trained (`TrainOnWhat.ALL_ASSISTANT_MESSAGES`). A rank-32 LoRA on the base model is trained with Adam at learning rate 1e-4, decaying linearly to 0, in batches of 32 for 4 epochs.\n4. **Checkpoints.** Sampler weights are saved every 25 steps and at the end, and each save rewrites `OUT/checkpoint.json`. If no step could be trained, the freshly created adapter is saved as step 0.\n\n## Budget and logs\n\n- Spend is estimated from token counts: Tinker at `offbook.method_api.TINKER_PRICES`, OpenRouter at the writer prices in `config.toml` (copied from Lab0's price list).\n",23745,"    log.info(\"budget: $%.2f Tinker, $%.2f OpenRouter, %.0f s, seed %d; %d skills\", budget.tinker_usd,\n             budget.openrouter_usd, budget.wall_s, budget.seed, len(skills))\n\n    from offbook.harness import Sampler\n    from offbook.sandbox import bubblewrap_support\n\n    workdir = \"/root\" if bubblewrap_support() is not None else \"the working directory\"\n    spend.stage = \"requests\"\n    requests = interleave(write_requests(skills, config, spend, clock, out, workdir), budget.seed)\n    spend.stage = \"answers\"\n    sampler = Sampler(budget.base_model)\n    kept = asyncio.run(answer_requests(requests, skills, config, spend, clock, out, sampler=sampler))\n    spend.stage = \"training\"\n    data = build_data(kept, sampler, config[\"train\"][\"max_length\"])\n    log.info(\"training data: %d examples from %d transcripts\", len(data), len(kept))\n    steps = train(data, config, spend, clock, out)\n    spend.stage = \"done\"\n    spend.notes[\"steps\"] = steps\n    spend.save()\n    log.info(\"done: %d steps, $%.2f Tinker, $%.2f OpenRouter, %.0f s\", steps, spend.tinker_usd,\n             spend.openrouter_usd, clock.elapsed)\n    return 0\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())"],"d":0.2}]},{"t":21.2,"$":0.1768,"c":[{"n":"bash","a":"cd /workspace/repo && cat method/openrouter.py && echo ------ && cat method/rehearse.py","e":0,"o":["\"\"\"A minimal OpenAI-compatible chat client (standard library only) for OpenRouter routes.\"\"\"\n\nfrom __future__ import annotations\n\nimport json\nimport os\nimport time\nimport urllib.error\nimport urllib.request\nfrom dataclasses import dataclass\n\n\nclass ChatError(RuntimeError):\n    pass\n\n\nclass EmptyReply(ChatError):\n    \"\"\"The route answered, with no text: a provider's content filter, a refusal, or a reply cut off (`length`) before\n    any content, such as one that spent its tokens on reasoning. `finish_reason` is the route's own word for why\n    (None if it gave none); the token counts are what the reply's usage reported, since such a reply can be billed.\"\"\"\n\n    def __init__(self, finish_reason: str | None, prompt_tokens: int, completion_tokens: int):\n        super().__init__(f\"the reply holds no text (finish_reason: {finish_reason or 'not given'}; \"\n                         f\"usage: {prompt_tokens} prompt and {completion_tokens} completion tokens)\")\n        self.finish_reason = finish_reason\n        self.prompt_tokens = prompt_tokens\n        self.completion_tokens = completion_tokens\n\n\n@dataclass(frozen=True)\nclass ChatResult:\n    text: str\n    prompt_tokens: int\n    completion_tokens: int\n    finish_reason: str | None = None   # the route's own: \"stop\", \"length\", \"content_filter\", ...\n\n\ndef endpoint_from_env() -> tuple[str, str]:\n    \"\"\"``OPENROUTER_BASE_URL``/``OPENROUTER_API_KEY`` (the final run), else ``OPENAI_BASE_URL``/``OPENAI_API_KEY``.\"\"\"\n    base = os.environ.get(\"OPENROUTER_BASE_URL\") or os.environ.get(\"OPENAI_BASE_URL\")\n    key = os.environ.get(\"OPENROUTER_API_KEY\") or os.environ.get(\"OPENAI_API_KEY\")\n    if not base or not key:\n        raise ChatError(\"set OPENROUTER_BASE_URL and OPENROUTER_API_KEY\")\n    return base.rstrip(\"/\"), key\n\n\nclass ChatClient:\n    def __init__(self, base_url: str | None = None, api_key: str | None = None, *, timeout_s: float = 600,\n                 retries: int = 2):\n        if base_url is None or api_key is None:\n            env_base, env_key = endpoint_from_env()\n            base_url = base_url or env_base\n            api_key = api_key or env_key\n        self.url = base_url.rstrip(\"/\") + \"/chat/completions\"\n        self.api_key = api_key\n        self.timeout_s = timeout_s\n        self.retries = retries\n\n    def chat(self, model: str, messages: list[dict], *, max_tokens: int, **params) -> ChatResult:\n        \"\"\"One completion. `params` are further request fields (temperature, seed, ...): send only those the route\n",9225,"    budget = Budget.final(seed=args.seed).scaled(fraction)\n    if args.wall_s is not None:\n        budget = Budget.from_dict({**budget.to_dict(), \"wall_s\": args.wall_s})\n    budget_path = budget.save(out / \"BUDGET.json\")\n    print(f\"rehearsing {args.method.name} on {len(chosen)} skills: ${budget.tinker_usd:.2f} Tinker, \"\n          f\"${budget.openrouter_usd:.2f} OpenRouter, {budget.wall_s:.0f} s\", flush=True)\n\n    run = run_method(args.method.resolve(), library, budget_path, out / \"out\", budget.wall_s, out / \"method.log\")\n    summary: dict = {\"skills\": [p.name for p in chosen], \"fraction\": fraction, \"budget\": budget.to_dict(), **run}\n    try:\n        summary[\"checkpoint\"] = read_checkpoint(out / \"out\").to_dict()\n    except CheckpointError as exc:\n        summary[\"checkpoint_error\"] = str(exc)\n    spend = out / \"out\" / \"spend.json\"\n    if spend.is_file():\n        summary[\"method_spend_estimate\"] = json.loads(spend.read_text())\n    (out / \"rehearsal.json\").write_text(json.dumps(summary, indent=2) + \"\\n\")\n    print(json.dumps(summary, indent=2))\n    return 0 if \"checkpoint\" in summary else 1\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())"],"d":0.4},{"n":"bash","a":"cd /workspace/repo && wc -l offbook/*.py && cat offbook/harness.py","e":0,"o":["    29 offbook/__init__.py\n    62 offbook/_text.py\n   424 offbook/harness.py\n   253 offbook/method_api.py\n   855 offbook/model_server.py\n   253 offbook/pi.py\n   725 offbook/sandbox.py\n   161 offbook/skills.py\n   437 offbook/tasks.py\n  3199 total\n\"\"\"The fixed agent harness, used in research, in a method's runs and in evaluating its result.\n\nThe agent is Pi 1.4.0, the coding-agent harness (github.com/earendil-works/pi, MIT; `offbook.pi`), with the\nsampler's model: in evaluation, Nemotron 3.5 Super with or without the adapter; in research, whichever Tinker model the\n`Sampler` names. One episode::\n\n    start a model server for this episode on 127.0.0.1 (`offbook.model_server.ModelServer`, around the `Sampler`)\n    write a fresh Pi configuration that points Pi at it (`offbook.pi.write_config`)\n    run Pi in the task's sandbox, in the task's working directory, with the instruction as its prompt: its tools\n        (read, bash, edit, write), its system prompt, its compaction; in the curated condition the task's skills\n        are its only skills, otherwise it has none\n    until Pi ends its run (the model answers without a tool call), the model server refuses a request at the call\n        limit, or the wall clock runs out; then everything Pi started is stopped\n    run the task's verifier, however the episode ended; reward = the number it writes to /logs/verifier/reward.txt\n\nLimits (`Limits`):\n\n- ``max_turns`` model calls per episode, Pi's compaction summaries included; the model server refuses the next one,\n  which ends Pi's run;\n- ``max_context_tokens``: the model server refuses a prompt that leaves less than the 8,192-token response allowance\n  in the context, with the error Pi recovers from by compacting and retrying once; Pi also compacts on its own when\n  the context passes 64,000 less 16,384 tokens;\n- ``wall_s`` for the whole episode (default: the task's agent timeout). Pi's commands have no time limit of their\n  own unless the model gives one (bash's ``timeout``).\n\n`EpisodeResult.stop` is ``done`` (Pi ended its run), ``max_turns``, ``wall_time`` or ``error`` (Pi ended on an\nerror, such as a context it could not compact enough, or exited abnormally). A sampling failure is not an outcome:\nit is raised, as is a Pi that cannot start. The verifier runs after every stop, as Harbor runs a task's verifier\nafter its agent: what the agent left in the environment is what is checked.\n\n",15970,"                         compactions=result.compactions, overflows=server.overflows, pi_exit_code=exit_code)\n        return result\n    except BaseException as exc:\n        transcript.write(\"error\", error=f\"{type(exc).__name__}: {exc}\",\n                         turns=server.calls if server else 0, tokens=dict(server.tokens) if server else {})\n        raise\n    finally:\n        if process is not None:\n            await process.stop()\n        if server is not None:\n            await server.close()\n        if config_dir is not None:\n            shutil.rmtree(config_dir, ignore_errors=True)\n        transcript.close()\n        if owns_sandbox and sandbox is not None:\n            await sandbox.cleanup()\n\n\ndef run_episode(\n    task: Task,\n    *,\n    sampler: SamplerLike,\n    skills: SkillsMode,\n    seed: int,\n    limits: Limits,\n    workdir: str | Path,\n    sandbox: Any = None,\n    verify: bool = True,\n) -> EpisodeResult:\n    \"\"\"Blocking form of `arun_episode` (runs its own event loop).\"\"\"\n    return asyncio.run(\n        arun_episode(task, sampler=sampler, skills=skills, seed=seed, limits=limits, workdir=workdir,\n                     sandbox=sandbox, verify=verify)\n    )"],"d":0}]},{"t":24.9,"$":0.2155,"c":[{"n":"bash","a":"cd /workspace/repo && cat offbook/model_server.py","e":0,"o":["\"\"\"A Nemotron model behind a local OpenAI-compatible endpoint: the model Pi talks to in every episode.\n\nPi (github.com/earendil-works/pi, `offbook.pi`) reaches its model through OpenAI's chat-completions API. The model\n(the sampler's base model, with or without an adapter: Super, with the adapter or without, in every evaluation\ncondition) is sampled through the Tinker SDK with the cookbook renderer ``nemotron3_ultra_preserve_thinking``\n(`Sampler`), so each episode starts a `ModelServer` on 127.0.0.1, on a random port, that serves `Sampler` to Pi, under\nthe model id ``nemotron`` whichever model it samples:\n\n- ``POST /v1/chat/completions``, streamed (server-sent events, as Pi asks) or not, and ``GET /v1/models``;\n- each request becomes a cookbook conversation: the system (or developer) messages first, then user, assistant (with\n  its ``reasoning_content`` as thinking and its ``tool_calls``) and tool messages, with the request's tool schemas as\n  cookbook tool specs, rendered by the renderer;\n- one response is sampled per request at the fixed settings (temperature 0.7, top-p 0.95, at most 8,192 tokens), with\n  a seed derived from the episode's seed and the request's place in the episode (`turn_seed`);\n- the response comes back as Pi's OpenAI provider parses it: the thinking as ``reasoning_content``, the visible text\n  as ``content``, and each of the renderer's XML tool calls as an OpenAI tool call whose JSON arguments take the\n  types of the tool's schema (`coerce_arguments`); ``finish_reason`` is ``tool_calls``, ``stop`` or ``length``, and\n  ``usage`` gives the exact rendered prompt length, the prompt-cache hits and the sampled tokens;\n- a prompt that leaves less than the per-response allowance in the context window (64,000 tokens) is refused with\n  HTTP 400 and the error Pi's compaction recovers from (\"maximum context length is 64000 tokens\");\n- after ``max_calls`` sampled responses (compaction summaries included) the next request is refused with HTTP 400,\n  which ends Pi's run; the harness records the episode as stopped at the limit;\n- a sampling failure is refused with HTTP 500 (not to be retried) and kept in `ModelServer.error`, which the harness\n  raises once the episode is over: an upstream fault is ours, not the episode's outcome. A malformed request (one the\n  renderer cannot build a prompt from, say) is refused with HTTP 400 and is not an error of ours;\n",26449,"    body = await reader.readexactly(length) if length else b\"\"\n    return method, path, headers, body\n\n\nasync def _write_response(writer: asyncio.StreamWriter, response: _Response, *, keep_alive: bool) -> None:\n    reason = _REASONS.get(response.status, \"OK\")\n    head = [f\"HTTP/1.1 {response.status} {reason}\"]\n    extra = [f\"{name}: {value}\" for name, value in response.headers.items()]\n    if response.events is not None:\n        payload = \"\".join(f\"data: {json.dumps(event, ensure_ascii=False)}\\n\\n\" for event in response.events)\n        payload += \"data: [DONE]\\n\\n\"\n        data = payload.encode(\"utf-8\")\n        head += [\"Content-Type: text/event-stream; charset=utf-8\", \"Cache-Control: no-cache\",\n                 f\"Content-Length: {len(data)}\", \"Connection: close\", *extra]\n    else:\n        data = json.dumps(response.body if response.body is not None else {}, ensure_ascii=False).encode(\"utf-8\")\n        head += [\"Content-Type: application/json\", f\"Content-Length: {len(data)}\",\n                 f\"Connection: {'keep-alive' if keep_alive else 'close'}\", *extra]\n    writer.write((\"\\r\\n\".join(head) + \"\\r\\n\\r\\n\").encode(\"latin-1\") + data)\n    await writer.drain()"],"d":0.2}]},{"t":29.3,"$":0.1925,"c":[{"n":"bash","a":"cd /workspace/repo && sed -n 300,460p offbook/model_server.py","e":0,"o":["    return found\n\n\ndef _json_or_text(raw: str) -> Any:\n    try:\n        return json.loads(raw)\n    except (json.JSONDecodeError, ValueError):\n        return raw\n\n\ndef coerce_value(raw: str, schema: Any) -> Any:\n    \"\"\"One raw XML parameter value as the type its JSON schema names; the raw text when it does not parse.\"\"\"\n    types = _schema_types(schema)\n    if not types:\n        return _json_or_text(raw)\n    if \"string\" in types:\n        return raw\n    text = raw.strip()\n    for kind in types:\n        if kind == \"integer\":\n            try:\n                return int(text)\n            except ValueError:\n                try:\n                    number = float(text)\n                except ValueError:\n                    continue\n                if number.is_integer():\n                    return int(number)\n        elif kind == \"number\":\n            try:\n                return int(text)\n            except ValueError:\n                try:\n                    return float(text)\n                except ValueError:\n                    continue\n        elif kind == \"boolean\" and text.lower() in (\"true\", \"false\"):\n            return text.lower() == \"true\"\n        elif kind == \"null\" and text in (\"null\", \"None\", \"\"):\n            return None\n        elif kind in (\"array\", \"object\"):\n            value = _json_or_text(text)\n            if isinstance(value, list if kind == \"array\" else dict):\n                return value\n    return raw\n\n\ndef coerce_arguments(arguments: dict[str, str], parameters: Any) -> dict[str, Any]:\n    \"\"\"Each raw argument as its tool's JSON schema types it (unknown arguments as JSON, or as text).\"\"\"\n    properties = parameters.get(\"properties\") if isinstance(parameters, dict) else None\n    properties = properties if isinstance(properties, dict) else {}\n    return {name: coerce_value(raw, properties.get(name)) if name in properties else _json_or_text(raw)\n            for name, raw in arguments.items()}\n\n\n@dataclass\nclass Reply:\n    \"\"\"A sampled response as Pi gets it, and as the transcript records it.\"\"\"\n\n    content: str\n    reasoning: str\n    #: OpenAI tool calls: {\"id\", \"type\": \"function\", \"function\": {\"name\", \"arguments\" (JSON)}}.\n    tool_calls: list[dict[str, Any]]\n    finish_reason: str\n    #: Tool-call blocks that did not parse; they stay in ``content`` as text.\n    unparsed: list[str] = field(default_factory=list)\n\n    def message(self) -> dict[str, Any]:\n",2966,"        finish = \"tool_calls\" if calls else \"stop\"\n    return Reply(content=content, reasoning=completion.thinking or \"\", tool_calls=calls, finish_reason=finish,\n                 unparsed=unparsed)\n\n\n# -- requests --------------------------------------------------------------------------------------\n\n\nclass RequestError(ValueError):\n    \"\"\"A request the server cannot serve (HTTP 400).\"\"\"\n\n\ndef _text(content: Any) -> str:\n    if content is None:\n        return \"\"\n    if isinstance(content, str):\n        return content\n    if isinstance(content, list):\n        return \"\".join(str(part.get(\"text\", \"\")) for part in content\n                       if isinstance(part, dict) and part.get(\"type\") in (\"text\", \"input_text\"))\n    return str(content)\n\n\ndef request_tools(body: dict[str, Any]) -> list[dict[str, Any]]:\n    \"\"\"The request's function tools as cookbook tool specs (name, description, JSON-schema parameters).\"\"\"\n    tools = []\n    for tool in body.get(\"tools\") or []:\n        function = tool.get(\"function\") if isinstance(tool, dict) else None\n        if not isinstance(function, dict) or not function.get(\"name\"):\n            raise RequestError(\"each tool must be a function with a name\")"],"d":0.4},{"n":"bash","a":"cd /workspace/repo && cat offbook/pi.py offbook/skills.py offbook/method_api.py offbook/__init__.py offbook/_text.py","e":0,"o":["\"\"\"Pi, the coding-agent harness every episode runs in: where it is installed, how it is configured and started.\n\nPi 1.4.0 (github.com/earendil-works/pi, MIT; npm ``@earendil-works/pi-coding-agent``) is installed with its own Node\nruntime at ``/opt/pi`` (``OFFBOOK_PI_HOME`` names another installation): ``node/bin/node`` and\n``node_modules/@earendil-works/pi-coding-agent``, from Pi's own install lockfile. Its source is in ``docs/pi/``.\n\nEach episode gets a fresh Pi configuration directory (`write_config`, Pi's ``PI_CODING_AGENT_DIR``):\n\n- ``models.json``: one custom provider, ``offbook``, using Pi's ``openai-completions`` API at the episode's\n  `offbook.model_server.ModelServer`, with one model, ``nemotron``, displayed under the sampler's base model's name\n  (`display_name`: \"Nemotron 3.5 Super\" when the episode samples Super, with or without an adapter): reasoning on, a\n  64,000-token context window and 8,192 tokens per response;\n- ``settings.json``: that model by default; automatic compaction on (Pi's defaults: it compacts when the context\n  passes the window less 16,384 tokens, keeping the latest 20,000 tokens of the conversation); no retries (a refused\n  request ends the run); no request or idle timeout short of the episode's own; telemetry, analytics, skill\n  commands and cache warming off; no packages, extensions, skills, prompt templates or themes.\n\nPi runs in JSON mode (`argv`): the task instruction is its prompt (on stdin), its default tools are on (``read``,\n``bash``, ``edit``, ``write``), the session is not saved, and nothing is discovered: no extensions, MCP, prompt\ntemplates, themes, context files (``AGENTS.md``, ``CLAUDE.md``) or skills, and project-local configuration is\nignored. In the curated condition, the task's skills directory is the one skill path it is given (``--skill``), so its\nsystem prompt lists exactly those skills. ``PI_OFFLINE``, ``PI_SKIP_VERSION_CHECK`` and ``PI_TELEMETRY=0`` keep it off\nthe network (`environment`).\n\"\"\"\n\nfrom __future__ import annotations\n\nimport functools\nimport json\nimport os\nimport re\nimport shutil\nimport subprocess\nfrom collections.abc import Sequence\nfrom dataclasses import dataclass, field\nfrom pathlib import Path\nfrom typing import Any\n\nfrom offbook import LIGHTNING, SUPER\n\n#: The Pi release every episode runs.\nPI_VERSION = \"1.4.0\"\nPI_PACKAGE = \"@earendil-works/pi-coding-agent\"\n#: Pi's own engine requirement (package.json ``engines.node``).\nNODE_MINIMUM = (22, 19, 0)\nDEFAULT_HOME = Path(\"/opt/pi\")\n",26365,"    body = text[match.end() :]\n    try:\n        data = yaml.safe_load(raw)\n    except yaml.YAMLError:\n        data = _lenient(raw)\n    if data is None:\n        data = {}\n    if not isinstance(data, dict):\n        raise ValueError(\"frontmatter is not a mapping\")\n    return data, body\n\n\ndef _lenient(raw: str) -> dict[str, Any]:\n    data: dict[str, Any] = {}\n    key: str | None = None\n    for line in raw.splitlines():\n        match = _KEY_LINE.match(line)\n        if match and not line[:1].isspace():\n            key, value = match.group(1), match.group(2).strip()\n            data[key] = _unquote(value)\n        elif key is not None and line.strip():\n            previous = data.get(key)\n            joined = f\"{previous} {line.strip()}\" if previous else line.strip()\n            data[key] = joined\n    return data\n\n\ndef _unquote(value: str) -> str:\n    if len(value) >= 2 and value[0] == value[-1] and value[0] in \"'\\\"\":\n        return value[1:-1]\n    if value in {\">\", \"|\", \">-\", \"|-\"}:\n        return \"\"\n    return value\n\n\ndef one_line(text: str) -> str:\n    \"\"\"Collapse runs of whitespace (including newlines) into single spaces.\"\"\"\n    return \" \".join(str(text).split())"],"d":0}]},{"t":34.9,"$":0.26,"c":[{"n":"bash","a":"cd /workspace/repo && sed -n 100,165p offbook/skills.py && cat envkit/README.md && ls -R envkit | head -80","e":0,"o":["    skills = []\n    for child in sorted(root.iterdir(), key=lambda p: p.name):\n        if child.name.startswith(\".\") or not child.is_dir() or not (child / SKILL_FILE).is_file():\n            continue\n        skills.append(load_skill(child))\n    return skills\n\n\n@dataclass(frozen=True)\nclass SkillsMode:\n    \"\"\"Which skills the model sees.\n\n    ``skills_dir`` is the host directory holding the packages. ``sandbox_path`` is where the packages\n    appear inside the environment the model works in; it defaults to ``skills_dir``.\n    \"\"\"\n\n    kind: str = NONE\n    skills_dir: Path | None = None\n    sandbox_path: str | None = None\n\n    def __post_init__(self) -> None:\n        if self.kind not in (NONE, CURATED):\n            raise ValueError(f\"kind must be {NONE!r} or {CURATED!r}, not {self.kind!r}\")\n        if self.kind == NONE:\n            if self.skills_dir is not None or self.sandbox_path is not None:\n                raise ValueError(\"the 'none' condition has no skills directory\")\n            return\n        if self.skills_dir is None:\n            raise ValueError(\"the 'curated' condition needs a skills directory\")\n        object.__setattr__(self, \"skills_dir\", Path(self.skills_dir).resolve())\n        if not self.skills_dir.is_dir():\n            raise ValueError(f\"{self.skills_dir}: not a directory\")\n        if self.sandbox_path is not None and not str(self.sandbox_path).startswith(\"/\"):\n            raise ValueError(\"sandbox_path must be absolute\")\n\n    @classmethod\n    def none(cls) -> SkillsMode:\n        return cls(NONE)\n\n    @classmethod\n    def curated(cls, skills_dir: str | Path, sandbox_path: str | None = None) -> SkillsMode:\n        return cls(CURATED, Path(skills_dir), sandbox_path)\n\n    @property\n    def location(self) -> str | None:\n        \"\"\"Where the skills appear to the model, or None in the 'none' condition.\"\"\"\n        if self.kind == NONE:\n            return None\n        return (self.sandbox_path or str(self.skills_dir)).rstrip(\"/\") or \"/\"\n\n    def skills(self) -> list[Skill]:\n        if self.kind == NONE:\n            return []\n        return load_skills(self.skills_dir)\n\n\ndef _first_paragraph(body: str) -> str:\n    for block in body.split(\"\\n\\n\"):\n        text = \" \".join(line.strip() for line in block.splitlines() if not line.lstrip().startswith(\"#\"))\n        if text.strip():\n            return text.strip()\n    return \"\"\n# envkit: tasks for the harness\n\n",8232,"envkit:\nREADME.md\nchecks.py\nexamples\ntemplate\n\nenvkit/examples:\nsensor-gaps\n\nenvkit/examples/sensor-gaps:\nenvironment\ninstruction.md\nsolution\ntask.toml\ntests\n\nenvkit/examples/sensor-gaps/environment:\nDockerfile\ndata\nskills\n\nenvkit/examples/sensor-gaps/environment/data:\nline3_temps.csv\n\nenvkit/examples/sensor-gaps/environment/skills:\ntimeseries-gaps\n\nenvkit/examples/sensor-gaps/environment/skills/timeseries-gaps:\nSKILL.md\nscripts\n\nenvkit/examples/sensor-gaps/environment/skills/timeseries-gaps/scripts:\nfind_gaps.py\n\nenvkit/examples/sensor-gaps/solution:\nsolve.sh\n\nenvkit/examples/sensor-gaps/tests:\nchecks.py\ntest.sh\ntest_outputs.py\n\nenvkit/template:\nenvironment\ninstruction.md\nsolution\ntask.toml\ntests\n\nenvkit/template/environment:\nDockerfile\ninput.csv\n\nenvkit/template/solution:\nsolve.sh\n\nenvkit/template/tests:\nchecks.py\ntest.sh\ntest_outputs.py"],"d":2.2},{"n":"bash","a":"cd /workspace/repo && cat skills/README.md | head -150 && for d in skills/*/; do n=$(find $d -type f | wc -l); s=$(wc -c < $d/SKILL.md); t=$(du -sk $d | cut -f1); echo \"$d files=$n skillmd_bytes=$s total_kb=$t\"; done","e":0,"o":["# Skills\n\nThis directory is a library of 43 agent skills that other people wrote and published under open licences. Each subdirectory is one skill, copied verbatim from its public source: a `SKILL.md` (YAML frontmatter with `name` and `description`, then instructions) and whatever references, scripts, templates or assets the skill bundles. The only files added are `SOURCE.md`, which gives the repository, commit and path the skill came from, and, where the source kept its licence at repository or plugin level, a copy of that licence as `LICENSE`.Some skills state the packages or tools they need in a `compatibility` frontmatter field.\n\n| Skill | Category | Type | Scripted | Source | Licence |\n|---|---|---|---|---|---|\n| [`bash-defensive-patterns`](bash-defensive-patterns/SKILL.md) | software-engineering | domain-procedure | no | `wshobson/agents` | MIT |\n| [`git-advanced-workflows`](git-advanced-workflows/SKILL.md) | software-engineering | tool-workflow | no | `wshobson/agents` | MIT |\n| [`jupyter-notebook`](jupyter-notebook/SKILL.md) | software-engineering | file-format-knowledge | yes | `openai/skills` | Apache-2.0 |\n| [`pandas-pro`](pandas-pro/SKILL.md) | software-engineering | library-api-usage | no | `Jeffallan/claude-skills` | MIT |\n| [`property-based-testing`](property-based-testing/SKILL.md) | software-engineering | evaluation-protocol | no | `trailofbits/skills` | CC-BY-SA-4.0 |\n| [`systematic-debugging`](systematic-debugging/SKILL.md) | software-engineering | debugging-heuristic | yes | `obra/superpowers` | MIT |\n| [`cantera`](cantera/SKILL.md) | industrial-physical-systems | domain-procedure | yes | `K-Dense-AI/scientific-agent-skills` | MIT |\n| [`capa-officer`](capa-officer/SKILL.md) | industrial-physical-systems | debugging-heuristic | yes | `alirezarezvani/claude-skills` | MIT |\n| [`embedded-systems`](embedded-systems/SKILL.md) | industrial-physical-systems | domain-procedure | no | `Jeffallan/claude-skills` | MIT |\n| [`pybamm`](pybamm/SKILL.md) | industrial-physical-systems | library-api-usage | yes | `K-Dense-AI/scientific-agent-skills` | MIT |\n| [`simpy`](simpy/SKILL.md) | industrial-physical-systems | library-api-usage | yes | `K-Dense-AI/scientific-agent-skills` | MIT |\n| [`data-quality-auditor`](data-quality-auditor/SKILL.md) | office-white-collar | data-cleaning-procedure | yes | `alirezarezvani/claude-skills` | MIT |\n",6134,"skills/pybamm/ files=6 skillmd_bytes=7285 total_kb=34\nskills/reconciliation/ files=3 skillmd_bytes=8144 total_kb=21\nskills/risk-metrics-calculation/ files=4 skillmd_bytes=2008 total_kb=22\nskills/sarif-parsing/ files=10 skillmd_bytes=17925 total_kb=85\nskills/scientific-critical-thinking/ files=11 skillmd_bytes=13146 total_kb=139\nskills/semgrep-rule-creator/ files=7 skillmd_bytes=7320 total_kb=48\nskills/simpy/ files=18 skillmd_bytes=14395 total_kb=168\nskills/slack-gif-creator/ files=8 skillmd_bytes=7841 total_kb=47\nskills/sql-queries/ files=3 skillmd_bytes=11126 total_kb=24\nskills/statistical-power/ files=8 skillmd_bytes=17834 total_kb=81\nskills/systematic-debugging/ files=13 skillmd_bytes=9530 total_kb=49\nskills/triage-nda/ files=3 skillmd_bytes=14583 total_kb=27\nskills/validate-data/ files=3 skillmd_bytes=14916 total_kb=28\nskills/variance-analysis/ files=3 skillmd_bytes=11232 total_kb=24\nskills/vulnerability-triage-brocards/ files=4 skillmd_bytes=8721 total_kb=42\nskills/writing-lean-proofs/ files=11 skillmd_bytes=15997 total_kb=117\nskills/yara-rule-authoring/ files=24 skillmd_bytes=24806 total_kb=217\nskills/zarr-python/ files=10 skillmd_bytes=8954 total_kb=47"],"d":0}]}],"parts":[83,116,126,100,119]}