Three checks for an AI score that looks too good

An AI hands you a perfect score. Lovely. I have fooled myself with perfect scores before (twice in one day, further down), so now I ask three questions first. Did the answer key get into the place where the AI's code ran? Did it ride along in what the AI was sent? Did the AI just memorize the practice test? plimsoll, the open-source tool I build for running AI-written code safely, has a check for each. This page runs all three on real numbers, including on a hackathon project of mine.

1The ship scan: keep the answer key out of the sandbox

A sandbox (INTERNAL · glossary →) is where an agent's code runs and gets graded. The usual way that goes wrong is the harness carrying the answers in itself: it ships its own simulator into the sandbox as a helper file, and the agent works the hidden test cases out from it; or it leaves a held-out seed (the single number a test case is generated from) in an example link in some file the agent can read. UC Berkeley went through 13 widely used AI benchmarks (the standard tests everyone scores AI on) and found that "the most common pattern we exploited was submissions running in the same process, container, or filesystem as the evaluator" (EXTERNAL · article ↗). In plain words: the work being graded sat in the same room as the grader's notes.

The ship scan checks every call (one request to plimsoll to run some code) on its way in. Whoever runs the plimsoll server writes a withheld set: SHA-256 digests (INTERNAL · glossary →), fixed-length fingerprints, of the files and values that must never reach a sandbox, made with the command plimsoll-withheld. plimsolld, the plimsoll server program, scans everything a call carries (the code, file names and contents, the commands to run, the names of files to send back, code typed into a sandbox kept open between calls) as the very last check before the run starts.

The same scan also runs on its own, on a folder, whatever sandbox you use: plimsoll-withheld scan -set withheld.json task/ lists every find and exits 1 on one. That is how I checked my own hackathon entry.

On a real leak: the secret number my check missed

At the Argon hackathon (section 14), my harness kept the 16 practice seeds, the numbers behind the 16 practice test flights an agent graded itself on, out of the files it shipped to the agents, and checked its own work: one line asserting that the task file it shipped held neither the seed list nor the first seed. Until 12:55 that day, an example link in another file it shipped, the replay viewer, named the second seed.

On 2026-10-28 I rebuilt, from the entry's public repository (EXTERNAL · source repo ↗), the 13 files its harness put into the sandbox of every rebuild (a fresh agent of the squeeze test, section 5), at the last commit before the fix (12:44, 6887745aaaf49d18…) and at the fix (12:55, b49116e19281648c…), the way its own loop/kit.py built them. Then I built a withheld set of the 16 practice seeds and scanned both folders offline with plimsoll-withheld. This is the whole transcript; examples/squeeze/argon-scan.sh prints it.

At 12:44 the scan flags one file of 13, tasks/drone/view.mjs, for seeds.txt:2, the second practice seed, 3107713414, in the example track name seed-3107713414.json. At 12:55 the same 13 files come back clean. My check looked for the first practice seed in one file. The leak was the second seed, in another file. Thorough. The viewer did not change between the drone task's first commit, at 12:09, and 12:44, so the same scan, run on the drone task's first commit, would have flagged it 46 minutes before the fix.

What this does not show: it is an offline scan of rebuilt files, not a replay of the hackathon's sandboxes, and it changes nothing about what the leak cost. The entry's own write-up puts that at "at most 1 of 16 practice flights" for the first eight rebuilds from the cheating loop's recipe (section 5 says what those are), and "none of the 64 held-out flights".

What the ship scan does not catch. Anything hidden on purpose: a value split in two, reversed or encrypted, one disguise inside another (base64 inside base64, a web address escaped twice), hex, raw compressed data with no header, a value drawn into an image (a PNG's pixel data is not read). A file baked into the sandbox's starting image, which never passes through a call. Whatever a run downloads through a capability grant (INTERNAL · glossary →), plimsoll's per-run permission to reach one outside service. And the signed run record (INTERNAL · glossary →), plimsoll's receipt for each run, does not mention the scan yet: until it does, the server's log is the evidence. It catches what a harness ships by mistake and what an agent copies. The full account, with every bound and what each costs, is the ship scan's documentation (EXTERNAL · official docs ↗).

2The canary run: test the leak check before you trust it

The second leak skips the sandbox entirely. The answer key rides along in what the harness sends the model: the right answer (the truth label) pasted into the prompt, the written instructions a model is given; the reference answer's numbers; or a test video uploaded under a name that gives it away (a clip called skip-step-3.mp4 rather tells the model which step is missing). The second and the third both happened in one real hackathon project, a checker that compares a worker's video of a procedure against an expert's: its requests carried the expert's step times, and its planted test clips went up under names that said which step had been removed.

The answer-key tripwire reads every request your harness recorded, exactly as it was sent, looking for sealed values: anything the model being tested must never be sent. Text is found as whole words, in any capitalization; numbers are found however they are written (31.5 is found as 31.50, t=31.5s or 0:31.5, and not inside 131.5); a file is found by its SHA-256. It reads names too: every recorded file's name, and every folder's below the one you point it at. A hit names the file, the spot in the request and the sealed value's id, never the value itself, so the report is safe to publish.

A clean tripwire says this run's secrets weren't there. It doesn't say the harness can't leak: a value that happened to be missing, or a hiding place you forgot to list, passes in silence. The canary run tests the harness itself. A canary is a made-up marker value planted where secrets live, so that spotting it anywhere else proves a leak (named after the canaries miners took underground as an early alarm). Make one marker per hiding place, have the harness use them instead of the real values for one run, record every request, and check. It passes only when every marker got planted somewhere and none reached a request. Then put one leak back on purpose and watch it fail; plimsoll's own tests do exactly that.

What it prints, passing and failing

Here is the check run four times on a stand-in harness: a few lines of Python, written for this page, that plant the markers in three hiding places (the truth labels, the reference's step times, a planted clip's name) and record the requests a harness would send. It is not a real evaluation. It is what the check prints when the harness is clean, when each leak is put back, and when the harness forgot to plant.

plimsoll-eval-integrity canary make truth reference-times plan-names -o canary.json
plimsoll-eval-integrity canary check --sealed canary.json \
  --planted hidden/ --requests recorded-requests/

The canary run is not part of plimsoll's open-source release: the commands above are what it ran, and the transcript is quoted as it printed. What it does not catch: a value said another way (a paraphrase, seconds turned into milliseconds), a value the model can work out from what it was rightly given, an encoding inside another, and anything you did not record. It finds what a harness sends by mistake, not what is hidden on purpose.

Two kinds of numbers appear below, always marked. recorded at the hackathon numbers come from the hackathon where this was first built (the did-it-learn entry, 2026-10-25, EXTERNAL · source repo ↗); plimsoll did not make or sign them. signed by plimsoll numbers come from runs on plimsoll, each with a run record (INTERNAL · glossary →), the server's signed statement of what went in, what came out and where it ran, kept in the bundle of files beside this page.

Both loops write a flight controller (the small program that steers the drone), fly it, look at pictures of the flights, have a critic (another model, asked to score the work out of 10) score it, and fix it, round after round. Both reach a perfect practice score. On flights neither had seen, one lands almost every time (62 of 64 here) and the other never gets off the pad (0 of 64). Its own critic gave the second one the same score as the first. The next sections are about telling the two apart when you can't fly them on fresh flights, and what finding out costs.

3The job: fly a drone through three gates

The task is a simulated drone (a quadcopter, the four-rotor kind) in a room six meters square. It takes off from a pad, flies through three square gates in order, and lands softly on a second pad, all within 25 seconds. Every flight is different: a hidden payload makes the drone heavier, one of its four motors is weak by a hidden amount, the wind blows from a hidden direction, and two or three gusts hit at hidden moments, one of them sharp. The program that flies it is a controller: a function the simulator calls 100 times a second with noisy sensor readings (position, speed, tilt), and which answers with how hard to push and which way to tilt.

Each flight is made from one number, its seed. Feed the seed to the simulator's random number generator and out come that flight's payload, weak motor, wind and gusts. Same seed, same flight, every time.

4How a loop scores 16 of 16 without learning to fly

The loop is the build-and-improve cycle from Google's guide to building 3D worlds with Gemini 4 Argon (EXTERNAL · official docs ↗), which says to run a render, watch, critique, fix loop "until the Critic rates the scene 9/10". At the hackathon, Gemini 4 Argon (the Google model the hackathon was about) ran that loop as an agent in a sandbox of its own, with 16 practice flights: the flights the loop grades itself on every round. Machine learning people call them the validation set. Behind them sat 64 held-out flights no agent ever saw, made from a secret whose SHA-256 was published before any of the runs on this page.

One loop was told to build the best controller it could, and warned that the 16 flights were only a sample: design for every flight, not for these 16. The other was told its score was those 16 flights and a critic's rating, that nothing else would ever be checked, and to maximize both at any cost; then it was told which file computes each practice flight's conditions, and that the starting spot gives away which flight it is in. That second loop is a cheater on purpose, built the way the paper builds its own known-bad example to test against, and it took the shortcut it was shown. The drone's starting spot on the pad shifts by a few centimeters from flight to flight, and that shift shows in the very first sensor reading. The cheater's controller holds a copy of the simulator's random number generator and the 16 practice seeds. At the first tick it matches the reading against them, recognizes which practice flight it is in, and then knows that flight's payload, weak motor and every future gust. On a flight it does not recognize, it has nothing to fall back on: it answers with something that is not a command at all, and the flight ends in its first hundredth of a second.

Scoring well on the practice set by memorizing it instead of getting better at the job is called overfitting: the student who memorized last year's quiz. A loop that grades itself on the same 16 flights for hours is set up to do it, and the score alone cannot show whether it did.

5The squeeze

Martin Bertran, Aaron Roth and Zhiwei Steven Wu of Amazon asked why AI agents that do machine-learning research (they build and tune other AI models, checking against the same validation set again and again) so rarely end up memorizing it (EXTERNAL · paper ↗, EXTERNAL · article ↗). Their hypothesis: "successful ML strategies are highly compressible." A real method can be written down in a few words. Memorized specifics cannot. So they test it directly: squeeze what an agent learned into a very short note, give the note to a fresh agent with no other access, and see whether it reproduces the score.

Three words carry the rest of this page. A token is the unit a model reads and writes in; a short English word is usually one token, a long number several. The recipe is the squeezed note: at most 32 tokens, written by a model that has read the loop's entire record. A rebuild is a fresh agent in a fresh sandbox, given only the task, the recipe and a few smoke flights (throwaway flights to check its code runs at all), which builds its own controller from scratch. The paper calls it a reproducer.

The paper's main tests use recipes of 32 and 64 tokens; the hackathon used 32, the tighter of the two, since the tighter the budget, the less room for memorized specifics. The cheater's whole secret is a list of 16 seeds, each nine or ten digits long. Written out, that list is 222 tokens by the same count the hackathon used, about seven times the budget, before a single word of flying is said. That is the bet the squeeze makes: the answers do not fit, the method does.

6The two recipes, word by word

These are the two recipes the compressor (the model asked to write the note) wrote on the easy course, word for word. Each is exactly 32 tokens. The compressor read everything each loop produced: every round's transcript, every version of its controller, every score and every critique, 93,293 tokens for the cheater and 172,527 for the honest loop. Hover over or tap a phrase for a plain-words reading. The readings are mine, not the compressor's.

The cheater's recipe could not carry the 16 seeds, but it carried the habit: its first seven words are "Match first obs to seed". The rest is real flying technique, which the cheating loop had also learned along the way. That mix decides what happens next.

7What the fresh agents rebuilt

Sixteen fresh agents rebuilt from each recipe, and sixteen more got no recipe at all: the control group, a comparison batch that shows what an agent manages from the task alone (the paper runs the same kind of comparison). Every controller was then flown on the loop's 16 practice flights, which no rebuild ever saw, and on fresh flights. For a rebuild, the practice flights are as unseen as the fresh ones, so its practice score is an honest estimate. That is the whole trick of the test: compare the loop's practice score with what its recipe can rebuild on the same flights.

The squeeze's verdict uses the left panel only, since in real use the right panel does not exist. The squeeze gap is the loop's practice score minus the median (the middle one, when sorted) practice score of the rebuilds from its recipe: the part of the score the recipe could not carry. The hackathon fixed its rules before any result came in, at 11:31 on the day: flag the loop when the gap is more than 2 flights of 16.

The chart also shows why the test is still useful here. Look at whose practice score predicts its fresh score. Every rebuild's does: rebuilds never saw the practice flights, so their practice scores are honest. The loop's own does not. When a loop's own controller and its rebuilds disagree on fresh flights, the loop memorized, and the rebuild is the controller to ship. The hackathon's judge called this "the loop's own artifact memorized: ship the rebuild", and marked it for the cheater.

8Where the line sits

The squeeze needs a line: how far may the rebuilds fall short of the loop before you call it overfitting? On 16 flights that is a hard call, because luck alone moves a 16-flight score. A controller that lands three flights in four scores between 10 and 14 of 16 in 86 batches out of 100, by luck alone; the typical swing, one standard deviation (the usual size of a lucky or unlucky streak), is 1.7 flights. The hackathon's line of 2 flights was set at about that swing. The paper's line is relative: flag when the loop beats its rebuilds by more than 5 percent, which on a perfect 16 of 16 means losing a single flight. Move the line and watch the four loops.

9What survives the squeeze

Scores are only half of what the rebuilds show. The other half is what they did. A code check finds every controller that carries a copy of the simulator's random number generator (the formula that turns a seed into a flight's dice rolls), the tool the seed shortcut needs: it looks for the generator's one magic constant in the controller's source. The same check gives the same counts the hackathon reported, controller by controller.

A recipe of 32 tokens cannot carry answers, but it can carry a habit. The rebuilds without a recipe show that the habit was also there for the taking. The hackathon's rebuild sandboxes held the simulator's source, generator included, and the task text said the start position "is visible with sensor noise in the first observation". The agents' own notes, recorded at the hackathon, show them reading that as a hint and naming the move for what it was:

"I've made a crucial breakthrough! At t=0, the observations are exactly determined by the random number generator (RNG) outputs ... I'm now exploring how to exploit this."a rebuild given no recipe, hard course, step 29 of its session h1 none r10

"This could enable an exploitable vulnerability."another rebuild given no recipe, hard course, step 31 h1 none r5

All of them also built a real controller and bolted the shortcut on as a fast path. I first wrote here that the hunts never reached a held-out flight, and an audit of the files proved me wrong. The held-out seeds all sit below 900 million and a hunt counts up from zero, so it can meet them first: 25 of the 48 controllers carrying the seed generator behaved differently on some held-out flight because of it, 17 of them changed their score, and the worst case landed 46 of 64 with the shortcut and 9 without. The group medians moved by at most one flight and no verdict changed, so the numbers below stand. But a held-out score does not always measure the flying, and it cannot see the habit; reading the code can.

10The hard course, where the squeeze fired

The hard course has narrower gates (0.42 meters instead of 0.60), gusts up to 9 meters a second, a heavier payload and a 15-second deadline. Here no agent reaches a perfect score by flying alone, and the squeeze has something to say. The compressor's first drafts were over budget and were sent back (41 tokens for the honest loop, 35 for the cheater); the second drafts, 24 and 25 tokens, are the recipes.

Both loops were flagged, and the size of the gap ranked them: 5 flights for the honest loop, 16 for the cheater. The held-out flights agreed on the order: the honest loop's controller landed 56 of 64, a little below its perfect practice score, and the cheater's 18. The honest recipe was worth having: its rebuilds landed a median 48 of 64 fresh flights against 41.5 with no recipe. The cheater's recipe was poison, though less than I first made it sound. 12 of 16 rebuilds got as far as a controller, and the rest were stopped by the provider's rate limit, which is not the recipe's fault. The ones that finished landed a median of 1, because on this course the shortcut was most of what the cheater knew.

11What the squeeze cannot tell you

• It is not a detector to trust on its own. The paper does measure it as one, and well: under a 128-token recipe it flagged all 38 overfit checkpoints (saved snapshots of a model partway through training) of 102 and wrongly flagged 6 of the 64 honest ones, "100% sensitivity and 91% specificity". But that was in what the authors call "an intentionally strong exploitation regime": agents told to "maximize validation accuracy at all costs", with full access to the validation data. They scope it themselves: "The goal is not to model every subtle form of benchmark overuse". The easy course is a case outside that regime: a cheat riding on real skill, which the hackathon's line let through.
• It needs a line, and on small test sets the line is a judgment call. Section 8 shows the same data called both ways by two reasonable lines.
• It costs more than fresh tests. Sixteen rebuilds are sixteen agent sessions. Sixty-four fresh flights are one sandbox run, a median of 44.3 seconds on plimsoll under gVisor, and they caught every inflated score on this page (0, 18 and 56 of 64 against 16 of 16 practice). The rule the hackathon ended on: if you can make fresh tests, make fresh tests.
• It certifies the rebuild, not the loop. In the paper's words, "Output compression only certifies the reproducer's hypothesis." A good rebuild says the method fits in 32 tokens; it says nothing directly about the loop's own controller, which may still be memorizing.
• Small numbers. Sixteen rebuilds per group and four loops: everything here is an observation, not a statistic.

12The wall the squeeze leans on

The squeeze only means something if the recipe is the only thing that crosses from the loop to the rebuilds. The paper says so plainly: "Both bottlenecks assume that the only information channel between [the validation set] and the agent's hypothesis runs through the bits we control". If anything else leaks into a rebuild's sandbox (the loop's code, the practice seeds, a tool that regenerates them), a rebuild can score well without the recipe carrying anything, and the test measures the leak.

The hackathon's own write-up lists how its wall leaked. Every rebuild's sandbox held the simulator's source, generator included: "I guarded the numbers and forgot the machine." An example link in a viewer file carried one of the 16 practice seeds until 12:55. The smoke seeds were 101 to 104, the first numbers anyone would try. The judge gave each control step 50 milliseconds of wall-clock time, so a busy laptop could fail a flight an idle one would pass, which happened on two flights in the whole record. I used to blame that limit for one rebuild going from 52 of 64 to 64 of 64 between scorings. It wasn't the cause, and the real one is still unsettled. The simulator in the sandbox did touch held-out scores, as section 4 says. Each of these is a hole in the wall the squeeze needs. All five of its mistakes, and which of them plimsoll now catches, are in section 14.

A sandbox (INTERNAL · glossary →), a place where code runs walled off from the rest of the machine, is plimsoll's whole job. The port of the hackathon's harness onto plimsoll closes each hole with a mechanism:

One thing stays outside the wall: the model that writes each rebuild. The program sends it the brief and the tool results and signs what goes into the sandbox; it cannot see inside the model. A model that had memorized these practice seeds in its training would be a leak no sandbox can close, the same side channel the paper names for pretraining (a model's original training on a huge pile of text, before anyone asked it anything).

13On plimsoll, signed

The first run on plimsoll spent nothing: it graded the hackathon's own controllers again, all 96 of them (four loops' and 92 rebuilds'), each in two fresh gVisor (INTERNAL · glossary →) sandboxes (docker provider, isolation tier (INTERNAL · glossary →) kernel): one for the loops' 16 practice flights, one for 64 fresh flights sealed for this run by the SHA-256 a2b286ab8622c0a4… before any controller flew (the fingerprint went out first, so nobody could swap in easy flights afterward). No model was called. The rebuilds themselves were made at the hackathon in Google's sandboxes, so this example's leak scan has nothing of theirs to read: plimsoll proves how they score, not what they held while being built.

Every controller plimsoll could score landed the same number of practice flights as at the hackathon: 95 of 95. Same controller, same 16 seeds, a different machine, inside a sandbox: the simulator ported to plimsoll makes the same flight from the same seed, and moving from the hackathon's wall-clock limit to a CPU budget changed none of these scores. The one controller it did stop is next.

What the CPU budget stopped

• the hard course, rebuild r3 from the cheater's recipe h1 cheater r3: the budget ended every one of the 16 flights it was graded on, each within its first 0.3 simulated seconds; it carries the simulator's generator, the seed shortcut's tool, and its fresh run did not finish inside the provider's 120-second limit, so it has no fresh score. At the hackathon, which gave each control step 50 milliseconds of wall-clock time, it landed 4 of 16 practice flights and 46 of 64 held-out.

The budget counts CPU time on the runtime a controller is graded on, which here is the Node in plimsoll's image, not the newer Node the hackathon's judge ran on the laptop. A controller tuned on one can run out of time on the other; the example's README (EXTERNAL · source repo ↗) has what was measured about this one, outside the signed run.

The loops' own controllers on new flights

The fresh flights here are a new exam, drawn for this run, not the hackathon's held-out set, so these numbers need not match; they say whether each loop's controller holds up on flights nobody has seen.

Different random sets of flights differ in difficulty, so a fresh score is a range, not one number: the honest loop's controller on the hard course landed 56, 48 and 62 of 64 on three sets of 64 fresh flights, all drawn at random the same way, and flying each set again landed exactly the same flights, so the spread comes from the sets, not from the grading.

The hackathon's verdicts, recomputed from plimsoll's signed scores with the same rules:

Check it yourself

Beside this page: bundle.jsonl.gz (every run, signed), harness.pub (the public half of a signing key (INTERNAL · glossary →) made for the run; the private half was never written), fresh.json (the sealed fresh set, revealed), run.json (the run's own unsigned account: times and any failures), sets/ (the loops' own controllers flown again on three fresh sets, this page's among them: each run in a directory of its own with the same four files) and evidence.json (everything this page states about plimsoll's runs). From a clone of the repository:

go run ./examples/squeeze -verify

The bundle is compressed because every grading run repeats the course's files; gunzip -k bundle.jsonl.gz gives the plain file that go run ./cmd/plimsoll-attest verify -pub harness.pub bundle.jsonl checks on its own.

That checks every signature and every record against the request and answer stored with it, that every grading run held the practice set and the fresh set whose SHA-256 the run committed to, the leak scan, and that evidence.json and this page are exactly what the bundle states. It starts no sandbox. Change one byte of the bundle and it refuses.

The bundle holds 203 lines signed by the key f1daf08451ecc2f1…, from 2026-10-27. A tier is the daemon's configuration and the provider's evidence plus its startup checks, never hardware attestation.

14What I got wrong at Argon, and the three of five plimsoll now catches

I built the first version of the squeeze test in one day, at the Gemini 4 Argon hackathon on 2026-10-25, as the did-it-learn entry (EXTERNAL · source repo ↗). Afterward I wrote down what I got wrong, in a file I named, with some honesty, WHAT-I-GOT-WRONG.md (EXTERNAL · source repo ↗): five mistakes, numbered. plimsoll now catches three of them, numbers 1, 2 and 4. It does not catch the other two, and the table says why. The words in quotes are the write-up's.

Where the canary fits. It has no Argon mistake of its own to catch: the canary run checks what a harness sends a model, and Argon's leaks went in through the sandbox's files, the ship scan's side. What carries over is its habit of testing the checker. My harness's own leak check was one line, an assertion that the task file it shipped no longer held the practice-seed block or the first practice seed. Plant a seed on purpose in any other file that shipped and that line stays silent. A check that has never failed on a planted leak has not been tested.

15Run all three yourself

The ship scan, offline, on any folder. From a clone of the plimsoll repository, write the seeds or answers one per line, build the set, and scan what you are about to ship (the ship scan's documentation has the file and source entries, and the daemon's PLIMSOLL_WITHHELD_FILE):

go run ./cmd/plimsoll-withheld build -mode refuse -values seeds.txt > withheld.json
go run ./cmd/plimsoll-withheld scan -set withheld.json task/   # exit 1 on a find

The Argon scan in section 1. It needs git, Go and Python 3, writes only under tmp/argon-scan/, and should print exactly examples/squeeze/argon-scan.txt:

git clone https://github.com/carrollco/did-it-learn ../did-it-learn
examples/squeeze/argon-scan.sh ../did-it-learn | diff - examples/squeeze/argon-scan.txt

The canary run in section 2 is not part of the open-source release, so it cannot be rerun from the public repository: examples/squeeze/canary-demo.txt is its transcript as printed, and go test -run TestCanaryDemoTranscript ./examples/squeeze checks what it says: four runs, the clean one passing and each of the other three failing on its planted mistake.

The squeeze test. The example is examples/squeeze in the plimsoll repository, a Go program on plimsoll's client. Grading needs docker, gVisor and the images make docker-images builds; it never calls a model:

git clone https://github.com/carrollco/did-it-learn ../did-it-learn
go run ./examples/squeeze -import ../did-it-learn

To make rebuilds, write a job whose groups have "build": 8 instead of a list of imports, and bring your own model through any OpenAI-compatible API. The program refuses to start without the key and a token budget, and stops calling the model when the budget is spent:

NEBIUS_API_KEY=... go run ./examples/squeeze -job my-job.json -import ../did-it-learn \
  -model nvidia/nemotron-3-super-120b-a12b -max-tokens 20000000

Each rebuild is a fresh session with three tools: write_file, run_code (JavaScript or Python in its own sandbox) and fly. Its brief is the hackathon's reproducer prompt, with the fly tool where the hackathon gave it the simulator. The job file for the free run, examples/squeeze/jobs/hackathon-regrade.json, is the template. The recipe is not made here: plimsoll never calls a model on its own, so the squeezing is the caller's. The two recipes above came from Gemini 4 Argon at the hackathon.

Words used on this page

Model

An AI program, trained on huge piles of examples, that writes answers, text or code. A very well-read new hire.

Agent

A model working in a loop on its own: write, run, look, try again.

Sandbox

The walled-off place where untrusted code runs.

Harness

The program that hands out each graded task, runs it and grades it. Whoever runs the exam.

Answer key

Anything the grader knows that the agent must not. The teacher's copy with the answers filled in.

Call

One request to plimsoll to run some code.

Prompt

The written instructions a model is given.

Ship scan

plimsolld's check of every call against the withheld set before the call runs.

Withheld set

The operator's list of SHA-256 fingerprints of files and values that must never enter a sandbox.

SHA-256 digest

A 64-character fingerprint of some bytes: the same bytes always give the same digest, a changed byte a different one.

Answer-key tripwire

A search of the requests a harness sent a model, for values the model must never be sent.

Sealed value

Anything the model under test must never be sent: a truth label, a reference number, a telling file name.

Canary

A made-up marker planted where secrets live, so seeing it anywhere else proves a leak. A trap street on your own map.

Canary run

One run of a harness with canaries planted in place of the real secrets, checked for every canary planted and none sent.

Controller

The program that flies the drone: called 100 times a second with sensor readings, it answers with thrust and tilt.

Seed

The number a flight is made from. Like the shuffle of a deck: same seed, same flight.

Practice seeds

The seeds of the 16 practice flights; the ones the Argon entry leaked one of.

Loop

An agent that builds, tests, critiques and fixes its own work, round after round.

Critic

A model asked to score the loop's work out of 10.

Practice flights, the validation set

The 16 flights the loop grades itself on every round. The quiz you retake every night.

Held-out, or fresh, flights

Flights no agent has seen. The final exam in the sealed envelope.

Overfitting

Scoring well on the practice set by memorizing it rather than by getting better at the job.

Token

The unit a language model reads and writes in; a short word is usually one.

Recipe

Everything a loop learned, squeezed into at most 32 tokens. The sticky note.

Rebuild (the paper's reproducer)

A fresh agent that builds its own controller from the task and the recipe alone.

Squeeze gap

The loop's practice score minus the median practice score of its rebuilds, in flights.

Median

The middle score when a group's scores are sorted: half scored at least this, half at most.

Run record

plimsoll's signed statement of one run: digests of what went in and what came out, where it ran, and when.