The ship scan
Keep the answer key out of the sandbox
When an AI agent's code is graded, the grader's own files must never reach the agent: its simulator, its hidden test cases, the seeds (single numbers each hidden case is generated from). The usual leak is not a break-in. It is a harness (INTERNAL · glossary →), the program that sets up each run and grades it, packing one of its own files into what it sends. plimsoll's ship scan checks every call on its way in, against a list of what must never get through, and stops the harness's mistake before any code runs.
The grader's notes in the same room as the work
UC Berkeley audited 13 widely used AI benchmarks and reported: “the most common pattern we exploited was submissions running in the same process, container, or filesystem as the evaluator” (EXTERNAL · article ↗). In plain words: the work being graded sat next to the grader's notes. A sandbox (INTERNAL · glossary →) keeps code from breaking out; it does nothing about what was put inside it.
So the operator of a plimsoll server, plimsolld, writes a withheld set: a file of digests (INTERNAL · glossary →), short fingerprints computed with SHA-256, of everything that must never reach a sandbox. A digest identifies a file or a value without containing it, so the set holds none of what it protects, and the daemon's log lines and refusals name an entry by its label, never by what it holds. The daemon does read each call's content, to compare it; and an audit-mode find leaves the run as it is, so code that prints a seed it was given returns that seed. The set can hold three kinds of entry:
- a file, matched whole, under any name;
- a file's source lines, so an edited or reformatted copy is still found once a third of its distinctive lines are there (and at least eight);
- a value, such as a seed or a hidden answer, found as a whole run of letters and digits anywhere in the call.
The scan is the last check before a call is admitted, after the request is validated, its grant (INTERNAL · glossary →) is resolved and its isolation tier (INTERNAL · glossary →) and its software rule (the caller's requirement on which software the sandbox runs, if it set one) are met, so it only reads calls that would otherwise run.
Refuse or audit: why a hidden seed is not refused
Each entry has a mode. In refuse mode a match stops the call before it is dispatched (INTERNAL · glossary →) and tells the caller which entry matched. In audit mode the call runs, the caller is told nothing, and the operator's log names the entry.
This page's set is built that way, with one command, from a toy harness of one simulator file and three seeds the program makes up for the run:
plimsoll-withheld build -mode refuse -file harness/ -source harness/ -source-mode refuse -values seeds.txt -values-mode audit > withheld.json
| Entry | Label | Mode |
|---|---|---|
| file | harness/sim.py | refuse |
| source | harness/sim.py | refuse |
| value | seeds.txt:1 | audit |
| value | seeds.txt:2 | audit |
| value | seeds.txt:3 | audit |
A label is the operator's own name for an entry: a file's path from the folder it was read from, a value's file and line. A refusal shows it to the caller, so a label should say which entry, never what it holds: seeds.txt:2, not the seed.
Five calls to a daemon with that set
The daemon ran with SANDBOX_PROVIDER=wasm: plimsoll's in-process JavaScript engine, process tier, chosen so this page needs no docker. It runs snippets only, and that changes nothing below: the three refused calls are projects (files plus steps), and the scan refuses them before the provider is asked. Sent clean, a project would have been refused here as unsupported, and would have run on the docker provider. Each call carried a trace id, the caller's own join key, which the daemon copies onto its log lines, so each call's lines are found below.
A clean controller
What was sent
An agent's controller as a JavaScript snippet (a PD loop: it steers by how far it is from the target and how fast it is moving): nothing in it is withheld.
What the caller was told
It ran. Exit code 0, output:
final altitude 9.958 (target 10)
What the operator's log says
{"msg":"code run","op":"javascript","caller":"agent","trace_id":"ship-scan-clean","exit_code":0,"withheld_scan":"scanned","withheld_set":"8300d5453293b7c9cc00c8da32ef33ea7dd9c36d95651aaa2d6f4ce7ed0d0916"}
Its run record (INTERNAL · glossary →), plimsoll's statement of what was sent, what came back and where it ran, checked by the client when the answer arrived: request b384c4af6f60, result a86f5be38418. The exchange is published beside this page (request, response): -verify recomputes the record from it and checks the answer shown here is the one in it.
The harness's simulator, shipped as a helper file
What was sent
A project whose one file, helpers/sim.py, is the harness's simulator byte for byte: the harness developer's mistake.
What the caller was told
Refused, never dispatched. Code permission_denied, marked not dispatched for reason permission, so the caller knows no code ran and can fix the call and send it again:
the request carries material this daemon's operator withholds from sandboxes: harness/sim.py (files[0].content)
The refusal names the entry by its label and the place in the request (files[0].content, the first file's content), never the caller's text.
What the operator's log says
{"msg":"withheld material refused","op":"project","caller":"agent","trace_id":"ship-scan-harness-file","withheld_scan":"scanned","withheld_set":"8300d5453293b7c9cc00c8da32ef33ea7dd9c36d95651aaa2d6f4ce7ed0d0916","withheld_found":["harness/sim.py"],"withheld_refused":true}
A refused call has no run record: nothing ran.
The simulator, edited
What was sent
The same simulator with its step function renamed and every indent turned into a tab, so its file entry no longer matches; its source entry does.
What the caller was told
Refused, never dispatched. Code permission_denied, marked not dispatched for reason permission, so the caller knows no code ran and can fix the call and send it again:
the request carries material this daemon's operator withholds from sandboxes: harness/sim.py (files[0].content)
The refusal names the entry by its label and the place in the request (files[0].content, the first file's content), never the caller's text.
What the operator's log says
{"msg":"withheld material refused","op":"project","caller":"agent","trace_id":"ship-scan-harness-edited","withheld_scan":"scanned","withheld_set":"8300d5453293b7c9cc00c8da32ef33ea7dd9c36d95651aaa2d6f4ce7ed0d0916","withheld_found":["harness/sim.py"],"withheld_refused":true}
A refused call has no run record: nothing ran.
The simulator, packed in an archive
What was sent
The simulator inside a .tar.gz, sent as base64 text in deps.tar.gz.b64, as a step that runs base64 -d would unpack it.
What the caller was told
Refused, never dispatched. Code permission_denied, marked not dispatched for reason permission, so the caller knows no code ran and can fix the call and send it again:
the request carries material this daemon's operator withholds from sandboxes: harness/sim.py (files[0].content)
The refusal names the entry by its label and the place in the request (files[0].content, the first file's content), never the caller's text and never the name of the file inside the archive.
What the operator's log says
{"msg":"withheld material refused","op":"project","caller":"agent","trace_id":"ship-scan-harness-archive","withheld_scan":"scanned","withheld_set":"8300d5453293b7c9cc00c8da32ef33ea7dd9c36d95651aaa2d6f4ce7ed0d0916","withheld_found":["harness/sim.py"],"withheld_refused":true}
A refused call has no run record: nothing ran.
A hidden seed in the agent's own code
What was sent
An agent's snippet with a hidden seed written into it as a constant: the seed reached the agent somehow, and its code carries it.
What the caller was told
It ran. Exit code 0, output:
gain 0.84
Nothing in the answer says a withheld value was found: the agent cannot tell this call from any other.
What the operator's log says
{"msg":"withheld material found","op":"javascript","caller":"agent","trace_id":"ship-scan-seed-in-code","withheld_scan":"scanned","withheld_set":"8300d5453293b7c9cc00c8da32ef33ea7dd9c36d95651aaa2d6f4ce7ed0d0916","withheld_found":["seeds.txt:2"],"withheld_refused":false}
{"msg":"code run","op":"javascript","caller":"agent","trace_id":"ship-scan-seed-in-code","exit_code":0,"withheld_scan":"scanned","withheld_set":"8300d5453293b7c9cc00c8da32ef33ea7dd9c36d95651aaa2d6f4ce7ed0d0916","withheld_found":["seeds.txt:2"]}
Its run record (INTERNAL · glossary →), plimsoll's statement of what was sent, what came back and where it ran, checked by the client when the answer arrived: request bb322cab71e0, result 5ac83f882ac1. Its request holds a hidden seed, so it is not published; its response is, and -verify recomputes the result digest from it and checks the answer shown here. (response)
The log lines are shown with only the fields this page is about; the daemon writes more (timings, the sandbox, the request's sizes), and none of them carries a request's content. The program checks the whole log, not only these fields, for every seed and every distinctive line of the simulator, and publishes nothing if it finds one.
The same set, before anything is sent
plimsoll-withheld scan runs the same scan over a folder, offline, with no daemon. A harness can run it over what it is about to ship. Here it scanned a folder holding the edited simulator and the agent's snippet, then a folder holding only the clean controller:
$ plimsoll-withheld scan -set withheld.json outbox/ harness/sim.py (refuse): outbox/helpers/plant.py seeds.txt:2 (audit): outbox/agent.js (exit 1)
$ plimsoll-withheld scan -set withheld.json clean/ clean: 1 files against set 8300d5453293b7c9cc00c8da32ef33ea7dd9c36d95651aaa2d6f4ce7ed0d0916 (exit 0)
A real case: a hackathon's leaked seed
At a hackathon on October 25, 2026, the did-it-learn entry (EXTERNAL · source repo ↗) shipped a replay viewer into every agent's sandbox whose example link named one of the 16 practice seeds, which the harness meant to keep from the agents, until a fix at 12:55. examples/squeeze/argon-scan.sh rebuilds the 13 files the harness shipped, from the entry's public repository at the last commit before the fix and at the fix, and scans both against a set of the 16 seeds. This is an offline reconstruction of what was shipped, not a replay of the hackathon's sandboxes. Before the fix:
seeds.txt:2 (refuse): tmp/argon-scan/before/workspace/tasks/drone/view.mjs
After it:
clean: 13 files against set 9d537b8d0bc5d3d42ade48900cb164a7648ed55de6ad3bc46bade7ad5d837449
The whole transcript, and how to rerun it, is on the squeeze page (INTERNAL · the hackathon scan →).
Build a set for your harness
plimsoll-withheld build -mode refuse -file harness/ -source harness/ -source-mode refuse \ -values seeds.txt -values answers.txt -values-mode audit -exempt grader > withheld.json plimsoll-withheld scan -set withheld.json outbox/ # offline: exit 1 on a find PLIMSOLL_WITHHELD_FILE=withheld.json plimsolld # with your provider's settings
- The grader's own calls must carry the harness, so its caller is named with
-exempt: calls from that caller are not scanned, and their log line says so. - A short value's digest is as secret as the value. A ten-digit seed's digest is found by trying every ten-digit number. Publish a set of files and long values if you like, as a statement of exactly what was looked for; keep a set of seeds as private as the seeds. This page publishes its set (withheld.json) only because its seeds were made up for this run, used nowhere else, and deleted when the run ended.
- The daemon refuses to start on a malformed set, since a set that loads with a typo scans for less than its author believes.
What it catches, and what it does not
- A withheld file under any name, and an edited, reformatted or bundled copy once a third of its distinctive lines are there.
- A withheld value in code, file contents, file names, steps and artifact names, as written, with escapes decoded, or base64-decoded once.
- Either one inside the archives it reads: gzip, bzip2, zlib, zip (stored, deflate, bzip2, zip64) and tar, nested.
- Archives it recognizes and cannot read (an encrypted zip member, xz, zstd, 7z, rar): a set with a refuse entry refuses the call; otherwise it runs and its log line says the scan was incomplete.
- Forms it does not recognize at all (raw deflate, lz4, an archive encoded twice, hex): they pass unseen, so material packed that way on purpose passes even a refuse entry.
- A value drawn into an image or packed into its pixels.
- Files baked into the sandbox's image, and whatever a run fetches through a grant: neither passes through a request.
- Code rewritten line by line, a few lines copied, or a model that has memorized the answer.
- Neither the answer nor the run record states the scan yet: a record shows what ran, not that it was scanned; the operator's log is the evidence.
The reference, with every bound and format: the ship scan (EXTERNAL · official docs ↗).
Reproduce
go run ./examples/ship-scan # build the tools, run the five calls, write this page go run ./examples/ship-scan -verify # check the published files offline
Run from the root of a plimsoll checkout; it builds plimsolld and plimsoll-withheld from source and needs no docker, no network and no credentials. -verify checks every claim above against evidence.json, the set's digest against withheld.json, recomputes the clean call's run record, and checks this page is what the evidence renders. The seeds differ on every run, so a rerun gives new digests and the same outcomes.
Made at plimsoll commit c9f4e98d03276b1bf518291e400c2da19532b608, provider wasm (process tier: plimsoll's engine runs inside the daemon's own process, with no operating-system isolation boundary, so it is not for hostile code in production), set SHA-256 8300d5453293b7c9cc00c8da32ef33ea7dd9c36d95651aaa2d6f4ce7ed0d0916.