Prime's own task ran unchanged on plimsoll, and every call left a record you can recheck

The headline numbers. A reward is verifiers' score for one attempt, from 0.0 (failed) to 1.0 (solved). runc (INTERNAL · glossary →) is docker's usual container runtime, and gVisor (INTERNAL · glossary →) is Google's stand-in kernel, which answers the code's requests to the operating system itself: the two walls the sandbox ran behind.

Run on 2026-10-29 with verifiers 0.4.1.dev97 (the version the repository's lock pins: verifiers==0.4.1.dev97) and Python 3.12.3, on one development machine.

The 30-second version

What ran

Every configuration below ran fresh, in one go, from one command (at the end of this page). On plimsoll every attempt asked for a floor (INTERNAL · glossary →), the weakest tier it accepts: container under runc and kernel under gVisor. vf-validate is verifiers' own check of an environment, run here on plimsoll-sum, a tiny test environment of plimsoll's: the gold (known correct) answer must pass, and the task left untouched must score 0.0 when graded in a fresh sandbox that received only its output files. A run where a script deliberately writes a wrong answer has to score 0.0 too, or a 1.0 means nothing.

One attempt, call by call

The uuid_ctf attempt under runc made 14 calls in one session, each with its own record. Each record names the one before it (the arrows), so a record removed from the middle, or two swapped, breaks the chain. The colors are verifiers' phases of an attempt, read from its own timing.

Each square: the call's number in the session, what kind of call it was (root finds the work directory, run runs a command, write writes a file), and the first characters of the record's own digest. Session fingerprint 6a3cc0377978: verifiers copies it into its trace, the record it saves of the attempt, so the trace and the log point at each other.

The scripted moves, and the record each one left

1. bash: ls /workspace; cat /workspace/corpus/ops/recovery_index.json
   → corpus (and 51 more lines)
   record 3695e10dbb60
2. bash: python3 /workspace/corpus/ops/extract_recovery_sources.py
   → "case": "INC-3575", (and 18 more lines)
   record 306f872d2b98
3. bash

   python3 - <<'PY'
   import hashlib, json, uuid
   r = json.load(open('/workspace/recovered_sources.json'))
   payload = r['domain_separator'].encode('utf-8') + b''.join(uuid.UUID(u).bytes for u in r['source_uuids'])
   derived = str(uuid.UUID(bytes=hashlib.sha256(payload).digest()[:16]))
   answer = {'result_uuid': 'PENDING', 'source_uuids': r['source_uuids'], 'evidence_paths': r['evidence_paths']}
   open('/workspace/answer.json', 'w').write(json.dumps(answer, indent=2) + '\n')
   print(derived)
   PY

   → 078fbbe4-adec-65a0-3629-496c56239d16
   record 88160f1a510a
4. edit: in /workspace/answer.json, replace "PENDING" with "078fbbe4-adec-65a0-3629-496c56239d16"
   → Edited /workspace/answer.json
   record 4e3e2c4afcd6
5. bash: cat /workspace/answer.json
   → "result_uuid": "078fbbe4-adec-65a0-3629-496c56239d16", (and 16 more lines)
   record 2f4eec0035bb

The same conversation as verifiers' own harness

Driven by the same script, plimsoll-bash on plimsoll and verifiers' shipped bash harness on verifiers' own docker runtime held the same conversation: 12 of 13 messages identical, character for character (the tool calls' random ids aside), with the same tool definitions. The one that differs:

• message 1 (system): only on plimsoll, "External network access is disabled."

verifiers itself adds that line to the system prompt when the runtime has no network.

Why plimsoll-bash, and not the shipped harness

verifiers' shipped bash harness, on plimsoll, never reached the task: it stopped in its own setup with wget: bad address 'astral.sh'. Every harness verifiers ships installs itself inside the runtime from the internet and then calls the model from inside it, and a plimsoll sandbox has no network at all. plimsoll-bash runs the same loop on the evaluator instead: verifiers' own chat loop from the shipped bash program, loaded unedited, with each bash or edit call sent to the sandbox as one session call. verifiers' harness contract allows a loop to run outside the runtime as long as every model call goes through verifiers' own endpoint, and this one does. For an environment's author it is one line of configuration, --env.agent.harness.id plimsoll-bash, not a change to the environment.

What each call costs

Each call through verifiers' own runtime API, timed on the evaluator with no model and no harness: 5 fresh runtimes per configuration, 20 commands each (the command is true, which does nothing), then a 1 KiB write and read. Bars are medians; the numbers in brackets are the fastest and slowest.

Read it plainly: a plimsoll call is slower than a docker exec. Each one is a full plimsoll project call: the server runs plimsoll's runner in the sandbox, reads the files back and sweeps away anything the command left running, and the client checks the record. The uuid_ctf attempt above made 14 calls, so at these medians plimsoll added about 3.5 s to it under runc (the difference per command times 14, simple arithmetic); a task that runs more commands pays proportionally more. Provisioning a runtime took longer on plimsoll (806 ms under runc, against 678 ms); tearing one down took less time on plimsoll (118 ms under runc, against 367 ms).

Whole attempts, phase by phase (seconds, one attempt each)

One attempt per configuration, so these are single readings, not averages. On verifiers' docker runtime the setup phase includes the shipped harness installing itself (its package installer and its program's packages) into each fresh container; that is why the whole attempt can come out longer there even though every single call is faster.

The limits, stated plainly

These come from the plugin's own documentation (linked at the end of this page). This run met two of them head on: the network, in the shipped harness's setup, and the work directory, in uuid_ctf's /workspace.

• One image. A plimsoll server runs the one image (the packaged file system a sandbox starts from) its operator configured. A task that names an image of its own is refused at start, before any of its code runs, with a message naming both. This run's image is plimsoll's Python image plus bash and a /workspace link, plimsoll-wip/verifiers-agent:dev (ad5b2477d638).
• No network, in any phase. A task that asks for a host on the internet is refused before anything runs; a setup or grading script that downloads something fails with its own error.
• One work directory: the session's /work, or a path the image links to it. uuid_ctf works in /workspace, which this image links to /work; the runtime checked that inside the sandbox before the task started. The rest of the file system is read-only.
• No background processes. Whatever a call leaves running is ended before the next call, so verifiers' run_background is refused, and with it a task's tool servers and long-running helpers.
• A call cut short ends the session's use. When verifiers' time limit cuts a call in flight, the client stops using that session (the call may have run), so the attempt ends as an error rather than a score; the record log lists the session as incomplete.
• Throughput. plimsolld's default limit is 30 runs a minute per caller, about two attempts a minute at this task's 14 calls each; this run raised it (SANDBOX_RATE_PER_MIN=600).

How much of Terminal-Bench 2 runs today: 0 of 89 tasks

Terminal-Bench 2 is a widely used benchmark of command-line tasks for AI agents, and verifiers can load it. This run read every task's files as data, nothing in them executed, at commit 2fd12b88aafd (EXTERNAL · source repo ↗):

The rules are plain text matches over each task's task.toml, tests/test.sh and environment/Dockerfile, written out in the program (census.go). Running Terminal-Bench needs plimsoll to serve a task's own image and either the network at grading or images with the test tools already inside.

What this proves, and what it doesn't

Run it yourself

From a clone of the repository (EXTERNAL · source repo ↗), with docker, gVisor installed as docker's runsc runtime, and Python 3.12:

make docker-images                         # plimsoll's sandbox images
make py-frameworks PY_FRAMEWORKS=verifiers # the hash-locked verifiers environment
go run ./examples/verifiers                # every configuration above, fresh; rewrites this page
go run ./examples/verifiers -verify        # recheck the files beside this page; starts nothing

The program fetches the two outside repositories at their pinned commits and refuses any other: prime-envs at e221a6514f2e, whose task files it also checks by SHA-256 because the task's own scoring code runs on the evaluator (as verifiers runs every task), and Terminal-Bench 2, which it only reads. It spends nothing: no model, no paid sandbox. Everything on this page comes from evidence.json and the record logs beside it:

• records/validate-runc.jsonl: 21 records in 3 sessions, container (21)
• records/validate-runsc.jsonl: 21 records in 3 sessions, kernel (21)
• records/uuid-gold-runc.jsonl: 14 records in 1 session, container (14)
• records/uuid-wrong-runc.jsonl: 10 records in 1 session, container (10)
• records/uuid-gold-runsc.jsonl: 14 records in 1 session, kernel (14)
• records/uuid-shipped-runc.jsonl: 8 records in 1 session, container (8)
• records/calls-runc.jsonl: 115 records in 5 sessions, container (115)
• records/calls-runsc.jsonl: 115 records in 5 sessions, kernel (115)

Recheck one with the Python client alone: check_record_log("records/uuid-gold-runc.jsonl") from plimsoll_client.verifiers_runtime. The runtime plugin, its limits and its configuration: the Python client's guide (EXTERNAL · official docs ↗). verifiers' runtime plugins: verifiers' runtimes guide (EXTERNAL · official docs ↗).