Grading a batch of controllers on every sandbox

These are fixed, hand-authored cart-pole controllers, tried as one batch: 64 candidate cart-pole controllers, each tried from 4 starting angles, 256 trials in all. Every trial was sent twice, as the same JavaScript program, to two different kinds of sandbox, 9 kinds in all, working at once. 256 of 256 trials got the same answer from both sandboxes, byte for byte, 57 of them across two different JavaScript engines, and 107 with a run in the cloud. Every run is signed into one bundle that anyone can check offline.

Fan out, fan in for a batch of controllers: 64 cart-pole controllers, each tried from 4 starting angles, sent across nine kinds of sandboxes through one plimsoll API (a WebAssembly engine, Docker locked down, Docker with gVisor, NVIDIA OpenShell, Kubernetes with runc, gVisor and Kata, a Docker Cloud microVM and an E2B microVM), and gathered into one scoreboard: 16 of 64 kept the pole up from every start.

The trial program computes the score for these fixed formulas. A model that writes its own program needs a separate checker, so it cannot award itself a good score.

Signed runs
512
one fresh sandbox each
Kinds of sandbox
9
each its own daemon, one pool
Isolation tiers
4 of 4
process, container, kernel, vm
Trials that agree
256 of 256
two sandboxes, same bytes
Wall time
2 min 2 s
start to last answer, operator's clock
Cloud cost
$0.0074
estimated: signed run time × list price

The scoreboard

spin gain kd (newtons per radian per second)0123.557.51015angle gain kp (N per radian)510152030406090kp 5, kd 0: balanced from 0 of 4 starting angles; the pole stayed up 409 steps in all0/4kp 5, kd 1: balanced from 0 of 4 starting angles; the pole stayed up 539 steps in all0/4kp 5, kd 2: balanced from 0 of 4 starting angles; the pole stayed up 602 steps in all0/4kp 5, kd 3.5: balanced from 0 of 4 starting angles; the pole stayed up 673 steps in all0/4kp 5, kd 5: balanced from 0 of 4 starting angles; the pole stayed up 767 steps in all0/4kp 5, kd 7.5: balanced from 0 of 4 starting angles; the pole stayed up 908 steps in all0/4kp 5, kd 10: balanced from 0 of 4 starting angles; the pole stayed up 906 steps in all0/4kp 5, kd 15: balanced from 0 of 4 starting angles; the pole stayed up 967 steps in all0/4kp 10, kd 0: balanced from 0 of 4 starting angles; the pole stayed up 534 steps in all0/4kp 10, kd 1: balanced from 0 of 4 starting angles; the pole stayed up 665 steps in all0/4kp 10, kd 2: balanced from 0 of 4 starting angles; the pole stayed up 806 steps in all0/4kp 10, kd 3.5: balanced from 0 of 4 starting angles; the pole stayed up 957 steps in all0/4kp 10, kd 5: balanced from 0 of 4 starting angles; the pole stayed up 1122 steps in all0/4kp 10, kd 7.5: balanced from 0 of 4 starting angles; the pole stayed up 1172 steps in all0/4kp 10, kd 10: balanced from 0 of 4 starting angles; the pole stayed up 1181 steps in all0/4kp 10, kd 15: balanced from 0 of 4 starting angles; the pole stayed up 1251 steps in all0/4kp 15, kd 0: balanced from 0 of 4 starting angles; the pole stayed up 592 steps in all0/4kp 15, kd 1: balanced from 0 of 4 starting angles; the pole stayed up 866 steps in all0/4kp 15, kd 2: balanced from 0 of 4 starting angles; the pole stayed up 1333 steps in all0/4kp 15, kd 3.5: balanced from 0 of 4 starting angles; the pole stayed up 1699 steps in all0/4kp 15, kd 5: balanced from 0 of 4 starting angles; the pole stayed up 1810 steps in all0/4kp 15, kd 7.5: balanced from 0 of 4 starting angles; the pole stayed up 1542 steps in all0/4kp 15, kd 10: balanced from 0 of 4 starting angles; the pole stayed up 1554 steps in all0/4kp 15, kd 15: balanced from 0 of 4 starting angles; the pole stayed up 1318 steps in all0/4kp 20, kd 0: balanced from 0 of 4 starting angles; the pole stayed up 663 steps in all0/4kp 20, kd 1: balanced from 1 of 4 starting angles; the pole stayed up 1375 steps in all1/4kp 20, kd 2: balanced from 2 of 4 starting angles; the pole stayed up 2000 steps in all2/4kp 20, kd 3.5: balanced from 4 of 4 starting angles; the pole stayed up 2000 steps in all4/4kp 20, kd 5: balanced from 4 of 4 starting angles; the pole stayed up 2000 steps in all4/4kp 20, kd 7.5: balanced from 1 of 4 starting angles; the pole stayed up 2000 steps in all1/4kp 20, kd 10: balanced from 1 of 4 starting angles; the pole stayed up 1671 steps in all1/4kp 20, kd 15: balanced from 0 of 4 starting angles; the pole stayed up 1364 steps in all0/4kp 30, kd 0: balanced from 0 of 4 starting angles; the pole stayed up 835 steps in all0/4kp 30, kd 1: balanced from 0 of 4 starting angles; the pole stayed up 1886 steps in all0/4kp 30, kd 2: balanced from 4 of 4 starting angles; the pole stayed up 2000 steps in all4/4kp 30, kd 3.5: balanced from 4 of 4 starting angles; the pole stayed up 2000 steps in all4/4kp 30, kd 5: balanced from 4 of 4 starting angles; the pole stayed up 2000 steps in all4/4kp 30, kd 7.5: balanced from 4 of 4 starting angles; the pole stayed up 2000 steps in all4/4kp 30, kd 10: balanced from 3 of 4 starting angles; the pole stayed up 2000 steps in all3/4kp 30, kd 15: balanced from 3 of 4 starting angles; the pole stayed up 1685 steps in all3/4kp 40, kd 0: balanced from 0 of 4 starting angles; the pole stayed up 806 steps in all0/4kp 40, kd 1: balanced from 0 of 4 starting angles; the pole stayed up 1923 steps in all0/4kp 40, kd 2: balanced from 4 of 4 starting angles; the pole stayed up 2000 steps in all4/4kp 40, kd 3.5: balanced from 4 of 4 starting angles; the pole stayed up 2000 steps in all4/4kp 40, kd 5: balanced from 4 of 4 starting angles; the pole stayed up 2000 steps in all4/4kp 40, kd 7.5: balanced from 4 of 4 starting angles; the pole stayed up 2000 steps in all4/4kp 40, kd 10: balanced from 4 of 4 starting angles; the pole stayed up 2000 steps in all4/4kp 40, kd 15: balanced from 3 of 4 starting angles; the pole stayed up 2000 steps in all3/4kp 60, kd 0: balanced from 0 of 4 starting angles; the pole stayed up 878 steps in all0/4kp 60, kd 1: balanced from 0 of 4 starting angles; the pole stayed up 1871 steps in all0/4kp 60, kd 2: balanced from 4 of 4 starting angles; the pole stayed up 2000 steps in all4/4kp 60, kd 3.5: balanced from 4 of 4 starting angles; the pole stayed up 2000 steps in all4/4kp 60, kd 5: balanced from 4 of 4 starting angles; the pole stayed up 2000 steps in all4/4kp 60, kd 7.5: balanced from 3 of 4 starting angles; the pole stayed up 2000 steps in all3/4kp 60, kd 10: balanced from 3 of 4 starting angles; the pole stayed up 2000 steps in all3/4kp 60, kd 15: balanced from 2 of 4 starting angles; the pole stayed up 2000 steps in all2/4kp 90, kd 0: balanced from 0 of 4 starting angles; the pole stayed up 808 steps in all0/4kp 90, kd 1: balanced from 0 of 4 starting angles; the pole stayed up 1627 steps in all0/4kp 90, kd 2: balanced from 4 of 4 starting angles; the pole stayed up 2000 steps in all4/4kp 90, kd 3.5: balanced from 4 of 4 starting angles; the pole stayed up 2000 steps in all4/4kp 90, kd 5: balanced from 3 of 4 starting angles; the pole stayed up 2000 steps in all3/4kp 90, kd 7.5: balanced from 3 of 4 starting angles; the pole stayed up 2000 steps in all3/4kp 90, kd 10: balanced from 2 of 4 starting angles; the pole stayed up 2000 steps in all2/4kp 90, kd 15: balanced from 2 of 4 starting angles; the pole stayed up 2000 steps in all2/4
Each square is one candidate controller: how many of its 4 starting angles (0.05, 0.1, 0.2, 0.3 radians from upright) it ended balanced from, after 500 steps of 10 ms. Hover a square for the steps the pole stayed up. A square outlined in red had a trial whose two answers did not match, and counts only the trials that did.
01234starting angles ended balanced

16 of 64 candidates balanced the pole from every starting angle. The pattern is the textbook one: too little angle gain (kp) and the pole falls whatever the damping; no spin gain (kd) and it wobbles until it falls; and the strongest gains do worse again, mostly from the larger starting angles.

The candidates are a grid of gains made by hand, from too weak to fight gravity, through no damping (it wobbles until it falls), to very strong. No model wrote them. Each is a linear state-feedback law: the force on the cart is kp times the pole's lean plus kd times its spin plus the cart's position and speed (gains 1 and 1), capped at 20 newtons. Balanced means the pole stayed up all 500 steps and ended within 0.05 radians of upright with the cart within half a metre of the middle.

The sandboxes

SandboxProviderTierWhereRunsMedian runEst. cost
WebAssembly enginewasmprocessinside the daemon, on one laptop570.04 sfree
Docker, locked downdockercontainerone laptop570.69 sfree
Docker with gVisordockerkernelone laptop570.76 sfree
NVIDIA OpenShellopenshellcontainera local OpenShell gateway, one laptop570.97 sfree
Kubernetes agent-sandbox, runcagentsandboxcontainera kind cluster, one laptop560.30 sfree
Kubernetes agent-sandbox, gVisoragentsandboxkernela kind cluster, one laptop570.62 sfree
Kubernetes agent-sandbox, Kata Containersagentsandboxvma kind cluster, one laptop571.95 sfree
E2B microVMe2bvmE2B's cloud571.35 s$0.0040
Docker Cloud Sandboxes microVMdockercloudvmDocker's cloud573.17 s$0.0034

These times describe this run. The rooms used different concurrency, locations and assigned work, so the median column is not a controlled provider-speed comparison.

A median run is the time from dispatch to answer that the run's signed record states, sandbox start included. Cost is that time multiplied by each cloud's list price per second; the provider's bill is the real figure. Where a sandbox is, is the operator's statement; the record states the provider and the tier.

Why the two answers should agree

The trial program uses only floating-point operations whose rounding the JavaScript standard specifies exactly (addition, subtraction, multiplication, division and comparison on 64-bit numbers), with sine and cosine as fixed polynomials instead of the engine's own, so every conforming engine should print the same bytes for a trial, a fingerprint of the whole trajectory included: node in most rooms, QuickJS in the WebAssembly engine. Agreement shows the answers match; it does not prove the simulator matches a real cart-pole or any sandbox secure.

What is signed, and what is stated

Signed by the harness key made for this run, after the client checked each run's record: every request (with the room it was sent to) and every answer, including the provider and the isolation tier the daemon reported and the run's start and end. A changed, missing or moved line fails the check below. Stated by the operator, in unsigned files beside the bundle: the batch and what each room is and where (rooms.json, with the engine and the list price), and when the run started and ended, which rooms sat out and why (run.json). A tier is the daemon's report: plimsoll's configuration and the provider's evidence plus a startup smoke test, never runtime attestation, and nothing here independently proves which machine ran the code.

Check it yourself

From a checkout of the repository:

go run ./examples/fanout-cartpole -verify

It checks every signature in bundle.jsonl under harness.pub (key 30f45e2d326e737ccc5a7f2f6f967255f14307076e885600af59cfde06059571), the chain that catches a line added, removed or moved, each signed record against the request and answer stored beside it, that each run's record states the provider and tier of the room it was sent to, and that this page is what the bundle and the two stated files render. It starts no sandbox.

Words on this page

Cart-pole
A cart on a rail with a pole hinged on top; the only thing a controller can do is push the cart left or right, and the job is to keep the pole up.
Controller
Code that decides, every step, how hard to push. Here every candidate is the same formula with different gains.
Gain
How strongly the controller reacts to something: kp to how far the pole leans, kd to how fast it is turning.
Sandbox
A place to run code someone else wrote without letting it reach the machine around it. Here each run gets a fresh one.
Isolation tier
How strong the wall is, weakest to strongest: process (a WebAssembly engine inside plimsoll's own process), container (Linux namespaces and a filter on system calls), kernel (gVisor, a separate kernel that answers the code's system calls), vm (a virtual machine with its own kernel).
Fan-out, fan-in
Sending pieces of one job to many workers at once, then gathering the answers into one result.

Run 2026-10-23 20:01 UTC. 64 candidates, 4 starting angles each. Rendered from the signed bundle by go run ./examples/fanout-cartpole.