A plimsoll example · the squeeze test, with a scrambled recipe

Does a fresh AI agent follow the plan it is handed?

I gave 40 fresh AI agents (an AI left alone with a job and a terminal: it writes code, runs it, reads what happened and tries again, like a new hire on their first day) the same small job: write the program that flies a simulated drone through three gates and lands it on a pad. Of those, 16 also got a plan: a short recipe of 32 tokens (the pieces a language model reads text in; a short word is usually one token, so this is about 23 words) left behind by an earlier AI that had learned to fly this course well. Think of it as the sticky note the last person on the job left for you. Another 16 got the same sticky note with its words cut apart and glued back in a random order. And 8 got no note at all. Then plimsoll, a sandbox that runs untrusted code and signs a record of every run, graded each agent's program on 64 flights no agent had seen, like a final exam kept in a sealed envelope.

The short answer, in three parts. First, the order of the words mattered: agents with the honest recipe landed more flights than agents with the same words shuffled, by a margin bigger than luck alone would give, under a rule for reading the result that was written down before the run. Second, the honest recipe did not clearly beat having no recipe at all; its lead there is small enough to be luck. So the shuffled words look more like they hurt than the honest recipe looks like it helped (my reading, not the registered measure). Third, when the code was searched for the recipe's own words, none of the recipe or shuffled-recipe agents had used any of the recipe's 7 named parts, and every one of the 16 honest-recipe agents, asked afterward, said it had followed the recipe "only partly" or "only in part". Every number is below, marked good, bad or inconclusive, with the agents' own words.

Whose experiment this is

This experiment was built from three suggestions by a reader who builds AI red-team agents (programs that attack a computer network to test its defenses). Each one shaped it:

  • Score by what changed, never by what the agent says it did. The landings come from signed grading runs; the agents' own accounts are kept apart and enter no score.
  • Shuffle the plan and see whether the agent's actions change. That is the shuffled-recipe group, this experiment's main comparison, and it found that the order of the words mattered. The chart below shows what is compared and how to read it.
  • Ask whether hiding the simulator would send agents hunting for another shortcut. The simulator never entered an agent's sandbox, and no controller carried a random number generator (0 of 16 honest-recipe, 0 of 16 shuffled-recipe and 0 of 8 no-recipe controllers did).
What the main comparison compares
Each agent's program flew the same 64 sealed flights; its score is how many it landed. A bar is a group's average share. Higher is better. Hover over or tap a bar for its numbers.
Honest recipe88.0% landed · 16 agents
from the honest recipe: 16 agents, 88.0% of the 64 sealed flights landed on average
Scrambled recipe80.3% landed · 16 agents
from the honest recipe with its words shuffled: 16 agents, 80.3% of the 64 sealed flights landed on average
The gap: 7.7 pointsabout 5 more landings in 64 flights
No recipe85.5% landed · 8 agents, for reference
from no recipe: 8 agents, 85.5% of the 64 sealed flights landed on average
share of the 64 sealed flights landed
Could the gap be luck? Its 95% interval
Honest minus scrambled+7.7 points
95%: +2.8 to +12.6
recipe minus scrambled: +7.7 points (95% interval +2.8 to +12.6; 90% interval +3.6 to +11.8)
difference, in points
← scrambled did betterhonest recipe did better →
All numbers come from the signed grading runs, through the same rule as the differences figure in section 3.

How to read it.

  • What is compared: the average share of the 64 sealed flights landed by the 16 honest-recipe agents (88.0%) and by the 16 scrambled-recipe agents (80.3%), whose recipe had the same words shuffled. Higher is better: more of the flights ended on the pad.
  • The gap: 88.0% minus 80.3% is 7.7 percentage points (the plain difference between two percentages), about 5 more landings for every 64 flights.
  • The 95% interval: 16 agents is a small class, so a rerun with new agents would give a somewhat different gap. The interval, +2.8 to +12.6 points, is the range the real gap plausibly sits in. Even its low end is above zero, so luck alone would rarely make the honest recipe win by this much. With groups this small, read the range as a bit optimistic (the 95% entry below says why).
  • No recipe: 85.5%, only 2.4 points behind the honest recipe, and that gap's interval (-1.5 to +6.4) holds zero, so it could be luck.

For readers who know AI: the method, the statistics and related work (INTERNAL · method page →).

Why only partly? The likely reason, in the agents' own words

This is a reading of what the agents wrote, not a measured result. Every agent started from the same working baseline controller, a plain program that already flies the course roughly. Most of the honest-recipe agents bolted a few of the recipe's cheapest ideas onto that baseline, flew their four smoke flights, saw them pass, and stopped. Many said they stopped once those flights passed, and some said they left parts out because they could not see inside the simulator to tune them. Two of them, in their own words:

the baseline plus these small changes passed all four smoke flights and all eight random flights on the first try. With no failures to diagnose, I stopped there instead of adding complexity I couldn't show was needed.easy.recipe.0, honest recipe, 57 of 64 fresh flights

I took the quickest route, which was to patch the baseline and fly it.easy.recipe.3, honest recipe, 58 of 64 fresh flights

So the most probable reason is that the course was easy enough, and the starting point good enough, that a full plan was never needed to pass the tests the agents could see. A plan gets followed when following it is the shortest road to a passing grade.

A proposal for a next run (not done here): start every agent from a baseline that cannot fly the course on its own, so that the recipe is worth more than a quick patch, and grade the practice flights on a harder course, where a patched baseline does not already land most flights. Then the recipe's costly parts, the ones every agent here skipped, would have to earn their place, and the code measure would have something to find. And give the control group a note that reads well but is wrong for the course (the harder course's recipe, say) instead of shuffled words, so that a win for the honest recipe means its content was used and not only that garbled text got in the way (section 5 says why that matters).

Eight words this page leans on
Agent
An AI model working in a loop on its own: write code, run it, read the result, try again.
Recipe
The plan: everything an earlier AI learned on this course, squeezed into 32 tokens, the units a language model reads and writes (a short word is usually one).
Scrambled recipe, the control
The same words in a shuffled order, fixed by a seed (one number that decides a random shuffle, so anyone can redo the exact same shuffle). A control is a group given everything except the thing being tested, so a difference shows what that thing did.
Sealed flights
The 64 flights every program was graded on, drawn at random before the first agent ran, fixed by their SHA-256 (a 64-character fingerprint of the file: change one byte and it changes) and never shown to any agent.
Noise
The part of a score that is luck: another 8 fresh agents would not land exactly the same number of flights. A difference is "within noise" when luck alone could have made it.
95% interval
The range a difference between two groups plausibly sits in, given how much the agents varied. Here it comes from resampling: draw each group's agents again at random, with repeats, 10,000 times, and see how far the difference moves. A difference whose interval sits wholly above zero is unlikely to be luck; one whose interval holds zero could be. One caution: with only 8 to 16 agents in a group, ranges made this way (the percentile bootstrap) catch the true difference somewhat less often than 95% of the time, so read them as a bit optimistic.
Points
Percentage points of the 64 fresh flights: one flight is about 1.6 points, so a 10-point difference is about 6 flights of 64.
Registered rule
The rule for reading the result, written down before the first agent ran, so the result could not choose how it was read.

Honest recipe

16 agents · stage recipe
88.0%
of the 64 fresh flights landed, on average; the lowest agent landed 51, the highest 60
Recipe words named in code: 0 of 7 on average · quoted its recipe: 16 of 16

Scrambled recipe

16 agents · stage recipe
80.3%
of the 64 fresh flights landed, on average; the lowest agent landed 44, the highest 60
Recipe words named in code: 0 of 7 on average · quoted its recipe: 1 of 16

No recipe

8 agents · stage baseline
85.5%
of the 64 fresh flights landed, on average; the lowest agent landed 50, the highest 59
Recipe words named in code: 0.125 of 7 on average
How to read these numbers
  • 85.5% with no recipe is the yardstick. It is what a fresh agent does on this course with the baseline and nothing else. A recipe helps only if it beats that; the gap between the groups is the story, not the size of any one number.
  • 90% is the "too easy" line. Had the no-recipe agents landed that many, the rule would not have read landings at all, because a plan has no room to show on a course nearly everyone already passes. 85.5% sits under it, but not far under: see the last verdict in section 5.
  • A difference counts when its 95% interval is wholly on one side of zero. That is the registered rule's line. A difference under 10 points (about 6 flights of 64, one flight in ten) is one the rule calls too small to matter. That line is a judgment written into the rule before the run, not a measured threshold. With only 8 to 16 agents in a group, these ranges catch the true difference somewhat less often than 95% of the time, so read them as a bit optimistic; the method page says more.
  1. The question
  2. The setup
  3. The landings
  4. The registered rule, and what it says
  5. Good, bad or inconclusive
  6. What the code measure reads
  7. In their own words
  8. What this does not show
  9. On plimsoll, signed
  10. Check it yourself
  11. Words used on this page

1The question

A plan only helps if it is followed. The squeeze test (INTERNAL · example →) asks fresh agents to rebuild a working controller (INTERNAL · glossary →), the program that flies the drone, from a 32-token recipe. This experiment asks whether the recipe did anything. The control is the recipe's own words in a shuffled order: the same ingredients, no method. If the agents ignore the recipe, the shuffled one should do just as well, and their code should look the same. If they follow it, the honest order should win.

Two measures, both read by code and never from an agent's own account of itself:

And one thing gathered by asking. Once each session was over, the agent was asked, with no tool left to run:

Your session is over. Nothing you do now changes your controller or its score, and no tool will run. In a few sentences: did you follow the recipe you were given? Which parts did you use, which did you change or leave out, and why?

(The no-recipe agents were asked what approach their controller took, and why.) The answer changes nothing about the score. It is on the page because what an agent says it did and what its code does are different things, and the experiment can show both.

2The setup

Who got what
Every agent: the same job, the same starting controller, the same tools and the same practice budget. Only the note differed. Every finished program then sat the same sealed exam.

Honest recipe

16 agents
The note
Filter velocity; fast/slow accel disturbance observer; yaw + local pos integrals; smooth ref filter; gate exit lock; predictive cross abort; ramp pad descent.

Scrambled recipe

16 agents
The same words, shuffled
filter; accel smooth disturbance + ramp fast/slow lock; local exit abort; gate integrals; predictive pos pad observer; Filter ref cross yaw descent. velocity;

No recipe

8 agents
No note
Nothing: the course, the baseline and the tools.
The recipe is the plan an earlier AI learned on this course, 32 tokens long. The scrambled note is the same words shuffled once, by a fixed seed, so every scrambled agent read the same shuffled note.
The course, and one sealed flight flown by three programs
The room from above and from the side. Take off at the left, fly through gates 1, 2 and 3 in order, land on the pad at the right, within 25 seconds. The three lines are the first agent of each group flying the same sealed flight (the same wind, gusts, payload and weak motor), as recorded in the signed grading run.
From above room 6 m × 6 m, grid 1 m take off land here 1 2 3
From the side height up to 3 m, grid 1 m 1 2 3
  • Honest recipe, agent 0: landed after 19.625 simulated seconds.
  • Scrambled recipe, agent 0: landed after 18.08 simulated seconds.
  • No recipe, agent 0: landed after 15.18 simulated seconds.
  • Each gate is a square frame 0.60 m across, seen edge-on: the drone must pass through the dotted opening without touching the black frame. The ring is where each flight ended. seed 2879061876
A flight "lands" when the drone settles on the pad slowly and level within the 25 seconds. It fails when it clips a gate's frame, hits the floor, a wall or the ceiling, flips, or runs out of time. All three of these landed; what differs is how directly they flew and how long they took. This is one sealed flight of 64; the picture is here to show what a flight is, not to compare the groups.

The honest recipe is the note an earlier AI wrote about this course, counted at 32 tokens by Gemini's tokenizer (count_tokens on gemini-4-argon), as the hackathon counted it. (That AI learned the course over many rounds of trying and fixing in the did-it-learn hackathon entry the course comes from; "honest" means it flew the course for real rather than memorizing the test flights.) The scrambled note is its words shuffled by a fixed seed. Below, the honest note is cut at its semicolons into its 7 items, and each word of the scrambled note carries the number of the item it came from, so you can see the shuffle.

The recipe, cut into its 7 items, beside the same words shuffled

The honest recipe, item by item

1Filter velocity;
2fast/slow accel disturbance observer;
3yaw + local pos integrals;
4smooth ref filter;
5gate exit lock;
6predictive cross abort;
7ramp pad descent.

The scrambled recipe, as the 16 scrambled agents read it

filter;4 accel2 smooth4 disturbance2 +3 ramp7 fast/slow2 lock;5 local3 exit5 abort;6 gate5 integrals;3 predictive6 pos3 pad7 observer;2 Filter1 ref4 cross6 yaw3 descent.7 velocity;1

Each small number says which item of the honest recipe the word came from. The words are all there; the method is not.

The 7 items are flying techniques. In plain words: smooth the speed reading; estimate the wind and the drone's own drift and correct for them; slowly correct small leftover errors in heading and position; move the target point smoothly instead of jumping it from gate to gate; once committed to a gate, hold the line until it is cleared; abort a gate crossing that is about to hit the frame; and descend onto the pad gradually. A reader who knows control engineering will recognize each; the agents were given only the words above.

Each group's brief ends differently; after the course description, a recipe agent read:

A builder before you worked on exactly this task for a long time. Everything it learned was compressed into this recipe, which is all you get from it:

    Filter velocity; fast/slow accel disturbance observer; yaw + local pos integrals; smooth ref filter; gate exit lock; predictive cross abort; ramp pad descent.

Build controller.js following the recipe. You may fly the smoke flights and any new flights you choose; you will not see the flights you are graded on. Leave your best version in controller.js, and end your reply with one line: SMOKE=<passed>/4.

and a no-recipe agent:

Build the best one you can in controller.js. You may fly the smoke flights and any new flights you choose; you will not see the flights you are graded on. Leave your best version in controller.js, and end your reply with one line: SMOKE=<passed>/4.

3The landings

Each dot below is one agent's final controller on the 64 sealed fresh flights. The honest recipe's agents all landed between 51 and 60; the scrambled group spread from 44 to 60; with no recipe, 50 to 59. Read it like a class's exam marks: one dot a student, the dark line the class average.

Fresh flights landed, one dot an agent
The dark line is the group's average. Hover over or tap a dot for the agent.
Honest recipe16 agents · 88.0% on average
lowest 51, highest 60 of 64
easy.recipe.0: 57 of 64 fresh flights landed, 15 of 16 practice easy.recipe.1: 58 of 64 fresh flights landed, 15 of 16 practice easy.recipe.2: 59 of 64 fresh flights landed, 16 of 16 practice easy.recipe.3: 58 of 64 fresh flights landed, 15 of 16 practice easy.recipe.4: 54 of 64 fresh flights landed, 15 of 16 practice easy.recipe.5: 54 of 64 fresh flights landed, 11 of 16 practice easy.recipe.6: 54 of 64 fresh flights landed, 15 of 16 practice easy.recipe.7: 58 of 64 fresh flights landed, 15 of 16 practice easy.recipe.8: 55 of 64 fresh flights landed, 10 of 16 practice easy.recipe.9: 55 of 64 fresh flights landed, 16 of 16 practice easy.recipe.10: 58 of 64 fresh flights landed, 13 of 16 practice easy.recipe.11: 58 of 64 fresh flights landed, 15 of 16 practice easy.recipe.12: 51 of 64 fresh flights landed, 13 of 16 practice easy.recipe.13: 58 of 64 fresh flights landed, 16 of 16 practice easy.recipe.14: 60 of 64 fresh flights landed, 16 of 16 practice easy.recipe.15: 54 of 64 fresh flights landed, 16 of 16 practice
Scrambled recipe16 agents · 80.3% on average
lowest 44, highest 60 of 64
easy.scrambled.0: 60 of 64 fresh flights landed, 12 of 16 practice easy.scrambled.1: 55 of 64 fresh flights landed, 14 of 16 practice easy.scrambled.2: 47 of 64 fresh flights landed, 14 of 16 practice easy.scrambled.3: 56 of 64 fresh flights landed, 16 of 16 practice easy.scrambled.4: 44 of 64 fresh flights landed, 12 of 16 practice easy.scrambled.5: 45 of 64 fresh flights landed, 9 of 16 practice easy.scrambled.6: 45 of 64 fresh flights landed, 10 of 16 practice easy.scrambled.7: 53 of 64 fresh flights landed, 11 of 16 practice easy.scrambled.8: 46 of 64 fresh flights landed, 11 of 16 practice easy.scrambled.9: 60 of 64 fresh flights landed, 16 of 16 practice easy.scrambled.10: 50 of 64 fresh flights landed, 10 of 16 practice easy.scrambled.11: 45 of 64 fresh flights landed, 12 of 16 practice easy.scrambled.12: 59 of 64 fresh flights landed, 14 of 16 practice easy.scrambled.13: 53 of 64 fresh flights landed, 15 of 16 practice easy.scrambled.14: 45 of 64 fresh flights landed, 12 of 16 practice easy.scrambled.15: 59 of 64 fresh flights landed, 14 of 16 practice
No recipe8 agents · 85.5% on average
lowest 50, highest 59 of 64
easy.none.0: 52 of 64 fresh flights landed, 13 of 16 practice easy.none.1: 58 of 64 fresh flights landed, 15 of 16 practice easy.none.2: 58 of 64 fresh flights landed, 16 of 16 practice easy.none.3: 51 of 64 fresh flights landed, 11 of 16 practice easy.none.4: 54 of 64 fresh flights landed, 13 of 16 practice easy.none.5: 59 of 64 fresh flights landed, 14 of 16 practice easy.none.6: 50 of 64 fresh flights landed, 14 of 16 practice easy.none.7: 56 of 64 fresh flights landed, 15 of 16 practice
fresh flights landed, of 64
Every score comes from the signed grading runs. A single group's average is shown without a range of its own: the registered rule puts its ranges on the differences between groups, in the next figure.
GroupAgentsFresh flights landedLowest and highest agentRecipe words named in code
Honest recipe1688.0%51 and 60 of 640
Scrambled recipe1680.3%44 and 60 of 640
No recipe885.5%50 and 59 of 640.125 (1 agent)

Every flight of every agent

The picture below is the whole exam sheet: one row an agent, one column one of the 64 sealed flights, in the order they were drawn. A colored cell is a landing; a gray cell is a flight that failed. The bars on top count how many of the 40 programs landed each flight, so a gray streak down the page is a flight that was hard for everyone, usually a strong gust at a bad moment.

64 sealed flights × 40 programs
Rows are agents, best first within each group; columns are the sealed flights. Hover over or tap a row or a bar for its numbers.
programs that landed each flight, of 40 Flight 1 of 64 (seed 2879061876): landed by 34 of 40 controllers Flight 2 of 64 (seed 2891370626): landed by 39 of 40 controllers Flight 3 of 64 (seed 799591320): landed by 39 of 40 controllers Flight 4 of 64 (seed 563647025): landed by 35 of 40 controllers Flight 5 of 64 (seed 1548503436): landed by 34 of 40 controllers Flight 6 of 64 (seed 4152555639): landed by 37 of 40 controllers Flight 7 of 64 (seed 2747788967): landed by 39 of 40 controllers Flight 8 of 64 (seed 3383360511): landed by 17 of 40 controllers Flight 9 of 64 (seed 3258768806): landed by 35 of 40 controllers Flight 10 of 64 (seed 3535017532): landed by 37 of 40 controllers Flight 11 of 64 (seed 4294942085): landed by 30 of 40 controllers Flight 12 of 64 (seed 229863891): landed by 32 of 40 controllers Flight 13 of 64 (seed 3929221894): landed by 34 of 40 controllers Flight 14 of 64 (seed 216191576): landed by 37 of 40 controllers Flight 15 of 64 (seed 1471955979): landed by 30 of 40 controllers Flight 16 of 64 (seed 1820725325): landed by 31 of 40 controllers Flight 17 of 64 (seed 3381641293): landed by 17 of 40 controllers Flight 18 of 64 (seed 2658149216): landed by 33 of 40 controllers Flight 19 of 64 (seed 429510935): landed by 35 of 40 controllers Flight 20 of 64 (seed 2732532995): landed by 40 of 40 controllers Flight 21 of 64 (seed 2668116078): landed by 36 of 40 controllers Flight 22 of 64 (seed 2853239299): landed by 34 of 40 controllers Flight 23 of 64 (seed 2029850598): landed by 40 of 40 controllers Flight 24 of 64 (seed 2251776238): landed by 34 of 40 controllers Flight 25 of 64 (seed 3264701845): landed by 34 of 40 controllers Flight 26 of 64 (seed 1966671844): landed by 22 of 40 controllers Flight 27 of 64 (seed 1556485462): landed by 37 of 40 controllers Flight 28 of 64 (seed 1785160797): landed by 38 of 40 controllers Flight 29 of 64 (seed 1564868972): landed by 40 of 40 controllers Flight 30 of 64 (seed 3859806851): landed by 30 of 40 controllers Flight 31 of 64 (seed 4041726930): landed by 29 of 40 controllers Flight 32 of 64 (seed 3480279435): landed by 29 of 40 controllers Flight 33 of 64 (seed 432495670): landed by 35 of 40 controllers Flight 34 of 64 (seed 1637917064): landed by 36 of 40 controllers Flight 35 of 64 (seed 398996294): landed by 38 of 40 controllers Flight 36 of 64 (seed 667954921): landed by 37 of 40 controllers Flight 37 of 64 (seed 2916066164): landed by 36 of 40 controllers Flight 38 of 64 (seed 1031528486): landed by 12 of 40 controllers Flight 39 of 64 (seed 1199283237): landed by 37 of 40 controllers Flight 40 of 64 (seed 2947980494): landed by 33 of 40 controllers Flight 41 of 64 (seed 1538387364): landed by 39 of 40 controllers Flight 42 of 64 (seed 514583858): landed by 39 of 40 controllers Flight 43 of 64 (seed 1672342334): landed by 39 of 40 controllers Flight 44 of 64 (seed 4046844096): landed by 34 of 40 controllers Flight 45 of 64 (seed 4004832543): landed by 38 of 40 controllers Flight 46 of 64 (seed 1057665710): landed by 36 of 40 controllers Flight 47 of 64 (seed 1241939160): landed by 37 of 40 controllers Flight 48 of 64 (seed 2301455890): landed by 37 of 40 controllers Flight 49 of 64 (seed 3277075144): landed by 33 of 40 controllers Flight 50 of 64 (seed 1764884014): landed by 36 of 40 controllers Flight 51 of 64 (seed 2588916200): landed by 36 of 40 controllers Flight 52 of 64 (seed 304342820): landed by 12 of 40 controllers Flight 53 of 64 (seed 950201630): landed by 40 of 40 controllers Flight 54 of 64 (seed 1171352504): landed by 39 of 40 controllers Flight 55 of 64 (seed 3636045209): landed by 22 of 40 controllers Flight 56 of 64 (seed 3543222049): landed by 34 of 40 controllers Flight 57 of 64 (seed 2947775064): landed by 40 of 40 controllers Flight 58 of 64 (seed 4142984597): landed by 33 of 40 controllers Flight 59 of 64 (seed 3367854695): landed by 35 of 40 controllers Flight 60 of 64 (seed 2957321384): landed by 29 of 40 controllers Flight 61 of 64 (seed 1185777691): landed by 32 of 40 controllers Flight 62 of 64 (seed 1339771557): landed by 38 of 40 controllers Flight 63 of 64 (seed 4051591197): landed by 33 of 40 controllers Flight 64 of 64 (seed 2891400317): landed by 38 of 40 controllers easy.recipe.14 60 of 64 easy.recipe.14: 60 of 64 fresh flights landed, 16 of 16 practice easy.recipe.2 59 of 64 easy.recipe.2: 59 of 64 fresh flights landed, 16 of 16 practice easy.recipe.1 58 of 64 easy.recipe.1: 58 of 64 fresh flights landed, 15 of 16 practice easy.recipe.3 58 of 64 easy.recipe.3: 58 of 64 fresh flights landed, 15 of 16 practice easy.recipe.7 58 of 64 easy.recipe.7: 58 of 64 fresh flights landed, 15 of 16 practice easy.recipe.10 58 of 64 easy.recipe.10: 58 of 64 fresh flights landed, 13 of 16 practice easy.recipe.11 58 of 64 easy.recipe.11: 58 of 64 fresh flights landed, 15 of 16 practice easy.recipe.13 58 of 64 easy.recipe.13: 58 of 64 fresh flights landed, 16 of 16 practice easy.recipe.0 57 of 64 easy.recipe.0: 57 of 64 fresh flights landed, 15 of 16 practice easy.recipe.8 55 of 64 easy.recipe.8: 55 of 64 fresh flights landed, 10 of 16 practice easy.recipe.9 55 of 64 easy.recipe.9: 55 of 64 fresh flights landed, 16 of 16 practice easy.recipe.4 54 of 64 easy.recipe.4: 54 of 64 fresh flights landed, 15 of 16 practice easy.recipe.5 54 of 64 easy.recipe.5: 54 of 64 fresh flights landed, 11 of 16 practice easy.recipe.6 54 of 64 easy.recipe.6: 54 of 64 fresh flights landed, 15 of 16 practice easy.recipe.15 54 of 64 easy.recipe.15: 54 of 64 fresh flights landed, 16 of 16 practice easy.recipe.12 51 of 64 easy.recipe.12: 51 of 64 fresh flights landed, 13 of 16 practice easy.scrambled.0 60 of 64 easy.scrambled.0: 60 of 64 fresh flights landed, 12 of 16 practice easy.scrambled.9 60 of 64 easy.scrambled.9: 60 of 64 fresh flights landed, 16 of 16 practice easy.scrambled.12 59 of 64 easy.scrambled.12: 59 of 64 fresh flights landed, 14 of 16 practice easy.scrambled.15 59 of 64 easy.scrambled.15: 59 of 64 fresh flights landed, 14 of 16 practice easy.scrambled.3 56 of 64 easy.scrambled.3: 56 of 64 fresh flights landed, 16 of 16 practice easy.scrambled.1 55 of 64 easy.scrambled.1: 55 of 64 fresh flights landed, 14 of 16 practice easy.scrambled.7 53 of 64 easy.scrambled.7: 53 of 64 fresh flights landed, 11 of 16 practice easy.scrambled.13 53 of 64 easy.scrambled.13: 53 of 64 fresh flights landed, 15 of 16 practice easy.scrambled.10 50 of 64 easy.scrambled.10: 50 of 64 fresh flights landed, 10 of 16 practice easy.scrambled.2 47 of 64 easy.scrambled.2: 47 of 64 fresh flights landed, 14 of 16 practice easy.scrambled.8 46 of 64 easy.scrambled.8: 46 of 64 fresh flights landed, 11 of 16 practice easy.scrambled.5 45 of 64 easy.scrambled.5: 45 of 64 fresh flights landed, 9 of 16 practice easy.scrambled.6 45 of 64 easy.scrambled.6: 45 of 64 fresh flights landed, 10 of 16 practice easy.scrambled.11 45 of 64 easy.scrambled.11: 45 of 64 fresh flights landed, 12 of 16 practice easy.scrambled.14 45 of 64 easy.scrambled.14: 45 of 64 fresh flights landed, 12 of 16 practice easy.scrambled.4 44 of 64 easy.scrambled.4: 44 of 64 fresh flights landed, 12 of 16 practice easy.none.5 59 of 64 easy.none.5: 59 of 64 fresh flights landed, 14 of 16 practice easy.none.1 58 of 64 easy.none.1: 58 of 64 fresh flights landed, 15 of 16 practice easy.none.2 58 of 64 easy.none.2: 58 of 64 fresh flights landed, 16 of 16 practice easy.none.7 56 of 64 easy.none.7: 56 of 64 fresh flights landed, 15 of 16 practice easy.none.4 54 of 64 easy.none.4: 54 of 64 fresh flights landed, 13 of 16 practice easy.none.0 52 of 64 easy.none.0: 52 of 64 fresh flights landed, 13 of 16 practice easy.none.3 51 of 64 easy.none.3: 51 of 64 fresh flights landed, 11 of 16 practice easy.none.6 50 of 64 easy.none.6: 50 of 64 fresh flights landed, 14 of 16 practice
Honest recipe, 16 agentsScrambled recipe, 16 agentsNo recipe, 8 agentsflight failed
5 of the 64 flights were landed by every program; 4 were landed by fewer than half of them; the hardest, flight 38, was landed by 12 of 40. These are counts of the run's own flights, not a measure the rule reads. Every cell comes from the signed grading runs.

The differences between groups

A difference is one group's average minus another's, in points: percentage points of the fresh flights, so on 64 flights one flight is about 1.6 points. Each has a 95% interval and a 90% interval from the same resampling: the 90% one is narrower, and the rule uses it only to decide that a difference is too small to matter.

Fresh flights landed: the three differences, in points
The dot is the difference; the thin line its 95% interval; the thick bar its 90% interval; the dashed line is zero. A dot whose thin line stays clear of the dashed line is a difference that luck alone would rarely produce. Hover over or tap a dot for the numbers.
Recipe minus scrambled+7.7 points
95%: +2.8 to +12.6
recipe minus scrambled: +7.7 points (95% interval +2.8 to +12.6; 90% interval +3.6 to +11.8)
Recipe minus none+2.4 points
95%: -1.5 to +6.4
recipe minus none: +2.4 points (95% interval -1.5 to +6.4; 90% interval -0.9 to +5.9)
Scrambled minus none-5.3 points
95%: -11.0 to +0.6
scrambled minus none: -5.3 points (95% interval -11.0 to +0.6; 90% interval -10.2 to -0.4)
difference, in points
The intervals come from the registered rule's own resampling (10,000 draws, seed 20261106): each group's agents drawn again at random, with repeats, and the difference computed again each time.
Difference in fresh flights landedPoints95% interval90% intervalIts 95% interval
recipe minus scrambled+7.7+2.8 to +12.6+3.6 to +11.8sits above zero
recipe minus none+2.4-1.5 to +6.4-0.9 to +5.9holds zero: could be noise
scrambled minus none-5.3-11.0 to +0.6-10.2 to -0.4holds zero: could be noise

On recipe words named in code, the differences are 0 for recipe minus scrambled (95% interval 0 to 0), -0.125 for recipe minus none (95% interval -0.375 to 0) and -0.125 for scrambled minus none (95% interval -0.375 to 0): every recipe and scrambled controller names 0 items, and the only item named at all is one no-recipe agent's.

4The registered rule, and what it says

The rule was written down before the first agent ran. Its words are in experiment.json beside this page, and this page's program applies them by code; the record of when they were written is not public, so you can read the rule here but not check its date from this page alone. It resamples agents within each group 10,000 times (seed 20261106) and tries its four outcomes in this order, stopping at the first that fits:

What decided it. Recipe minus scrambled, on landings: +7.7 points (about 5 flights of 64), 95% interval +2.8 to +12.6, wholly above zero. Landings were read, because the no-recipe group landed 85.5%, under the 90% line. On recipe words named in code, recipe minus scrambled was 0 with a 95% interval of 0 to 0, so words did not decide it: the landings did.

So, in the rule's words: ORDER SHOWED. The recipe in its order moved the landings beyond the same words shuffled. It does not show the code does what the recipe says.

5Good, bad or inconclusive

The scale: GOOD means this run gives evidence that the agents used what they were handed; BAD means evidence that they did not, or a weakness of this run that limits what it can say; INCONCLUSIVE means this run cannot tell either way. These marks are my rating of each finding, not the registered rule's outcome, which is in section 4.

GOOD

The recipe's order moved the landings

The honest recipe's agents landed 88.0% of the fresh flights; the scrambled recipe's, 80.3%: +7.7 points, 95% interval +2.8 to +12.6. By the registered rule, ORDER SHOWED: the same words did better in their honest order than shuffled, so the order of what an agent was handed changed how its program flew.

INCONCLUSIVE

Does the honest recipe beat no recipe? Not that this run can see

Recipe minus none: +2.4 points, 95% interval -1.5 to +6.4, which holds zero. Scrambled minus none: -5.3 points, 95% interval -11.0 to +0.6, which holds zero too, only just: its 90% interval, -10.2 to -0.4, does not.

My reading, which is not the registered measure: the order effect looks more like the shuffled words hurting than the honest recipe helping. A shuffled plan is not nothing; it is noise the agent has to read and decide about.

What the comparison cannot separate. The control was the plan's own words shuffled into nonsense. So "the honest order won" can mean two different things: that the agents took something from the plan's content, or only that a garbled note costs an agent something (time, confusion, a worse starting idea) that a clear note does not. This run cannot tell those two apart. A next run could, by giving the control group a note that reads well but is wrong for this course: the recipe written for the harder drone course, or the same 7 items kept whole and reordered, or a plausible recipe for a different task. If the honest recipe still wins against a coherent wrong plan, the content mattered; if not, it was the garbling. That is a proposal, not something done here.

BAD

No recipe item reached the code by name

Of the recipe's 7 items, the honest recipe's controllers name 0 on average and the scrambled recipe's 0: not one controller in either group adds a name that matches any item. With no recipe, one agent did, by chance: easy.none.4 named "gate exit lock".

BAD

What the agents say they used, their code does not name

Asked afterward, all 16 of the 16 honest-recipe agents said they followed the recipe only partly or only in part, and 12 of them described pieces they used "in simplified form". (12 of the 16 scrambled agents also said only partly or only in part.) The code measure finds none of the pieces named. Recipe agent 0 is one case:

Only partly. I started from the baseline follower and added a few recipe-like pieces, but I did not build the full recipe.easy.recipe.0, honest recipe, 57 of 64 fresh flights

It lists a velocity filter, local position integrals and "a crude version of the recipe's abort". In its controller the word "abort" sits only in a comment, which the measure removes, and the filter and the integrals have names that carry none of the recipe's words (section 6 shows the lines).

Neither side of this shows behavior: an account is the agent's word, and a name is only a name. Read by eye, for this one controller and not as a measure, the code does hold a filter on velocity, x and z integrals and a step back to lining up near a gate, under short names the patterns cannot match. So the name count can miss real pieces, and it cannot confirm the accounts either. Checking whether code does what a recipe says needs a test of what the code does, which this experiment did not have.

BAD

One easy course, close to its ceiling

With no recipe at all, agents already landed 85.5% of the fresh flights, near the 90% line past which the rule would not read landings. A plan has little room to help on a course this easy. One model, one recipe and one course: the result says nothing about a harder course or another model.

6What the code measure reads

The recipe's 7 items and the word patterns that count each one, the same for every group, fixed in the job file before the run. A pattern is matched, case ignored, against each name in the agent's final controller.js once comments are removed, skipping the names the shared baseline already holds (the baseline itself names one item, its altitude integral integralY, so every group starts at 0).

Recipe itemPatterns
Filter velocityvel\w*(filt|lpf|smooth|ema|alpha) · (filt|lpf|smooth|ema)\w*vel
fast/slow accel disturbance observerdisturb · observer · \bdob\b
yaw + local pos integralsintegr · errsum|sumerr · \bki\b|\bki_
smooth ref filter(ref|target|setpoint)\w*(filt|smooth|lpf) · \bsp_?(filt|smooth|lpf) · (filt|smooth|lpf)\w*(ref|target|setpoint)
gate exit lockexit\w*lock|lock\w*exit · gate\w*(lock|latch)|(lock|latch)\w*gate · latch
predictive cross abortabort · predict\w*cross|cross\w*predict
ramp pad descentramp · descen\w*rate

Recipe agent 0's controller (2,698 characters) as graded, the lines for its velocity filter, its integrals and its abort:

fv.x += 0.4 * (obs.vel.x - fv.x); fv.y += 0.4 * (obs.vel.y - fv.y); fv.z += 0.4 * (obs.vel.z - fv.z);
// abort if lateral error too large near the gate plane
if (p.x > gate.x - 0.3 && p.x < gate.x && (Math.abs(p.z - gate.z) > 0.1 || Math.abs(p.y - gate.y) > 0.13)) {
if (Math.abs(ex) < 0.4) ix = clamp(ix + ex * 0.01, -0.6, 0.6);
if (Math.abs(ez) < 0.4) iz = clamp(iz + ez * 0.01, -0.6, 0.6);

fv is a filtered velocity, ix and iz are integrals, and the if under the comment sends the drone back to line up again. None of those names matches a pattern, and the one word that does, "abort", is in the comment.

7In their own words

Quoted exactly from each agent's answer to the question asked after its session, with the agent's score. The agents' answers are the harness's record of what the model said, outside plimsoll and unsigned; each controller's SHA-256 ties its transcript to its signed grading runs.

From the honest recipe

The change that made the difference was my own, not the recipe's. The baseline's descent ramp barely moved the drone. Replacing it with a velocity-commanded descent fixed the flights that timed out sitting over the pad.easy.recipe.2, 59 of 64 fresh flights

I can't say the recipe's parts would have helped or hurt, because I didn't try them. The result is a lightly modified baseline, not an implementation of the recipe.easy.recipe.14, 60 of 64 fresh flights

I should have said up front that I was dropping parts of the recipe rather than only listing them afterward.easy.recipe.6, 54 of 64 fresh flights

From the scrambled recipe

I only followed the recipe loosely. Its terms were cryptic, so I started from the baseline waypoint follower and iterated on the failures I saw.easy.scrambled.8, 46 of 64 fresh flights

No, I mostly didn't follow the recipe. I took the baseline stabilizer and added one thing: an integral term on x and z position error.easy.scrambled.6, 45 of 64 fresh flights

Most of my gains came from fixing my own mistakes, not from the recipe.easy.scrambled.9, 60 of 64 fresh flights

With no recipe

Landing: the altitude integrator kept growing during descent, so the drone climbed instead of landing. Freezing it during the landing phase fixed this. It was the biggest single fix.easy.none.5, 59 of 64 fresh flights

I chose to patch the baseline rather than redesign it, because I had no simulator and could only see pass/fail results from fly.easy.none.6, 50 of 64 fresh flights

I did not use the advance knowledge of gusts at all.easy.none.4, 54 of 64 fresh flights

Every agent, and every answer

All 40, in the order they were built. Open one to read its whole answer, exactly as it came back (the model writes markdown; its bold and lists are kept, nothing else).

8What this does not show

9On plimsoll, signed

Each stage is one run of the same job, with its own signed bundle. Every grading run held exactly its share of the practice set or the sealed fresh set, and a leak scan read every call into an agent's sandbox for the practice and fresh seeds and the simulator's files, and every practice run for the seeds.

Stage baseline: no recipe, 8 agents, 12:31 to 12:42 UTC
327 lines signed
by the key e3a616d81578a635…. 80 grading runs (median 8.6 s, longest 10.4 s), 640 graded flights, 0 ended by the CPU budget. 176 calls into agents' sandboxes and 61 practice runs scanned for 83 forbidden strings: none found. Every grading run held exactly its sets: yes.
Stage recipe: honest recipe and scrambled recipe, 32 agents, 12:46 to 13:22 UTC
1,058 lines signed
by the key 544dd61f42445ee7…. 320 grading runs (median 8.5 s, longest 12.2 s), 2,560 graded flights, 0 ended by the CPU budget. 536 calls into agents' sandboxes and 167 practice runs scanned for 83 forbidden strings: none found. Every grading run held exactly its sets: yes.

The design. Both stages record the SHA-256 of the design they were made under, 21e2f11c9c62cab0…: the job, the system prompt, each group's brief and after-session question, the tools, the budgets, the request settings, the scrambled recipe and the code checks. This page's program writes that same design from the public job file and checks the digest, so every prompt the agents were given can be rebuilt from the repository:

go run ./examples/squeeze -job examples/squeeze/jobs/recipe-scramble-easy.json -api messages -model claude-sonnet-5-5 -design design.json

The cost. The 40 agents' 478 model calls cost $6.24 at the list prices checked on 2026-11-07 (a million tokens: $2 of prompt read fresh, $2.50 written to the prompt cache, $0.10 read from it, $10 written by the model; EXTERNAL · official docs ↗), computed from the token counts each stage's run.json records. A 2-call check before the run that the prompt cache worked brings it to $6.25. 92% of the prompt tokens were read from the prompt cache (the model's provider keeps the start of a long conversation it has already read, and charges a twentieth of the fresh price to read it again), which is what kept the cost low.

A tier is the daemon's configuration and the provider's evidence plus its startup checks, never hardware attestation.

10Check it yourself

Beside this page: experiment.json (the stages in order, the registered rule, the list prices), evidence.json (every number this page states) and, under stages/baseline/ and stages/recipe/, each stage's run as it wrote it: bundle.jsonl.gz (every run, signed), harness.pub (the public half of the stage's signing key), fresh.json (the sealed set, revealed), run.json (the run's own unsigned account: times, token counts, each agent's turns; its model_base is the local relay that enforced the spending cap, not where the model ran), transcripts.jsonl.gz (every agent's whole conversation, its reasoning summaries and tool calls included), and the evidence.json and agents.json derived from them. From a clone of the repository:

go run ./examples/squeeze -job examples/squeeze/jobs/recipe-scramble-easy.json -staged -verify -out docs/examples/squeeze-recipe

That checks every signature in both bundles and every record against the request and answer stored with it, the leak scans, that every grading run held the sealed set whose SHA-256 the run committed to, that both stages were made under the design the program writes and flew the same set, that each agent's record follows from its transcript and its signed grading runs, and that evidence.json, this page and the method page are exactly what those files state: the intervals and the rule's outcome are computed again, not read. It starts no sandbox and calls no model. Change one byte of a bundle and it refuses.

Run it again. It needs docker with gVisor and the images make docker-images builds, an Anthropic API key in ANTHROPIC_API_KEY, and a token budget the program enforces. Seal a fresh set first, then run each stage on it; -paid is the program's consent to spend, and it refuses to start without it:

J=examples/squeeze/jobs/recipe-scramble-easy.json
go run ./examples/squeeze -job $J -seal fresh.json
go run ./examples/squeeze -job $J -api messages -model claude-sonnet-5-5 -paid -max-tokens 90000000 \
  -fresh fresh.json -fresh-sha256 <its SHA-256> -groups none -out baseline
go run ./examples/squeeze -job $J -api messages -model claude-sonnet-5-5 -paid -max-tokens 90000000 \
  -fresh fresh.json -fresh-sha256 <its SHA-256> -groups recipe,scrambled -build 16 -out recipe

Words used on this page

Model
An AI program, trained on huge piles of examples, that writes answers, text or code.
Model id
The exact version name of a model that a program asks for, so the same one can be asked for again.
Agent
A model working in a loop on its own: write, run, look, try again.
Sandbox
The walled-off place where untrusted code runs.
Harness
The program that hands out each task, runs it and grades it. Here, examples/squeeze.
Controller
The program that flies the drone: called 100 times a simulated second with sensor readings, it answers with thrust, tilt and turn.
Baseline
The plain controller every agent started from, the same for every group.
Token
The unit a language model reads and writes in; a short word is usually one.
Recipe
An earlier AI's whole method on this course, in 32 tokens.
Scrambled recipe
The same words in a shuffled order: the control for "does the order matter?".
Control group
A group given everything except the thing being tested.
Seed
One number that fixes a random draw, so the same shuffle or the same flight can be made again exactly.
Practice flights
The 16 flights the earlier AI tuned itself on; graded here too, but not the measure.
Smoke flights
The 4 flights the course names for a quick check, open to every agent.
Fresh, or sealed, flights
The 64 flights no agent saw, drawn at random and fingerprinted before the run.
SHA-256
A 64-character fingerprint of some bytes: the same bytes always give the same one, a changed byte a different one.
Points
Percentage points of the fresh flights; one flight of 64 is about 1.6.
Noise
The part of a score that is luck; a difference "within noise" is one luck alone could have made.
Resampling
Drawing a group's agents again at random, with repeats, many times, to see how far a difference could move.
95% and 90% intervals
The ranges that hold the middle 95% and 90% of those resampled differences.
Registered rule
The rule for reading the result, fixed before the run.
Messages API
The way a program sends the model a request over the internet and reads its answer.
Prompt cache
The provider's copy of the start of a conversation it has already read, charged at a twentieth of the fresh price.
Bundle
One file holding every run's signed record for a stage, one line a record.
Run record
plimsoll's signed statement of one run: digests of what went in and what came out, where it ran, and when (INTERNAL · glossary →).

Sources. The squeeze test and its drone course (INTERNAL · example →). The hackathon entry the course and the recipe come from, did-it-learn (EXTERNAL · source repo ↗). The harness, examples/squeeze (EXTERNAL · source repo ↗). The model's list prices (EXTERNAL · official docs ↗). The shuffled-recipe test: the suggestion of a reader who builds AI red-team agents. The method, the statistics and related work: INTERNAL · method page →.

This page is written by examples/squeeze -staged from the files beside it; -verify checks that it still is.