Method, statistics and related work

The results page (INTERNAL · results page →) is written for a reader who knows no AI. This page is for one who does: the design as preregistered, the statistics and their limits at these group sizes, the threats to validity, and where the question sits in the published literature. Every number here is computed from the same files as the results page by the same program, and -verify refuses either page if a number drifts.

1. The design
2. The statistics, and their limits
3. An exploratory check: exact permutation tests
4. Threats to validity
5. Related work
6. Small preregistered experiments: what the literature asks for
7. Every agent, in numbers
8. Files and verification

1The design

Question. Does a fresh coding agent use the plan it is handed? The treatment is a 32-token recipe distilled by a model from an earlier agent's full record of learning this course (the squeeze test's "honest" recipe). The primary control is the same recipe with its words shuffled: lexical content held fixed, structure removed. A second control is no recipe. The job file (examples/squeeze/jobs/recipe-scramble-easy.json) states it this way:

Does a fresh agent follow the plan it is handed, or ignore it? Fresh agents on the easy course: from the hackathon's honest 32-token recipe, from the same recipe with its words shuffled by a fixed seed (the same words, no plan), and from no recipe. Run in two stages: the eight no-recipe agents first; then the recipe and scrambled groups, equal in size, up to the 16 each listed here, sized (-build) from the first stage's measured cost alone. Every agent has the same tools and the same practice budget (fly calls are limited, and the reason is cost); each controller is graded in sandboxes of its own on the loop's 16 practice flights and on 64 fresh flights, every run signed. Scores come from the signed grading runs only; agents.json reads each agent's recipe use, its controller's code and its answer to the question asked after.

Arms and sizes

The no-recipe arm ran first (8 agents, 12:31 to 12:42 UTC on 2026-11-07) as a ceiling check: had its mean landings reached 90% of the fresh set, the rule would not have read landings, since a course that easy leaves no headroom for a plan to show. It landed 85.5%, so the two recipe arms ran (16 and 16 agents, 12:46 to 13:22 UTC, 4 minutes after the first stage ended), interleaved: rebuilds were started in turn across the two groups, so neither group ran wholly before the other. The recipe arms' size was fixed by the first stage's measured cost alone (the job's -build), never by its score; the job lists 16 as the most each may have, and both reached it.

What was fixed before the first agent ran

• The sealed set. 64 flights drawn at random (fresh.json, a nonce and 64 32-bit seeds), committed to by SHA-256 d81c7b18c6cf2ade2d4cbab4178a4957b2a0fc78a23ee59db7c16ce5d230819b. Both stages flew exactly this set; -verify recomputes each grading run's part of it from the set and refuses a run holding anything else.
• The design document, SHA-256 21e2f11c9c62cab0ce7e63f8b567a2e663f56e3bd9a3c4ef053c02959210f2e5: the job, the system prompt, each arm's brief and after-session question, the tool definitions, the budgets, the request settings, the scrambled recipe and the code checks. Each stage's run.json records this digest; the program recomputes it from the public job file.
• The registered rule (experiment.json): two measures per agent, three pairwise differences of means with 95% and 90% percentile bootstrap intervals from resampling agents within arm (10,000 draws, seed 20261106, CPython's random.Random, reproduced in Go), a ceiling rule on the untreated arm (90%), equivalence margins (10% of the fresh set on landings, 1 item on words), and four outcomes tried in order. The date the rule was written is recorded privately, not on these pages; a reader can check the rule's words and its application, not its timestamp.
• The scramble. The honest recipe's whitespace-separated words, punctuation attached, Fisher-Yates shuffled by a splitmix64 generator from seed 20261106, written out in job.go so no library's generator is involved. One shuffle, the same for every scrambled agent.

The two measures, and the one thing asked

• landed: fresh flights landed, of 64: the agent's final controller graded in sandboxes of its own on the 64 sealed fresh flights, read from the signed grading runs.
• words: recipe words named in code: how many of the recipe's 7 items have a pattern that matches a name (an identifier) in the agent's final controller, comments removed, that the shared baseline controller does not already hold; a measure of the recipe's words taken up, not of what the code does.
• Asked, not measured. After the last turn, with no tool available, each agent was asked whether and how it followed its recipe (no-recipe agents: what approach it took and why). The answer is recorded and quoted; it enters no score. Separating the agent's account from the measured behavior was a reader's suggestion and is the point of the comparison in the results page's verdict "What the agents say they used, their code does not name".

The four outcomes, in the order the rule tries them

• ORDER SHOWEDthis run
  recipe minus scrambled has its 95% interval above zero on recipe words named in code, or on landings when they are read: the recipe in its order moved the words the code names, or the landings, beyond the same words shuffled. It does not show the code does what the recipe says.
• WORDS ONLY
  not that, but recipe minus none and scrambled minus none both have their 95% intervals above zero on recipe words named in code: the agents took up the words, and the order they came in did not show.
• NO RECIPE EFFECT SEEN
  recipe minus scrambled has its 90% interval inside the margin (1 item of words; 10 points of landings, when read), and so does recipe minus none on words: the recipe moved nothing these two measures can see.
• INCONCLUSIVE
  anything else.

Equal affordances

Every arm had the same model (claude-sonnet-5-5, Messages API, effort medium, at most 16,000 output tokens a turn, a 150,000-token prompt cap), the same system prompt, the same baseline controller, the same three tools, and the same budget: 30 turns and 20 fly calls of up to 8 new random flights, plus the course's 4 smoke flights. The simulator was never in an agent's sandbox: fly ran the agent's controller.js in a separate sandbox and returned per-flight results only, so an agent could practice but not read the generator or the sealed seeds. The brief's tail for each arm:

A builder before you worked on exactly this task for a long time. Everything it learned was compressed into this recipe, which is all you get from it:

    Filter velocity; fast/slow accel disturbance observer; yaw + local pos integrals; smooth ref filter; gate exit lock; predictive cross abort; ramp pad descent.

Build controller.js following the recipe. You may fly the smoke flights and any new flights you choose; you will not see the flights you are graded on. Leave your best version in controller.js, and end your reply with one line: SMOKE=<passed>/4.

Build the best one you can in controller.js. You may fly the smoke flights and any new flights you choose; you will not see the flights you are graded on. Leave your best version in controller.js, and end your reply with one line: SMOKE=<passed>/4.

The system prompt, the same for every agent:

You are building a flight controller in a Linux sandbox. You have three tools. write_file writes a file into your work directory. run_code runs JavaScript or Python in your sandbox; files persist between calls. fly flies controller.js from your work directory in a separate sandbox that holds the simulator and answers with each flight's result; it flies controller.js only as your last write_file of it left it, so write controller.js with write_file. Read API.md first. Build, fly, read the results, fix, and fly again.

2The statistics, and their limits

Unit of analysis. The agent. Each agent's score is the fraction of the 64 sealed flights its final controller landed; the 64 flights of one controller are not independent trials (they share the controller), so the rule resamples agents, not flights. With 16, 16 and 8 agents the effective sample is small, and everything below follows from that.

The intervals. Percentile bootstrap: for a pair, draw each arm's agents with replacement (10,000 times), recompute the difference of means, and take the 2.5th and 97.5th percentiles (95%) or the 5th and 95th (90%). Known limits at these sizes: the percentile interval tends to be too narrow in small samples, so its nominal 95% is optimistic (Hesterberg, below); the bootstrap distribution of a mean of 8 values has few distinct resamples; and landings are bounded above (several agents sit at 60 of 64), so the arm distributions are skewed toward the ceiling. The result page's reading "wholly above zero" is therefore a little less than a 95% claim, and the registered rule was written knowing a run this small could only show a large effect.

What the primary pair shows. Recipe minus scrambled on landings: +7.7 points, 95% interval +2.8 to +12.6, 90% interval +3.6 to +11.8. The interval's width, about 10 points, is the run's resolution: a true effect of a few points could not have shown. On recipe words named in code the pair's difference is 0 with interval 0 to 0: every agent in both arms scored 0, so the bootstrap has nothing to resample and the interval is degenerate, which is the "too little variety" failure of the bootstrap, not evidence of a tight estimate.

The secondary pairs. Recipe minus none +2.4 points (95% -1.5 to +6.4); scrambled minus none -5.3 points (95% -11.0 to +0.6; 90% -10.2 to -0.4). Both hold zero at 95%. The rule names them "others": they are read, not decided on, and the results page labels the reading that the scrambled words hurt more than the recipe helped as the author's, not the rule's.

Equivalence and the 90% interval. The outcome NO RECIPE EFFECT SEEN is a two one-sided tests reading: a difference whose 90% interval lies inside the margin (10% on landings, 1 item on words) is declared too small to matter. The 90% interval is the standard choice for a 5% equivalence test. It did not fire here.

Multiplicity. Three pairs and two measures give six intervals, but the rule orders its outcomes so that one comparison decides ORDER SHOWED (the primary pair, on words, or on landings when read), and the ceiling rule chose the measure before any interval was computed. No correction is applied to the secondary pairs, and none is claimed; they are descriptive.

Why the per-arm means carry no interval. An earlier draft showed each arm's mean with its own bootstrap interval. They were removed: the rule puts its intervals on differences, a lone arm's interval invites the reader to compare intervals by eye (which is not a test of the difference), and with 8 agents such an interval is especially unreliable.

3An exploratory check: exact permutation tests

EXPLORATORY The registered rule used bootstrap intervals and nothing else. The tests below were not registered and decide nothing; they are here because a reader who knows the percentile bootstrap's small-sample habits will ask what a method with no resampling assumption says about the same data. They are reported in full so that they can be read as a check on the registered reading, not as a second result.

For each pair, the exact permutation (shuffle) test of the difference in mean fresh flights landed: of every way to deal the pooled agents of the two arms into two groups of the observed sizes, the share whose difference of means is at least as far from zero as the observed one (two-sided). Scores are whole flights of the same total, so the count is exact, by a dynamic program over subset size and summed landings; nothing is sampled. The null hypothesis is exchangeability: that the arm labels carry no information.

Read beside the registered result: the primary pair's permutation p of 0.007 is on the same side of the conventional line as the bootstrap interval that excludes zero, so the registered reading does not rest on the bootstrap's small-sample optimism alone. The two secondary pairs' values (0.234 and 0.179) sit where their bootstrap intervals, which hold zero, put them. These numbers were computed after the result was known, from the published scores, and should be weighed as such.

4Threats to validity

• The control cannot separate "followed the plan's content" from "garbled text cost something". The scrambled recipe is nonsense an agent must still read and decide about. The honest order's win is consistent with agents using the plan, and equally with a coherent note costing nothing while an incoherent one costs time or seeds a worse first idea. The natural follow-up is a coherent but wrong control: the harder course's recipe, the same seven items kept whole and reordered, or a plausible recipe for a different task. If the honest recipe still wins, the content mattered. That is a proposal, not something done here.
• Ceiling. The untreated arm landed 85.5%, under the 90% rule but close to it. Headroom for a plan to help was about 14 points; the test is more sensitive to a plan that hurts than to one that helps.
• The words measure is a lexical proxy. "Recipe words named in code" counts identifier patterns, comments removed, baseline identifiers excluded. It has false negatives (a filter named fv is a filter) and the registered outcomes say so. A measure of what the code does (unit tests on the controller's behavior, or an instrumented simulator) was not available; its absence is why the words measure decided nothing here and why the rule reads landings first when it may.
• Two stages, 4 minutes apart. The untreated arm ran before the two treated arms, with the same model id and sealed set. A hosted model can change between calls; host load differed; the grading's median wall time was 8.6 s in stage 1 and 8.5 s in stage 2. The primary pair ran within one stage, interleaved, and is not exposed to this; the secondary pairs are.
• The baseline does most of the work. Every agent started from a working controller. The agents' own accounts say they patched it and stopped when the smoke flights passed; a recipe competes with "the baseline already flies" for the agent's attention. A weaker baseline or a harder course would raise the value of a plan, at the cost of comparability with the squeeze test.
• Self-report is unsigned and post hoc. The after-session answers are the model's text, recorded by the harness outside plimsoll. They are tied to the graded controller by its SHA-256 but are not evidence of what the code does, and the literature below gives reasons to expect them to be unfaithful.
• One model, one course, one recipe, one shuffle. A single fixed shuffle means the scrambled arm's result is a property of that permutation, not of shuffling in general; a word order that happened to read almost sensibly, or one that put a misleading phrase first, would move the result. Nothing here generalizes to another model or a harder course.
• Practice budget. 20 fly calls of 8 flights is a cost-driven limit. Three of the 40 agents (easy.recipe.12, honest recipe, easy.scrambled.8, scrambled recipe, easy.scrambled.12, scrambled recipe) used every call; the other 37 used fewer. An agent that would have followed the plan given more iterations is indistinguishable here from one that would not.
• Sizes were set by cost. The treated arms' size came from stage 1's spend, not from a power analysis; the intervals above are the honest statement of what the run could resolve.

5Related work

Each source below was opened and its abstract read before it was cited; the one-line summaries are of the abstracts, and a reader should go to the paper for the detail. The question this experiment asks sits at the meeting of four literatures.

Do agents execute the plans they declare or are given?

• Oota, Herrera, Cabot Sagrera, López de Prado and Khan (2026), "Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution" (EXTERNAL · paper ↗). The abstract reports that under a generic plan-then-act executor only 22% to 45% of trajectories across three benchmarks preserved the planning structure the model had declared, and that executors built to enforce a pattern do enforce it ("pattern-specific executors enforce the intended structure"). The closest published cousin of this experiment's question: here the plan was handed to the agent rather than declared by it, and the agents' accounts of partial use are the same gap between stated and executed plan.
• Valmeekam, Marquez, Olmo, Sreedharan and Kambhampati (2022, revised 2023), "PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change" (EXTERNAL · paper ↗). A benchmark drawn from automated-planning domains on which, per its abstract, "LLM performance falls quite short" at plan generation. Relevant as the other side of the coin: this experiment did not ask the agents to plan, it handed them one, and asked whether it was used.
• Zhou, Lu, Mishra, Brahma, Basu, Luan, Zhou and Hou (2023), "Instruction-Following Evaluation for Large Language Models" (EXTERNAL · paper ↗). IFEval scores instruction following with "verifiable instructions", checked by code rather than by a judge. The principle this experiment borrows: whether a plan was followed is read from the artifact by code, never from the model's own account.

Is the model's account of what it used faithful?

• Turpin, Michael, Perez and Bowman (2023), "Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting" (EXTERNAL · paper ↗). Biasing features in a prompt (such as the order of answer options) shift a model's answers while its step-by-step explanation never mentions them; explanations can "systematically misrepresent the true reason for a model's prediction". The direct reason this page treats the agents' after-session answers as text to be quoted, not as evidence.
• Lanham and others (2023), "Measuring Faithfulness in Chain-of-Thought Reasoning" (EXTERNAL · paper ↗). Perturbs a model's stated reasoning and measures whether its answer moves; finds reliance on stated reasoning varies widely by task and that larger models tend to produce less faithful reasoning on most tasks.
• Chen and others (2025), "Reasoning Models Don't Always Say What They Think" (EXTERNAL · paper ↗). Inserts hints into prompts and checks whether the model's reasoning acknowledges using a hint it used; the abstract reports reveal rates "often below 20%". The nearest published analogue to "did the agent say it used the recipe": here every honest-recipe agent volunteered partial use, which is the opposite failure and still not a measurement.

How much does word order matter to a language model?

• Sinha, Jia, Hupkes, Pineau, Williams and Kiela (2021), "Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little" (EXTERNAL · paper ↗). Models pretrained on word-shuffled text still do well on many downstream tasks, which the authors attribute to "higher-order word co-occurrence statistics". A reason to have expected the scrambled recipe to carry much of the honest one's value: the words alone carry a lot.
• Pham, Bui, Mai and Nguyen (2020, revised 2021), "Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?" (EXTERNAL · paper ↗). Shuffling input words often leaves a classifier's predictions unchanged on GLUE tasks; the models lean on "superficial cues". Same expectation, from the input side. Against both, the agents here did worse with the shuffled recipe, which is why the garbled-text explanation above has to be taken seriously: an agent is not a classifier, and it must act on what it reads.
• Lu, Bartolo, Moore, Riedel and Stenetorp (2021; ACL 2022), "Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity" (EXTERNAL · paper ↗). The order of a prompt's examples can swing a model between near state-of-the-art and near chance. Order sensitivity at the prompt level, with the same content, is established; this experiment is a small instance of it inside an agent loop.
• Sclar, Choi, Tsvetkov and Suhr (2023; ICLR 2024), "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design" (EXTERNAL · paper ↗). Models are "extremely sensitive to subtle changes in prompt formatting"; the authors argue for reporting a range over plausible formats rather than one number. A single fixed shuffle, as here, is one point of such a range.

Where the design's choices came from

Three suggestions from a reader who builds AI red-team agents shaped the design: score by what changed in the sandbox, never by what the agent says it did (so landings come from signed grading runs, the words measure from the code, and the agents' accounts are kept apart); scramble the plan and see whether the agent's actions change (the primary comparison); and ask whether hiding the simulator would send agents hunting for another shortcut (the simulator never entered an agent's sandbox, and each controller was checked for a random number generator and for the simulator's own generator constant: 0, 0 and 0 controllers carried a generator and 0, 0 and 0 the constant, by arm).

6Small preregistered experiments: what the literature asks for

• Nosek, Ebersole, DeHaven and Mellor (2018), "The preregistration revolution", PNAS (EXTERNAL · paper ↗). Preregistration separates generating a hypothesis from testing it, against "hindsight bias" and the temptation to read a result in whatever way fits. This experiment's rule, measures, pairs, resampling and margins were fixed in experiment.json before the first agent ran; its exploratory section is labeled as such; what it could not do, prove its own timestamp to a reader, is stated.
• Miller (2024), "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations" (EXTERNAL · paper ↗). Treats a model evaluation as an experiment: report intervals, choose the unit of analysis, compare models with paired designs where possible, and plan the run to reduce noise. Here the unit is the agent and the comparisons are unpaired across arms, which is the main source of width in the intervals.
• Agarwal, Schwarzer, Castro, Courville and Bellemare (2021), "Deep Reinforcement Learning at the Edge of the Statistical Precipice" (EXTERNAL · paper ↗). With few runs, point estimates can mislead and "reliable evaluation in the few run deep RL regime cannot ignore the uncertainty in results"; the authors recommend interval estimates and robust aggregates. The few-run regime is exactly where 16, 16 and 8 agents sit.
• Hesterberg (2014), "What Teachers Should Know about the Bootstrap: Resampling in the Undergraduate Statistics Curriculum" (EXTERNAL · paper ↗). The percentile bootstrap interval "badly under-covers in small samples"; better intervals exist. The reason this page states the intervals' optimism and adds the exploratory permutation check rather than leaning on the percentile interval alone. A next run would do well to register a bootstrap-t or BCa interval, or an exact test, in the rule itself.

7Every agent, in numbers

From evidence.json: each agent's fresh and practice landings from its signed grading runs, and its session's turns and fly calls from the harness's record. Sorted as built.

8Files and verification

Beside this page: experiment.json (stages in order, the registered rule, list prices), evidence.json (every number both pages state, including each agent's per-flight bits), and under stages/baseline/ and stages/recipe/ each stage's bundle.jsonl.gz, harness.pub, fresh.json, run.json, transcripts.jsonl.gz, evidence.json and agents.json. run.json's model_base (http://127.0.0.1:37749 and http://127.0.0.1:42949, one a stage) is a local relay on the experimenter's machine that enforced the spending cap and forwarded each call to the model's provider; the model itself ran on its maker's servers. From a clone of the repository:

go run ./examples/squeeze -job examples/squeeze/jobs/recipe-scramble-easy.json -staged -verify -out docs/examples/squeeze-recipe

That checks every signature in both bundles, every record against its stored request and answer, the leak scans, that every grading run held the sealed set the run committed to, that both stages were made under the design the program writes, that each agent's record follows from its transcript and signed grading runs (the transcript's controller text must hash to the graded controller; its system and user prompts must be the design's for its arm), and that evidence.json, the results page and this page are byte for byte what those files render. The bootstrap intervals, the rule's outcome and the permutation counts are computed again, not read. It starts no sandbox and calls no model.

The cost of the run, $6.25 over 478 model calls at list prices, is stated on the results page with its breakdown.