The spec said the export should include archived records. It shouldn't have. The agent implemented it exactly, wrote tests for it, and the tests passed — because they were tests of the spec. Review approved it, because the diff did what the ticket said. It shipped, and three weeks later somebody in support worked out why customers were seeing data they had asked to have deleted.
Nothing failed. Every detector we own was green, and all of them were pointed at the wrong rung.
A warning, before the argument rather than after it: I am the author of the method described here, and of the document that graded it. Nothing below has been reviewed by anyone with an interest in its being wrong. The specific consequence is stated at the end, and one claim in this piece is deliberately weaker than I would like to make it because of this paragraph.
The whole picture, quickly
Work descends a ladder: methodology → method → spec → task → code. How you decide how to work, the procedure you actually use, what the thing should be, what to do next, and the doing.
A bad line costs an hour. A bad task costs a day. A bad spec costs a quarter and everything built on it. A bad methodology costs years, and the people who paid usually never find out that was what happened.
Now price the feedback. At the bottom it is instant, mechanical and nearly free: a type error, an exit code. By the top, the feedback loop on was this methodology right is a career, and you get one sample from it. Cost of verification runs one way along the ladder; consequence runs the other.
So the honest description of our situation is not that engineers are unscientific. It is that we automated verification exactly where it was already cheapest, and left it absent exactly where it costs the most. Every detector most teams own — types, tests, lint, CI, alerts — lives on the bottom two rungs.
An objection arrives immediately, and it is correct: most teams' bottom-rung verification is theatre. Quite so — and notice the shape of it. At the bottom, verification is cheap enough that its absence is a choice rather than a price, which is exactly why the fakes there are so brazen. Nobody had to build them, so nobody checks them. At the top there is nothing to fake, because there was never anything to build.
That was defensible while writing code was the expensive part. It isn't any more. Generation got cheap, the bottom of the ladder got faster, and the binding constraint moved up. An agent will implement a wrong spec faster, more completely, and with better test coverage than any human ever could. A wrong spec now produces immaculately verified wrong software, and produces it in an afternoon.
This piece is about the spec rung, because that is where the pain is arriving and where you can do something about it on Monday. But the argument is the same all the way up.
Why the top of the ladder was never mapped
Not bad faith. Absence of a closed output.
A line of code returns something you can compare. Run it, get a value, diff it. A spec returns prose. A method returns a way of working. A methodology returns an opinion. When the only instrument available is a person who has seen a lot of projects, the person is the instrument — and seniority becomes a reasonable proxy for calibration, because nothing better was on offer.
That is what "best practice" means. It is what a field says when it has no denominator.
Bacon complained in 1620 that the mind left to itself finds what it already believes, and asked for weights on the intellect rather than wings. (I am working from the standard rendering and have not checked the Latin.) Whatever became of his programme elsewhere, it never arrived at the top of our ladder, for a reason that is entirely economic: to settle one question about one team's way of working, you needed two readers who had never spoken, a control run with the load-bearing input withheld, and three more moves of that kind. Person-weeks, per question. Nobody was ever going to pay that to find out whether their spec template was any good.
Old maps wrote here be dragons over the water nobody had sounded. What fills unmapped territory is monsters and experts, and neither of them is fake. Some of those dragons are real. The point of taking soundings is not that the experts are wrong — it is finding out which dragons are on your coast.
What DORA actually did
One programme sailed out and did it properly, and it deserves precision rather than a wave.
DORA measured engineering practice instead of asserting it: validated survey instruments, latent constructs tested for reliability and discriminant validity, results published with their limits attached. Nearly every argument in this piece stands on somebody having established first that these questions are answerable at all.
And then they handed the instrument over. The culture items are printed in the book. There is a five-question self-check built for a single team. The text says plainly: "teams can quickly and easily measure their organizational culture", and "By surveying everyone on your team, you can help avoid problems associated with having a few overly positive or overly negative responses." Their current guidance warns against using the continental map as a local ladder — against competing between teams, against comparing applications that share nothing. Worth noting honestly that this is a shift: the 2019 report sold cluster analysis as a way to "benchmark against the industry." They moved, and they moved in the right direction.
They drew the atlas and handed out the compass.
What nobody could hand over was the ability to check whether your way of working does anything — running your own procedure, withholding pieces of it, comparing the outputs enough times to know what the floor is. At the old prices that never happened. That the price is why it never happened is my inference, not something the field has published about itself; I could not find anyone who has tested it.
That is the part that changed. The atlas exists, the compass was distributed years ago, and the survey is now an afternoon.
Two published positions disagree with that sentence, and a piece about not grading your own homework had better say so.
The first is DORA's own current guidance, which warns that building integrations to get precise local delivery data "might not be worth the initial investment," and still prefers surveys over telemetry even for measuring a single team. The second is harder: software-engineering researchers have measured the cost of LLM-based studies and got the opposite answer to mine — budgets running to thousands of dollars, annual spend from a few hundred to tens of thousands, and artifact-availability rates roughly half those of everything else at the same conference.
My reconciliation, offered as a claim and not a proof: both of those price a model as the subject of a published study at population scale. This piece prices a model as the apparatus on one repository, where the ground truth is already on disk and nothing has to generalise. Different unit, different bill. If you are doing the former, none of what follows applies to you.
What actually got cheap
The tests are in the next section, and the striking thing about all of them is how little programming they involve: every one is a pipeline that eats text and emits structured text.
Note precisely what changed, because it is not what it looks like. A spec did not become a closed output — it is still prose. What became mechanical is the comparison. Reading two specs against each other clause by clause, and reporting where they differ, used to be a person's afternoon; it is now a procedure you can run three times before lunch. Closure was never a property of the artifact. It was a property of the operator you could afford to apply to it, and that is the thing that got cheap.
With a boundary worth knowing. METR's RE-Bench measured agents against 61 human experts on open-ended research engineering: the best agents scored about four times the humans at a two-hour budget, humans narrowly overtook them at eight hours, and humans scored twice the agents at thirty-two. The cost curve collapsed at the short-horizon, closed-output end and nowhere else. Every test in the next section is short-horizon and closed-output.
Build a method. Then test the method.
You do not have to adopt anybody's methodology. You have to write down how your team actually writes a spec, and then find out whether that way of writing works.
Write it down first. Not a document — four lines. Ours, for a while, was: a spec names the user-visible change, the data it touches, what must not change, and how we will know it worked. If yours is three bullets on a whiteboard, that is a method. The moment it is written down it is testable. While it lives in people's heads it is not.
Now test it the way you would test code. Take a brief you are about to build from. I will use the one from the top of this piece: "Let users export their account data."
Test 1 — write it three times, blind. Same brief, three independent runs, no shared context. Then read the three side by side, clause by clause.
Run A: "The export includes every record the user can see in the app."
Run B: "The export includes the user's active records."
Run C: "The export includes all records not permanently deleted."
Three answers, all reasonable, all different, and the whole difference is one word nobody defined: archived. A human reviewer reads straight past that, because a human reviewer supplies the missing definition silently and never notices having done it. That is precisely the defect that shipped in the story at the top of this piece — findable on a Tuesday, for the price of writing the brief out twice more.
Two rules make this test work rather than flatter you. Compare clause by clause, never by counting requirements: three documents can each carry fourteen requirements and agree on none of them. And where they disagree, adjudicate — do not merge. You cannot use agreement to manufacture the answer and then cite agreement as evidence the answer is good.
Of everything here, this test has cost us the least and killed the most.
Test 2 — delete a section and see whether anything moves. Take the two pages of Background and Context at the front of the brief. Pull them out. Regenerate. Diff the requirement list.
If the requirements come back the same, those two pages are not reaching the output. They may still be worth keeping — someone has to onboard — but they are not doing the job the brief claims they do, and nobody had any way of knowing that on Friday. Make the comparison something that executes: count changed requirement IDs, diff the clause list. Not "it feels similar."
One warning, and it is the sharpest practical danger in this whole method: removing one thing only tells you about one thing. Two sections that say the same thing in different words will each test as decoration, and deleting both will break the spec. So before you delete anything, gather the candidates and re-run once with all of them out.
Test 3 — plant a contradiction and require a refusal. Put both of these in the brief:
The export must include archived records.
Data a user has asked us to delete must never leave the system.
Both read like policy. They cannot both hold. A spec process worth anything stops and names the conflict. A process that quietly picks a side and writes fluent prose over the top has just shown you exactly what it does with the contradictions you did not plant on purpose.
Read the result asymmetrically. Passing proves very little — you wrote the plant, and your own plants are always easier to spot than the real ones. Failing proves a great deal.
Test 4 — sound a channel you have already charted. Before you believe any number this apparatus gives you, hand it two briefs you already have firm opinions about: last quarter's spec that everyone thought was clean, and the one people still tell stories about. Run the procedure on both.
If it cannot separate those two, it cannot separate anything, and every number it has ever handed you was noise with a decimal point in it.
Test 5 — run the napkin. The cheapest possible replacement for the entire apparatus: one experienced person, one day, producing a plain traceability report. Score the napkin on the same substrate as the method and publish both numbers next to each other.
We did this at the end of three weeks of building a method. The napkin scored 3 of 13 as specified, and 7 of 13 steelmanned. Those two numbers disagree, and we had written down in advance that if they disagreed we would claim nothing. So we claim nothing.
It was allowed to lose. The napkin did not obviously lose. Almost nobody runs the cheap baseline, for a reason everyone understands and nobody says out loud: it might win.
Then apply the ladder to whatever you found. A gap you can delete outranks a gap you can design out, which outranks a check that catches it, which outranks a person who promises to remember. That part is not new and does not need re-deriving.
A note on the trap that ships in the box
The tempting move, once generation is cheap, is to run everything three times and trust whatever the runs agree on. Agreement feels exactly like evidence and carries almost none of it.
Our central instrument reported 0.905 agreement with a key node intact, and 0.905 with that node deleted — identical to three decimal places, in a condition that had lost fifty-five percent of its rows. Not weak signal. None. A grouping step elsewhere shipped with a merge condition that could never fire: three runs, 23 groups, zero merges, perfect agreement every time. The perfect agreement was the symptom. For weeks we also quoted 54 of 63 as though the floor were zero; when we finally ran the same procedure with the key text withheld, the controls agreed with each other 42 times out of 63. "54 against a floor of 42" is a much smaller and much more honest claim. And thirty-six claims rested on "three runs agreed" — re-checked, ten were orphaned outright, and the orphans lay in an unbroken line along the method's spine. (Our recount of that audit sums to 37, not 36. It is unreconciled, and a piece arguing that you should count things has no business quietly fixing its own count.)
It agreed with itself.
None of which is our discovery, and that is the most useful thing about it. Cohen's 1960 paper opens with two psychiatrists agreeing on 82% of cases purely by chance. Campbell and Fiske showed in 1959 that measures of different things by the same method routinely outscore measures of the same thing by different methods — which is why a panel of language models is one instrument sampled repeatedly, not many raters. The modern numbers say it louder: nine frontier judges drawn from seven model families supply about 2.2 independent votes' worth of information (Kohli, 2026), and error correlation rises with capability, across providers and architectures alike (Kim et al., 2025). Artstein and Poesio close it in a sentence: "achieving good agreement cannot ensure validity: Two observers of the same event may well share the same prejudice while still being objectively wrong."
Generation got cheap. The apparatus got cheap. Verification did not.
Running a check costs nothing now. Knowing which check can discriminate costs exactly what it always did.
What this buys you, and what it never will
Set the expectation correctly, on purpose, before you start.
It will never tell you your practices are good. A single team's favourable verdict on its own practice is unavailable at any level of rigour. That is not a gap in the tooling; it is what "one team" means. Anyone selling you the favourable version out of a single codebase is selling what they have always been selling.
What it will tell you is what is dead. Refutation is cheap. Endorsement is not, and never will be. And the useful thing about dead machinery is that everybody recognises it the moment somebody names it out loud:
- The gate that has never once bounced anything. Not rarely — never, since the day it went in.
- The test command that runs, prints something reassuring, and exits 0 without having asserted anything.
- The check wired so that it only fires in a situation your team never actually produces, so an ordinary push gets no pass, no fail, nothing at all.
- The step whose removal changes nothing downstream that anyone can name.
- The metric that returns the same value for every input you can hand it, including the one it was built to replace.
- The absence wearing the costume of a finding. "63 of 63 items have nothing watching them" reads like a discovery about a product. It is a restatement of nobody wrote an objective down anywhere in this input — and two separate steps counted that one absence separately, which downstream read as two measurements corroborating each other.
Notice what every one of those is: an absence, not a percentage. That distinction is worth carrying, because it survived contact with our own audit and the percentages did not. A figure computed against a field most rows never carried. An overlap that read 41.2%, or about 20%, or 29.4%, depending on a denominator nobody had ever stated. A subtotal printed as a total, above a table with nine more rows in it.
It was allowed to lose. That is the only reason any of it is worth anything, and it is the filter for deciding which cheap checks to run at all: run the ones with a column in which your favourite can come out badly.
Four rules, if you are going to survey
- Count effective votes, not agents. Agreement inside one model family is one sounding, however many times you take it. If you need independence, you need a genuinely different method — a person, an execution, a ground truth — not another run.
- No number without its floor. Run the degenerate version on purpose. And know what beating the floor proves: that the procedure used something in the input. Not that it used the right thing.
- Re-run controls per batch. The rater changes underneath you between Tuesday and Thursday. A control run once, at design time, certifies nothing about next month.
- Prefer answers that execute. A diff, a count, an exit code, over any answer that is an agent's judgement. This is the ordinary detector ladder, turned on your own instruments — and it is the move the older sciences could never make, because their ground truth was not sitting on disk. Ours is.
Before the conclusions
The document underneath this piece was supposed to carry the warning at the top of this one. It didn't, for a day — and a day was enough for it to be cited as grounds for restructuring the very thing it graded.
So the honest downgrade: the sharpest claims in this piece are the ones I did not derive. The agreement failure is documented by people with no stake in our work. The floor baseline's limits, the ablation confounds, the transportability result — all published, all older than the project. What is ours is the assembly, not the instruments: running this apparatus at n=1, on a working codebase, as ordinary engineering rather than as a study. Whether that assembly is worth anything is exactly the kind of favourable single-team verdict this piece says you cannot have, so I am not going to claim it. It is the claim I would most like to make, and the warning at the top is why I am not making it.
One number on ourselves, and it is a worse number than it looks. In one audit, three checkable claims turned out to be false — among them a tag we reported missing that had been correct for months, and a contradiction between two requirements that turned out not to contradict. The denominator we published was "roughly forty," which is a bound written as prose, which our own first rule forbids. Nobody counted. So: three claims false out of an uncounted number, which is not a rate — and we only know even that because somebody re-ran every claim against its source before sending. An error rate you have not measured is not low. It is unknown, and every corpus that has never measured its own is quietly assuming zero.
The whole three weeks produced a method that is not usable in the form we built it. Everything still standing came from the machinery built to check the work, not from the work.
Your coast, not the continent
The atlas is real and it was drawn honestly. It is also a map of the continent, and your team is one coastline on it.
The reason you can now survey it yourself is the same reason your survey will never be an atlas: your soundings are valid for your water, which is exactly what you needed and precisely what you may not publish as a general truth. The continental map doesn't answer your question. Your soundings don't overturn the continental map.
For the first time the instruments are free and the ship is in the harbour. What still costs is the judgement about where to sail — and one habit: writing down, before you look, the depth that would make you turn back.
There is a serious objection to that habit, and it deserves an answer rather than silence. An economist has argued that pre-registration stops working once experiments become nearly free: a registry is incentive-compatible only because running the experiment is expensive, and when a run costs a dollar and thirty seconds, nothing stops you running fifty more.
He is right, about registries. But two instruments have been sharing one word. A registry defends against selective reporting across many runs, and that defence genuinely does lean on runs being costly. Writing down the disconfirming outcome defends against something else — retelling whatever came back as the thing you expected — and that has never depended on cost. Cheap runs make it worse, not better, because you get more outcomes to narrate. The distinction is drawn explicitly in the literature, between analyses that came from predictions and analyses that came from postdictions; and the sharpest work on it concerns the single-run case, where no fishing happened at all and the result is still not what it appears.
Note also that the proposed replacement — make everything push-button reproducible so anyone can re-run it — is a many-runs remedy. It leaves postdiction untouched. Reproducible and retold-as-expected is a perfectly available state, and it is the one our corpus was in.
Where the map says dragons, take a sounding.
No comments yet