Why explaining the work finds what writing it hid
Part 4 of the series · ← Part 3: The Knowledge Base
Part 2 described AI-first work as sculpting: you and the agent shape a block of material over many sessions. Part 3 gave that work a shared knowledge base, so neither of you starts from zero again and what you learn accumulates as the base grows. This part is about what happens when the work has to be judged — when the sculpture leaves the studio and someone else has to look at it and say whether it is right.
For a genuinely small change — a rename, a guard, a one-file fix — a diff is the perfect review artifact. It is precise, it is complete, and the reviewer holds the whole thing in their head. Above that, it starts losing ground, and the threshold is lower than most teams assume. A medium feature touching five files across two subsystems already carries trade-offs the lines cannot show. At the far end — a rewrite of ninety-six production files across twelve subsystems — the diff is something nobody reads at all. Not because reviewers are lazy, but because a diff answers exactly one question: what changed? The questions that decide whether the work is correct are different ones:
Every non-obvious line is the residue of a decision. The diff shows the residue and throws away the decision. The reviewer is left to reverse-engineer intent from syntax.
The alternative that was tried and abandoned is invisible. Reviewers re-propose it, and the author re-explains it — in a comment thread, weeks later, from memory.
A diff is uniformly flat. A hundred trivial renames and one subtle concurrency bug have the same visual weight and the same claim on attention.
Nothing in a diff distinguishes "this is verified against production data" from "this compiles and looks plausible."
Which makes the useful test not size but whether decisions were made. If a change carries reasoning a reviewer cannot reconstruct from the lines — a trade-off, a rejected alternative, an invariant that has to hold, a fix whose correctness depends on something two files away — it earns a retrospective, whether that is four files or four hundred. Only genuinely small, self-evident changes are better served by the diff alone. In practice that puts almost everything above a quick fix inside the net, which is the opposite of how these documents are usually treated: as a ceremony reserved for the one big rewrite a team does every few years.
So we started writing the answers down, as a document that renders the code and the reasoning together: whole classes with syntax highlighting, and inline annotation bubbles in the style of a code-review comment, attached to the exact lines they talk about. Static, self-contained HTML — open it in a browser, no tooling, no server, no account.
The first version was built from small snippets of code with commentary around them. It was rejected outright, and the correction mattered more than anything else in the design: whole classes, or it does not work. Without the surrounding code, an annotation is an assertion the reader has to take on faith. With it, the reader can check the claim against the very lines it describes. That single constraint is what separates a document reviewers trust from a document they skim.
We built the retrospective for human reviewers. It turned out to have a second reader.
For the human, the gain is straightforward. The document puts the important implemented points in front of them with the reasoning attached: why it was built this way, what was fixed, how it was verified, and, where it matters, an explicit marker saying this is the spot to concentrate on, and this is the question to settle. That inverts the economics of review. Instead of spending attention finding the places that deserve it, they arrive with the map drawn and spend it on judgement: the one part a reviewer can do and an author cannot.
Here is the part we did not design and would not have predicted. Producing a retrospective forces the agent, or the workflow driving several of them, to look at its own work from a completely different angle. And at that angle, it sees defects it could not see while writing the code. Not only bugs: design mistakes, invariants that were never actually enforced, contracts documented in a javadoc that the implementation quietly violates.
This is not a hypothetical. Writing the retrospective for that pipeline rewrite surfaced three confirmed bugs in code that had been reviewed, tested, and was already running. The most serious was a silent data-loss path: an error classified as retryable was completed exceptionally, but the committer configuration in use committed the offset anyway, so the record was never delivered and never reached the dead-letter queue. Every piece was correct; the composition was not. The other two were quieter: a configuration parameter that was read, documented, and never actually used anywhere, and a category of image that ended up in the dead-letter queue, whereas the previous implementation had silently discarded it.
An obvious objection: any careful second pass might have found those, so the document is incidental. The honest answer is that a retrospective is not the only thing that can find them; what it does is make finding them structural rather than fortunate. A plain re-review has no forcing function and tends to stop where the code looks familiar. Explaining a subsystem end to end has a completion condition, and all three bugs lived where two subsystems met, which is the place a file-by-file pass has no reason to visit.
The mechanism, once named, can be used deliberately.
A prose explanation fails loudly when it is wrong. A method can compile, pass its tests, and be subtly incorrect. It is much harder to write the sentence "a failure here retries, so no record is lost" without noticing that you cannot finish it honestly, because tracing it leads to a committed offset and a vanished record.
The retrospective is not documentation produced after the review. It is an instrument that changes what the author — human or agent — is able to notice, and it does so before any reviewer opens it. The document is the deliverable; the altitude shift is the payload.
A retrospective that is only annotated code is a retrospective of the code. What a reviewer needs is a retrospective of the task, with the code as the last layer. That was the biggest gap in our first working version: the context existed, scattered across annotations on individual lines, where nobody could read it whole.
The first layer is the task overview: motivation and tickets, what is in scope and explicitly what is deliberately not, completion criteria and how each was verified, plus key decisions as a compact table — decision, rejected alternative, why, how reversible. The second is architecture: per chapter, the role of the subsystem, its inputs and outputs, the invariants the reviewer should police. The third is the annotated code itself, grouped into thematic chapters rather than one per file, each with a two-line summary and a note on test coverage. The fourth is findings: confirmed bugs on their own page, open runtime questions in one table with how to check each, and deferred fixes with an explicit trigger.
When a change is big enough to split, chapters are assigned by risk, not file count: one reviewer owns data correctness, another concurrency, a third the edge cases. A medium task is one chapter and one reviewer, and the four layers still apply.
Concretely: a file header with its summary, then source lines with annotation bubbles.
Everything that follows in this article — the taxonomy, the gates, the whole data model — exists because the first attempt broke. The break is worth describing honestly: the numbers are unusually clear about what went wrong, and the instinct that caused it is the natural one.
The first real retrospective used seven labels. Two of them — call them Problem ("a subtlety the code deals with") and Caution ("a latent risk, keep an eye on this") — were written the way they sound: speculatively, while reading the code, as notes for later. They went straight into the document handed to reviewers. Months afterwards we audited every one of them against the actual code. This is a different tally from the three bugs above: those came out of writing the document, while this is an audit of the annotations themselves. The result:
| Label | In the document | Deleted as noise | Real findings |
|---|---|---|---|
| Problem | 69 | 67 | 2 |
| Caution | 5 | 5 | 0 |
| Verify | 52 | 8 | 12 fixable · 32 genuinely runtime |
Seventy-two of seventy-four Problem and Caution annotations were noise. Ninety-seven percent. Meanwhile Verify, the label whose questions could not be answered by reading, held up: 32 of its 52 entries genuinely needed runtime, another 12 pointed at something worth fixing, and only 8 were deleted.
That contrast is the whole diagnosis. The healthy label was the one whose subject matter was genuinely undecidable from the source. The two toxic ones covered things that could have been decided by reading the code — and were not. They were places where the author noticed something and deferred the thinking, wrapped in a label that made deferral look like diligence.
The direct cost was hours of cleanup across twelve chapters, most of it spent deleting things that should never have been written.
The real cost was trust. The reviewer read a label whose legend said "a subtlety the code handles" and reasonably asked whether it was, in fact, handled. Once that question is live, it applies to every annotation in the document — including all the correct ones. A review artifact's value rests entirely on the reader believing its claims; a single category of unreliable claims discounts the whole thing.
The lesson is not "be more careful." We already intended to be careful. The lesson is that an intention is not a mechanism. As long as there existed a place to put an unverified worry inside the reviewer-facing document, unverified worries went there, dozens of them. The fix had to be structural: remove the place, and give the worries somewhere else to live.
Annotations need categories, so the reader can tell "here is context" from "here is a defect" at a glance. The taxonomy that produced those numbers had seven labels. The excess was not decorative; it was the mechanism. The replacement has four:
| Kind | What it is | Gate — what it cannot exist without |
|---|---|---|
| Why | Why the code is designed this way, including the subtlety it handles | Describes a verified current state. May not contain a claim of a defect or an invitation to check something. |
| Solution | How this specific code handles that subtlety | Must point at a construct in the file shown, not a general principle. |
| Verify | A question answerable only at runtime or externally | Requires why it cannot be settled from the code and a concrete way to settle it. Anything decidable by reading is forbidden here. |
| Bug | A confirmed defect | Requires a concrete failure scenario and a verification chain. Plus adjudication by a second model and a human reading it against the code. |
What matters is not the count but this: there is no label into which an unverified worry can be filed. Each of the four either describes something known to be true or carries the proof of its own claim. A hunch has nowhere to land, so it has to go somewhere else entirely — the subject of the next section.
Our earlier taxonomy had a label called Resolved. That was a category error worth naming, because it is easy to repeat: resolved is not a kind of annotation but a stage in an annotation's life. As a label it silently meant two things — "the worry was unfounded" and "the bug was fixed" — and duplicated what belongs in a status field. So: four kinds, and a small state machine orthogonal to them.
| Status | Meaning | Required field |
|---|---|---|
| open | Still outstanding — for the reviewer, or for a fix | — |
| verified_ok | Checked; the concern was unfounded | evidence |
| fixed | Repaired | fix — commit or MR |
| deferred | Consciously postponed | trigger — when it comes due |
| withdrawn | A false positive, retracted | withdrawn_reason |
A withdrawn annotation is not rendered, but it stays in the source data. That distinction is deliberate: retracted findings leave an audit trail instead of vanishing. When someone asks in three months why a concern disappeared, the answer is in the file rather than in somebody's memory.
Removing the toxic labels was necessary but not sufficient. The instinct that filled them does not disappear because a label did: notice something, defer the thinking, write it down for later. It needs somewhere legitimate to go, and the reviewer-facing document is the wrong place. So the speculation gets a home of its own, and four gates stand between it and the reader.
Candidate findings go into a separate file that is never rendered. Speculation is allowed there, and required to state how it was found. An annotation is written only after adjudication. Expect to drop more than nine in ten; that is the gate working, not failing.
A Verify without both proof fields, or a Bug without a failure scenario and a verification chain, does not build. The same check catches category confusion in context annotations: if a Why bubble contains "missing", "could fail", "risk", or "worth verifying", the build stops and asks which it is. We added that after catching it in our own document, in a Why annotation that explained an invariant and then trailed off into "worth verifying that this holds for future implementations too."
The context pass is explicitly forbidden from emitting findings; anything noticed goes to the inbox. Hunting defects is a different pass with a different posture. Mixing them is how context turns into pseudo-findings.
Every "critical" is read personally against the code before it is recorded: in our audit, five findings came back marked critical and two survived that reading. And the scoping decision — which files, which chapters, who owns what — is approved by a human early, while it is still cheap to change.
Gates only work if there is something to gate. The first retrospective had a quieter design flaw that made every rule above unenforceable: the HTML was the source of truth. It came from an ad-hoc script nobody saved, so chapters could not be regenerated, every change was surgery on a minified document, and hand-written values like the per-file annotation count drifted out of sync.
Invert it. Annotations become data; HTML becomes a build artifact. Re-labelling an annotation is a one-field edit and a rebuild, and every derived value — counts, the legend, the bug page, the open-items table — is computed at build time, so none of it can drift.
Which is the moment to be precise about which half does what. The rendered page is for the human: it is what makes a hundred files navigable and shows a reviewer where to look. The schema is what acts on the agent. Having to fill in why a question cannot be settled from the code, or what concrete inputs produce the wrong output, is the thing that forces the altitude — and it would force it even if the output were plain text. The visualization earns its place because a reviewer will actually read it; the constraint behind it is what makes it worth reading.
An annotation pinned only to a line number is a time bomb. Fix a bug three files up and every line number below shifts; the bubble slides onto an unrelated statement and now says something false about code it was never about. Our first document "solved" this by freezing the source it displayed, which meant reconciling the document with reality by hand after every fix.
The fix is one required field. Every annotation carries an anchor: a substring of the
line it belongs to. The build checks that the substring is actually on that line, and if it is not,
the build fails — printing what the line really contains and, when it can find the
anchor elsewhere, where it moved to:
Drift becomes a build failure instead of a false statement in a document a reviewer trusts.
Which yields a small, ordinary pipeline — two data files in, validation in the middle, a static site out:
All of this is packaged as a skill: a folder of instructions and scripts an agent loads on request. The shape is deliberately plain, so you can rebuild or adapt it rather than adopt it verbatim.
In practice the loop is three commands — validate while writing, gate before handing over, render:
The --release gate refuses to hand anything over while the inbox holds unadjudicated
candidates, or while a confirmed bug sits at status open. The bundle is built from the output
directory only, never by adding a whole folder to git, which is how internal notes reach a reviewer.
That folder listing makes the process look heavier than it is, so it is worth saying plainly what a
medium change looks like. Five files, one subsystem: one chapter, one reviewer, a manifest of maybe
thirty lines. The inbox is a scratch list you clear in a single sitting, not an adjudication panel.
The second-model pass is worth paying for on a confirmed bug and is overkill for anything else. The
--release gate still refuses to ship an unadjudicated candidate, which at that size takes
a minute to satisfy. What scales with the change is the tooling, not the discipline —
the same four layers and the same four gates, in an afternoon instead of a week.
Keep the load-bearing parts: annotations as data with HTML as an artifact, the anchor field, a taxonomy with no home for unverified worries, the quarantine inbox, derived values computed at build time, whole files rather than snippets. Change freely the manifest format, the palette, the chapter split, how reviewers are assigned. Keep out project-specific verification commands: which cluster, which metrics backend, which storage belong in the project's knowledge base from Part 3. The skill only requires that a verification playbook be loaded, and carries two rules that each cost us a day: a deployed value is not a default (we read a config default in the repo and concluded a feature was off, while the deployed override had it on), and a cumulative log is not current state (two months of accumulated findings, mostly from one catch-up on day one, read as "we are losing data right now" while the live gauge sat at zero).
Part 1 framed the agent as a regulator inside a feedback loop. This is that loop closing on the work itself: a measurement that changes the thing measured, in the useful direction.
The mechanics are unremarkable in isolation: a TOML file, a few hundred lines of Python, a stylesheet, four labels, a quarantine file. The leverage is in the constraints. A taxonomy with no hiding place for a guess. A build that refuses to render a claim without its proof. An anchor that turns code drift into a failed build instead of a quiet falsehood. Take the constraints away and you have a document that looks the same and means much less — we know, because we shipped that version first.