Chapter 12. Reviews: Inspecting Evidence, Not Diffs

Learning objectives

After this chapter you should be able to:

Plan a Fagan-style inspection (roles, operations, rates, checklists, data) and compute its cost for a given change.
State inspection's load-bearing assumption and use the negation test to show which of its outputs become unreliable when the author is an agent.
Separate what has been measured from what is projected about reviewing agent-written code.
Assemble an evidence package for a change and inspect it with a checklist rebuilt for agent defect classes.
Budget review hours for a project using the review terms of the agent-aware cost model (Chapter 6).
Design and analyze a three-arm inspection that compares raw-diff inspection, evidence-package inspection and an AI reviewer against seeded ground truth, reporting what the AI reviewer adds beyond the human arms.

Opening case

A composite case, not a real engagement.

Aiko and Marcus are the two engineers reviewing agent work on the dispatch dashboard from Chapter 6. On Tuesday an agent delivers a pull request titled "Let dispatchers reassign jobs." It touches 14 files and about 640 lines. The description is clear and confident. It explains the new reassign button, the new endpoint, the role check and the tests, and ends: "All tests pass. Access restricted to dispatchers."

The chapter continues from here.

What the chapter covers

12.1 The classical method
- 12.1.1 Where inspection came from
- 12.1.2 The method
- 12.1.3 What the data show, and a figure to audit
- 12.1.4 From inspection to modern code review
- 12.1.5 A worked example: pricing an inspection
- 12.1.6 Why inspection worked
12.2 The assumption it depends on
12.3 What agents change
- 12.3.1 Volume
- 12.3.2 The explanation is not evidence
- 12.3.3 Agent defect classes
- 12.3.4 Review as an audit of evidence
- 12.3.5 Who is right, Fagan or Hassan?
- 12.3.6 Self-correction and a separate reviewer
12.4 The verdict
12.5 What replaces it: inspecting the evidence package
- 12.5.1 What the package is
- 12.5.2 The procedure
- 12.5.3 A checklist rebuilt for agent defect classes
- 12.5.4 The AI reviewer as one arm
- 12.5.5 The review budget
- 12.5.6 Dispatch: the access-check defect, replayed
12.6 Lab: Three-arm inspection
12.7 What would make this verdict false

Lab and prediction

Lab: Three-arm inspection

Feeds milestone M5. How the labs fit the project

The lab tests this prediction from the working paper the book grows from (Smith 2026a):

P9 Evidence-package inspection finds more seeded defects per reviewer-hour than raw-diff inspection; the AI reviewer finds a different defect set

A result that contradicts a prediction, honestly reported, is a result. You are graded on accurate records and reasoned interpretation, not on agreeing with the book.

Summary

Fagan's yields are measurements on one code base at one pace (82% in one case, about two thirds in another); the 60 to 90% range attributed to him is not in the 1976 paper.
Load-bearing assumption: rates and checklists are calibrated to human authors. Agents break the rate (volume), the checklist (different defects) and the author's account (generated after the fact). Verdict: recalibrate. The discipline holds; what is inspected, the rate and the checklist change.
Evidence: review time and pull request size are up sharply in industry data; no study measures inspection yield on agent code.
Replacement: a six-part evidence package inspected with Fagan's operations, a checklist for six agent defect classes, and H_review(package) = P × t_p + (λγL)/r. Dispatch effort is 105 hours with raw-diff review at r = 400, 144 at r = 200, and about 106 with package inspection at r = 200.
The AI reviewer is one arm, judged by incremental detection and joint misses, not by assumed independence. A separately configured reviewer is not the same arrangement as self-correction.
The three-arm lab tests P9 against seeded ground truth.

Key terms

Fagan inspection: a formal review with a moderator, defined roles, preparation, a meeting that finds but does not fix errors, rework and follow-up.Also used in 1
Error detection efficiency: errors found by an inspection divided by total errors in the product before it, times 100.
Evidence package: the six-part set of artifacts, each produced by someone other than the author or marked as the author's statement, that an inspector audits instead of the raw diff.Also used in 2, 3, 14, 15, 16
Bound property: an acceptance property linked to the check that tests it.
Non-change manifest: a list of what was deliberately left unchanged and why (Hassan's "Changes and Non-Changes Manifest").
Surprise log: the agent's record of anything unexpected it found; a lead for the inspector.
Delta-first review: reviewing only what changed since the last approval (Hassan 2026, §7.3.3, pp. 221–222).
Agent defect classes: nonexistent APIs, phantom dependencies (imports of packages that do not exist or are not needed), tests that assert nothing, edited tests, plausible handling of ambiguity, and missing server-side checks.
Consistency without correctness: agreement among reviewers that share blind spots.Also used in 16
Three-arm inspection: the lab comparing raw-diff inspection, evidence-package inspection and an AI reviewer against ground truth.

Connections

Where this chapter and another point to each other most often.

Chapter 2: The Assumption Audit: A Method That Outlasts the Tools
Inspection's load-bearing assumption is: > Rates and checklists are calibrated to human authors. Apply the negation test (Section 2.3).
Chapter 4: Requirements as the Primary Artifact
Intent lives in the specification and acceptance properties a human wrote before generation (Chapter 4), and review should check the change against those, not the agent's story. ### 12.3.3 Agent defect classes Agents make some defects people rarely make, and make them look right.
Chapter 6: Cost and Schedule Estimation After COCOMO
Budget review hours for a project using the review terms of the agent-aware cost model (Chapter 6). 6.
Chapter 9: Risk Management: Project Risk and Adversarial Risk
(2025) found that 19.7% of packages recommended by code-generating models in one large study did not exist (package suggestions, not agent runs; Chapter 9 treats the attack side).
Chapter 11: Construction When Code Is Cheap
Take the Dispatch parameters (illustrative, Section 6.5.3): P = 40, γ = 2.0, L = 6,000, specification 30 hours, steering 10 hours (30 tasks at 1/3 hour; Section 11.5.1), verification 20 hours and ρ = 0.3.
Chapter 13: Testing, Evaluation, and Tests as Steering
Verification replaces reading. If executable checks covered most defect classes with joint misses near zero, people could stop reading code line by line and audit the verification system instead (Chapters 13 and 15), and the verdict would become "breaks." This flips the verdict.
Chapter 14: Configuration Management When Agents Commit
Provenance | The commit trailers of Section 14.5.2 | Who or what produced this, under what instructions? | Harness (Chapter 14) | Part 6 uses Chapter 14's eight trailer keys: Mission, Agent-Model, Agent-Harness, Baseline, Autonomy, Evidence, Directed-by and Approved-by.
Chapter 15: Reliability of Nondeterministic Systems
Independence is the stronger claim, needed only to multiply the layers' miss rates (Chapter 15).
Chapter 16: Roles and the Four Conditions for Assurance
The failure to avoid is what Hassan calls compliance by declaration: accepting that a required check happened because the actor says so (§9.9.6, p. 277; Chapter 16, second condition).

See it on the map

12 Reviews: Inspecting Evidence, Not Diffs