A method that outlasts the tools

This appendix collects the apparatus of Chapter 2 on one set of pages, so you can keep it open while you audit. It adds nothing new. If a definition here and a definition in Chapter 2 ever differ, Chapter 2 governs.

A.1 The audit card

The audit on one card

1. Method: What specific mechanism or activity (the unit of analysis)? Learn it from the primary source.
2. Assumption: What, if false, makes its output unreliable? (Negation test. Worker, artifact or process.)
3. Evidence: Does that still hold with agents? How strong is the evidence?
4. Verdict: Holds, recalibrate, breaks or inverts? (Decision rule, Section A.4.)
5. Replacement: What do you do instead? What does it keep?
6. Prediction: What observation would show the verdict wrong?

Then ask: what would have to change about agents for this to stop being true?

Table A.1 pairs each step with the mistake students make most often there (Chapter 2, Section 2.4).

Table A.1. The six steps and their common mistakes.

Part II chapters follow one pattern for Steps 1 to 5: what the method actually requires; what teams often do instead; what agents change; what the revised procedure must enforce.

A.2 A blank audit form

Copy this onto one page. Fill it in before you collect data, and date it.

• Date and author:
• Method and unit of analysis (specific, with its primary source):
• Candidate assumptions (at least three) and the negation test result for each:
• Load-bearing assumption (one sentence; worker, artifact or process):
• Evidence found, with its rung on the ladder and its relevance, measurement quality, uncertainty and replication (Section A.5):
• Verdict, and the question of the decision rule at which you stopped (Section A.4):
• Replacement, and what it keeps from the classical method:
• Prediction (what you will observe, from which data, by when):
• What would have to change about agents for this conclusion to stop being true:

A.3 The negation test

State the candidate assumption. Negate it. Ask what happens to the method's output.

• If the output becomes unreliable, the assumption is load-bearing.
• If the output is unaffected, or only less precise, the assumption is incidental.

Look for load-bearing assumptions in three places (Chapter 2, Section 2.3).

• The worker: who does the work and what is true of them (writes slowly, understands what they wrote, asks when unsure, coordinates by talking, can be held accountable).
• The artifact: what is produced and what is true of it (code is scarce, a program stays fixed between fixes, behavior is determined by versioned source).
• The process: what is expensive, what is cheap, and what sets the pace (iteration is expensive, completed work is progress).

The test asks what would happen if the assumption failed. It does not ask whether the assumption has failed. Settle the first question about the method before you take up the second question about the world.

A.4 The verdict tests

Name the unit of analysis first. A verdict is a claim about that unit, and verdicts are comparable only for like units: a model, a design principle, an activity and a family of practices answer different questions. The same practice can earn different verdicts at different units. Then apply the decision rule of Chapter 2, Section 2.5, in order, and stop at the first yes.

The decision rule

• Q1. Is the unit an activity whose purpose was set by a scarcity that has reversed? If yes: Inverts, and also report the verdict for the activity's main mechanism using Q2 to Q4.
• Q2. Does the load-bearing assumption still describe agent-assisted work (its negation remains hypothetical)? If yes: Holds.
• Q3. Can the mechanism produce reliable output again by changing only parameter values, rates, thresholds, or the definition of an existing quantity, keeping its procedure and outputs? If yes: Recalibrate.
• Q4. Otherwise (new variables, records or steps that change what is computed or controlled): Breaks.

A new input artifact to an unchanged procedure is Q3, not Q4.

Table A.2. The four verdicts, the question that yields each, and an example.

Q1 comes first because an inverted activity can contain mechanisms that hold or recalibrate. Q2 asks about the world and is answered from evidence; Q3 and Q4 ask about the method and are answered by trying the fix. Two readers who apply the rule to the same unit and assumption should reach the same verdict. When they do not, the disagreement is about the unit or the assumption, and that is where to argue.

Two cautions keep the hardest call, Q3 against Q4, honest.

• A new artifact is not a break. A method can keep its structure while the thing it operates on changes: inspection of an evidence package is still inspection (recalibrate).
• A "holds" can come with new artifacts. Information hiding holds for agents precisely because design knowledge must now be written into convention files and decision records. Ask whether the method's logic changed, not whether its paperwork did.

Some areas get two verdicts. Risk management recalibrates for project risks and breaks for the simple unconditional probability-times-loss formulation applied to adaptive threats. Size estimation breaks for lines of code as a stand-in for effort and survives for function points. Say exactly where the line falls. Every verdict is a dated judgment: write the date next to yours.

A.5 The evidence ladder

Grade a source in two passes: by where its design puts it on the ladder, then by how well it fits your question (Chapter 2, Section 2.6).

Table A.3. The evidence ladder, strongest design first.

A rung is where a source starts, not where it ends. Four qualities move it.

• Relevance. Do the people, tasks, tools and period match your question? A lower rung with high relevance can outweigh a higher rung with low.
• Measurement quality. Was the outcome measured or self-reported, and is it what you care about or a proxy? Pull requests merged is an output; time to finish a task is effort.
• Uncertainty. How large is the sample, how wide the interval, what selection effects could produce the result? Prefer ranges to points.
• Replication. Has someone else found it in another setting? Several chapters citing one study is repetition, not replication.

Three rules go with the ladder.

1. Separate measurement from projection. A study measured 2025 tools. A claim about what agents "will" do is a projection. Say which one a verdict relies on.
2. When evidence conflicts, ask what each source measured. Outputs, effort and end-of-pipeline outcomes are different quantities, and studies of each can all be right.
3. Keep the chain visible. For every claim write four links: the observation (what was measured), the inference (what you conclude), the proposed mechanism (why it would be so) and the test (what would refute it). Each link is weaker than the one before. Appendix H applies this to the book's own claims.

A.6 The thirteen verdicts

Table A.4 reproduces Table 2.1. Each chapter gives the full argument.

Table A.4. Verdicts for the thirteen areas of a traditional software engineering course (reproduced from Table 2.1).

The rows audit different units, so they are not thirteen measurements on one scale: Chapter 6 audits one model, Chapters 4 and 11 audit activities, Chapter 10 a design principle, and the rest families of methods or practices. Each chapter states its own unit at the start of Section N.2, and that statement governs.

For the two activities that invert, the chapters also report a verdict for the main mechanism (Q1): the specification mechanism recalibrates (Chapter 4), and the authorship-based construction practices break (Chapter 11).

The tally is two hold, five recalibrate (counting risk once), four break and two invert. These thirteen judgments summarize the audits. Their value lies in the assumptions and proposed tests behind them, rather than in the count assigned to each category.

A.7 The four conditions

Chapter 1 gives these in brief and Chapter 16 develops and tests them. They are my synthesis, unvalidated, and the claim is necessity, not sufficiency: a role played by an agent (project management, risk, red team, reviewer) that fails a condition adds the appearance of assurance, and one that meets all four has only earned the chance to be useful.

1. Incremental detection. A checking role adds assurance when it detects defects the generator's own checks miss. Measure its incremental detection and the joint miss rate; do not assume statistical independence.
2. Evidence, not self-report. Roles read artifacts (CI results, merged state, test output), not other agents' claims.
3. Explicit channels. What roles must coordinate on travels through logged messages or structured handoffs.
4. Human ownership. Every role has a named human who owns its decisions. Accountability does not transfer to an agent.

Keep three things apart under condition 1. Implementation separation (a fresh session, a different prompt) is cheap and establishes nothing about correlation. Diversity (a different model family, a different method, a person) makes coincident misses less likely, but a different family does not establish independence, and different authorship alone does not either (Chapter 13). Measured statistical independence means the observed joint miss rate matches the product of the miss rates. Independence is what licenses multiplying miss probabilities (p^k, Chapter 15). It is not a prerequisite for useful checking: a correlated second layer still removes some of the defects the first layer missed (Chapter 15, Section 15.5.3).

A.8 The closing question

End every audit, and every chapter, with this question, and write your answer down:

What would have to change about agents for this conclusion to stop being true?

If you can answer it, you understand both the method and the setting it works in. If you cannot, you have memorized a verdict, and verdicts expire.