Appendix H. Claim-to-Evidence Ledger
H.1 How to read the ledger
Several chapters rest on the same few studies, and repeated use of one study is repetition, not replication. This ledger lays out, for each major claim, the chain of Chapter 2, Section 2.6: the observation (what was measured, and by whom), the inference I draw, the mechanism I propose, and the test that could refute the claim. Each link is weaker than the one before.
The Observation column holds the only measured results, and it says what kind each is.
- Measured: a result from a cited study, with its source. The study measured its own population and tools, not necessarily the claim.
- Illustrative: a number I chose or computed for a worked example. It is not data.
- Proved or Derived: a mathematical result, or my own derivation from one.
- Not measured: I found no study. The claim is a hypothesis.
The Inference, Mechanism and Test columns are my synthesis, and none of it is validated. Sources are named by author and the section that cites them in full; the full references are in that chapter's list. A prediction (P1 to P10) is the test that Appendix G assigns to a lab. Use the ledger two ways: to see how much of a verdict rests on measurement, and as a model for the ledger of your own post-mortem, where each claim you make about your project gets a row.
H.2 Foundations
Table H.1. Claims of Chapters 1 and 2.
| Claim | Observation (source) | Inference | Proposed mechanism | Test that could refute it |
|---|---|---|---|---|
| Effort moves from writing to reviewing; it is not simply cut (Section 2.6) | Measured: Faros, review time +91%; METR trial, 19% slower (dated, early 2025); DORA, lower throughput and stability | Review and integration absorb the saving | Writing got cheap; reading and checking did not | P1: review plus verification hours exceed construction hours |
| The decision rule Q1 to Q4 gives independent readers the same verdict (Section 2.5) | Not measured. Two worked cases in the chapter | Disagreement traces to the unit or the assumption | An ordered rule removes the label dispute | Two readers, one fixed unit and assumption (Chapter 2 lab): different verdicts refute it |
| Software engineering is a coordination discipline (Section 1.3) | Historical record (Garmisch, Conway, Parnas, Brooks); no measurement | Coordination is the durable problem | Work divides; parts must agree through channels | Table 18.2: costly events in your log do not trace to channels |
| Estimation, life cycles, risk, configuration and reliability are missing from agent-era courses (Section 1.5) | Measured: none of 18 courses with public topic lists coded them (Geng et al.); a preliminary analysis of 8 courses (Smith 2026a) | Absent from public lists; not shown untaught | Tool-first course design (proposed) | Survey syllabi and assessments, not topic lists |
H.3 Part II, Chapters 3 to 9
Table H.2. One claim per chapter, Chapters 3 to 9.
| Claim | Observation (source) | Inference | Proposed mechanism | Test that could refute it |
|---|---|---|---|---|
| 3: Where review binds, review capacity sizes the loop (Section 3.5.1) | Not measured: no study relates loop length to reviewer capacity | Loop length follows W ≤ h × T ÷ k | Human review is the binding resource | P1 and the Chapter 3 lab: loop length not within 25% of the inequality, with review not binding |
| 4: Agents fill gaps silently, so the specification is the primary input (Sections 4.3, 4.4) | Measured, benchmarks: over 60% of responses were code, not questions; correctness fell 35% to 52% (Wu and Fard); asking helped by up to 74% (Vijayvargiya et al.) | Specification quality may bound output quality | Tacit intent reaches the model only if written down | P2: agent errors, not spec gaps and ambiguity, are the largest root-cause category |
| 5: A fixed delivered-lines rate stops predicting effort; verification size matters (Section 5.3) | Illustrative: 6,000 delivered lines, 9,000 or 18,000 presented. No project data | A second size, (R, P), is needed | Review scales with lines presented; verification with properties | P3 and the chapter's own: lines track hours as well as function points |
| 6: COCOMO II has no terms for lines presented, review rate, properties, steering, rework (Sections 6.4, 6.5.6) | Illustrative: Dispatch 105 h (90 to 200); COCOMO II 119 to 3,206 h by setting. No outcome data | Its mechanisms move the level, not the response | Review-dominated effort has no input in the model | P4: COCOMO II at its strongest ties the agent-aware model on held-out projects |
| 7: Credit at verified merge; the generated-to-merged gap leads slippage (Sections 7.5.3, 7.5.4) | Illustrative: Dispatch week 4, 76 h by generation, 126 by merge, 118 actual. No study computes both | Credit at generation overstates progress | Agents generate faster than people review | P5: the two earned-value curves track each other, even in high-output weeks |
| 8: Shared context carries only what is fixed beforehand (Sections 8.3.5, 8.4) | Proved for the game (Section 17.2). Its mapping to software is a conjecture | Mid-task information needs a message | Context is fixed before the task; messages carry what arises during it | Chapter 8's own: most coordination defects trace to declared channels |
| 9: Unconditional P × L cannot be grounded for attacker-chosen events (Sections 9.4, 9.5.3) | Measured: 19.7% of recommended packages did not exist (Spracklen et al.), an error rate, not an attack rate. No injection incident data | Flag each path by its legs; do not invent P(attack) | Removing a leg works whatever the probability | P7: unconditional scores rank incidents as well as structural flags do |
H.4 Part II, Chapters 10 to 15
Table H.3. One claim per chapter, Chapters 10 to 15.
| Claim | Observation (source) | Inference | Proposed mechanism | Test that could refute it |
|---|---|---|---|---|
| 10: Design knowledge must be written down for agents (Sections 10.3, 10.4) | Measured: long contexts used unevenly (Liu et al.); convention files help for non-standard practice, not overviews (Gloaguen et al.). Retention not measured | Written rationale replaces lost memory | Where the harness persists nothing, files carry design knowledge | P8: package length predicts comprehension hours as well as fullness does |
| 11: Code is abundant; understanding is scarce (Section 11.3.1) | Measured: quiz scores about 17% lower with AI help, 52 learners, preprint (Shen and Tamkin). Inherited agent code not measured | Reading and decomposition are the bottleneck | Delegated writing skips the understanding writing produced | P1 and the chapter's own: comprehension cost falls toward generation cost |
| 12: An evidence package finds more seeded defects per reviewer-hour than a raw diff (Section 12.5) | Measured: review time +91%, pull request size +154% (Faros). No yield study on agent code | Raw-diff inspection does not scale | Checks the author cannot alter show the reviewer where to look | P9: no gain per reviewer-hour, or the AI arm's misses fully overlap the humans' |
| 13: Agent-written tests can encode the bug (Sections 13.3, 13.5) | Measured: LLM oracles mostly capture actual behavior, 24 Java repositories (Konstantinou et al., preprint); graders tampered with in 39 of 128 runs, one suite (METR) | A green pipeline can mean nothing | The oracle is derived from the code it checks | Chapter 13's own: agent suites match spec-derived suites on mutation score, with no defect-encoding tests |
| 14: The history no longer records how a change was produced; a hosted model is recordable, not restorable (Sections 14.3, 14.4) | Measured: one service, March to June 2023, executable code 52% to 10% (Chen et al.); provider notes on serving changes. Incident rates not measured | The baseline must hold model, prompts and settings | Behavior changes arrive without a commit | Chapter 14's own: a model-only swap stays within the control's run-to-run variation |
| 15: Layers on one model fail together, so p^k overstates; a correlated layer still adds detection (Sections 15.3.4, 15.5.3) | Measured elsewhere: Knight and Leveson (27 versions); Kim et al. (same wrong answer 60% of the time when both were wrong). Illustrative: Dispatch, 12 joint misses of 100 against 4 expected | Measure the joint miss rate; do not multiply | Shared training gives shared blind spots | P10: joint misses at or below the product, with power to see a departure |
H.5 Part III: the four conditions and the coordination bound
Table H.4. The four conditions and the coordination bound (Chapters 16 and 17).
| Claim | Observation (source) | Inference | Proposed mechanism | Test that could refute it |
|---|---|---|---|---|
| Condition 1: a checking role adds assurance when it detects defects the generator's checks miss (Section 16.2.1) | Measured elsewhere: Knight and Leveson; Kim et al.; a preprint on 48 agent versions, majority vote cut mean failures 387 to 131 (Ron et al.) | Judge a checker by what it adds, not by its separation | Shared training correlates misses; diversity of method helps | Seeded flaws (Chapter 16 lab): a role with no incremental detection still lowers escaped defects |
| Condition 2: roles read artifacts, not claims (Section 16.2.2) | Measured: developers misjudged their own speed (METR); tampered graders (Section 13.3.2). Compliance by declaration is a practitioner term | A report is a claim, not evidence | Agents narrate; artifacts do not | Claim-to-artifact audit: reports match artifacts every time, so a sample would do |
| Condition 3: coordination travels through logged channels (Section 16.2.3) | Not measured on production stacks. The bound is proved for a game | Mid-task dependencies need a declared channel | Unlisted paths carry dependencies nobody reviews | Channel inventory and leak test: converge-or-split decisions prove rare |
| Condition 4: every role has a named human owner (Section 16.2.4) | Not measured. Practice statements only (Hassan; Tantithamthavorn) | Accountability stays human, whoever typed the change | Authorship, approval and accountability are separate | Reconstruction test: roles with owners do no better at explaining and suspending than roles without |
| On converge-or-split decisions, agents sharing only prior information win at most 75% (Section 17.2) | Proved: the classical value of the CHSH game (Clauser et al.; Cleve et al.). Derived: 16 strategies enumerated | Shared context buys a convention, not communication | Everything fixed in advance acts as one shared string | A flaw in the proof, or a classical strategy above 3 of 4 rounds |
| The shared-interface decision obeys the bound as a necessary condition (Sections 17.3.1 to 17.3.3) | Derived: truth table; every specification win is a parity win; brute force over 16 strategies | Specification win rate ≤ CHSH win rate ≤ 75% | The specification game is stricter than parity | Independent check of the truth table and the enumeration |
| Software has decisions of this kind often enough to matter (Section 17.3.4) | Not measured. A conjecture | Some concurrent collisions may have this structure | Two tickets, one interface, no message | Mine concurrent changes in agent-built repositories: the structure is rare |
| The leak test detects a planted channel (Section 17.6.1) | Measured, instrument demonstration: 200-round pilots, 74.5% clean, 84.5% planted leak (Smith 2026b). Intervals and counts are my calculations | The test can flag a leak of effective fraction about 0.38 | A leak fraction q lifts the win rate to 0.75 + 0.25q | The 1,000-round runs miss a small planted leak at the planned power |
| Reasoning, labels, or both fixed the reflex failure (Section 17.6.2) | Measured, confounded: 26.5% became 74.5% after reasoning was enabled and lanes were relabeled A and B | The pilot does not separate the two changes | Reasoning may apply the rule; neutral labels may remove a pull toward "1" | A 2 × 2 factorial, 200 rounds per cell, 1,600 calls |
| Without explicit messages, win rates stay at or below 75%, and any excess traces to a channel (P6) | Not measured on a real stack. The pilots are not production evidence | An excess triggers a rig audit; a later finding names the channel | A hidden channel raises the rate above the convention's | An excess survives audit and ablation with no channel found |
H.6 Part IV: the project and its synthesis
Table H.5. Claims of Chapter 18.
| Claim | Observation (source) | Inference | Proposed mechanism | Test that could refute it |
|---|---|---|---|---|
| A refit that contains the actual is compatible with the model; it does not validate it (Section 18.6) | Illustrative: a composite student's refit gives about 80 h against 83 h actual, after the fact | An accounting identity can be fitted to anything | Parameters from the same log reproduce its hours | P4 held out: a model that fits but loses to the alternatives on held-out projects |
| Most questions a package should have answered ask for reasons or states, not rules (Section 18.4.5) | Not measured. A composite case has 6 of 9 questions the package should have answered (illustrative) | Packages hold rules well and private decision context badly | A package is a message with a structured body | Chapter 18's own: most should-have questions ask for rules |
| Coordination is the bottleneck, ahead of the loop and of decisions (Section 18.7) | My judgment. Two rival claims (Tantithamthavorn; Hassan) are practitioner statements | Each claim is right about a layer | What lies beyond the bound must travel through a designed channel | Your log, Table 18.2: product decisions dominate your escalations |
H.7 Extending the ledger
Add a row for each claim in your own post-mortem. Write the observation first, from your log, with its entry numbers. Write the test before you look at the result, and date it. If your row's observation is a single number from a single project, say so in the Inference column; that is an anecdote with a log entry, not a finding.