Experiment and artifact chain
Sections
Reading settings
Natural voice recorded for this course, the same in every browser.
No voice for this language in this browser.
More voices
Code and English terms
Off: the page’s voice reads everything, with no pause between two voices.
No voice for this language in this browser.
More voices
More voices: Microsoft Edge has natural online voices, and Windows (Settings › Time & language › Speech), macOS and Android install others.
An experiment is useful when another person or agent can determine exactly what happened without trusting the author.
Minimum chain
Section titled “Minimum chain”| Artifact | Required content |
|---|---|
| Observation | Concrete repository pain, pinned source or measured symptom |
| Hypothesis | Directional, falsifiable claim written before measurement |
| Baseline | Current quality, cost, latency or change surface |
| Protocol | Fixed corpus/workload, exact commands, limits and stop conditions |
| Raw evidence | Outputs, usage, hashes and environment; secrets excluded |
| Verdict | Confirmed, refuted or inconclusive against predeclared criteria |
| Promotion receipt | Accountable authority, exact evidence and next reversible step |
Classical and agentic metrics
Section titled “Classical and agentic metrics”For classical software engineering, prefer defect rate, changed-file surface, deterministic test setup, p50/p95 latency, recovery time and operational burden.
For agentic systems, add accepted-result quality, false positives, calibration, escalation, retries, provider-specific input/cached/output tokens and dollars. Do not sum token counts from unlike tokenizers and call the total a saving.
Independent refutation
Section titled “Independent refutation”The reviewer tries to invalidate the conclusion: leakage in the corpus, a cheaper deterministic alternative, shifted human cost, missing failure cases, or an artifact that does not match the tested revision.
Exercise: define a 50% token-saving experiment
Fix one corpus and downstream model. Compare calling it on every case with calling it only after a Jev gate. Keep the quality gate identical. Reject the hypothesis on any false authority-sensitive support, or when retries and human review erase the dollar saving.