Minimum record spec: what must be filled in for a run to count
Measured on 2026-11-14 — numbers from real runs
The raw data here comes from real runs; token counts and timings were parsed from the session files with tools/usage.py. Sample size and what does not generalise are stated on the page, with no averaging.
Question to answer
How detailed must a record be before it is reproducible? Which missing field disqualifies a claim?
Control groups
Work backwards from four kinds of invalid record seen on this site (conclusion without raw values, totalTokens without breakdown, one run treated as evidence, no version recorded), each with why it fails and what to add.
Environment & version
- Host version
- 0.1.2-alpha.5
- Model ID
- not applicable
- Mode / permissions
- not applicable — this page is a record spec
- Input files
- Cache-miss input, cache-hit input, and output, <strong>in three separate columns</strong>. Recording only totalTokens records nothing: totalTokens equals the three summed, their unit prices differ by 50 to 200 times, and merging them is irreversible
Fields to log
| # | Field | Result |
|---|---|---|
| 1 | The three mandatory environment items | Host version (verbatim `dsh --version`), model ID and gateway, the three permission tiers (sandbox mode plus approval policy), OS and node version |
| 2 | The three token items that must be separate | Cache-miss input, cache-hit input, and output, <strong>in three separate columns</strong>. Recording only totalTokens records nothing: totalTokens equals the three summed, their unit prices differ by 50 to 200 times, and merging them is irreversible |
| 3 | Sample size that must be stated | How many runs. A single run is not evidence — in the cache measurements run one showed 16,210 cache-miss and run two 628, a 25.8-fold gap |
| 4 | Time of sampling that must be recorded | Time of sampling. Prices split peak and off-peak (peak is 01-04 and 06-10 UTC, Monday to Friday), so the window decides the rate; model versions and channel pointers also move over time |
| 5 | The 'does not generalise' section that must exist | The 'does not generalise' section on this page is mandatory. If you cannot write it, you do not yet know the conclusion's boundary, and it should not ship |
| 6 | Identifying marks of the four invalid kinds | One: conclusion with no raw values. Two: totalTokens without breakdown. Three: one run treated as evidence. Four: no version or time. Any one disqualifies the record as evidence |
| 7 | Minimum field set for a valid record | Four environment fields, three token items, sample size, sampling time, a boundary section, and an input fingerprint (hash) — ten in total. Miss one and the record is marked partially verified, not verified |
What the runs showed
- Three rules became hard rules here because we paid for them. One: tokens must be three columns.My first pass took the last usage per turn and produced 28,827; summing per step gives 126,551, a 4.4-fold difference. Treat inputTokens as total input (ignoring 89,216 cache hits) and you get a third, different number.
- Two: the first run must be discarded.Measured cache effect over four runs: cache-miss 16,210 then 628, 599, 893. Run one is cold cache; using it as baseline makes everything after look like a 96 percent saving.
- Three: the 'does not generalise' section is mandatory, not a courtesy.In the PTC comparison PTC clearly looked cheaper, but the task was counting lines in four files, which suits being written as one program. Drop that boundary and readers conclude PTC is always better.
- One counter-intuitive addition: record whether it was right, too.Run four in the cache group cost 2.9 times run three yet was the only one to get the edge case right (199 versus 200 lines). Track cost alone and the conclusion becomes 'run four wasted money'.
Conclusion
A run usable as evidence needs at least ten fields: four environment items, three token items, sample size, sampling time, a boundary section, and an input hash. The four commonest invalid records are conclusion without raw values, totalTokens alone, one run, and no version or time. None of that is pedantry — each one can invert the conclusion.
What does not generalise
- This is our own spec, not anyone's standard. Other sites reach usable conclusions with fewer fields when their question is narrow enough.
- Two of the ten (sampling time and input hash) are often skipped in manual work. Skipping is not an error, but the record must then be marked partially verified.
Why we don’t write results yet
This page is a rule and produces no numbers. Its job is to let anyone reproduce a record instead of having to trust the author.