Cache hit or miss: the same task reads 10 times apart
Measured on 2026-11-14 — numbers from real runs
Question to answer
Run the same task several times: why does the first run cost so much more in tokens than the second? How much does a cache hit save, and on which line item?
Control groups
Same task, model, and input, run four times without changing input; plus a control group that clears the session each time (cold cache). Record cache-miss input, cache-hit input, and output separately, then price each.
Environment & version
- Host version
- 0.1.2-alpha.5
- Model ID
- host default route (no model ID recorded)
- Mode / permissions
- --profile headless / workspace-write / Windows 11 / node 22.22.2
- Input files
- Task: report the line count of x.txt and big.txt. x.txt has 4 lines; big.txt has 200 but its last line has no trailing newline. Directory fingerprint unchanged before and after
Fields to log
| # | Field | Result |
|---|---|---|
| 1 | Host version and model ID | 0.1.2-alpha.5; model via the host default route (no ID in the session header) |
| 2 | Task and input hash | Task: report the line count of x.txt and big.txt. x.txt has 4 lines; big.txt has 200 but its last line has no trailing newline. Directory fingerprint unchanged before and after |
| 3 | Cache-miss input tokens per run | 16,210 / 628 / 599 / 893 |
| 4 | Cache-hit input tokens per run | 15,872 / 47,872 / 47,872 / 65,408 |
| 5 | Output tokens per run | 137 / 319 / 308 / 1,493 (reasoning 0 / 65 / 95 / 902) |
| 6 | Cost per line item and total at official prices | Cache-hit $0.00005/0.00014/0.00014/0.00020; cache-miss $0.00243/0.00009/0.00009/0.00013; output $0.00008/0.00019/0.00018/0.00090. Totals $0.002561/$0.000429/$0.000418/$0.001226 |
| 7 | Ratio between run 1 and run 4 | Cache-miss input on run one was 25.8 times run two; cost on run one was 6.0 times run two. Yet total tokens on run one (32,219) were lower than run two (48,819) |
| 8 | Whether clearing the session returns to run 1 levels | No cleared-session control was run, but run one is equivalent to a cold cache: 16,210 cache-miss against 599 to 893 afterwards |
What the runs showed
- Caching saves the cache-miss input line, and drastically: 16,210 on run one, 628 and 599 after — down to about one twenty-fifth. The later three all sit between 600 and 900, meaning the cache is hitting steadily.
- More total tokens does not mean more expensive. Run two used 48,819 total tokens, 51 percent more than run one's 32,219, yet cost a sixth as much: 47,872 of run two was cache hits at $0.003 per million, while 16,210 of run one was cache-miss input at $0.15 per million, fifty times dearer. Any dashboard reporting only totalTokens ranks these two backwards.
- Run four cost $0.001226, 2.9 times run three, not because the cache failed but because it thought more: 1,493 output tokens against 137 to 319, with 902 reasoning.
- Run four was also the only one that got the edge case right: it reported big.txt as 199 lines and explained that the last line has no trailing newline. The first three all said 200. The extra three-fold cost bought a more correct answer.
- Wall clock ran 3.5, 5.3, 5.5, 11.4 seconds, tracking step count (2, 3, 3, 4) rather than token volume.
Conclusion
A cache hit saves the cache-miss input line, down to a twenty-fifth on repeat runs, and cuts the total bill to about a sixth. But the three items must be read separately: the run with fewer total tokens can be the dearer one. And the pricier run was the one that got the edge case right — cheapness and correctness are separate questions, so never judge by the bill alone.
What does not generalise
- Only four runs, with no cleared-session control. Treating run one as the cold case is an approximation, not a controlled condition.
- Hit rate depends on server-side cache policy and expiry, which this site cannot observe or control — only the outcome is measurable.
- The task holds about 200 lines of text, a small input. With tens of thousands of tokens the absolute gap between hit and miss grows, and the ratios may not hold.
Why we don’t write results yet
The question is which line item caching saves and by how much, not whether caching helps. All three items must be recorded separately; totalTokens alone cannot answer it.