Context plugins: do they actually save tokens
Experiment design set, results await raw records
This page gives only the experiment design, not the results. Numbers we haven’t run are never written; paste raw records into the table after running, then write the conclusion.
Question to answer
With and without a context-management plugin (dsh-context, billion-context), how much do tokens, compaction count and answer completeness differ on the same long-document task?
Control groups
Three arms: no context plugin, plugin A only, A plus B. Same model, same input, same instruction, three runs each, raw values recorded with no cosmetic averaging.
Environment & version
- Host version
- To fill
- Model ID
- To fill
- Mode / permissions
- Keep both groups identical
- Input files
- Record hashes so both runs use the same files
Fields to log
| # | Field | Result |
|---|---|---|
| 1 | Host version and plugin versions | to be tested |
| 2 | Model ID and gateway | to be tested |
| 3 | Input document size and hash | to be tested |
| 4 | Number of context compactions | to be tested |
| 5 | Input / output tokens | to be tested |
| 6 | Whether all ten key points survived (checked one by one) | to be tested |
| 7 | Wall-clock time | to be tested |
| 8 | Errors and retries | to be tested |
Conclusion
To be tested. Any conclusion must also state what it can't generalize to.
Why we don’t write results yet
Community claims about context plugins saving tokens conflict; some runs cost more because of an extra summarisation call. No conclusion is assumed here. Numbers come only from run records; empty until measured.