Local model vs API model: quality and time on the same task
Experiment design set, results await raw records
This page gives only the experiment design, not the results. Numbers we haven’t run are never written; paste raw records into the table after running, then write the conclusion.
Question to answer
Same coding / writing task, run once on a locally deployed model and once on a cloud API model — how much do output quality, elapsed time, and per-task cost differ?
Control groups
Local model group / API model group; task, input, and mode identical, 3 runs each, raw records kept.
Environment & version
- Host version
- To fill
- Model ID
- To fill
- Mode / permissions
- Keep both groups identical
- Input files
- Record hashes so both runs use the same files
Fields to log
| # | Field | Result |
|---|---|---|
| 1 | Host version and model ID (local and API respectively) | to be tested |
| 2 | Task and input-file hash | to be tested |
| 3 | Tool calls per turn | to be tested |
| 4 | Input / output tokens | to be tested |
| 5 | Wall-clock time | to be tested |
| 6 | Whether the final output matches | to be tested |
| 7 | Failure and retry records | to be tested |
Conclusion
To be tested. Any conclusion must also state what it can't generalize to.
Why we don’t write results yet
Community expects local models to save cost but slightly lower quality, API models the reverse; this site assumes nothing and only collects raw data.