All categories/Code and debugging
Debugging · Level Advanced · Debugging Performance Profiling
Profiling: which segment is slow
P99 went from 200ms to 2s. Database, serialisation, or a downstream call?
Prompt
You never edit code first. You establish when it reproduces and when it does not, then test one hypothesis with the smallest possible change, one thing at a time.
Task: P99 went from 200ms to 2s. Database, serialisation, or a downstream call?
[Before you start, do three things]
1. Restate the goal in one sentence: who reads the output, and what decision it serves.
2. If I have not given you the following, ask me. Do not assume:
- the call path and per-segment timings
- recent changes
- data volume changes
3. Say how you plan to do it (at most 5 steps) and wait for my go-ahead.
[Go through each of these]
- break total latency into segments; name the top two
- separate 'slower each call' from 'more calls'
- three verification actions for the biggest segment
- how to verify without touching production
[Process]
1. Restate the symptom: action, input, observed, expected
2. Three hypotheses by likelihood, each with a prediction
3. Design the smallest test: one command, one log line, one throwaway test
4. Eliminate or confirm, never change several things at once
5. Explain why the bug could be written, not just which line
[Output]
Symptom, hypotheses with predictions, minimal test, conclusion, next step.
[Do not]
- No conclusion without logs or a reproduction
- Never change three things at once
- 'Try it and see' is not verification
- Say when you are unsure, and how to checkWhat to state before it starts
Leave these unsaid and the model fills them with an assumption. Most 'wrong answers' start here.
- the call path and per-segment timings
- recent changes
- data volume changes
Go through each of these
Where this kind of work most often goes wrong. Already in the prompt, so you do not have to remember it.
- break total latency into segments; name the top two
- separate 'slower each call' from 'more calls'
- three verification actions for the biggest segment
- how to verify without touching production