Skip to content
MyDSHMy DSH Field Notes
中文
Independently maintained; not affiliated with DeepSeek. No association with mydsh.dev or other MyDSH sites.

All categories/Code and debugging

Debugging · Level Advanced · Debugging Testing CI

Flaky tests: passes locally, fails in CI

A test passes locally and fails randomly in CI.

Prompt

You never edit code first. You establish when it reproduces and when it does not, then test one hypothesis with the smallest possible change, one thing at a time.

Task: A test passes locally and fails randomly in CI.

[Before you start, do three things]
1. Restate the goal in one sentence: who reads the output, and what decision it serves.
2. If I have not given you the following, ask me. Do not assume:
   - failure message and frequency
   - environment differences
   - dependence on time, network, randomness
3. Say how you plan to do it (at most 5 steps) and wait for my go-ahead.

[Go through each of these]
- decide flaky versus real defect first
- four common causes: timing, concurrency, external deps, shared state
- run fifty times and record the failure distribution
- propose a fix direction, not a weakened assertion

[Process]
1. Restate the symptom: action, input, observed, expected
2. Three hypotheses by likelihood, each with a prediction
3. Design the smallest test: one command, one log line, one throwaway test
4. Eliminate or confirm, never change several things at once
5. Explain why the bug could be written, not just which line

[Output]
Symptom, hypotheses with predictions, minimal test, conclusion, next step.

[Do not]
- No conclusion without logs or a reproduction
- Never change three things at once
- 'Try it and see' is not verification
- Say when you are unsure, and how to check

What to state before it starts

Leave these unsaid and the model fills them with an assumption. Most 'wrong answers' start here.

  • failure message and frequency
  • environment differences
  • dependence on time, network, randomness

Go through each of these

Where this kind of work most often goes wrong. Already in the prompt, so you do not have to remember it.

  • decide flaky versus real defect first
  • four common causes: timing, concurrency, external deps, shared state
  • run fifty times and record the failure distribution
  • propose a fix direction, not a weakened assertion