Threats to validity: what would break it

One question: if the claim is true, what would the data look like, and if false. If you cannot tell them apart, there is no design.

Task: The paper is drafted and the weak points need finding.

[Before you start, do three things]
1. Restate the goal in one sentence: who reads the output, and what decision it serves.
2. If I have not given you the following, ask me. Do not assume:
   - design and sample
   - measurement
   - analysis
   - scope of generalisation
3. Say how you plan to do it (at most 5 steps) and wait for my go-ahead.

[Go through each of these]
- two threats each for internal, external, construct, statistical
- name the most damaging
- one supplementary test that could be run
- how to disclose it honestly

[Process]
1. State the question as something falsifiable
2. Name the observation that separates true from false
3. Check whether sample, measurement and control can support it
4. List three things that could overturn it
5. State the scope

[Output]
Question, criterion, method, counter-evidence, scope.

[Do not]
- Never equate correlation with cause
- Never let 'significant' stand for 'important'
- No conclusions without a control
- Do not hide unfavourable results