All categories/Data and analysis
Business analysis · Level Advanced · Analysis Cleaning Data
Data cleaning: which rows can go
An export has duplicates, gaps and messy formats. I need rules, not silent deletion.
Prompt
You write the definition down before touching the data, because more than half of 'the numbers are wrong' is a definition mismatch. You never give a conclusion without a comparison.
Task: An export has duplicates, gaps and messy formats. I need rules, not silent deletion.
[Before you start, do three things]
1. Restate the goal in one sentence: who reads the output, and what decision it serves.
2. If I have not given you the following, ask me. Do not assume:
- what the data must answer
- the primary key
- business meaning of missing fields
3. Say how you plan to do it (at most 5 steps) and wait for my go-ahead.
[Go through each of these]
- start with a health check: rows, duplicates, missing rate, outliers
- each rule states what is removed, why, how many rows
- never default missing to zero; explain the effect of each option
- keep the raw copy and a cleaning log
[Process]
1. Write the definition, time range and denominator of every metric and get confirmation
2. Ask what decision this supports
3. Check data completeness and definition consistency before concluding
4. Every conclusion carries a comparison
5. End with the condition under which the conclusion fails
[Output]
Definitions, then conclusion, then limits. One sentence for the conclusion, a table for evidence.
[Do not]
- Never say rose or fell without a comparison
- Do not present correlation as cause
- Do not use 'significant' unless a test was run
- If the numbers do not reconcile, stop and say soWhat to state before it starts
Leave these unsaid and the model fills them with an assumption. Most 'wrong answers' start here.
- what the data must answer
- the primary key
- business meaning of missing fields
Go through each of these
Where this kind of work most often goes wrong. Already in the prompt, so you do not have to remember it.
- start with a health check: rows, duplicates, missing rate, outliers
- each rule states what is removed, why, how many rows
- never default missing to zero; explain the effect of each option
- keep the raw copy and a cleaning log