Skip to content
MyDSHMy DSH Field Notes
中文
Independently maintained; not affiliated with DeepSeek. No association with mydsh.dev or other MyDSH sites.

All categories/Data and analysis

Data extraction and SQL · Level Advanced · SQL ETL Incremental

ETL: daily incremental logic

A daily incremental job needs idempotency and backfill.

Prompt

You state the question before choosing tables. The hard part is not syntax but definitions and edges: time ranges, deduplication, nulls, and row multiplication from joins.

Task: A daily incremental job needs idempotency and backfill.

[Before you start, do three things]
1. Restate the goal in one sentence: who reads the output, and what decision it serves.
2. If I have not given you the following, ask me. Do not assume:
   - source and update timestamp field
   - share of late-arriving data
   - how to rerun after failure
3. Say how you plan to do it (at most 5 steps) and wait for my go-ahead.

[Go through each of these]
- state the incremental key: updated_at, id, partition
- idempotent: rerunning a date must not change the result
- handle late data with a backfill window
- a monitor on daily row count

[Process]
1. State the question and the output grain in one line: what does one row mean
2. Confirm table structure, keys and cardinality before joining
3. Write the SQL with a comment on the business meaning of each filter
4. Self-check: dedup, time boundaries, nulls, denominator
5. Give a validation: which known total should match

[Output]
Approach and grain first, then the SQL, then a validation query.

[Do not]
- No SELECT *
- Do not wrap indexed columns in functions without saying so
- Never assume LEFT JOIN preserves row count
- Check the data dictionary rather than guessing a field's meaning

What to state before it starts

Leave these unsaid and the model fills them with an assumption. Most 'wrong answers' start here.

  • source and update timestamp field
  • share of late-arriving data
  • how to rerun after failure

Go through each of these

Where this kind of work most often goes wrong. Already in the prompt, so you do not have to remember it.

  • state the incremental key: updated_at, id, partition
  • idempotent: rerunning a date must not change the result
  • handle late data with a backfill window
  • a monitor on daily row count