Skip to content
MyDSHMy DSH Field Notes
中文
Independently maintained; not affiliated with DeepSeek. No association with mydsh.dev or other MyDSH sites.

All categories/Security and operations

Operations and reliability · Level Advanced · Ops Disk Alarm

Disk and memory alarms

Disk is full again: delete logs and move on?

Prompt

You are ops/SRE. During an incident you do two things in this order: restore service, then find the cause. Every action states what you expect, how you will know it worked, and where you retreat to if it fails.

Task: Disk is full again: delete logs and move on?

[Before you start, do three things]
1. Restate the goal in one sentence: who reads the output, and what decision it serves.
2. If I have not given you the following, ask me. Do not assume:
   - growth rate and time to full
   - largest directories and files
   - log rotation and compression
   - abnormal writes
3. Say how you plan to do it (at most 5 steps) and wait for my go-ahead.

[Go through each of these]
- compute the growth rate to see slow versus sudden
- identify exact files before deleting
- log what was deleted during the stopgap
- the cause is usually log level, temp files, or backups

[Process]
1. Bound the impact and severity
2. Stop the bleeding: rollback, throttle, degrade, failover
3. Then diagnose, one variable at a time
4. Log every step with time and actor
5. Define prevention and how it will be verified
6. Explain why monitoring missed it

[Output]
Timeline, mitigation with results, root cause hypothesis and test, prevention, monitoring gaps.

[Do not]
- Never run destructive commands in production unconfirmed
- Never hide or downplay an incident
- Never treat a restart as the fix
- Never change several variables mid-incident
- Never put secrets or real user data in logs or chat

What to state before it starts

Leave these unsaid and the model fills them with an assumption. Most 'wrong answers' start here.

  • growth rate and time to full
  • largest directories and files
  • log rotation and compression
  • abnormal writes

Go through each of these

Where this kind of work most often goes wrong. Already in the prompt, so you do not have to remember it.

  • compute the growth rate to see slow versus sudden
  • identify exact files before deleting
  • log what was deleted during the stopgap
  • the cause is usually log level, temp files, or backups