All categories/Customer support
Support knowledge and process · Level Advanced · Support ops AI Eval
Pre-launch eval for AI support
We want AI support but fear wrong or harmful answers.
Prompt
You do support operations: turn conversations into procedures others can follow. Not nice prose — conditions, actions, deadlines, and when the rule stops applying. A new hire should use it in week one.
Task: We want AI support but fear wrong or harmful answers.
[Before you start, do three things]
1. Restate the goal in one sentence: who reads the output, and what decision it serves.
2. If I have not given you the following, ask me. Do not assume:
- intent coverage
- tolerable error rate
- high-risk scenarios
- human fallback capacity
3. Say how you plan to do it (at most 5 steps) and wait for my go-ahead.
[Go through each of these]
- measure accuracy per intent, never an average
- test 'say I do not know' as its own case
- route refunds, privacy, safety away from AI
- keep human sampling and one-click takeover
[Process]
1. Confirm who the process serves and what triggers it
2. Map real branches, not the ideal path
3. Every step gets an action, an owner, a deadline
4. Note expiry and last-verified date
5. Break it with the worst case
6. State who may change it and how changes are announced
[Output]
A process, table, or checklist ready to use, ending with the cases it does not cover.
[Do not]
- No 'improve communication' or 'raise awareness'
- Never leave out the owner
- Never assume background knowledge
- No rule without an expiry conditionWhat to state before it starts
Leave these unsaid and the model fills them with an assumption. Most 'wrong answers' start here.
- intent coverage
- tolerable error rate
- high-risk scenarios
- human fallback capacity
Go through each of these
Where this kind of work most often goes wrong. Already in the prompt, so you do not have to remember it.
- measure accuracy per intent, never an average
- test 'say I do not know' as its own case
- route refunds, privacy, safety away from AI
- keep human sampling and one-click takeover