Pick the pattern before the model
“Smarter search,” “draft replies,” and “flag anomalies” are product jobs. Match the pattern to the job so you don’t overbuild.
| Pattern | Best when | Watch for |
|---|---|---|
| RAG / retrieval | Answers must cite your docs or tickets | Chunking quality and stale sources |
| Classification | Stable labels exist (spam, urgency, topic) | Drift when categories change |
| Assistive generation | Humans edit before send | Hallucinated facts without citations |
| Custom training | Narrow domain, high volume, clear labels | Cost, latency, and ownership of retrains |
Start from the user job
Define what “good” looks like and what happens when the model is wrong—before you wire an API key into the product.
- Write the job in one sentence and who reviews the output.
- Pick a success metric operators can see (time saved, precision on a sample, escalation rate).
- Design a non-ML fallback so the feature can fail closed.
Prefer managed models + your data layer
RAG over your docs, classification on labeled tickets, and assistive generation with citations usually beat custom training for early releases.
Ship with evals and kill switches
Golden sets, offline scores, and a way to disable the feature beat silent regressions. Operators need visibility when the model is wrong.
- Build a golden set from real messy examples, not only demos.
- Score offline before every prompt or retrieval change.
- Log samples for review; keep PII out of marketing and shared channels.
- Ship a flag to turn the feature off without a full deploy.
