skylar payneFailure taxonomy
The map of what breaks, why it breaks, and which failures are worth preventing.
Topic hub
AI Evals
Writing about reviewing AI outputs, measuring failures, and testing whether a change improves the system.
Start here
- 40% Better, 75% Faster
A case study of query sets, trace annotations, and retrieval and latency changes at Frigade.
Core concepts
Review UI
The place humans inspect traces, label failures, and create the ground truth for improvement.
Judge calibration
Comparing a model judge with human labels on held-out examples, including the mistakes it misses.
Regression sets
Examples of past failures used to check whether a later change reintroduces them.
Related writing
Open the Practical AI Evals series →-
A streamlined system for AI evaluation that closes the gap between seeing problems and fixing them.
-
How Frigade Slashed Latency & Boosted User Helpfulness
-
An AI Maturity Model
-
A Practical Guide to Evaluation-Driven Improvement