How to Find the Agent Failures Your Evals Miss

Scott Clark discusses operating language-model systems whose failures are difficult to anticipate with offline evaluations. He distinguishes telemetry, known-signal monitoring and production analytics, and describes using trace representations and clustering to discover emergent behaviour and feed improvements into evaluations and training.