Topic: Emergent Behaviour

Scott Clark discusses operating language-model systems whose failures are difficult to anticipate with offline evaluations. He distinguishes telemetry, known-signal monitoring and production analytics, and describes using trace representations and clustering to discover emergent behaviour and feed improvements into evaluations and training.