Topic: language-model systems

Scott Clark discusses operating language-model systems whose failures are difficult to anticipate with offline evaluations. He distinguishes telemetry, known-signal monitoring and production analytics, and describes using trace representations and clustering to discover emergent behaviour and feed improvements into evaluations and training.

Sebastian Gehrmann explains why retrieval augmentation can introduce or worsen safety problems in language-model systems. The discussion covers unsafe-output examples, evaluation approaches and possible mechanisms, then considers domain-specific risks in financial services, governance, prompt design and layered mitigation strategies.