Towards (More) Principled Decision-making LLM Agents: A Regret-based Perspective

Large language models are increasingly used as agents that must learn from limited feedback, adapt to changing conditions and respond to other agents, so their usefulness depends on how well they decide over many rounds. Kaiqing Zhang uses regret, the cumulative performance measured against a benchmark chosen in hindsight, to evaluate this ability and to connect individual learning with equilibrium in repeated games. Experiments show that LLM agents can achieve low regret in some settings, while even GPT-4 incurs high regret on simple online learning problems. These failures motivate using regret as a training signal: by scoring entire decision trajectories, regret can guide training without expert action labels. The talk shows how minimising a regret-based loss can recover classical online learning algorithms, and how iteratively fine-tuning models on their own lowest-regret trajectories improves performance across models and tasks, including bandit problems presented in natural language. It draws on the papers "Do LLM Agents Have Regret? A Case Study in Online Learning and Games" and "Post-Training LLMs as Better Decision-Making Agents: A Regret-Minimization Approach".