47 min
Seminar, Hosted by Simons Institute for the Theory of Computing
Understanding Language Model Hallucinations Through Parametric Knowledge Integration
UC Riverside
Recording
Abstract
Language model hallucinations arise not only from gaps in knowledge but also from failures in how models retrieve, select and integrate information. This talk examines these failures from two complementary perspectives. It first presents a mechanistic account of non-factual hallucinations, identifying distinct failures in knowledge enrichment and answer extraction within the model's factual recall process. It then extends the analysis to long-context language models, examining how parametric knowledge interacts, and sometimes conflicts, with information provided in context. Stronger contextual retrieval does not necessarily lead to more reliable knowledge use and may even interfere with parametric recall. Together, these studies characterize hallucination as a failure of knowledge coordination, with implications for model interpretability, evaluation, mitigation and design.
Topics
- hallucination
- Language Model Hallucinations, broader topic: hallucinations
- Parametric Knowledge, search World Wide
- knowledge enrichment, search World Wide
- answer extraction, search World Wide
- factual recall, search World Wide
- long-context language models, broader topic: language models
- contextual retrieval, broader topic: retrieval
- knowledge coordination, broader topic: coordination
- model interpretability