Human-like capacity limits in working memory models result from naturalistic sensory constraints

Working memory (WM) allows us to hold information temporarily and make complex decisions beyond reflexive response to stimuli. One prominent feature of WM is its capacity limit. Despite decades of study, the root of this limit is still not well-understood. Most previous accounts for this limit assume various forms of memory constraints and make strong, often oversimplified, assumptions about sensory representations of stimuli. In this work, we built visual-cognitive neural network models of WM that process raw sensory stimuli. In contrast to intuitions that capacity limit results from memory constraints, we found that pre-training the sensory region of our models with natural images poses enough constraints on models to exhibit human-like behavior patterns across a wide range of WM capacity tasks. In change detection tasks, the detection accuracy decreases rapidly when the number of stimuli to be remembered increases. In continuous report tasks, the fidelity of the report again decreases rapidly when more stimuli are shown. In contrast, models without realistic constraints on the sensory regions produce super-human performance in these tasks. Human-like behavior cannot be restored simply by restricting the size of these models or adding processing noise. Unlike phenomenological accounts of WM capacity, our neural network models allow us to test the neural mechanisms of capacity limitation. We found that the average neural activation in our model increases and then plateaus when more stimuli are presented, and capacity limitation appears to arise in a bottom-up fashion; both are broadly consistent with previous fMRI and electrophysiological studies. Our work suggests that many phenomena about WM capacity can be explained by sensory constraints. Our models offer a fresh perspective in our understanding of the origin of the WM capacity limit and highlight the importance of building models with realistic sensory processing even when studying memory and other high-level cognitive phenomena.