Towards a general model of human reward-based learning
Traditional work in the study of human reward-based learning involves designing an experimental task---often inspired by Reinforcement Learning (RL) theory---and fits a small set of computational models---often inspired by RL algorithms---to that dataset. For example, researchers often model human behavior on bandit tasks using variants of Q-learning. While this approach has been highly productive, leading to landmark discoveries such as the dopamine reward prediction error hypothesis, it also has limitations. This talk focuses on the lack of generalizability of such models: Even if they closely fit behavior on the original task, models derived from the one-task-one-model paradigm usually predict behavior on other tasks quite poorly. I argue that this lack of generalizability is a fundamental problem for the cognitive sciences: we intuitively expect our models to be robust to superficial task differences, such as variations in the number of choice options, reward probabilities, or the exact kind of non-stationarity. I will propose potential solutions to this problem along two dimensions: the behavioral dataset and the computational model. Regarding computational models, I will introduce work in which we moved beyond the limitations of hand-crafted one-off models by employing flexible, data-driven methods. These methods allowed us to compare classes of models instead of individual model instances, allowing us to cover the space of possible models more exhaustively, and innovate cognitive mechanisms very efficiently. For the behavioral dataset, we move from using single learning tasks to a comprehensive task space that encompasses most existing paradigms in the literature, while closing the gaps between them in a near-continuous fashion. Our results suggest that more general models in conjunction with broader datasets can pave the road toward increasingly general models of human reward-based learning and decision making, and a persistent departure from many aspects of RL theory. Presented in the van Vreeswijk Theoretical Neuroscience Seminar series (formerly WWTNS) on 2026-05-20. Recording duration: 00:51:19.
Decision and Behavior
This webinar addressed computational perspectives on how animals and humans make decisions, spanning normative, descriptive, and mechanistic models. Sam Gershman (Harvard) presented a capacity-limited reinforcement learning framework in which policies are compressed under an information bottleneck constraint. This approach predicts pervasive perseveration, stimulus‐independent “default” actions, and trade-offs between complexity and reward. Such policy compression reconciles observed action stochasticity and response time patterns with an optimal balance between learning capacity and performance. Jonathan Pillow (Princeton) discussed flexible descriptive models for tracking time-varying policies in animals. He introduced dynamic Generalized Linear Models (Sidetrack) and hidden Markov models (GLM-HMMs) that capture day-to-day and trial-to-trial fluctuations in choice behavior, including abrupt switches between “engaged” and “disengaged” states. These models provide new insights into how animals’ strategies evolve under learning. Finally, Kenji Doya (OIST) highlighted the importance of unifying reinforcement learning with Bayesian inference, exploring how cortical-basal ganglia networks might implement model-based and model-free strategies. He also described Japan’s Brain/MINDS 2.0 and Digital Brain initiatives, aiming to integrate multimodal data and computational principles into cohesive “digital brains.”
Connectome-based models of neurodegenerative disease
Neurodegenerative diseases involve accumulation of aberrant proteins in the brain, leading to brain damage and progressive cognitive and behavioral dysfunction. Many gaps exist in our understanding of how these diseases initiate and how they progress through the brain. However, evidence has accumulated supporting the hypothesis that aberrant proteins can be transported using the brain’s intrinsic network architecture — in other words, using the brain’s natural communication pathways. This theory forms the basis of connectome-based computational models, which combine real human data and theoretical disease mechanisms to simulate the progression of neurodegenerative diseases through the brain. In this talk, I will first review work leading to the development of connectome-based models, and work from my lab and others that have used these models to test hypothetical modes of disease progression. Second, I will discuss the future and potential of connectome-based models to achieve clinically useful individual-level predictions, as well as to generate novel biological insights into disease progression. Along the way, I will highlight recent work by my lab and others that is already moving the needle toward these lofty goals.
Modeling the Navigational Circuitry of the Fly
Navigation requires orienting oneself relative to landmarks in the environment, evaluating relevant sensory data, remembering goals, and convert all this information into motor commands that direct locomotion. I will present models, highly constrained by connectomic, physiological and behavioral data, for how these functions are accomplished in the fly brain.
Computational models of spinal locomotor circuitry
To effectively move in complex and changing environments, animals must control locomotor speed and gait, while precisely coordinating and adapting limb movements to the terrain. The underlying neuronal control is facilitated by circuits in the spinal cord, which integrate supraspinal commands and afferent feedback signals to produce coordinated rhythmic muscle activations necessary for stable locomotion. I will present a series of computational models investigating dynamics of central neuronal interactions as well as a neuromechanical model that integrates neuronal circuits with a model of the musculoskeletal system. These models closely reproduce speed-dependent gait expression and experimentally observed changes following manipulation of multiple classes of genetically-identified neuronal populations. I will discuss the utility of these models in providing experimentally testable predictions for future studies.
Computational models and experimental methods for the human cornea
The eye is a multi-component biological system, where mechanics, optics, transport phenomena and chemical reactions are strictly interlaced, characterized by the typical bio-variability in sizes and material properties. The eye’s response to external action is patient-specific and it can be predicted only by a customized approach, that accounts for the multiple physics and for the intrinsic microstructure of the tissues, developed with the aid of forefront means of computational biomechanics. Our activity in the last years has been devoted to the development of a comprehensive model of the cornea that aims at being entirely patient-specific. While the geometrical aspects are fully under control, given the sophisticated diagnostic machinery able to provide a fully three-dimensional images of the eye, the major difficulties are related to the characterization of the tissues, which require the setup of in-vivo tests to complement the well documented results of in-vitro tests. The interpretation of in-vivo tests is very complex, since the entire structure of the eye is involved and the characterization of the single tissue is not trivial. The availability of micromechanical models constructed from detailed images of the eye represents an important support for the characterization of the corneal tissues, especially in the case of pathologic conditions. In this presentation I will provide an overview of the research developed in our group in terms of computational models and experimental approaches developed for the human cornea.
Decision Making and the Brain
In this talk, we will examine human behavior from the perspective of the choices we make every day. We will study the role of the brain in enabling these decisions and discuss some simple computational models of decision making and the neural basis. Towards the end, we will have a short, interactive session to engage in some easy decisions that will help us discover our own biases.
Why would we need Cognitive Science to develop better Collaborative Robots and AI Systems?
While classical industrial robots are mostly designed for repetitive tasks, assistive robots will be challenged by a variety of different tasks in close contact with humans. Hereby, learning through the direct interaction with humans provides a potentially powerful tool for an assistive robot to acquire new skills and to incorporate prior human knowledge during the exploration of novel tasks. Moreover, an intuitive interactive teaching process may allow non-programming experts to contribute to robotic skill learning and may help to increase acceptance of robotic systems in shared workspaces and everyday life. In this talk, I will discuss recent research I did on interactive robot skill learning and the remaining challenges on the route to human-centered teaching of assistive robots. In particular, I will also discuss potential connections and overlap with cognitive science. The presented work covers learning a library of probabilistic movement primitives from human demonstrations, intention aware adaptation of learned skills in shared workspaces, and multi-channel interactive reinforcement learning for sequential tasks.
Homeostatic structural plasticity of neuronal connectivity triggered by optogenetic stimulation
Ever since Bliss and Lømo discovered the phenomenon of long-term potentiation (LTP) in rabbit dentate gyrus in the 1960s, Hebb’s rule—neurons that fire together wire together—gained popularity to explain learning and memory. Accumulating evidence, however, suggests that neural activity is homeostatically regulated. Homeostatic mechanisms are mostly interpreted to stabilize network dynamics. However, recent theoretical work has shown that linking the activity of a neuron to its connectivity within the network provides a robust alternative implementation of Hebb’s rule, although entirely based on negative feedback. In this setting, both natural and artificial stimulation of neurons can robustly trigger network rewiring. We used computational models of plastic networks to simulate the complex temporal dynamics of network rewiring in response to external stimuli. In parallel, we performed optogenetic stimulation experiments in the mouse anterior cingulate cortex (ACC) and subsequently analyzed the temporal profile of morphological changes in the stimulated tissue. Our results suggest that the new theoretical framework combining neural activity homeostasis and structural plasticity provides a consistent explanation of our experimental observations.
Computational Models of Fine-Detail and Categorical Information in Visual Working Memory: Unified or Separable Representations?
When we remember a stimulus we rarely maintain a full fidelity representation of the observed item. Our working memory instead maintains a mixture of the observed feature values and categorical/gist information. I will discuss evidence from computational models supporting a mix of categorical and fine-detail information in working memory. Having established the need for two memory formats in working memory, I will discuss whether categorical and fine-detailed information for a stimulus are represented separately or as a single unified representation. Computational models of these two potential cognitive structures make differing predictions about the pattern of responses in visual working memory recall tests. The present study required participants to remember the orientation of stimuli for later reproduction. The pattern of responses are used to test the competing representational structures and to quantify the relative amount of fine-detailed and categorical information maintained. The effects of set size, encoding time, serial order, and response order on memory precision, categorical information, and guessing rates are also explored. (This is a 60 min talk).
Space and its computational challenges
How our senses work both separately and together involves rich computational problems. I will discuss the spatial and representational problems faced by the visual and auditory system, focusing on two issues. 1. How does the brain correct for discrepancies in the visual and auditory spatial reference frames? I will describe our recent discovery of a novel type of otoacoustic emission, the eye movement related eardrum oscillation, or EMREO (Gruters et al, PNAS 2018). 2. How does the brain encode more than one stimulus at a time? I will discuss evidence for neural time-division multiplexing, in which neural activity fluctuates across time to allow representations to encode more than one simultaneous stimulus (Caruso et al, Nat Comm 2018). These findings all emerged from experimentally testing computational models regarding spatial representations and their transformations within and across sensory pathways. Further, they speak to several general problems confronting modern neuroscience such as the hierarchical organization of brain pathways and limits on perceptual/cognitive processing.
Seeing things clearly: Image understanding through hard-attention and reasoning with structured knowledges
In this talk, Jonathan aims to frame the current challenges of explainability and understanding in ML-driven approaches to image processing, and their potential solution through explicit inference techniques.
Microorganism locomotion in viscoelastic fluids
Many microorganisms and cells function in complex (non-Newtonian) fluids, which are mixtures of different materials and exhibit both viscous and elastic stresses. For example, mammalian sperm swim through cervical mucus on their journey through the female reproductive tract, and they must penetrate the viscoelastic gel outside the ovum to fertilize. In micro-scale swimming the dynamics emerge from the coupled interactions between the complex rheology of the surrounding media and the passive and active body dynamics of the swimmer. We use computational models of swimmers in viscoelastic fluids to investigate and provide mechanistic explanations for emergent swimming behaviors. I will discuss how flexible filaments (such as flagella) can store energy from a viscoelastic fluid to gain stroke boosts due to fluid elasticity. I will also describe 3D simulations of model organisms such as C. Reinhardtii and mammalian sperm, where we use experimentally measured stroke data to separate naturally coupled stroke and fluid effects. We explore why strokes that are adapted to Newtonian fluid environments might not do well in viscoelastic environments.
Computational psychophysics at the intersection of theory, data and models
Behavioural measurements are often overlooked by computational neuroscientists, who prefer to focus on electrophysiological recordings or neuroimaging data. This attitude is largely due to perceived lack of depth/richness in relation to behavioural datasets. I will show how contemporary psychophysics can deliver extremely rich and highly constraining datasets that naturally interface with computational modelling. More specifically, I will demonstrate how psychophysics can be used to guide/constrain/refine computational models, and how models can be exploited to design/motivate/interpret psychophysical experiments. Examples will span a wide range of topics (from feature detection to natural scene understanding) and methodologies (from cascade models to deep learning architectures).
Inferring brain-wide interactions using data-constrained recurrent neural network models
Behavior arises from the coordinated activity of numerous distinct brain regions. Modern experimental tools allow access to neural populations brain-wide, yet understanding such large-scale datasets necessitates scalable computational models to extract meaningful features of inter-region communication. In this talk, I will introduce Current-Based Decomposition (CURBD), an approach for inferring multi-region interactions using data-constrained recurrent neural network models. I will first show that CURBD accurately isolates inter-region currents in simulated networks with known dynamics. I will then apply CURBD to understand the brain-wide flow of information leading to behavioral state transitions in larval zebrafish. These examples will establish CURBD as a flexible, scalable framework to infer brain-wide interactions that are inaccessible from experimental measurements alone.
Time perception: how our judgment of time is influenced by the regularity and change in stimulus distribution?
To organize various experiences in a coherent mental representation, we need to properly estimate the duration and temporal order of different events. Yet, our perception of time is noisy and vulnerable to various illusions. Studying these illusions can elucidate the mechanism by which the brain perceives time. In this talk, I will review a few studies on how the brain perceives duration of events and the temporal order between self-generated motion and sensory feedback. Combined with computational models at different levels, these experiments illustrated that the brain incorporates the prior knowledge of the statistical distribution of the duration of stimuli and the decay of memory when estimating duration of an individual event, and adjusts its perception of temporal order to changes in the statistics of the environment.
“Models for Liquid-liquid Phase Separation of Intrinsically Disordered Proteins”
Intrinsically disordered proteins (IDPs), lack of a well-defined folded structure, have been recently shown to be critical to forming membrane-less organelles via liquid-liquid phase separation (LLPS). Due to the flexible conformations of IDPs, it could be challenging to investigate IDPs with solely experimental techniques. Computational models can therefore provide complementary views at several aspects, including the fundamental physics underlying LLPS and the sequence determinants contributing to LLPS. In this presentation, I will start with our coarse-grained computational framework that can help generate sequence dependent phase diagrams. The coarse-grained model further led to the development of a polymer model with empirical parameters to quickly predict LLPS of IDPs. At last, I will show our preliminary efforts on addressing molecular interactions within LLPS of IDPs using all-atom explicit-solvent simulations.
Computational models of neural development
Unlike even the most sophisticated current forms of artificial intelligence, developing biological organisms must build their neural hardware from scratch. Furthermore they must start to evade predators and find food before this construction process is complete. I will discuss an interdisciplinary program of mathematical and experimental work which addresses some of the computational principles underlying neural development. This includes (i) how growing axons navigate to their targets by detecting and responding to molecular cues in their environment, (ii) the formation of maps in the visual cortex and how these are influenced by visual experience, and (iii) how patterns of neural activity in the zebrafish brain develop to facilitate precisely targeted hunting behaviour. Together this work contributes to our understanding of both normal neural development and the etiology of neurodevelopmental disorders.
Towards hybrid models of retinal circuits - integrating biophysical realism, anatomical constraints and predictive performance
Visual processing in the retina has been studied in great detail at all levels such that a comprehensive picture of the retina's cell types and the many neural circuits they form is emerging. However, the currently best performing models of retinal function are black-box CNN models which are agnostic to such biological knowledge. Here, I present two of our recent attempts to develop computational models of processing in the inner retina, which both respect biophysical and anatomical constraints yet provide accurate predictions of retinal activity
Computational Models of Large-Scale Brain Networks - Dynamics & Function
Theoretical and computational models of neural systems have been traditionally focused on small neural circuits, given the lack of reliable data on large-scale brain structures. The situation has started to change in recent years, with novel recording technologies and large organized efforts to describe the brain at a larger scale. In this talk, Professor Mejias from the University of Amsterdam will review his recent work on developing anatomically constrained computational models of large-scale cortical networks of monkeys, and how this approach can help to answer important questions in large-scale neuroscience. He will focus on three main aspects: (i) the emergence of functional interactions in different frequency regimes, (ii) the role of balance for efficient large-scale communication, and (iii) new paradigms of brain function, such as working memory, in large-scale networks.