Finding Meaning in Memories
Daphna Shohamy and Ashok Litwin-Kumar bridge experimental and computational neuroscience to explain how the brain assigns significance to memories, how dopamine shapes memory-guided decisions, and how bodily signals influence learning. Isabel Low moderates a public discussion and question session.
How to Improve Motivation & Overcome Procrastination | Dr. Masud Husain
Masud Husain explains the neuroscience of motivation, dopamine, effort, and reward, together with research on apathy, dementia, Parkinson's disease, Alzheimer's disease, and cognitive resilience.
What is So Interesting About Reinforcement Learning?
This talk aims to answer these questions along four dimensions. First is history. RL was the basis of AI long before the term AI was introduced in 1956. The first machine learning (ML) systems were based on RL even before digital computers existed. Despite notable early successes of ML based on RL, RL essentially disappeared from ML until relatively recently. A second reason for renewed interest in RL is the clarification of some misunderstandings that have been prevalent in the ML community. A third, and most important, reason for this resurgence is that new, or rediscovered, algorithms and connections to well developed mathematical and engineering methods have been worked out. Finally, a fourth reason for the renewed interest in RL is its strong links to animal reward systems, in particular, to the role that dopamine plays in motivation and learning. VVTNS Sixth Season Opening Lecture. Presented in the van Vreeswijk Theoretical Neuroscience Seminar series (formerly WWTNS) on 2025-10-29. Recording duration: 00:55:21.
Understanding reward-guided learning using large-scale datasets
Understanding the neural mechanisms of reward-guided learning is a long-standing goal of computational neuroscience. Recent methodological innovations enable us to collect ever larger neural and behavioral datasets. This presents opportunities to achieve greater understanding of learning in the brain at scale, as well as methodological challenges. In the first part of the talk, I will discuss our recent insights into the mechanisms by which zebra finch songbirds learn to sing. Dopamine has been long thought to guide reward-based trial-and-error learning by encoding reward prediction errors. However, it is unknown whether the learning of natural behaviours, such as developmental vocal learning, occurs through dopamine-based reinforcement. Longitudinal recordings of dopamine and bird songs reveal that dopamine activity is indeed consistent with encoding a reward prediction error during naturalistic learning. In the second part of the talk, I will talk about recent work we are doing at DeepMind to develop tools for automatically discovering interpretable models of behavior directly from animal choice data. Our method, dubbed CogFunSearch, uses LLMs within an evolutionary search process in order to "discover" novel models in the form of Python programs that excel at accurately predicting animal behavior during reward-guided learning. The discovered programs reveal novel patterns of learning and choice behavior that update our understanding of how the brain solves reinforcement learning problems.
Understanding reward-guided learning using large-scale datasets
Understanding the neural mechanisms of reward-guided learning is a long-standing goal of computational neuroscience. Recent methodological innovations enable us to collect ever larger neural and behavioral datasets. This presents opportunities to achieve greater understanding of learning in the brain at scale, as well as methodological challenges. In the first part of the talk, I will discuss our recent insights into the mechanisms by which zebra finch songbirds learn to sing. Dopamine has been long thought to guide reward-based trial-and-error learning by encoding reward prediction errors. However, it is unknown whether the learning of natural behaviours, such as developmental vocal learning, occurs through dopamine-based reinforcement. Longitudinal recordings of dopamine and bird songs reveal that dopamine activity is indeed consistent with encoding a reward prediction error during naturalistic learning. In the second part of the talk, I will talk about recent work we are doing at DeepMind to develop tools for automatically discovering interpretable models of behavior directly from animal choice data. Our method, dubbed CogFunSearch, uses LLMs within an evolutionary search process in order to "discover" novel models in the form of Python programs that excel at accurately predicting animal behavior during reward-guided learning. The discovered programs reveal novel patterns of learning and choice behavior that update our understanding of how the brain solves reinforcement learning problems.
A Geometric Approach for the Study of Functional Connectivity Dynamics
Functional connectivity has been the focus of many research groups aiming to study the interaction between cells and brain regions. A standard method for analyzing connectivity is to statistically compare pairwise interactions between cells or brain regions across behavioral states or conditions. This methodology ignores the intrinsic properties of functional connectivity as a multivariate and dynamic signal, expressing the correlational configuration of the network. In this talk, I will present a geometric approach, combining Graph Theory and Riemannian Geometry to build "a graph of graphs" and extract the latent dynamics of the overall correlational structure. Using this approach, we formulate the statistical relations between network dynamics and spontaneous behavior as a second-order Taylor’s expansion. Our analysis shows that fast fluctuations in functional connectivity of large-scale cortical networks are closely linked to variations in behavioral metrics related to the arousal state. We further expand this methodology to longer time scales to study the effect of dopamine on network dynamics in the primary motor cortex (M1) during learning. We developed a series of analysis methods indicating that as animals learn to perform a motor task, the network of pyramidal neurons in layer 2-3 gradually and monotonically reorganizes toward an "expert" configuration. Our results highlight the critical role of dopamine in driving synaptic plasticity: Blocking dopaminergic neurotransmission locally in M1 prevented motor learning at the behavioral level and concomitantly halted plasticity changes in network activity and in functional connectivity. Presented in the van Vreeswijk Theoretical Neuroscience Seminar series (formerly WWTNS) on 2025-01-29. Recording duration: 00:27:08.
The Neurobiology of the Addicted Brain
Personalized medicine and predictive health and wellness: Adding the chemical component
Wearable sensors that detect and quantify biomarkers in retrievable biofluids (e.g., interstitial fluid, sweat, tears) provide information on human dynamic physiological and psychological states. This information can transform health and wellness by providing actionable feedback. Due to outdated and insufficiently sensitive technologies, current on-body sensing systems have capabilities limited to pH, and a few high-concentration electrolytes, metabolites, and nutrients. As such, wearable sensing systems cannot detect key low-concentration biomarkers indicative of stress, inflammation, metabolic, and reproductive status. We are revolutionizing sensing. Our electronic biosensors detect virtually any signaling molecule or metabolite at ultra-low levels. We have monitored serotonin, dopamine, cortisol, phenylalanine, estradiol, progesterone, and glucose in blood, sweat, interstitial fluid, and tears. The sensors are based on modern nanoscale semiconductor transistors that are straightforwardly scalable for manufacturing. We are developing sensors for >40 biomarkers for personalized continuous monitoring (e.g., smartwatch, wearable patch) that will provide feedback for treating chronic health conditions (e.g., perimenopause, stress disorders, phenylketonuria). Moreover, our sensors will enable female fertility monitoring and the adoption of more healthy lifestyles to prevent disease and improve physical and cognitive performance.
Social and non-social learning: Common, or specialised, mechanisms? (BACN Early Career Prize Lecture 2022)
The last decade has seen a burgeoning interest in studying the neural and computational mechanisms that underpin social learning (learning from others). Many findings support the view that learning from other people is underpinned by the same, ‘domain-general’, mechanisms underpinning learning from non-social stimuli. Despite this, the idea that humans possess social-specific learning mechanisms - adaptive specializations moulded by natural selection to cope with the pressures of group living - persists. In this talk I explore the persistence of this idea. First, I present dissociations between social and non-social learning - patterns of data which are difficult to explain under the domain-general thesis and which therefore support the idea that we have evolved special mechanisms for social learning. Subsequently, I argue that most studies that have dissociated social and non-social learning have employed paradigms in which social information comprises a secondary, additional, source of information that can be used to supplement learning from non-social stimuli. Thus, in most extant paradigms, social and non-social learning differ both in terms of social nature (social or non-social) and status (primary or secondary). I conclude that status is an important driver of apparent differences between social and non-social learning. When we account for differences in status, we see that social and non-social learning share common (dopamine-mediated) mechanisms.
Light-driven dopamine release in the adult and developing retina
Obesity and Brain – Bidirectional Influences
The regulation of body weight relies on homeostatic mechanisms that use a combination of internal signals and external cues to initiate and terminate food intake. Homeostasis depends on intricate communication between the body and the hypothalamus involving numerous neural and hormonal signals. However, there is growing evidence that higher-level cognitive function may also influence energy balance. For instance, research has shown that BMI is consistently linked to various brain, cognitive, and personality measures, implicating executive, reward, and attentional systems. Moreover, the rise in obesity rates over the past half-century is attributed to the affordability and widespread availability of highly processed foods, a phenomenon that contradicts the idea that food intake is solely regulated by homeostasis. I will suggest that prefrontal systems involved in value computation and motivation act to limit food overconsumption when food is scarce or expensive, but promote over-eating when food is abundant, an optimum strategy from an economic standpoint. I will review the genetic and neuroscience literature on the CNS control of body weight. I will present recent studies supporting a role of prefrontal systems in weight control. I will also present contradictory evidence showing that frontal executive and cognitive findings in obesity may be a consequence not a cause of increased hunger. Finally I will review the effects of obesity on brain anatomy and function. Chronic adiposity leads to cerebrovascular dysfunction, cortical thinning, and cognitive impairment. As the most common preventable risk factor for dementia, obesity poses a significant threat to brain health. I will conclude by reviewing evidence for treatment of obesity in adults to prevent brain disease.
Hallucinating mice, dopamine and immunity; towards mechanistic treatment targets for psychosis
Hallucinations are a core symptom of psychotic disorders and have traditionally been difficult to study biologically. We developed a new behavioral computational approach to measure hallucinations-like perception in humans and mice alike. Using targeted neural circuit manipulations, we identified a causal role for striatal dopamine in mediating hallucination-like perception. Building on this, we currently investigate the neural and immunological upstream regulators of these dopaminergic circuits with the goal to identify new biological treatment targets for psychosis
Integration of 3D human stem cell models derived from post-mortem tissue and statistical genomics to guide schizophrenia therapeutic development
Schizophrenia is a neuropsychiatric disorder characterized by positive symptoms (such as hallucinations and delusions), negative symptoms (such as avolition and withdrawal) and cognitive dysfunction1. Schizophrenia is highly heritable, and genetic studies are playing a pivotal role in identifying potential biomarkers and causal disease mechanisms with the hope of informing new treatments. Genome-wide association studies (GWAS) identified nearly 270 loci with a high statistical association with schizophrenia risk; however each locus confers only a small increase in risk therefore it is difficult to translate these findings into understanding disease biology that can lead to treatments. Induced pluripotent stem cell (iPSC) models are a tractable system to translate genetic findings and interrogate mechanisms of pathogenesis. Mounting research with patient-derived iPSCs has proposed several neurodevelopmental pathways altered in SCZ, such as neural progenitor cell (NPC) proliferation, imbalanced differentiation of excitatory and inhibitory cortical neurons. However, it is unclear what exactly these iPS models recapitulate, how potential perturbations of early brain development translates into illness in adults and how iPS models that represent fetal stages can be utilized to further drug development efforts to treat adult illness. I will present the largest transcriptome analysis of post-mortem caudate nucleus in schizophrenia where we discovered that decreased presynaptic DRD2 autoregulation is the causal dopamine risk factor for schizophrenia (Benjamin et al, Nature Neuroscience 2022 https://doi.org/10.1038/s41593-022-01182-7). We developed stem cell models from a subset of the postmortem cohort to better understand the molecular underpinnings of human psychiatric disorders (Sawada et al, Stem Cell Research 2020). We established a method for the differentiation of iPS cells into ventral forebrain organoids and performed single cell RNAseq and cellular phenotyping. To our knowledge, this is the first study to evaluate iPSC models of SZ from the same individuals with postmortem tissue. Our study establishes that striatal neurons in the patients with SCZ carry abnormalities that originated during early brain development. Differentiation of inhibitory neurons is accelerated whereas excitatory neuronal development is delayed, implicating an excitation and inhibition (E-I) imbalance during early brain development in SCZ. We found a significant overlap of genes upregulated in the inhibitory neurons in SCZ organoids with upregulated genes in postmortem caudate tissues from patients with SCZ compared with control individuals, including the donors of our iPS cell cohort. Altogether, we demonstrate that ventral forebrain organoids derived from postmortem tissue of individuals with schizophrenia recapitulate perturbed striatal gene expression dynamics of the donors’ brains (Sawada et al, biorxiv 2022 https://doi.org/10.1101/2022.05.26.493589).
Classification of Dopamine Cells
Chemistry of the adaptive mind: lessons from dopamine
The human brain faces a variety of computational dilemmas, including the flexibility/stability, the speed/accuracy and the labor/leisure tradeoff. I will argue that striatal dopamine is particularly well suited to dynamically regulate these computational tradeoffs depending on constantly changing task demands. This working hypothesis is grounded in evidence from recent studies on learning, motivation and cognitive control in human volunteers, using chemical PET, psychopharmacology, and/or fMRI. These studies also begin to elucidate the mechanisms underlying the huge variability in catecholaminergic drug effects across different individuals and across different task contexts. For example, I will demonstrate how effects of the most commonly used psychostimulant methylphenidate on learning, Pavlovian and effortful instrumental control depend on fluctuations in current environmental volatility, on individual differences in working memory capacity and on opportunity cost respectively.
Dyskinesia: the failure of dopamine-dependent motor control
Learning in/about/from the basal ganglia
The basal ganglia are a collection of brain areas that are connected by a variety of synaptic pathways and are a site of significant reward-related dopamine release. These properties suggest a possible role for the basal ganglia in action selection, guided by reinforcement learning. In this talk, I will discuss a framework for how this function might be performed and computational results using an upward mapping to identify putative low-dimensional control ensembles that may be involved in tuning decision policy. I will also present some recent experimental results and theory – related to effects of extracellular ion dynamics -- that run counter to the classical view of basal ganglia pathways and suggest a new interpretation of certain aspects of this framework. For those not so interested in the basal ganglia, I hope that the upward mapping approach and impact of extracellular ion dynamics will nonetheless be of interest!
Mechanisms and Roles of Fast Dopamine Signaling
Dopamine is a neuromodulator that codes information on various time scales. I will discuss recent progress on the identification of fast release mechanisms for dopamine in the mouse striatum. I will present data on triggering mechanisms of dopamine release and evaluate its roles in striatal regulation. In the long-term, our work will allow for a better understanding of the mechanisms and time scales of dopamine coding in health and disease.
Brain and behavioural impacts of early life adversity
Abuse, neglect, and other forms of uncontrollable stress during childhood and early adolescence can lead to adverse outcomes later in life, including especially perturbations in the regulation of mood and emotional states, and specifically anxiety disorders and depression. However, stress experiences vary from one individual to the next, meaning that causal relationships and mechanistic accounts are often difficult to establish in humans. This interdisciplinary talk considers the value of research in experimental animals where stressor experiences can be tightly controlled and detailed investigations of molecular, cellular, and circuit-level mechanisms can be carried out. The talk will focus on the widely used repeated maternal separation procedure in rats where rat offspring are repeatedly separated from maternal care during early postnatal life. This early life stress has remarkably persistent effects on behaviour with a general recognition that maternally-deprived animals are susceptible to depressive-like phenotypes. The validity of this conclusion will be critically appraised with convergent insights from a recent longitudinal study in maternally separated rats involving translational brain imaging, transcriptomics, and behavioural assessment.
Inter-individual variability in reward seeking and decision making: role of social life and consequence for vulnerability to nicotine
Inter-individual variability refers to differences in the expression of behaviors between members of a population. For instance, some individuals take greater risks, are more attracted to immediate gains or are more susceptible to drugs of abuse than others. To probe the neural bases of inter-individual variability we study reward seeking and decision-making in mice, and dissect the specific role of dopamine in the modulation of these behaviors. Using a spatial version of the multi-armed bandit task, in which mice are faced with consecutive binary choices, we could link modifications of midbrain dopamine cell dynamics with modulation of exploratory behaviors, a major component of individual characteristics in mice. By analyzing mouse behaviors in semi-naturalistic environments, we then explored the role of social relationships in the shaping of dopamine activity and associated beahviors. I will present recent data from the laboratory suggesting that changes in the activity of dopaminergic networks link social influences with variations in the expression of non-social behaviors: by acting on the dopamine system, the social context may indeed affect the capacity of individuals to make decisions, as well as their vulnerability to drugs of abuse, in particular nicotine.
Dynamic dopaminergic signaling probabilistically controls the timing of self-timed movements
Human movement disorders and pharmacological studies have long suggested molecular dopamine modulates the pace of the internal clock. But how does the endogenous dopaminergic system influence the timing of our movements? We examined the relationship between dopaminergic signaling and the timing of reward-related, self-timed movements in mice. Animals were trained to initiate licking after a self-timed interval following a start cue; reward was delivered if the animal’s first lick fell within a rewarded window (3.3-7 s). The first-lick timing distributions exhibited the scalar property, and we leveraged the considerable variability in these distributions to determine how the activity of the dopaminergic system related to the animals’ timing. Surprisingly, dopaminergic signals ramped-up over seconds between the start-timing cue and the self-timed movement, with variable dynamics that predicted the movement/reward time, even on single trials. Steeply rising signals preceded early initiation, whereas slowly rising signals preceded later initiation. Higher baseline signals also predicted earlier self-timed movement. Optogenetic activation of dopamine neurons during self-timing did not trigger immediate movements, but rather caused systematic early-shifting of the timing distribution, whereas inhibition caused late-shifting, as if dopaminergic manipulation modulated the moment-to-moment probability of unleashing the planned movement. Consistent with this view, the dynamics of the endogenous dopaminergic signals quantitatively predicted the moment-by-moment probability of movement initiation. We conclude that ramping dopaminergic signals, potentially encoding dynamic reward expectation, probabilistically modulate the moment-by-moment decision of when to move. (Based on work from Hamilos et al., eLife, 2021).
A role for dopamine in value-free learning
Recent success in training artificial agents and robots derives from a combination of direct learning of behavioral policies and indirect learning via value functions. Policy learning and value learning employ distinct algorithms that depend upon evaluation of errors in performance and reward prediction errors, respectively. In mammals, behavioral learning and the role of mesolimbic dopamine signaling have been extensively evaluated with respect to reward prediction errors; but there has been little consideration of how direct policy learning might inform our understanding. I’ll discuss our recent work on classical conditioning in naïve mice (https://www.biorxiv.org/content/10.1101/2021.05.31.446464v1) that provides multiple lines of evidence that phasic dopamine signaling regulates policy learning from performance errors in addition to its well-known roles in value learning. This work points towards new opportunities for unraveling the mechanisms of basal ganglia control over behavior under both adaptive and maladaptive learning conditions.
Dopaminergic modulation of synaptic plasticity in learning and psychiatric disorders
Transient changes in dopamine activity in response to reward and punishment have been known to regulate reward-related learning. However, the cellular basis that detects the transient dopamine signaling has long been unclear. Using two-photon microscopy and optogenetics, I have shown that transient increases and decreases of dopamine modulate plasticity of dopamine D1 and D2 receptor-expressing cells in the nucleus accumbens, respectively. At the behavioral level, I characterized that these D1 and D2 cells cooperatively tune learning by generalization and discrimination learning. Interestingly, disturbance of the dopamine signaling impaired D2 cell plasticity and discrimination learning, which was analogous to salience misattribution seen in subjects with schizophrenia.
New tools for monitoring & manipulating cellular function
Dr. Looger will discuss reagents for tracking Ca2+, membrane potential ("voltage"), glutamate, GABA, acetylcholine, serotonin, dopamine, etc. He will also cover optogenetics tools and methods for correlative light/electron microscopy. They make all tools freely available to everyone and work to get them in the hands of people that have limited resources.
Dopamine
Hallucinating mice and dopamine – towards mechanistic treatment targets for psychosis
Psychotic disorders are devastating conditions without any mechanistic treatment available. One major hurdle in the biological study of psychosis is the challenge of rigorously probing this condition in pre-clinical animal models. The goal of our research is to develop and exploit innovative frameworks for the study of psychosis in mice. In our present work, where we developed a cross-species computational psychiatry approach to probe hallucination-like perception. This enabled us to directly relate human and mouse behavior, and to demonstrate and dissect the causal role of striatal dopamine in hallucination-like perception. Our results suggest a neural circuit mechanism for the long-standing dopamine hypothesis of psychosis, and provide a new translational framework for the biological study of psychosis. This opens up exciting possibilities for advancing the biological understanding of psychosis and to identify mechanistic treatment targets.
Conflict or complement: Parallel memories control behaviour in Drosophila
Drosophila can learn to associate odours with reward or punishment and the resulting memories direct odour-specific approach or avoidance behaviours. Recent progress has revealed a straightforward model for learning in which reinforcing dopaminergic neurons assign valence to odour representations in the neural ensemble of the mushroom bodies. Dopamine directed synaptic depression alters the route of odour-driven activity through the mushroom body output network. This circuit configuration and influence of internal state guide the expression of appropriate behaviour. Importantly, learned behaviour is flexible and can be updated as the fly accumulates additional experience. Our latest studies demonstrate that well-informed behaviour is guided by combining parallel conflicting and complementary memories of opposite valence.
Human cognitive biases and the role of dopamine
Cognitive bias is a "subjective reality" that is uniquely created in the brain and affects our various behaviors. It may lead to what is widely called irrationality in behavioral economics, such as inaccurate judgment and illogical interpretation, but it also has an adaptive aspect in terms of mental hygiene. When such cognitive bias is regarded as a product of information processing in the brain, the approach to clarify the mechanism in the brain will play a part in finding the direct relations between the brain and the mind. In my talk, I will introduce our studies investigating the neural and molecular bases of cognitive biases, especially focusing on the role of dopamine.
The role of spatiotemporal waves in coordinating regional dopamine decision signals
The neurotransmitter dopamine is essential for normal reward learning and motivational arousal processes. Indeed these core functions are implicated in the major neurological and psychiatric dopamine disorders such as schizophrenia, substance abuse disorders/addiction and Parkinson's disease. Over the years, we have made significant strides in understanding the dopamine system across multiple levels of description, and I will focus on our recent advances in the computational description, and brain circuit mechanisms that facilitate the dual role of dopamine in learning and performance. I will specifically describe our recent work with imaging the activity of dopamine axons and measurements of dopamine release in mice performing various behavioural tasks. We discovered wave-like spatiotemporal activity of dopamine in the striatal region, and I will argue that this pattern of activation supports a critical computational operation; spatiotemporal credit assignment to regional striatal subexperts. Our findings provide a mechanistic description for vectorizing reward prediction error signals relayed by dopamine.
The cellular basis of Parkinson’s disease
Parkinson’s disease is affects millions of people around the world. The disease is characterized by typical movement defects that are caused by the loss of dopaminergic neurons, but several very debilitating non-motor symptoms occur more than 10 years before the motor symptoms. I will discuss how we study these non-motor symptoms including sleep disturbances and olfactory defects using large collections of knock in fruit flies that model the numerous familial forms of Parkinson’s disease as well as using human iPS cells from patients. A common emerging theme are defects in protein homeostasis that in specific neuronal cell types, cause cellular defects that explain the Parkinson-relevant phenotypes. Our work reveals the mechanisms that cause early defects in Parkinson’s disease and it opens therapeutic avenues to start tackling this disease.
Recurrent corticothalamic feedback in the auditory system: perceptual salience and dopaminergic modulation
The Dopamine Synapse and Learning
The actions of dopamine within the striatum are central to the selection of cortical and perhaps thalamic inputs that mediate learning throughout life, including during operant conditioning, reward and avoidance learning and the establishment of motor patterns. Dysfunction of these synaptic circuits during maturation or aging underlies many neurological, psychiatric and neurodevelopment disorders. We will discuss the biological sequences by which these synapses are altered as an animal interacts with the environment.
Male songbirds turn off their self-evaluation systems when they sing to females
Attending to mistakes while practicing alone provides opportunities for learning but self-evaluation during audience-directed performance could distract from ongoing execution. It remains unknown how animals switch between practice and performance modes, and how evaluation systems process errors across distinct performance contexts. We recorded from striatal-projecting dopamine (DA) neurons as male songbirds transitioned from singing alone to singing female-directed courtship song. In the presence of the female, singing-related performance error signals were reduced or gated off and DA neurons were instead phasically activated by female vocalizations. Mesostriatal DA neurons can thus dynamically change their tuning with changes in social context.