Developmental emergence of personality
The Nature versus Nurture debate has generally been considered from the lens of genome versus experience dichotomy and has dominated our thinking about behavioral individuality and personality traits. In contrast, the role of nonheritable noise during brain development in behavioral variation is understudied. Using the Drosophila melanogaster visual system, I will discuss our efforts to dissect how individuality in circuit wiring emerges during development, and how that helps generate individual behavioral variation.
Insights into vision from interpreting a neuronal wiring diagram
In 2023, the FlyWire Consortium released the neuronal wiring diagram of an adult fly brain. This contains as a corollary the first complete wiring diagram of a visual system, which has been used to identify all 200+ cell types that are intrinsic to the Drosophila optic lobe. About half of these cell types were previously unknown, and less than 20% have ever been recorded by a physiologist. I will argue that plausible functions for many cell types can be guessed by interpreting the wiring diagram. VVTNS Fifth Season Closing Lecture. Presented in the van Vreeswijk Theoretical Neuroscience Seminar series (formerly WWTNS) on 2025-06-25. Recording duration: 00:42:39.
Plasticity of the adult visual system
Using ML tools in neuroscience to define optimality in complex natural behavior
Biological systems must selectively encode partial information about the environment, as dictated by the capacity constraints at work in all living organisms. For example, we cannot see every feature of the light field that reaches our eyes; temporal resolution is limited by transmission noise and delays, and spatial resolution is limited by the finite number of photoreceptors and output cells in the retina. Classical efficient coding theory describes how sensory systems can maximize information transmission given such capacity constraints, but it treats all input features equally. Not all inputs are, however, of equal value to the organism. Our work quantifies whether and how the brain selectively encodes stimulus features, specifically predictive features, that are most useful for fast and effective movements. We have shown that efficient predictive computation starts at the earliest stages of the visual system in the retina. We borrow techniques from machine learning, statistical physics, and information theory to assess how we get terrific, predictive vision from these imperfect (lagged and noisy) component parts. In broader terms, we aim to build a more complete theory of efficient encoding in the brain, and along the way have found some intriguing connections between approaches to coarse graining in biology, machine learning, and physics. Presented in the van Vreeswijk Theoretical Neuroscience Seminar series (formerly WWTNS) on 2024-06-05. Recording duration: 00:41:40.
Modelling the fruit fly brain and body
Through recent advances in microscopy, we now have an unprecedented view of the brain and body of the fruit fly Drosophila melanogaster. We now know the connectivity at single neuron resolution across the whole brain. How do we translate these new measurements into a deeper understanding of how the brain processes sensory information and produces behavior? I will describe two computational efforts to model the brain and the body of the fruit fly. First, I will describe a new modeling method which makes highly accurate predictions of neural activity in the fly visual system as measured in the living brain, using only measurements of its connectivity from a dead brain [1], joint work with Jakob Macke. Second, I will describe a whole body physics simulation of the fruit fly which can accurately reproduce its locomotion behaviors, both flight and walking [2], joint work with Google DeepMind.
Restructuring cortical feedback circuits
We hardly notice when there is a speck on our glasses, the obstructed visual information seems to be magically filled in. The mechanistic basis for this fundamental perceptual phenomenon has, however, remained obscure. What enables neurons in the visual system to respond to context when the stimulus is not available? While feedforward information drives the activity in cortex, feedback information is thought to provide contextual signals that are merely modulatory. We have made the discovery that mouse primary visual cortical neurons are strongly driven by feedback projections from higher visual areas when their feedforward sensory input from the retina is missing. This drive is so strong that it makes visual cortical neurons fire as much as if they were receiving a direct sensory input. These signals are likely used to predict input from the feedforward pathway. Preliminary results show that these feedback projections are strongly influenced by experience and learning.
Flexible codes and loci of visual working memory
Neural correlates of visual working memory have been found in early visual, parietal, and prefrontal regions. These findings have spurred fruitful debate over how and where in the brain memories might be represented. Here, I will present data from multiple experiments to demonstrate how a focus on behavioral requirements can unveil a more comprehensive understanding of the visual working memory system. Specifically, items in working memory must be maintained in a highly robust manner, resilient to interference. At the same time, storage mechanisms must preserve a high degree of flexibility in case of changing behavioral goals. Several examples will be explored in which visual memory representations are shown to undergo transformations, and even shift their cortical locus alongside their coding format based on specifics of the task.
Feedback controls what we see
We hardly notice when there is a speck on our glasses, the obstructed visual information seems to be magically filled in. The visual system uses visual context to predict the content of the stimulus. What enables neurons in the visual system to respond to context when the stimulus is not available? In cortex, sensory processing is based on a combination of feedforward information arriving from sensory organs, and feedback information that originates in higher-order areas. Whereas feedforward information drives the activity in cortex, feedback information is thought to provide contextual signals that are merely modulatory. We have made the exciting discovery that mouse primary visual cortical neurons are strongly driven by feedback projections from higher visual areas, in particular when their feedforward sensory input from the retina is missing. This drive is so strong that it makes visual cortical neurons fire as much as if they were receiving a direct sensory input.
What does time of day mean for vision?
Profound changes in the visual environment occur over the course of the day-night cycle. There is therefore a profound pressure for cells and circuits within the visual system to adjust their function over time, to match the prevailing visual environment. Here, I will discuss electrophysiological data collected from nocturnal and diurnal rodents that reveal how the visual code is ‘temporally optimised’ by 1) the retina’s circadian clock, and 2) a change in behavioural temporal niche.
Dynamic spatial processing in insect vision
How does the visual system of insects function in vastly different light intensities, process separate parts of the visual field in parallel, and cope with eye sizes that differ between individuals? This talk will give you the answers we receive from our unique(ly adorable) model system: hawkmoths.
A Panoramic View on Vision
Statistics of natural scenes are not uniform - their structure varies dramatically from ground to sky. It remains unknown whether these non-uniformities are reflected in the large-scale organization of the early visual system and what benefits such adaptations would confer. By deploying an efficient coding argument, we predict that changes in the structure of receptive fields across visual space increase the efficiency of sensory coding. To test this experimentally, developed a simple, novel imaging system that is indispensable for studies at this scale. In agreement with our predictions, we could show that receptive fields of retinal ganglion cells change their shape along the dorsoventral axis, with a marked surround asymmetry at the visual horizon. Our work demonstrates that, according to principles of efficient coding, the panoramic structure of natural scenes is exploited by the retina across space and cell-types.
A novel form of retinotopy in area V2 highlights location-dependent feature selectivity in the visual system
Topographic maps are a prominent feature of brain organization, reflecting local and large-scale representation of the sensory surface. Traditionally, such representations in early visual areas are conceived as retinotopic maps preserving ego-centric retinal spatial location while ensuring that other features of visual input are uniformly represented for every location in space. I will discuss our recent findings of a striking departure from this simple mapping in the secondary visual area (V2) of the tree shrew that is best described as a sinusoidal transformation of the visual field. This sinusoidal topography is ideal for achieving uniform coverage in an elongated area like V2 as predicted by mathematical models designed for wiring minimization, and provides a novel explanation for stripe-like patterns of intra-cortical connections and functional response properties in V2. Our findings suggest that cortical circuits flexibly implement solutions to sensory surface representation, with dramatic consequences for large-scale cortical organization. Furthermore our work challenges the framework of relatively independent encoding of location and features in the visual system, showing instead location-dependent feature sensitivity produced by specialized processing of different features in different spatial locations. In the second part of the talk, I will propose that location-dependent feature sensitivity is a fundamental organizing principle of the visual system that achieves efficient representation of positional regularities in visual input, and reflects the evolutionary selection of sensory and motor circuits to optimally represent behaviorally relevant information. The relevant papers can be found here: V2 retinotopy (Sedigh-Sarvestani et al. Neuron, 2021) Location-dependent feature sensitivity (Sedigh-Sarvestani et al. Under Review, 2022)
Space and its computational challenges
How our senses work both separately and together involves rich computational problems. I will discuss the spatial and representational problems faced by the visual and auditory system, focusing on two issues. 1. How does the brain correct for discrepancies in the visual and auditory spatial reference frames? I will describe our recent discovery of a novel type of otoacoustic emission, the eye movement related eardrum oscillation, or EMREO (Gruters et al, PNAS 2018). 2. How does the brain encode more than one stimulus at a time? I will discuss evidence for neural time-division multiplexing, in which neural activity fluctuates across time to allow representations to encode more than one simultaneous stimulus (Caruso et al, Nat Comm 2018). These findings all emerged from experimentally testing computational models regarding spatial representations and their transformations within and across sensory pathways. Further, they speak to several general problems confronting modern neuroscience such as the hierarchical organization of brain pathways and limits on perceptual/cognitive processing.
Mature retina is resilient to partial photoreceptor loss
I will discuss recent findings from our lab about the effects of partial photoreceptor loss on the retinal circuit’s structure and function. I will relate this work to the question of whether the visual system can distinguish between changes in light level and photoreceptor number.
Demystifying the richness of visual perception
Human vision is full of puzzles. Observers can grasp the essence of a scene in an instant, yet when probed for details they are at a loss. People have trouble finding their keys, yet they may be quite visible once found. How does one explain this combination of marvelous successes with quirky failures? I will describe our attempts to develop a unifying theory that brings a satisfying order to multiple phenomena. One key is to understand peripheral vision. A visual system cannot process everything with full fidelity, and therefore must lose some information. Peripheral vision must condense a mass of information into a succinct representation that nonetheless carries the information needed for vision at a glance. We have proposed that the visual system deals with limited capacity in part by representing its input in terms of a rich set of local image statistics, where the local regions grow — and the representation becomes less precise — with distance from fixation. This scheme trades off computation of sophisticated image features at the expense of spatial localization of those features. What are the implications of such an encoding scheme? Critical to our understanding has been the use of methodologies for visualizing the equivalence classes of the model. These visualizations allow one to quickly see that many of the puzzles of human vision may arise from a single encoding mechanism. They have suggested new experiments and predicted unexpected phenomena. Furthermore, visualization of the equivalence classes has facilitated the generation of testable model predictions, allowing us to study the effects of this relatively low-level encoding on a wide range of higher-level tasks. Peripheral vision helps explain many of the puzzles of vision, but some remain. By examining the phenomena that cannot be explained by peripheral vision, we gain insight into the nature of additional capacity limits in vision. In particular, I will suggest that decision processes face general-purpose limits on the complexity of the tasks they can perform at a given time.
Analyzing Retinal Disease Using Electron Microscopic Connectomics
John DowlingJohn E. Dowling received his AB and PhD from Harvard University. He taught in the Biology Department at Harvard from 1961 to 1964, first as an Instructor, then as assistant professor. In 1964 he moved to Johns Hopkins University, where he held an appointment as associate professor of Ophthalmology and Biophysics. He returned to Harvard as professor of Biology in 1971, was the Maria Moors Cabot Professor of Natural Sciences from 1971-2001, Harvard College professor from 1999-2004 and is presently the Gordon and Llura Gund Professor of Neurosciences. Dowling was chairman of the Biology Department at Harvard from 1975 to 1978 and served as associate dean of the faculty of Arts and Sciences from 1980 to 1984. He was Master of Leverett House at Harvard from 1981-1998 and currently serves as president of the Corporation of The Marine Biological Laboratory in Woods Hole. He is a Fellow of the American Academy of Arts and Sciences, a member of the National Academy of Sciences and a member of the American Philosophical Society. Awards that Dowling received include the Friedenwald Medal from the Association of Research in Ophthalmology and Vision in 1970, the Annual Award of the New England Ophthalmological Society in 1979, the Retinal Research Foundation Award for Retinal Research in 1981, an Alcon Vision Research Recognition Award in 1986, a National Eye Institute's MERIT award in 1987, the Von Sallman Prize in 1992, The Helen Keller Prize for Vision Research in 2000 and the Llura Ligget Gund Award for Lifetime Achievement and Recognition of Contribution to the Foundation Fighting Blindness in 2001. He was granted an honorary MD degree by the University of Lund (Sweden) in 1982 and an honorary Doctor of Laws degree from Dalhousie University (Canada) in 2012. Dowling's research interests have focused on the vertebrate retina as a model piece of the brain. He and his collaborators have long been interested in the functional organization of the retina, studying its synaptic organization, the electrical responses of the retinal neurons, and the mechanisms underlying neurotransmission and neuromodulation in the retina. Dowling became interested in zebrafish as a system in which one could explore the development and genetics of the vertebrate retina about 20 years ago. Part of his research team has focused on retinal development in zebrafish and the role of retinoic acid in early eye and photoreceptor development. A second group has developed behavioral tests to isolate mutations, both recessive and dominant, specific to the visual system.
The role of motion in localizing objects
Everything we see has a location. We know where things are before we know what they are. But how do we know where things are? Receptive fields in the visual system specify location but neural delays lead to serious errors whenever targets or eyes are moving. Motion may be the problem here but motion can also be the solution, correcting for the effects of delays and eye movements. To demonstrate this, I will present results from three motion illusions where perceived location differs radically from physical location. These help understand how and where position is coded. We first look at the effects of a target’s simple forward motion on its perceived location. Second, we look at perceived location of a target that has internal motion as well as forward motion. The two directions combine to produce an illusory path. This “double-drift” illusion strongly affects perceived position but, surprisingly, not eye movements or attention. Even more surprising, fMRI shows that the shifted percept does not emerge in the visual cortex but is seen instead in the frontal lobes. Finally, we report that a moving frame also shifts the perceived positions of dots flashed within it. Participants report the dot positions relative to the frame, as if the frame were not moving. These frame-induced position effects suggest a link to visual stability where we see a steady world despite massive displacements during saccades. These motion-based effects on perceived location lead to new insights concerning how and where position is coded in the brain.
Vision outside of the visual system (in Drosophila)
We seek to understand the control of behavior – by animals, their brains, and their neurons. Reiser and his team are focused on the fly visual system, using modern methods from the Drosophila toolkit to understand how visual pathways are involved in specific behaviors. Due to the recent connectomics explosion, they now study the brain-wide networks organizing visual information for behavior control. The team combines explorations of visually guided behaviors with functional investigations of specific cell types throughout the fly brain. The Reiser lab actively develops and disseminates new methods and instruments enabling increasingly precise quantification of animal behavior.
Smart perception?: Gestalt grouping, perceptual averaging, and memory capacity
It seems we see the world in full detail. However, the eye is not a camera nor is the brain a computer. Incredible metabolic constraints render us unable to encode more than a fraction of information available in each glance. Instead, our illusion of stable and complete perception is accomplished by parsimonious representation relying on natural order inherent in the surrounding environment. I will begin by discussing previous behavioral work from our lab demonstrating one such strategy by which the visual system represents average properties of Gestalt-grouped sets of individual objects, warping individual object representations toward the Gestalt-defined mean. I will then discuss on-going work using a behavioral index of averaging Gestalt-grouped information established in our previous work in conjunction with an ERP-index of VSTM capacity (the CDA) to measure whether the Gestalt-grouping and perceptual averaging strategy acts to boost memory capacity above the classic “four-item” limit. Finally, I will outline our pre-registered study to determine whether this perceptual strategy is indeed engaged in a “smart” manner under normal circumstances, or compromises fidelity for capacity by perceptually-averaging in trials with only four items that could otherwise be individually represented.
Imaging the influences of sensory experience on visual system circuit development
Using a combination of in vivo imaging of neuronal circuit functional and structural dynamics, we have investigated the mechanisms by which patterned neural activity and sensory experience alter connectivity in the developing brain. We have identified, in addition to the long-hypothesized Hebbian structural plasticity mechanisms, a kind of plasticity induced by the absence of correlated firing that we dubbed “Stentian plasticity”. In the talk I will discuss the phenomenology and some mechanistic insights regarding Stentian mechanisms in brain development. Further, I will show how glia may have a key role in circuit remodeling during development. These studies have led us to an appreciation of the importance of neuron-glia interactions in early development and the ability of patterned activity to guide circuit wiring.
The generation of neural diversity
Claude Desplan is a Silver Professor of Biology and Neuroscience at NYU. He was born in Algeria and was trained at Ecole Normale Supérieure St. Cloud, France. He received his DSc at INSERM in Paris in 1983 and joined Pat O’Farrell at UCSF as a postdoc. There he demonstrated that the homeodomain, a conserved signature of many developmental genes, is a DNA binding motif. Currently, Dr. Desplan works at NYU where he investigates the generation of neural diversity using the Drosophila visual system.
Connectivity and computation in the cerebral cortex
Neuroscientists believe that perception, action and cognition arise from brain’s activity. A major challenge in neuroscience is to understand how brain’s complex circuits give rise to activity patterns that support these different functions. I will discuss different ways of mapping neural circuits in the brain, and how we can relate the structure of neural circuits to the computations that take place within them, with an emphasis on the visual system.
The Blurry Beginnings: What nature’s strangest eyes tell us about the evolution of vision
Our study reveals the most elaborate opsin expression patterns ever described in any animal eye. In mantis shrimp, a pugnacious crustacean renowned for its visual sophistication, we found unexpected retinal expression patterns highlighting the potential for cryptic photoreceptor functional diversity, including single photoreceptors that coexpress opsins from different spectral clades and a single opsin with a putative nonvisual function important in color vision. This study demonstrates the evolutionary potential for increasing visual system functional diversity through opsin gene duplication and diversification, as well as changes in patterns of gene coexpression among photoreceptors and retinula cells. These results have significant implications for the function of other visual systems, particularly in arthropods where large numbers of retinally expressed opsins have been documented.
Vision for escape and pursuit
We want to understand how the visual system detects and tracks salient stimuli in the environment to initiate and guide specific behaviors (i.e., visual neuroethology). Predator avoidance and prey capture are central selection pressures of animal evolution. Mice use vision to detect aerial predators and hunt insects. I will discuss studies from my group that identify specific circuits and pathways in the early visual system (i.e., the retina and its subcortical targets) mediating predator avoidance and prey capture in mice. Our results highlight the importance of subcellular visual processing in the retina and the alignment of viewing strategies with region- and cell-type-specific retinal ganglion cell projection patterns to the brain.
Arousal modulates retinal output
Neural responses in the visual system are usually not purely visual but depend on behavioural and internal states such as arousal. This dependence is seen both in primary visual cortex (V1) and in subcortical brain structures receiving direct retinal input. In this talk, I will show that modulation by behavioural state arises as early as in the output of the retina.To measure retinal activity in the awake, intact brain, we imaged the synaptic boutons of retinal axons in the superficial superior colliculus (sSC) of mice. The activity of about half of the boutons depended not only on vision but also on running speed and pupil size, regardless of retinal illumination. Arousal typically reduced the boutons’ visual responses to preferred direction and their selectivity for direction and orientation.Arousal may affect activity in retinal boutons by presynaptic neuromodulation. To test whether the effects of arousal occur already in the retina, we recorded from retinal axons in the optic tract. We found that, in darkness, more than one third of the recorded axons was significantly correlated with running speed. Arousal had similar effects postsynaptically, in sSC neurons, independent of activity in V1, the other main source of visual inputs to colliculus. Optogenetic inactivation of V1 generally decreased activity in collicular neurons but did not diminish the effects of arousal. These results indicate that arousal modulates activity at every stage of the visual system. In the future, we will study the purpose and the underlying mechanisms of behavioural modulation in the early visual system
Using marmosets for the study of the visual cortex: unique opportunities, and some pitfalls
Marmosets (Callithrix jacchus) are small South American monkeys which are being increasingly becoming adopted as animal models in neuroscience. Knowledge about the marmoset visual system has developed rapidly over the last decade. But what are the comparative advantages, and disadvantages involved in adopting this emerging model, as opposed to the more traditionally used macaque monkey? In this talk I will present case studies where the simpler brain morphology and short developmental cycle of the marmoset have been key factors in facilitating discoveries about the anatomy and physiology of the visual system. Although no single species provides the “ideal” animal model for invasive studies of the neural bases of visual processing, I argue that the development of robust methodologies for the study of the marmoset brain provides exciting opportunities to address long-standing problems in neuroscience.
Dimensions of variability in circuit models of cortex
Cortical circuits receive multiple inputs from upstream populations with non-overlapping stimulus tuning preferences. Both the feedforward and recurrent architectures of the receiving cortical layer will reflect this diverse input tuning. We study how population-wide neuronal variability propagates through a hierarchical cortical network receiving multiple, independent, tuned inputs. We present new analysis of in vivo neural data from the primate visual system showing that the number of latent variables (dimension) needed to describe population shared variability is smaller in V4 populations compared to those of its downstream visual area PFC. We successfully reproduce this dimensionality expansion from our V4 to PFC neural data using a multi-layer spiking network with structured, feedforward projections and recurrent assemblies of multiple, tuned neuron populations. We show that tuning-structured connectivity generates attractor dynamics within the recurrent PFC current, where attractor competition is reflected in the high dimensional shared variability across the population. Indeed, restricting the dimensionality analysis to activity from one attractor state recovers the low-dimensional structure inherited from each of our tuned inputs. Our model thus introduces a framework where high-dimensional cortical variability is understood as ``time-sharing’’ between distinct low-dimensional, tuning-specific circuit dynamics.
Cones with character: An in vivo circuit implementation of efficient coding
In this talk I will summarize some of our recent unpublished work on spectral coding in the larval zebrafish retina. Combining 2p imaging, hyperspectral stimulation, computational modeling and connectomics, we take a renewed look at the spectral tuning of cone photoreceptors in the live eye. We find that already cones optimally rotate natural colour space in a PCA-like fashion to disambiguate greyscale from "colour" information. We then follow this signal through the retinal layers and ultimately into the brain to explore the major spectral computations performed by the visual system at its consecutive stages. We find that by and large, zebrafish colour vision can be broken into three major spectral zones: long wavelength grey-scale-like vision, short-wavelength prey capture circuits, and spectrally diverse mid-wavelength circuits which possibly support the bulk of "true colour vision" in this tetrachromate vertebrate.
Mechanism(s) of negative feedback from horizontal cells to cones and its consequence for (color) vision
Vision starts in the retina where images are transformed and coded into neuronal activity relevant for the brain. These coding steps function optimally over a wide range of conditions: from bright day on the beach to a moonless night. Under these very different conditions, specific retinal mechanisms continue to select relevant aspects of the visual world and send this information to the brain. We are studying the neuronal processing involved in these selection and adaptation processes. This knowledge is essential for understanding how the visual system works and forms the basis for research dedicated to restoring vision in blind people.
Duke VisionFest Symposium Keynote: Applications of High-Resolution Retinal Imaging
Duke VisionFest is a virtual research symposium conceived to bring together the many Duke groups studying aspects of the visual system. Keynote by Dr. Joseph Carroll, Medical College of Wisconsin, and featuring talks ranging from photoreceptor biology to visual system evolution to clinical diagnosis. Pre-registration is required to attend but open to anyone interested.
Natural visual stimuli for mice
During the course of evolution, a species’ environment shapes its sensory abilities, as individuals with more optimized sensory abilities are more likely survive and procreate. Adaptations to the statistics of the natural environment can be observed along the early visual pathway and across species. Therefore, characterising the properties of natural environments and studying the representation of natural scenes along the visual pathway is crucial for advancing our understanding of the structure and function of the visual system. In the past 20 years, mice have become an important model in vision research, but the fact that they live in a different environment than primates and have different visual needs is rarely considered. One particular challenge for characterising the mouse’s visual environment is that they are dichromats with photoreceptors that detect UV light, which the typical camera does not record. This also has consequences for experimental visual stimulation, as the blue channel of computer screens fails to excite mouse UV cone photoreceptors. In my talk, I will describe our approach to recording “colour” footage of the habitat of mice – from the mouse’s perspective – and to studying retinal circuits in the ex vivo retina with natural movies.
Wiring up direction selective circuits in the retina
The development of neural circuits is profoundly impacted by both spontaneous and sensory experience. This is perhaps most well studied in the visual system, where disruption of early spontaneous activity called retinal waves prior to eye opening and visual deprivation after eye opening leads to alterations in the response properties and connectivity in several visual centers in the brain. We address this question in the retina, which comprises multiple circuits that encode different features of the visual scene, culminating in over 40 different types of retinal ganglion cells. Direction-selective ganglion cells respond strongly to an image moving in the preferred direction and weakly to an image moving in the opposite, or null, direction. Moreover, as recently described (Sabbah et al, 2017) the preferred directions of direction selective ganglion cells cluster along four directions that align along two optic flow axes, causing variation of the relative orientation of preferred directions along the retinal surface. I will provide recent progress in the lab that addresses the role of visual experience and spontaneous retinal waves in the establishment of direction selective tuning and direction selectivity maps in the retina.