Reliability
reliability
Thomas Krak
The Uncertainty in Artificial Intelligence (UAI) group is looking for a highly motivated and skilled PhD candidate to work in the area of probabilistic machine learning. The position is fully funded for a term of four years. The research direction will be determined together with the successful candidate and in line with the NWO Perspectief Project Personalised Care in Oncology (www.personalisedcareinoncology.nl). The research topics may include, but are not restricted to: Probabilistic graphical models (Markov, Bayesian, credal networks), Causality: Theory and application, Cautious AI, including imprecise probabilities, Robust stochastic processes, Tractable models and decision-making, Online/continual learning with evolving data.
A Novel Neurophysiological Approach to Assessing Distractibility within the General Population
Vulnerability to distraction varies across the general population and significantly affects one’s capacity to stay focused on and successfully complete the task at hand, whether at school, on the road, or at work. In this talk, I will begin by discussing how distractibility is typically assessed in the literature and introduce our innovative ERP approach to measuring it. Since distractibility is a cardinal symptom of ADHD, I will introduce its most widely used paper-and-pencil screening tool for the general population as external validation. Following that, I will present the Load Theory of Attention and explain how we used perceptual load to test the reliability of our neural marker of distractibility. Finally, I will highlight potential future applications of this marker in clinical and educational settings.
An open-source miniature two-photon microscope for large-scale calcium imaging in freely moving mice
Due to the unsuitability of benchtop imaging for tasks that require unrestrained movement, investigators have tried, for almost two decades, to develop miniature 2P microscopes-2P miniscopes–that can be carried on the head of freely moving animals. In this talk, I would first briefly review the development history of this technique, and then report our latest progress on developing the new generation of 2P miniscopes, MINI2P, that overcomes the limits of previous versions by both meeting requirements for fatigue-free exploratory behavior during extended recording periods and satisfying demands for further increasing the cell yield by an order of magnitude, to thousands of neurons. The performance and reliability of MINI2P are validated by recordings of spatially tuned neurons in three brain regions and in three behavioral assays. All information about MINI2P is open access, with instruction videos, code, and manuals on public repositories, and workshops will be organized to help new users getting started. MINI2P permits large-scale and high-resolution calcium imaging in freely-moving mice, and opens the door to investigating brain functions during unconstrained natural behaviors.
The Secret Bayesian Life of Ring Attractor Networks
Efficient navigation requires animals to track their position, velocity and heading direction (HD). Some animals’ behavior suggests that they also track uncertainties about these navigational variables, and make strategic use of these uncertainties, in line with a Bayesian computation. Ring-attractor networks have been proposed to estimate and track these navigational variables, for instance in the HD system of the fruit fly Drosophila. However, such networks are not designed to incorporate a notion of uncertainty, and therefore seem unsuited to implement dynamic Bayesian inference. Here, we close this gap by showing that specifically tuned ring-attractor networks can track both a HD estimate and its associated uncertainty, thereby approximating a circular Kalman filter. We identified the network motifs required to integrate angular velocity observations, e.g., through self-initiated turns, and absolute HD observations, e.g., visual landmark inputs, according to their respective reliabilities, and show that these network motifs are present in the connectome of the Drosophila HD system. Specifically, our network encodes uncertainty in the amplitude of a localized bump of neural activity, thereby generalizing standard ring attractor models. In contrast to such standard attractors, however, proper Bayesian inference requires the network dynamics to operate in a regime away from the attractor state. More generally, we show that near-Bayesian integration is inherent in generic ring attractor networks, and that their amplitude dynamics can account for close-to-optimal reliability weighting of external evidence for a wide range of network parameters. This only holds, however, if their connection strengths allow the network to sufficiently deviate from the attractor state. Overall, our work offers a novel interpretation of ring attractor networks as implementing dynamic Bayesian integrators. We further provide a principled theoretical foundation for the suggestion that the Drosophila HD system may implement Bayesian HD tracking via ring attractor dynamics.
The Problem of Testimony
The talk will detail work drawing on behavioural results, formal analysis, and computational modelling with agent-based simulations to unpack the scale of the challenge humans face when trying to work out and factor in the reliability of their sources. In particular, it is shown how and why this task admits of no easy solution in the context of wider communication networks, and how this will affect the accuracy of our beliefs. The implications of this for the shift in the size and topology of our communication networks through the uncontrolled rise of social media are discussed.
Improving reliability through design and reporting
As part of the BNA's ongoing Credibility in Neuroscience work, this series of three short webinars will provide neuroscience researchers working in an in vivo setting with tips on how to improve the credibility of their work. Each webinar will be hosted by Emily Sena, member of the BNA's Credibility Advisory Board, with the opportunity for questions.
From natural scene statistics to multisensory integration: experiments, models and applications
To efficiently process sensory information, the brain relies on statistical regularities in the input. While generally improving the reliability of sensory estimates, this strategy also induces perceptual illusions that help reveal the underlying computational principles. Focusing on auditory and visual perception, in my talk I will describe how the brain exploits statistical regularities within and across the senses for the perception space, time and multisensory integration. In particular, I will show how results from a series of psychophysical experiments can be interpreted in the light of Bayesian Decision Theory, and I will demonstrate how such canonical computations can be implemented into simple and biologically plausible neural circuits. Finally, I will show how such principles of sensory information processing can be leveraged in virtual and augmented reality to overcome display limitations and expand human perception.
Commonly used face cognition tests yield low reliability and inconsistent performance: Implications for test design, analysis, and interpretation of individual differences data
Unfamiliar face processing (face cognition) ability varies considerably in the general population. However, the means of its assessment are not standardised, and selected laboratory tests vary between studies. It is also unclear whether 1) the most commonly employed tests are reliable, 2) participants show a degree of consistency in their performance, 3) and the face cognition tests broadly measure one underlying ability, akin to general intelligence. In this study, we asked participants to perform eight tests frequently employed in the individual differences literature. We examined the reliability of these tests, relationships between them, consistency in participants’ performance, and used data driven approaches to determine factors underpinning performance. Overall, our findings suggest that the reliability of these tests is poor to moderate, the correlations between them are weak, the consistency in participant performance across tasks is low and that performance can be broadly split into two factors: telling faces together, and telling faces apart. We recommend that future studies adjust analyses to account for stimuli (face images) and participants as random factors, routinely assess reliability, and that newly developed tests of face cognition are examined in the context of convergent validity with other commonly used measures of face cognition ability.
Synapses, Shadows and Stress Contagion
Survival is predicated on the ability of an organism to respond to stress. The reliability of this response is ensured by a synaptic architecture that is relatively inflexible (i.e. hard-wired). Our work has shown that in naive animals, synapses on CRH neurons in the paraventricular nucleus of the hypothalamus are very reluctant to modification. If animals are stressed, however, these synapses become willing to learn. This seminar will focus on mechanisms linking acute stress to metaplastic changes at glutamate synapses, and also show how stress, and these synaptic changes can be transmitted from one individual to another.
The role of high- and low-level factors in smooth pursuit of predictable and random motions
Smooth pursuit eye movements are among our most intriguing motor behaviors. They are able to keep the line of sight on smoothly moving targets with little or no overt effort or deliberate planning, and they can respond quickly and accurately to changes in the trajectory of motion of targets. Nevertheless, despite these seeming automatic characteristics, pursuit is highly sensitive to high-level factors, such as the choices made about attention, or beliefs about the direction of upcoming motion. Investigators have struggled for decades with the problem of incorporating both high- and low-level processes into a single coherent model. This talk will present an overview of the current state of efforts to incorporate high- and low-level influences, as well as new observations that add to our understanding of both types of influences. These observations (in contrast to much of the literature) focus on the directional properties of pursuit. Studies will be presented that show: (1) the direction of smooth pursuit made to pursue fields of noisy random dots depends on the relative reliability of the sensory signal and the expected motion direction; (2) smooth pursuit shows predictive responses that depend on the interpretation of cues that signal an impending collision; and (3) smooth pursuit during a change in target direction displays kinematic properties consistent with the well-known two-thirds power law. Implications for incorporating high- and low-level factors into the same framework will be discussed.
Acetylcholine modulation of short-term plasticity is critical to reliable long-term plasticity in hippocampal synapses
CA3-CA1 synapses in the hippocampus are the initial locus of episodic memory. The action of acetylcholine alters cellular excitability, modifies neuronal networks, and triggers secondary signaling that directly affects long-term plasticity (LTP) (the cellular underpinning of memory). It is therefore considered a critical regulator of learning and memory in the brain. Its action via M4 metabotropic receptors in the presynaptic terminal of the CA3 neurons and M1 metabotropic receptors in the postsynaptic spines of CA1 neurons produce rich dynamics across multiple timescales. We developed a model to describe the activation of postsynaptic M1 receptors that leads to IP3 production from membrane PIP2 molecules. The binding of IP3 to IP3 receptors in the endoplasmic reticulum (ER) ultimately causes calcium release. This calcium release from the ER activates potassium channels like the calcium-activated SK channels and alters different aspects of synaptic signaling. In an independent signaling cascade, M1 receptors also directly suppress SK channels and the voltage-activated KCNQ2/3 channels, enhancing post-synaptic excitability. In the CA3 presynaptic terminal, we model the reduction of the voltage sensitivity of voltage-gated calcium channels (VGCCs) and the resulting suppression of neurotransmitter release by the action of the M4 receptors. Our results show that the reduced initial release probability because of acetylcholine alters short-term plasticity (STP) dynamics. We characterize the dichotomy of suppressing neurotransmitter release from CA3 neurons and the enhanced excitability of the postsynaptic CA1 spine. Mechanisms underlying STP operate over a few seconds, while those responsible for LTP last for hours, and both forms of plasticity have been linked with very distinct functions in the brain. We show that the concurrent suppression of neurotransmitter release and increased sensitivity conserves neurotransmitter vesicles and enhances the reliability in plasticity. Our work establishes a relationship between STP and LTP coordinated by neuromodulation with acetylcholine.
The Jena Voice Learning and Memory Test (JVLMT)
The ability to recognize someone’s voice spans a broad spectrum with phonagnosia on the low end and super recognition at the high end. Yet there is no standardized test to measure the individual ability to learn and recognize newly-learnt voices with samples of speech-like phonetic variability. We have developed the Jena Voice Learning and Memory Test (JVLMT), a 20 min-test based on item response theory and applicable across different languages. The JVLMT consists of three phases in which participants are familiarized with eight speakers in two stages and then perform a three-alternative forced choice recognition task, using pseudo sentences devoid of semantic content. Acoustic (dis)similarity analyses were used to create items with different levels of difficulty. Test scores are based on 22 Rasch-conform items. Items were selected and validated in online studies based on 232 and 454 participants, respectively. Mean accuracy is 0.51 with an SD of .18. The JVLMT showed high and moderate correlations with convergent validation tests (Bangor Voice Matching Test; Glasgow Voice Memory Test) and a weak correlation with a discriminant validation test (Digit Span). Empirical (marginal) reliability is 0.66. Four participants with super recognition (at least 2 SDs above the mean) and 7 participants with phonagnosia (at least 2 SDs below the mean) were identified. The JVLMT is a promising screen too for voice recognition abilities in a scientific and neuropsychological context.
Learning to perceive with new sensory signals
I will begin by describing recent research taking a new, model-based approach to perceptual development. This approach uncovers fundamental changes in information processing underlying the protracted development of perception, action, and decision-making in childhood. For example, integration of multiple sensory estimates via reliability-weighted averaging – widely used by adults to improve perception – is often not seen until surprisingly late into childhood, as assessed by both behaviour and neural representations. This approach forms the basis for a newer question: the scope for the nervous system to deploy useful computations (e.g. reliability-weighted averaging) to optimise perception and action using newly-learned sensory signals provided by technology. Our initial model system is augmenting visual depth perception with devices translating distance into auditory or vibro-tactile signals. This problem has immediate applications to people with partial vision loss, but the broader question concerns our scope to use technology to tune in to any signal not available to our native biological receptors. I will describe initial progress on this problem, and our approach to operationalising what it might mean to adopt a new signal comparably to a native sense. This will include testing for its integration (weighted averaging) alongside the native senses, assessing the level at which this integration happens in the brain, and measuring the degree of ‘automaticity’ with which new signals are used, compared with native perception.
Searching for the Super-Searchers
A striking range of individual differences has been reported in a variety of visual search tasks, which naturally leads to the idea that some people are better at finding things than others. However, this conclusion appears to be premature. We carried out a replication of three recent visual search experiments and found that each task showed a wide range of individual differences as predicted, and observed good test-retest reliability in all three. However, performance on any one task was not correlated with the performance in the others: participants who naturally adopt efficient search strategies in one paradigm may perform at chance in another! Furthermore, we also show that behaviour in different versions of the same paradigm can be radically different: When simple line segments are used for search items, a large range of search strategies are found. If we instead use more complex search items, all our participants effortlessly adopt an optimal strategy. These results suggest search strategies are stable over time, but context-specific. To understand visual search we, therefore, need to account not only for differences between individuals but also how individuals interact with the search task and context.
Error correction and reliability timescale in converging cortical networks
Rapidly changing inputs such as visual scenes and auditory landscapes are transmitted over several synaptic interfaces and perceived with little loss of detail, but individual neurons are typically “noisy” and cortico-cortical connections are typically “weak”. To understand how information embodied in spike train is transmitted in a lossless manner, we focus on a single synaptic interface: between pyramidal cells and putative interneurons. Using arbitrary white noise patterns injected intra-cortically as photocurrents to freely-moving mice, we find that directly-activated cells exhibit precision of several milliseconds, but post-synaptic, indirectly-activated cells exhibit higher precision. Considering multiple identical messages, the reliability of directly-activated cells peaks at a timescale of dozens of milliseconds, whereas indirectly-activated cells exhibit an order-of-magnitude faster timescale. Using data-driven modelling, we find that error correction is consistent with non-linear amplification of coincident spikes.
Algorithmic advances in face matching: Stability of tests in atypical groups
Face matching tests have traditionally been developed to assess human face perception in the neurotypical range, but methods that underlie their development often make it difficult for these measures to be applied in atypical populations (developmental prosopagnosics, super recognizers) due to unadjusted difficulty. We have recently presented the development of the Oxford Face Matching Test, a measure that bases individual item-difficulty on algorithmically derived similarity of presented stimuli. The measure seems useful as it can be given online or in-laboratory, has good discriminability and high test-retest reliability in the neurotypical groups. In addition, it has good validity in separating atypical groups at either of the spectrum ends. In this talk, I examine the stability of the OFMT and other traditionally used measures in atypical groups. On top of the theoretical significance of determining whether reliability of tests is equivalent in atypical population, this is an important question because of the practical concerns of retesting the same participants across different lab groups. Theoretical and practical implications for further test development and data sharing are discussed.
Multisensory Perception: Behaviour, Computations and Neural Mechanisms
Our senses are constantly bombarded with a myriad of diverse signals. Transforming this sensory cacophony into a coherent percept of our environment relies on solving two computational challenges: First, we need to solve the causal inference problem - deciding whether signals come from a common cause and thus should be integrated, or come from different sources and be treated independently. Second, when there is a common cause, we should integrate signals across the senses weighted in proportion to their sensory reliabilities. I discuss recent research at the behavioural, computational and neural systems level that investigates how the brain addresses these two computational challenges in multisensory perception.
Uncertainty in perceptual decision-making
Whether we are deciding about Covid-related restrictions, estimating a ball’s trajectory when playing tennis, or interpreting radiological images – most any choice we make is based on uncertain evidence. How do we infer that information is more or less reliable when making these decisions? How does the brain represent knowledge of this uncertainty? In this talk, I will present recent neuroimaging data combined with novel analysis tools to address these questions. Our results indicate that sensory uncertainty can reliably be estimated from the human visual cortex on a trial-by-trial basis, and moreover that observers appear to rely on this uncertainty when making perceptual decisions.
A human-specific modifier of synaptic development, cortical circuit connectivity and function
The remarkable cognitive abilities characterizing humans has been linked to unique patterns of connectivity characterizing the neocortex. Comparative studies have shown that human cortical pyramidal neurons (PN) receive a significant increase of synaptic inputs when compared to other mammals, including non-human primates and rodents, but how this may relate to changes in cortical connectivity and function remained largely unknown. We previously identified a human-specific gene duplication (HSGD), SRGAP2C, that, when induced in mouse cortical PNs drives human-specific features of synaptic development, including a correlated increase in excitatory (E) and inhibitory (I) synapse density through inhibition of the ancestral SRGAP2A protein (Charrier et al. 2012; Fossatti et al. 2016; Schmidt et al. 2019). However, the origin and nature of this increased connectivity and its impact on cortical circuit function was unknown. I will present new results exploring these questions (see Schmidt et al. (2020) https://www.biorxiv.org/content/10.1101/852970v1). Using a combination of transgenic approaches and quantitative monosynaptic tracing, we discovered that humanization of SRGAP2C expression in the mouse cortex leads to a specific increase in local and long-range cortico-cortical inputs received by layer 2/3 cortical PNs. Moreover, using in vivo two-photon imaging in the barrel cortex of awake mice, we show that humanization of SRGAP2C expression increases the reliability and selectivity of sensory- evoked responses in layer 2/3 PNs. We also found that mice humanized for SRGAP2C in all cortical pyramidal neurons and throughout development are characterized by improved behavioural performance in a novel whisker-based sensory discrimination task compared to control wild-type mice. Our results suggest that the emergence of SRGAP2C during human evolution underlie a new substrate for human brain evolution whereby it led to increased local and long-range cortico-cortical connectivity and improved reliability of sensory-evoked cortical coding. References cited Charrier C.*, Joshi K. *, Coutinho-Budd J., Kim, J-E., Lambert N., de Marchena, J., Jin W-L., Vanderhaeghen P., Ghosh A., Sassa T, and Polleux F. (2012) Inhibition of SRGAP2 function by its human-specific paralogs induces neoteny of spine maturation. Cell 149:923-935. * Co-first authors. Fossati M, Pizzarelli R, Schmidt ER, Kupferman JV, Stroebel D, Polleux F*, Charrier C*. (2016) SRGAP2 and Its Human-Specific Paralog Co-Regulate the Development of Excitatory and Inhibitory Synapses. Neuron. 91(2):356-69. * Co-senior corresponding authors. Schmidt E.R.E., Kupferman J.V., Stackmann M., Polleux F. (2019) The human-specific paralogs SRGAP2 and SRGAP2C differentially modulate SRGAP2A-dependent synaptic development. Scientific Rep. 9(1):18692. Schmidt E.R.E, Zhao H.T., Hillman E.M.C., Polleux F. (2020) Humanization of SRGAP2C expression increases cortico-cortical connectivity and reliability of sensory-evoked responses in mouse brain. Submitted. See also: https://www.biorxiv.org/content/10.1101/852970v1
A synaptic plasticity rule based on presynaptic variance to infer input reliability
COSYNE 2022
A synaptic plasticity rule based on presynaptic variance to infer input reliability
COSYNE 2022
Forming and updating pain expectations: Influence of sequence volatility and test-retest reliability
FENS Forum 2024
The influence of pulse shape and current direction of TMS on test-retest reliability and variability of single pulse TMS protocols
FENS Forum 2024
Optimizing electrical stimulation parameters for human-derived neuronal networks: An investigation into the reliability of evoked responses
FENS Forum 2024
Reliability of reduced inter-subject functional connectivity during naturalistic movie-watching fMRI in autism — comparison of German and Finnish samples
FENS Forum 2024
Test-retest reliability analysis of EEG power measures and their association with L2 vocabulary learning in children and adults
FENS Forum 2024
Test-retest reliability in the effort learning task
FENS Forum 2024