Validity
validity
Use case determines the validity of neural systems comparisons
Deep learning provides new data-driven tools to relate neural activity to perception and cognition, aiding scientists in developing theories of neural computation that increasingly resemble biological systems both at the level of behavior and of neural activity. But what in a deep neural network should correspond to what in a biological system? This question is addressed implicitly in the use of comparison measures that relate specific neural or behavioral dimensions via a particular functional form. However, distinct comparison methodologies can give conflicting results in recovering even a known ground-truth model in an idealized setting, leaving open the question of what to conclude from the outcome of a systems comparison using any given methodology. Here, we develop a framework to make explicit and quantitative the effect of both hypothesis-driven aspects—such as details of the architecture of a deep neural network—as well as methodological choices in a systems comparison setting. We demonstrate via the learning dynamics of deep neural networks that, while the role of the comparison methodology is often de-emphasized relative to hypothesis-driven aspects, this choice can impact and even invert the conclusions to be drawn from a comparison between neural systems. We provide evidence that the right way to adjudicate a comparison depends on the use case—the scientific hypothesis under investigation—which could range from identifying single-neuron or circuit-level correspondences to capturing generalizability to new stimulus properties
Time perception in film viewing as a function of film editing
Filmmakers and editors have empirically developed techniques to ensure the spatiotemporal continuity of a film's narration. In terms of time, editing techniques (e.g., elliptical, overlapping, or cut minimization) allow for the manipulation of the perceived duration of events as they unfold on screen. More specifically, a scene can be edited to be time compressed, expanded, or real-time in terms of its perceived duration. Despite the consistent application of these techniques in filmmaking, their perceptual outcomes have not been experimentally validated. Given that viewing a film is experienced as a precise simulation of the physical world, the use of cinematic material to examine aspects of time perception allows for experimentation with high ecological validity, while filmmakers gain more insight on how empirically developed techniques influence viewers' time percept. Here, we investigated how such time manipulation techniques of an action affect a scene's perceived duration. Specifically, we presented videos depicting different actions (e.g., a woman talking on the phone), edited according to the techniques applied for temporal manipulation and asked participants to make verbal estimations of the presented scenes' perceived durations. Analysis of data revealed that the duration of expanded scenes was significantly overestimated as compared to that of compressed and real-time scenes, as was the duration of real-time scenes as compared to that of compressed scenes. Therefore, our results validate the empirical techniques applied for the modulation of a scene's perceived duration. We also found interactions on time estimates of scene type and editing technique as a function of the characteristics and the action of the scene presented. Thus, these findings add to the discussion that the content and characteristics of a scene, along with the editing technique applied, can also modulate perceived duration. Our findings are discussed by considering current timing frameworks, as well as attentional saliency algorithms measuring the visual saliency of the presented stimuli.
On the link between conscious function and general intelligence in humans and machines
In popular media, there is often a connection drawn between the advent of awareness in artificial agents and those same agents simultaneously achieving human or superhuman level intelligence. In this talk, I will examine the validity and potential application of this seemingly intuitive link between consciousness and intelligence. I will do so by examining the cognitive abilities associated with three contemporary theories of conscious function: Global Workspace Theory (GWT), Information Generation Theory (IGT), and Attention Schema Theory (AST), and demonstrating that all three theories specifically relate conscious function to some aspect of domain-general intelligence in humans. With this insight, we will turn to the field of Artificial Intelligence (AI) and find that, while still far from demonstrating general intelligence, many state-of-the-art deep learning methods have begun to incorporate key aspects of each of the three functional theories. Given this apparent trend, I will use the motivating example of mental time travel in humans to propose ways in which insights from each of the three theories may be combined into a unified model. I believe that doing so can enable the development of artificial agents which are not only more generally intelligent but are also consistent with multiple current theories of conscious function.
Do we measure what we think we are measuring?
Tests used in the empirical sciences are often (implicitly) assumed to be representative of a target mechanism in the sense that similar tests should lead to similar results. In this talk, using resting-state electroencephalogram (EEG) as an example, I will argue that this assumption does not necessarily hold true. Typically EEG studies are conducted selecting one analysis method thought to be representative of the research question asked. Using multiple methods, we extracted a variety of features from a single resting-state EEG dataset and conducted correlational and case-control analyses. We found that many EEG features revealed a significant effect in the case-control analyses. Similarly, EEG features correlated significantly with cognitive tasks. However, when we compared these features pairwise, we did not find strong correlations. A number of explanations to these results will be discussed.
Brain and behavioural impacts of early life adversity
Abuse, neglect, and other forms of uncontrollable stress during childhood and early adolescence can lead to adverse outcomes later in life, including especially perturbations in the regulation of mood and emotional states, and specifically anxiety disorders and depression. However, stress experiences vary from one individual to the next, meaning that causal relationships and mechanistic accounts are often difficult to establish in humans. This interdisciplinary talk considers the value of research in experimental animals where stressor experiences can be tightly controlled and detailed investigations of molecular, cellular, and circuit-level mechanisms can be carried out. The talk will focus on the widely used repeated maternal separation procedure in rats where rat offspring are repeatedly separated from maternal care during early postnatal life. This early life stress has remarkably persistent effects on behaviour with a general recognition that maternally-deprived animals are susceptible to depressive-like phenotypes. The validity of this conclusion will be critically appraised with convergent insights from a recent longitudinal study in maternally separated rats involving translational brain imaging, transcriptomics, and behavioural assessment.
Metabolic spikes: from rogue electrons to Parkinson's
Conventionally, neurons are thought to be cellular units that process synaptic inputs into synaptic spikes. However, it is well known that neurons can also spike spontaneously and display a rich repertoire of firing properties with no apparent functional relevance e.g. in in vitro cortical slice preparations. In this talk, I will propose a hypothesis according to which intrinsic excitability in neurons may be a survival mechanism to minimize toxic byproducts of the cell’s energy metabolism. In neurons, this toxicity can arise when mitochondrial ATP production stalls due to limited ADP. Under these conditions, electrons deviate from the electron transport chain to produce reactive oxygen species, disrupting many cellular processes and challenging cell survival. To mitigate this, neurons may engage in ADP-producing metabolic spikes. I will explore the validity of this hypothesis using computational models that illustrate the implications of synaptic and metabolic spiking, especially in the context of substantia nigra pars compacta dopaminergic neurons and their degeneration in Parkinson's disease.
Commonly used face cognition tests yield low reliability and inconsistent performance: Implications for test design, analysis, and interpretation of individual differences data
Unfamiliar face processing (face cognition) ability varies considerably in the general population. However, the means of its assessment are not standardised, and selected laboratory tests vary between studies. It is also unclear whether 1) the most commonly employed tests are reliable, 2) participants show a degree of consistency in their performance, 3) and the face cognition tests broadly measure one underlying ability, akin to general intelligence. In this study, we asked participants to perform eight tests frequently employed in the individual differences literature. We examined the reliability of these tests, relationships between them, consistency in participants’ performance, and used data driven approaches to determine factors underpinning performance. Overall, our findings suggest that the reliability of these tests is poor to moderate, the correlations between them are weak, the consistency in participant performance across tasks is low and that performance can be broadly split into two factors: telling faces together, and telling faces apart. We recommend that future studies adjust analyses to account for stimuli (face images) and participants as random factors, routinely assess reliability, and that newly developed tests of face cognition are examined in the context of convergent validity with other commonly used measures of face cognition ability.
Categories, language, and visual working memory: how verbal labels change capacity limitations
The limited capacity of visual working memory constrains the quantity and quality of the information we can store in mind for ongoing processing. Research from our lab has demonstrated that verbal labeling/categorization of visual inputs increases its retention and fidelity in visual working memory. In this talk, I will outline the hypotheses that explain the interaction between visual and verbal inputs in working memory, leading to the boosts we observed. I will further show how manipulations of the categorical distinctiveness of the labels, the timing of their occurrence, to which item labels are applied, as well as their validity modulate the benefits one can draw from combining visual and verbal inputs to alleviate capacity limitations. Finally, I will discuss the implications of these results to our understanding of working memory and its interaction with prior knowledge.
Inclusive Human Participant Research
Human participant research is somehow both antithetical and complementary to science. On the one hand, working with human participants provides incredibly rich and complex data with ‘real-world’ ecological validity. On the other, this richness is due to the incredible number of variables which uncontrollably become intertwined with your research interest, potentially limiting the conclusions you can draw from your work. Historical over-representation of white men as research participants, coupled with often overly-stringent exclusion criteria has led to a diversity crisis in human participant research. For our research to be truly inclusive, representative and generalisable to the rest of the population, our data must be collected from diverse individuals. This session will explore common barriers to diversity in studies with human participants, and will provide guidance on how to make sure your own research is accessible and inclusive.
Psychological mechanisms and functions of 5-HT and SSRIs in potential therapeutic change: Lessons from the serotonergic modulation of action selection, learning, affect, and social cognition
Uncertainty regarding which psychological mechanisms are fundamental in mediating SSRI treatment outcomes and wide-ranging variability in their efficacy has raised more questions than it has solved. Since subjective mood states are an abstract scientific construct, only available through self-report in humans, and likely involving input from multiple top-down and bottom-up signals, it has been difficult to model at what level SSRIs interact with this process. Converging translational evidence indicates a role for serotonin in modulating context-dependent parameters of action selection, affect, and social cognition; and concurrently supporting learning mechanisms, which promote adaptability and behavioural flexibility. We examine the theoretical basis, ecological validity, and interaction of these constructs and how they may or may not exert a clinical benefit. Specifically, we bridge crucial gaps between disparate lines of research, particularly findings from animal models and human clinical trials, which often seem to present irreconcilable differences. In determining how SSRIs exert their effects, our approach examines the endogenous functions of 5-HT neurons, how 5-HT manipulations affect behaviour in different contexts, and how their therapeutic effects may be exerted in humans – which may illuminate issues of translational models, hierarchical mechanisms, idiographic variables, and social cognition.
Precision and Temporal Stability of Directionality Inferences from Group Iterative Multiple Model Estimation (GIMME) Brain Network Models
The Group Iterative Multiple Model Estimation (GIMME) framework has emerged as a promising method for characterizing connections between brain regions in functional neuroimaging data. Two of the most appealing features of this framework are its ability to estimate the directionality of connections between network nodes and its ability to determine whether those connections apply to everyone in a sample (group-level) or just to one person (individual-level). However, there are outstanding questions about the validity and stability of these estimates, including: 1) how recovery of connection directionality is affected by features of data sets such as scan length and autoregressive effects, which may be strong in some imaging modalities (resting state fMRI, fNIRS) but weaker in others (task fMRI); and 2) whether inferences about directionality at the group and individual levels are stable across time. This talk will provide an overview of the GIMME framework and describe relevant results from a large-scale simulation study that assesses directionality recovery under various conditions and a separate project that investigates the temporal stability of GIMME’s inferences in the Human Connectome Project data set. Analyses from these projects demonstrate that estimates of directionality are most precise when autoregressive and cross-lagged relations in the data are relatively strong, and that inferences about the directionality of group-level connections, specifically, appear to be stable across time. Implications of these findings for the interpretation of directional connectivity estimates in different types of neuroimaging data will be discussed.
Algorithmic advances in face matching: Stability of tests in atypical groups
Face matching tests have traditionally been developed to assess human face perception in the neurotypical range, but methods that underlie their development often make it difficult for these measures to be applied in atypical populations (developmental prosopagnosics, super recognizers) due to unadjusted difficulty. We have recently presented the development of the Oxford Face Matching Test, a measure that bases individual item-difficulty on algorithmically derived similarity of presented stimuli. The measure seems useful as it can be given online or in-laboratory, has good discriminability and high test-retest reliability in the neurotypical groups. In addition, it has good validity in separating atypical groups at either of the spectrum ends. In this talk, I examine the stability of the OFMT and other traditionally used measures in atypical groups. On top of the theoretical significance of determining whether reliability of tests is equivalent in atypical population, this is an important question because of the practical concerns of retesting the same participants across different lab groups. Theoretical and practical implications for further test development and data sharing are discussed.
Cross-validation under different sensory conditions reveals the practical validity of the MVUE model
FENS Forum 2024