Learning Relational Rules from Rewards

Humans perceive the world in terms of objects and relations between them. In fact, for any given pair of objects, there is a myriad of relations that apply to them. How does the cognitive system learn which relations are useful to characterize the task at hand? And how can it use these representations to build a relational policy to interact effectively with the environment? In this paper we propose that this problem can be understood through the lens of a sub-field of symbolic machine learning called relational reinforcement learning (RRL). To demonstrate the potential of our approach, we build a simple model of relational policy learning based on a function approximator developed in RRL. We trained and tested our model in three Atari games that required to consider an increasingly number of potential relations: Breakout, Pong and Demon Attack. In each game, our model was able to select adequate relational representations and build a relational policy incrementally. We discuss the relationship between our model with models of relational and analogical reasoning, as well as its limitations and future directions of research.

NMC4 Short Talk: What can deep reinforcement learning tell us about human motor learning and vice-versa ?

In the deep reinforcement learning (RL) community, motor control problems are usually approached from a reward-based learning perspective. However, humans are often believed to learn motor control through directed error-based learning. Within this learning setting, the control system is assumed to have access to exact error signals and their gradients with respect to the control signal. This is unlike reward-based learning, in which errors are assumed to be unsigned, encoding relative successes and failures. Here, we try to understand the relation between these two approaches, reward- and error- based learning, and ballistic arm reaches. To do so, we test canonical (deep) RL algorithms on a well-known sensorimotor perturbation in neuroscience: mirror-reversal of visual feedback during arm reaching. This test leads us to propose a potentially novel RL algorithm, denoted as model-based deterministic policy gradient (MB-DPG). This RL algorithm draws inspiration from error-based learning to qualitatively reproduce human reaching performance under mirror-reversal. Next, we show MB-DPG outperforms the other canonical (deep) RL algorithms on a single- and a multi- target ballistic reaching task, based on a biomechanical model of the human arm. Finally, we propose MB-DPG may provide an efficient computational framework to help explain error-based learning in neuroscience.

Deep kernel methods

Deep neural networks (DNNs) with the flexibility to learn good top-layer representations have eclipsed shallow kernel methods without that flexibility. Here, we take inspiration from deep neural networks to develop a new family of deep kernel method. In a deep kernel method, there is a kernel at every layer, and the kernels are jointly optimized to improve performance (with strong regularisation). We establish the representational power of deep kernel methods, by showing that they perform exact inference in an infinitely wide Bayesian neural network or deep Gaussian process. Next, we conjecture that the deep kernel machine objective is unimodal, and give a proof of unimodality for linear kernels. Finally, we exploit the simplicity of the deep kernel machine loss to develop a new family of optimizers, based on a matrix equation from control theory, that converges in around 10 steps.

What neural oscillations can(not) do for syntactic structure building

The question of how syntactic structure can be built at the neural level has come to the forefront of cognitive neuroscience in the last decade. Neural oscillations have been widely recognised as playing an important role in building syntactic representations. In this talk I will review existing oscillatory approaches to syntactic structure building and assess their functionality in light of basic properties of a hierarchical syntactic structure, such as varied length of syntactic phrases, nesting of constituents, overlap in length between different levels of the syntactic hierarchy and others. I will also briefly discuss key requirements on neural structure building mechanisms from the perspective of a real-time parser.

Plasticity of interneuron networks within the hippocampus

Swarms for people

As tiny robots become individually more sophisticated, and larger robots easier to mass produce, a breakdown of conventional disciplinary silos is enabling swarm engineering to be adopted across scales and applications, from nanomedicine to treat cancer, to cm-sized robots for large-scale environmental monitoring or intralogistics. This convergence of capabilities is facilitating the transfer of lessons learned from one scale to the other. Cm-sized robots that work in the 1000s may operate in a way similar to reaction-diffusion systems at the nanoscale, while sophisticated microrobots may have individual capabilities that allow them to achieve swarm behaviour reminiscent of larger robots with memory, computation, and communication. Although the physics of these systems are fundamentally different, much of their emergent swarm behaviours can be abstracted to their ability to move and react to their local environment. This presents an opportunity to build a unified framework for the engineering of swarms across scales that makes use of machine learning to automatically discover suitable agent designs and behaviours, digital twins to seamlessly move between the digital and physical world, and user studies to explore how to make swarms safe and trustworthy. Such a framework would push the envelope of swarm capabilities, towards making swarms for people.

Cholinergic modulation of the cerebellum

Many studies have investigated the major glutamatergic inputs to the cerebellum, mossy fibres and climbing fibres, however far less is known about its neuromodulatory inputs. In particular, anatomical studies have described cholinergic input to the cerebellum, yet little is known about its role(s). In this talk, I will present our recent findings which demonstrate that manipulating acetylcholine receptors in the cerebellum causes effects at both a cellular and behavioural level. Activating acetylcholine receptors alters the intrinsic properties and synaptic inputs of cerebellar output neurons, and blocking these receptors results in deficits in a range of behavioural tasks.

Inclusive Human Participant Research

Human participant research is somehow both antithetical and complementary to science. On the one hand, working with human participants provides incredibly rich and complex data with ‘real-world’ ecological validity. On the other, this richness is due to the incredible number of variables which uncontrollably become intertwined with your research interest, potentially limiting the conclusions you can draw from your work. Historical over-representation of white men as research participants, coupled with often overly-stringent exclusion criteria has led to a diversity crisis in human participant research. For our research to be truly inclusive, representative and generalisable to the rest of the population, our data must be collected from diverse individuals. This session will explore common barriers to diversity in studies with human participants, and will provide guidance on how to make sure your own research is accessible and inclusive.

Inclusive Data Science

A single person can be the source of billions of data points, whether these are generated from everyday internet use, healthcare records, wearable sensors or participation in experimental research. This vast amount of data can be used to make predictions about people and systems: what is the probability this person will develop diabetes in the next year? Will commit a crime? Will be a good employee? Is of a particular ethnicity? Predictions are simply represented by a number, produced by an algorithm. A single number in itself is not biased. How that number was generated, interpreted and subsequently used are all processes deeply susceptible to human bias and prejudices. This session will explore a philosophical perspective of data ethics and discuss practical steps to reducing statistical bias. There will be opportunity in the last section of the session for attendees to discuss and troubleshoot ethical questions from their own analyses in a ‘Data Clinic’.

Finding the Fault Lines: Detecting Urban Social Boundaries using Social Data Science

In urban environments, social boundaries are the areas that emerge from processes of economic inequality and social segregation. These boundaries are important, as they serve both as areas of interaction and conflict. By applying geographical thinking to classic methods in data science, we can better understand where these boundaries emerge and how they delineate communities. In this talk, I’ll explain a bit about the basics of “boundary detection” in urban analytics. I’ll present a new method, the “geosilhouette,” that builds on previous methods of identifying the boundaries between clusters. And, finally, I’ll show how this can change our understanding of urban community.

Modelling affective biases in rodents: behavioural and computational approaches

My research focuses, broadly speaking, on how emotions impact decision making. Specifically, I am interested in affective biases, a phenomenon known to be important in depression. Using a rodent decision-making task, combined with computational modelling I have investigated how different antidepressant and pro-depressant manipulations that are known to alter mood in humans alter judgement bias, and provided insight into the decision processes that underlie these behaviours. I will also highlight how the combination of behaviour and modelling can provide a truly translation approach, enabling comparison and interpretation of the same cognitive processes between animal and human research.

‘Optimistic’ and ‘pessimistic’ decision-making as an indicator of animal emotion and welfare

Reliable and validated measures of emotion in animals are of great import; they are crucial to better understanding and developing treatments for human mood disorders, and they are necessary for ensuring good animal welfare. We have developed a novel measure of emotion in animals that is grounded in theory and psychological research – decision-making under ambiguity. Specifically, we consider that more ‘optimistic’ decisions about ambiguous stimuli reflect more positive emotional states, while the opposite is true for more ‘pessimistic’ decisions. In this talk, we will outline the background behind and implementation of this measure, meta-analyses that have been conducted to validate the measure, and discuss how computational modelling has been used to further understand the cognitive processes underlying ‘optimistic’ and ‘pessimistic’ decision-making as an indicator of animal emotion and welfare.

Investigating the impact of the pandemic on adolescent anxiety and cognitive function

Meg was awarded funding to look into how the coronavirus pandemic has affected children's mental health and wellbeing.

Flow, fluctuate and freeze: Epithelial cell sheets as soft active matter

Epithelial cell sheets form a fundamental role in the developing embryo, and also in adult tissues including the gut and the cornea of the eye. Soft and active matter provides a theoretical and computational framework to understand the mechanics and dynamics of these tissues.I will start by introducing the simplest useful class of models, active brownian particles (ABPs), which incorporate uncoordinated active crawling over a substrate and mechanical interactions. Using this model, I will show how the extended ’swirly’ velocity fluctuations seen in sheets on a substrate can be understood using a simple model that couples linear elasticity with disordered activity. We are able to quantitatively match experiments using in-vitro corneal epithelial cells.Adding a different source of activity, cell division and apoptosis, to such a model leads to a novel 'self-melting' dense fluid state. Finally, I will discuss a direct application of this simple particle-based model to the steady-state spiral flow pattern on the mouse cornea.

Spinners, not swimmers: how sperm flagella fooled us for 350 years - now in 3D!

In the 17th century, Antonie van Leeuwenhoek used one of the earliest microscopes to see how sperm swim. He described the sperm as a “living animalcule” with a “tail, which, when swimming, lashes with a snakelike movement, like eels in water”. Strikingly, this perception of how sperm moves has not changed since. Indeed, anyone today with a modern microscope would make the same observation: sperm swim forward by wiggling their tail symmetrically side-to-side. Our new research using 3D microscopy shows that we have all been victims of a sperm deception, an illusion. Only now we can see that for 350 years we have been wrong about how sperm actually swims.