Control Theory seminars

Multi-agent reinforcement learning (MARL) has recently shown great promise as an approach to networked system control. Arguably, one of the most difficult and important tasks for which large scale networked system control is applicable is common-pool resource (CPR) management. Crucial CPRs include arable land, fresh water, wetlands, wildlife, fish stock, forests and the atmosphere, of which proper management is related to some of society’s greatest challenges such as food security, inequality and climate change. This talk will consist of three parts. In the first, we will briefly look at climate change and how it poses a significant threat to life on our planet. In the second, we will consider the potential of multi-agent systems for climate change mitigation and adaptation. And finally, in the third, we will discuss recent research from InstaDeep into better understanding the behaviour of networked MARL systems used for CPR management. More specifically, we will see how the tools from empirical game-theoretic analysis may be harnessed to analyse the differences in networked MARL systems. The results give new insights into the consequences associated with certain design choices and provide an additional dimension of comparison between systems beyond efficiency, robustness, scalability and mean control performance.

Brain-machine interfaces (BMIs) change how the brain sends and receives information from the environment, opening new ways to probe brain function. For instance, motor BMIs allow us to precisely define and manipulate the sensorimotor loop which has enabled new insights into motor control and learning. In this talk, I’ll first present an example study where sensory-motor loop manipulations in BMI allowed us to probe feed-forward and feedback control mechanisms in ways that are not possible in the natural motor system. This study shed light on sensorimotor processing, and in turn led to state-of-the-art neural interface performance. I’ll then survey recent work that highlights the likelihood that BMIs, much like natural motor learning, engages multiple distributed learning mechanisms that can be carefully interrogated with BMI.

Learning is a population phenomenon. Since it is the organized activity of populations of neurons that cause movement, learning a new skill must involve reshaping those population activity patterns. Seeing how the brain does this has been elusive, but a brain-computer interface approach can yield new insight. We presented monkeys with novel BCI mappings that we knew would be difficult for them to learn how to control. Over several days, we observed the emergence of new patterns of neural activity that endowed the animals with the ability to perform better at the BCI task. We speculate that there also exists a direct relationship between new patterns of neural activity and new abilities during natural movements, but it is much harder to see in that setting.

Brain-computer interfaces (BCIs) have afforded paralysed users “mental control” of computer cursors and robots, and even of electrical stimulators that reanimate their own limbs. Most existing BCIs map the activity of hundreds of motor cortical neurons recorded with implanted electrodes into control signals to drive these devices. Despite these impressive advances, the field is facing a number of challenges that need to be overcome in order for BCIs to become widely used during daily living. In this talk, I will focus on two such challenges: 1) having BCIs that allow performing a broad range of actions; and 2) having BCIs whose performance is robust over long time periods. I will present recent studies from our group in which we apply neuroscientific findings to address both issues. This research is based on an emerging view about how the brain works. Our proposal is that brain function is not based on the independent modulation of the activity of single neurons, but rather on specific population-wide activity patters —which mathematically define a “neural manifold”. I will provide evidence in favour of such a neural manifold view of brain function, and illustrate how advances in systems neuroscience may be critical for the clinical success of BCIs.

Model-based approaches to control and decision making have long held the promise of being more powerful and data efficient than model-free counterparts. However, success with model-based methods has been limited to those cases where a perfect model can be queried. The game of Go was mastered by AlphaGo using a combination of neural networks and the MCTS planning algorithm. But planning required a perfect representation of the game rules. I will describe new algorithms that instead leverage deep neural networks to learn models of the environment which are then used to plan, and update policy and value functions. These new algorithms offer hints about how brains might approach planning and acting in complex environments.