Deep reinforcement learning

Most walking animals can maintain motor control after limb injury or amputation, where coordinating movements of a drastically altered body requires dynamic interactions between the nervous system, the biomechanics of the body, and the physical environment. Until recently, it has been challenging to link neural and biomechanical models to investigate adaptive motor recovery because it requires coupling neural control in closed-loop with the environment. Further, while previous approaches have been able to simulate walking in biomechanically realistic bodies, the joint kinematics and ground rea

The vast majority of natural animal behavior involves far more than passive perception and reflexive responses. Indeed, animals actively move themselves in the world and systematically collect sensory observations over time; such active sensing allows them to infer environmental states that are not directly measurable, which ultimately guide ethologically important behaviors. One clear example of such a behavior is odor plume localization in flying insects. By changing their speed and turning, insects produce variations in their multisensory inputs that, once integrated over time, can resolve

Maintaining physiological homeostasis is essential for survival. To achieve this, brains integrate various inputs, such as internal physiological and external environmental signals, to guide behavior. Inspired by brain-body literature, we consider three key axes along which the dynamics of these inputs can differ: their timescales, dependence on actions, and contribution to error signals that drive behavior. To understand how these axes impact physiological homeostasis, we consider an agent that can influence its internal physiology through interactions with the environment. We assume that env

…of visual conditions after only a few learning flights, (a feat so far unrivalled by any artificial system). Furthermore, they perform complex navigational tasks, such as ‘computing’ novel shortcuts between previously known but unrelated food sources, supporting claims about the existence of a ‘cognitive map’ in honeybees. This raises the question which internal representations of the world (and the animal’

…We use a deep reinforcement learning model to study sensory integration in a navigation task in two different scenarios. In the first scenario, there is

…obscures the actual dynamics of learning, which can occur within few trials and differ between subjects. For instance, reexamination of previous data (Uengoer and Lachnit, 2006) reveals that during learning subjects do not change their responses gradually, but abruptly (Fig. A). These “jumps” occur at different trials for each subject and averaging therefore paints a misleading picture of gradual learning curves. These data are not easily accounted for with simple associative lea

…Here, we investigate the potential of deep reinforcement learning to capture sensorimotor representations. By training artificial agents to imitate natural grasp movements using imitation learning, we develop stimul

How does the motor cortex (MC) achieve generalizable and purposeful movements from the complex non-linear musculoskeletal system? This understanding can help us uncover how does the cortical activity relates to the movement. To elucidate this relationship, we build a model of the MC with realistic properties. To mimic the recurrent connections between cortical neurons, we use recurrent neural networks (RNNs) to model the MC. The MC model receives sensory feedback and delivers muscle excitations to anatomically accurate musculoskeletal models to produce experimentally observed movements. The se

…In deep reinforcement learning (RL), artificial agents can extract re-usable skills from past experience and recombine them in a hierarchical manner. It remains largely unknown, however, whether the brain similarly composes a novel behavior. Here we trained deep RL agents with the soft actor-critic (SAC) algorithm and studied their representation of RL variables during hierarchical learning. The objective of SAC is to maximize future cumulative rewards and policy

…using artificial agents trained with reinforcement learning to develop an integrated understanding of the behaviors and neural comput

…Here, we use a deep reinforcement learning model to investigate whether a navigation strategy could arise spontaneously during spatial learning without imposing a bias onto the

…methods in 3D pose estimation and deep reinforcement learning, we trained a latent variable model to control a virtual rodent body to imitate the natural behaviors of real rats in a physics simulator. Once trained, the model faithfully replicated diverse rat behaviors and gen

How does the motor cortex achieve generalizable and purposeful movements from the complex, nonlinear musculoskeletal system? ​​Previous research in this field has focused on developing dimensionality reduction and modeling techniques to elucidate the structure in high-dimensional neural activity, and relate this directly to kinematic outcomes. However, these models typically do not consider the biophysical underpinnings of the musculoskeletal system, nor do they allow us to understand the role of sensory feedback in motor control. These models thus fail to elucidate the computational role of n

Deep reinforcement learning (RL) has successfully trained machines to excel in video games, board games, and robotics. It has been argued that together, deep RL and robotics can be used to study biological learning by virtue of being embodied intelligence in the real world (Tan et al., 2021). In fact, RL itself was originally formulated to model animal behavior (Sutton & Barto, 1998). Here we go one step further and directly interface deep RL agents with a living neural network: that of the nematode C. elegans. We present a hybrid deep RL - C. elegans closed-loop computational system where

…or how ToM could emerge through learning to act in a social environment. We sought to identify network mechanisms of ToM in a recurrent neural network trained to play a competitive neuroscience task that requires ToM. The network, trained using deep reinforcement learni

The representation of “what happened when” is central to encoding episodic and working memories. Recently discovered hippocampal “time cells” are theorized to provide the neural substrate for such representations by forming distinct sequences that encode both time elapsed and sensory content. However, while multiple neurophysiological studies have presented contradictory results on the role of “time cells” in memory, little work has directly addressed this discrepancy. Here, we hypothesize that this discrepancy is a result of different studies using tasks that involve different cognitive deman

…cannot be used to infer within-trial learning trends. Behavioral features such as speed or trial duration capture trends in the agent's decision-making, but do not necessarily indicate that the agent is getting better at the task. We address this issue by modeling the task as a Markov Decision Process (MDP). The Q values of the optimal policy measu

In the deep reinforcement learning (RL) community, motor control problems are usually approached from a reward-based learning perspective. However, humans are often believed to learn motor control through directed error-based learning. Within this learning setting, the control system is assumed to have access to exact error signals and their gradients with respect to the control signal. This is unlike reward-based learning, in which errors are assumed to be unsigned, encoding relative successes and failures. Here, we try to understand the relation between these two approaches, reward- and erro

…have, to date, focused largely on deep neural networks trained using supervised learning, in tasks such as image classification. However, there is another area of recent AI work which has so far received less attention from neuroscientists, but which may have more profound neuroscientific implications: Deep reinforcement learning. Deep RL offers a rich framework for studying the interplay among learning, representation