The internship aims to explore the usefulness of the Fisher-Ráo metric combined with deep probabilistic models. The main question is whether or not this metric has some relationship with the training of deep generative models. In plain, we would like to understand if the training and/or fine-tuning of such probabilistic models follow optimal paths on the manifold of probability distributions. Your task will be to design and implement an experimental framework allowing to measure what kind of paths are followed on the manifold of probability distributions when such deep probabilistic models are trained. To that aim, one must first be able to measure distances in this manifold, and here is where the Fisher-Ráo metric comes in the game. The candidate does not need to be familiar with the specific concepts of Fisher-Ráo metric, but needs to be open to learning new mathematical concepts. The implementation of these experiments will require knowledge in Python and in PyTorch.

The Cyprus Institute invites applications for a Post-Doctoral Fellow to pursue research in Machine Learning. The successful candidate will be actively engaged in cutting-edge research in terms of core problems in ML and AI such as developing efficient and interpretable deep nets, continual learning, neuro-inspired ML, self-supervised learning, and other cutting-edge topics. The candidate should have deep understanding of machine learning fundamentals (e.g., linear algebra, probability theory, optimization) as well as broad knowledge of the state-of-the-art in AI and machine and learning. Additionally, the candidate should have extensive experience with ML programming frameworks (e.g., PyTorch). The candidate will be working primarily with two PIs: Prof. Constantine Dovrolis and Prof. Mihalis Nicolaou. The appointment is for a period of 2 years, with the option of renewal subject to performance and the availability of funds.

Spiking recurrent neural networks (RNNs) are a promising tool for solving a wide variety of complex cognitive and motor tasks, due to their rich temporal dynamics and sparse processing. However training spiking RNNs on dedicated neuromorphic hardware is still an open challenge. This is due mainly to the lack of local, hardware-friendly learning mechanisms that can solve the temporal credit assignment problem and ensure stable network dynamics, even when the weight resolution is limited. These challenges are further accentuated, if one resorts to using memristive devices for in-memory computing to resolve the von-Neumann bottleneck problem, at the expense of a substantial increase in variability in both the computation and the working memory of the spiking RNNs. In this talk, I will present our recent work where we introduced a PyTorch simulation framework of memristive crossbar arrays that enables accurate investigation of such challenges. I will show that recently proposed e-prop learning rule can be used to train spiking RNNs whose weights are emulated in the presented simulation framework. Although e-prop locally approximates the ideal synaptic updates, it is difficult to implement the updates on the memristive substrate due to substantial device non-idealities. I will mention several widely adapted weight update schemes that primarily aim to cope with these device non-idealities and demonstrate that accumulating gradients can enable online and efficient training of spiking RNN on memristive substrates.

We introduce Norse: An open-source library for gradient-based training of spiking neural networks. In contrast to neuron simulators which mainly target computational neuroscientists, our library seamlessly integrates with the existing PyTorch ecosystem using abstractions familiar to the machine learning community. This has immediate benefits in that it provides a familiar interface, hardware accelerator support and, most importantly, the ability to use gradient-based optimization. While many parallel efforts in this direction exist, Norse emphasizes flexibility and usability in three ways. Users can conveniently specify feed-forward (convolutional) architectures, as well as arbitrarily connected recurrent networks. We strictly adhere to a functional and class-based API such that neuron primitives and, for example, plasticity rules composes. Finally, the functional core API ensures compatibility with the PyTorch JIT and ONNX infrastructure. We have made progress to support network execution on the SpiNNaker platform and plan to support other neuromorphic architectures in the future. While the library is useful in its present state, it also has limitations we will address in ongoing work. In particular, we aim to implement event-based gradient computation, using the EventProp algorithm, which will allow us to support sparse event-based data efficiently, as well as work towards support of more complex neuron models. With this library, we hope to contribute to a joint future of computational neuroscience and neuromorphic computing.

Norse aims to exploit the advantages of bio-inspired neural components, which are sparse and event-driven - a fundamental difference from artificial neural networks. Norse expands PyTorch with primitives for bio-inspired neural components, bringing you two advantages: a modern and proven infrastructure based on PyTorch and deep learning-compatible spiking neural network components.

In this workshop you'll learn the fundamentals of PyTorch using an incremental, from-first-principles approach. We'll start with tensors, autograd, and the dynamic computation graph, and then move on to developing and training a simple model using PyTorch's model classes, datasets, data loaders, optimizers, and more. You should be comfortable using Python, Jupyter notebooks, Google Colab, Numpy and, preferably, object oriented programming.