ePoster

Elucidating the hierarchical organization of natural behavior with masked autoencoders

Lucas Stoffland 3 co-authors

Presenting Author

Conference
FENS Forum 2024 (2024)
Messe Wien Exhibition & Congress Center, Vienna, Austria

Conference

FENS Forum 2024

Messe Wien Exhibition & Congress Center, Vienna, Austria

Resources

Authors & Affiliations

Lucas Stoffl, Andy Bonnetto, Stéphane d'Ascoli, Alexander Mathis

Abstract

Understanding the hierarchical organization of natural behavior is a fundamental objective in neuroscience. Despite the availability of unsupervised methods for behavioral analysis, hierarchical variants are still lacking. Here, we introduce hBehaveMAE, a Masked Autoencoder framework designed to elucidate the hierarchical nature of motion capture data in an unsupervised fashion. Our hierarchical framework offers interpretability, with lower encoder levels capturing fine-grained movements, while higher discerning complex actions. To validate the effectiveness of hBehaveMAE, we present results from its application to two distinct datasets. First, we introduce Shot7M2, a synthetic basketball playing benchmark annotated with movemes, actions, and activities across 7.2 million frames. hBehaveMAE demonstrates strong performance on this synthetic dataset, highlighting its ability to capture hierarchical behavioral structures. Moving beyond synthetic data, we validate hBehaveMAE on MABe22, a benchmark representing short and long-term behavioral states of interacting mice. hBehaveMAE achieves state-of-the-art performance without the need for preprocessing, showcasing its efficacy in real-world experimental settings. In summary, our work contributes valuable insights to neuroscience, offering a powerful framework for studying behavioral hierarchies in synthetic and experimental datasets.

Unique ID: fens-24/elucidating-hierarchical-organization-e5b932e9