SKELETON MOTION WORDS FOR UNSUPERVISED SKELETON-BASED TEMPORAL ACTION SEGMENTATION

Segmenting continuous behaviour into meaningful actions is a central step in the quantitative analysis of animal and human behaviour. In practice, this segmentation has often relied on manual annotation by human observers, which is time-consuming, subjective, and difficult to scale to long recordings and large behavioural datasets. These challenges have motivated the development of unsupervised behavioural segmentation methods. However, existing approaches often impose explicit models of behavioural state dynamics and are primarily applied within individual recordings, rather than to identifying consistent actions across datasets and subjects.

To address these limitations, we present Skeleton Motion Quantization (SMQ), an unsupervised framework for segmenting long, untrimmed skeleton recordings into globally consistent action segments across multiple sequences and subjects. SMQ encodes joint-level temporal dynamics using a dilated temporal convolutional autoencoder, while keeping information from different joints disentangled to avoid dominance by a subset of joints. The resulting latent sequences are divided into short, non-overlapping temporal patches and discretized via vector quantization, yielding a set of prototypical skeleton motion words that capture recurring patterns of movement. Assigning each patch to its closest motion word directly produces a temporally coherent segmentation of behaviour.

We evaluate SMQ on multiple human motion datasets spanning wearable sensors and full-body motion capture. Across datasets, SMQ consistently outperforms existing unsupervised action segmentation approaches, producing less fragmented and more coherent segments. By identifying action segments directly from kinematic data, SMQ provides a scalable and objective alternative to manual annotation, with potential applications in behavioural neuroscience and human movement analysis.