Skip to contentSkip to filters

We use cookies for analytics.

An Egocentric Approach to Social AI

Jim Rehg

University of Illinois Urbana-Champaign

Recording

54 min

Abstract

While computer vision and NLP have made tremendous progress in extracting semantics from images, video and text, computational understanding of human social behavior is still in its infancy. Face-to-face communication, using a rich repertoire of visual, acoustic and linguistic channels, is the foundation of all social interaction, and the acquisition of social skill in infancy is a critical developmental milestone whose disruption, as in autism, has life-long consequences. Current AI largely performs surface-level analyses, such as action detection, recognition and prediction from video or sentiment inference from utterances; a major challenge is to address the core constructs of social interaction, including joint attention, theory of mind and social appraisals. A key hypothesis is that computational social understanding is enabled by an egocentric perspective: capturing social signals from the viewpoint of each social partner through head- and body-worn sensors, an approach aligned with commercial work in augmented and virtual reality and with the child-development literature.

The talk gives background on egocentric perception and summarizes progress toward egocentric social understanding. A key technical challenge is inferring social attention from multimodal sensor data by analysing video of naturalistic interactions with machine learning models, without eye tracking. It reviews recent progress in estimating visual and auditory attention from egocentric data, describes a benchmark dataset for multimodal social understanding built on multi-person social deduction games such as One Night Werewolf, and reviews progress toward modeling social attention to improve the diagnosis and treatment of autism.

Topics