54 min
Seminar, Hosted by Toyota Technological Institute at Chicago (TTIC)
An Egocentric Approach to Social AI
University of Illinois Urbana-Champaign
Recording
Abstract
While computer vision and NLP have made tremendous progress in extracting semantics from images, video and text, computational understanding of human social behavior is still in its infancy. Face-to-face communication, using a rich repertoire of visual, acoustic and linguistic channels, is the foundation of all social interaction, and the acquisition of social skill in infancy is a critical developmental milestone whose disruption, as in autism, has life-long consequences. Current AI largely performs surface-level analyses, such as action detection, recognition and prediction from video or sentiment inference from utterances; a major challenge is to address the core constructs of social interaction, including joint attention, theory of mind and social appraisals. A key hypothesis is that computational social understanding is enabled by an egocentric perspective: capturing social signals from the viewpoint of each social partner through head- and body-worn sensors, an approach aligned with commercial work in augmented and virtual reality and with the child-development literature.
The talk gives background on egocentric perception and summarizes progress toward egocentric social understanding. A key technical challenge is inferring social attention from multimodal sensor data by analysing video of naturalistic interactions with machine learning models, without eye tracking. It reviews recent progress in estimating visual and auditory attention from egocentric data, describes a benchmark dataset for multimodal social understanding built on multi-person social deduction games such as One Night Werewolf, and reviews progress toward modeling social attention to improve the diagnosis and treatment of autism.