ExpressEar : Sensing Fine-Grained Facial Expressions with Earables

Abstract

Employing my competency in affect detection and ML, I worked on sensing facial expressions with earables. Traditional, vision-based facial expression recognition methods are vulnerable to external factors like occlusion and lighting while also raising privacy concerns coupled with the impractical requirement of positioning the camera in front of the user at all times. To bridge this gap, we propose ExpressEar, an ear-mounted Inertial Measurement Unit (IMU) sensing technique capable of identifying fine-grained facial actions by leveraging the fact that different facial muscle movements produce distinct signal patterns in the IMU data stream. Following the Facial Action Coding System (FACS), which encodes every possible expression in terms of constituent movements called Action Units (AUs), ExpressEar identifies facial expressions at the atomic level employing a Temporal Convolutional Network (TCN) architecture. We conducted a user study (N=12) to evaluate the performance of our approach and found that ExpressEar can detect and distinguish between 32 Facial Action Units AUs (including 2 variants of asymmetric AUs) with an average accuracy of 89.9% for any given user. To inspect the effect on performance in free-living conditions, we evaluated the system under challenging noise and user mobility conditions. While ExpressEar was able to differentiate facial AUs from other ear canal deformations with an accuracy of 99.2%, we witnessed a slight drop in performance along with less extensive coverage of facial AUs for classification in mobile settings. We achieved an average per-user accuracy of 83.85% (SD: 1.25, Max: 85.61%, Chance: 7.143%) across 14 AUs performed in a moving rapid-transit train, and 84.34% (SD: 0.63, Max: 84.89%, Chance: 25%) across 4 AUs performed while walking. We envision that ExpressEar can not only yield information about the user’s affect and behaviour, but can also enable a unique opportunity to create a novel input space leveraging these expressions for intuitive human-machine interaction. It was a joint collaboration with WEAVE Lab, IIIT-Delhi.

Team Members:

D. Verma (IIIT Delhi), S. Bhalla (IIIT Delhi), D. Sahnan (IIIT Delhi)