title: IMUSIC: IMU-based Facial Expression Capture
summary: For facial motion capture and analysis, the dominated solutions are generally
  based on visual cues, which cannot protect privacy and are vulnerable to
  occlusions. Inertial measurement units (IMUs) serve as potential rescues yet
  are mainly adopted for full-body motion capture. In this paper, we propose
  IMUSIC to fill the gap, a novel path for facial expression capture using purely
  IMU signals, significantly distant from previous visual solutions.The key
  design in our IMUSIC is a trilogy. We first design micro-IMUs to suit facial
  capture, companion with an anatomy-driven IMU placement scheme. Then, we
  contribute a novel IMU-ARKit dataset, which provides rich paired IMU/visual
  signals for diverse facial expressions and performances. Such unique
  multi-modality brings huge potential for future directions like IMU-based
  facial behavior analysis. Moreover, utilizing IMU-ARKit, we introduce a strong
  baseline approach to accurately predict facial blendshape parameters from
  purely IMU signals. Specifically, we tailor a Transformer diffusion model with
  a two-stage training strategy for this novel tracking task. The IMUSIC
  framework empowers us to perform accurate facial capture in scenarios where
  visual methods falter and simultaneously safeguard user privacy. We conduct
  extensive experiments about both the IMU configuration and technical components
  to validate the effectiveness of our IMUSIC approach. Notably, IMUSIC enables
  various potential and novel applications, i.e., privacy-protecting facial
  capture, hybrid capture against occlusions, or detecting minute facial
  movements that are often invisible through visual cues. We will release our
  dataset and implementations to enrich more possibilities of facial capture and
  analysis in our community.
paper_authors._id: 65c307f3e272bf45aa1b45da
paper_authors.name: Youjia Wang
paper_authors._id: 65c307f3e272bf45aa1b45db
paper_authors.name: Yiwen Wu
paper_authors._id: 65c307f3e272bf45aa1b45dc
paper_authors.name: Ruiqian Li
paper_authors._id: 65c307f3e272bf45aa1b45dd
paper_authors.name: Hengan Zhou
paper_authors._id: 65c307f3e272bf45aa1b45de
paper_authors.name: Hongyang Lin
paper_authors._id: 65c307f3e272bf45aa1b45df
paper_authors.name: Yingwenqi Jiang
paper_authors._id: 65c307f3e272bf45aa1b45e0
paper_authors.name: Yingsheng Zhu
paper_authors._id: 65c307f3e272bf45aa1b45e1
paper_authors.name: Guanpeng Long
paper_authors._id: 65c307f3e272bf45aa1b45e2
paper_authors.name: Jingya Wang
paper_authors._id: 65c307f3e272bf45aa1b45e3
paper_authors.name: Lan Xu
paper_authors._id: 65c307f3e272bf45aa1b45e4
paper_authors.name: Jingyi Yu
paper_title: IMUSIC: IMU-based Facial Expression Capture
paper_summary: For facial motion capture and analysis, the dominated solutions are generally
  based on visual cues, which cannot protect privacy and are vulnerable to
  occlusions. Inertial measurement units (IMUs) serve as potential rescues yet
  are mainly adopted for full-body motion capture. In this paper, we propose
  IMUSIC to fill the gap, a novel path for facial expression capture using purely
  IMU signals, significantly distant from previous visual solutions.The key
  design in our IMUSIC is a trilogy. We first design micro-IMUs to suit facial
  capture, companion with an anatomy-driven IMU placement scheme. Then, we
  contribute a novel IMU-ARKit dataset, which provides rich paired IMU/visual
  signals for diverse facial expressions and performances. Such unique
  multi-modality brings huge potential for future directions like IMU-based
  facial behavior analysis. Moreover, utilizing IMU-ARKit, we introduce a strong
  baseline approach to accurately predict facial blendshape parameters from
  purely IMU signals. Specifically, we tailor a Transformer diffusion model with
  a two-stage training strategy for this novel tracking task. The IMUSIC
  framework empowers us to perform accurate facial capture in scenarios where
  visual methods falter and simultaneously safeguard user privacy. We conduct
  extensive experiments about both the IMU configuration and technical components
  to validate the effectiveness of our IMUSIC approach. Notably, IMUSIC enables
  various potential and novel applications, i.e., privacy-protecting facial
  capture, hybrid capture against occlusions, or detecting minute facial
  movements that are often invisible through visual cues. We will release our
  dataset and implementations to enrich more possibilities of facial capture and
  analysis in our community.
paper_ai_summary: A novel IMU-based approach for facial expression capture, IMUSIC, uses a tailored micro-IMU architecture, a custom dataset, and a Transformer diffusion model to achieve high accuracy in privacy-protected scenarios and under occlusions.
paper_ai_keywords: IMU IMUSIC micro-IMUs anatomy-driven placement IMU-ARKit facial blendshape parameters Transformer diffusion model two-stage training strategy