title: MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition
summary: In this paper, we study a novel problem in egocentric action recognition,
  which we term as "Multimodal Generalization" (MMG). MMG aims to study how
  systems can generalize when data from certain modalities is limited or even
  completely missing. We thoroughly investigate MMG in the context of standard
  supervised action recognition and the more challenging few-shot setting for
  learning new action categories. MMG consists of two novel scenarios, designed
  to support security, and efficiency considerations in real-world applications:
  (1) missing modality generalization where some modalities that were present
  during the train time are missing during the inference time, and (2)
  cross-modal zero-shot generalization, where the modalities present during the
  inference time and the training time are disjoint. To enable this
  investigation, we construct a new dataset MMG-Ego4D containing data points with
  video, audio, and inertial motion sensor (IMU) modalities. Our dataset is
  derived from Ego4D dataset, but processed and thoroughly re-annotated by human
  experts to facilitate research in the MMG problem. We evaluate a diverse array
  of models on MMG-Ego4D and propose new methods with improved generalization
  ability. In particular, we introduce a new fusion module with modality dropout
  training, contrastive-based alignment training, and a novel cross-modal
  prototypical loss for better few-shot performance. We hope this study will
  serve as a benchmark and guide future research in multimodal generalization
  problems. The benchmark and code will be available at
  https://github.com/facebookresearch/MMG_Ego4D.
paper_authors._id: 64619c09846a6c8c8304b7aa
paper_authors.name: Xinyu Gong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-15T15:28:10.203Z
paper_authors._id: 63ab45de8a357d5057ab7ce5
paper_authors.avatarUrl: /avatars/47c8cd29253ed8f1937c8223da30b80b.svg
paper_authors.fullname: Xinyu Gong
paper_authors.type: user
paper_authors.user: neogong
paper_authors._id: 64619c09846a6c8c8304b7ab
paper_authors.name: Sreyas Mohan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T12:53:54.656Z
paper_authors._id: 63ad24a5073766d0850d01d8
paper_authors.avatarUrl: /avatars/9d4662e639949c8ef7b431b09dd6d69d.svg
paper_authors.fullname: Sreyas Mohan
paper_authors.type: user
paper_authors.user: sreyasmohan
paper_authors._id: 64619c09846a6c8c8304b7ac
paper_authors.name: Naina Dhingra
paper_authors._id: 64619c09846a6c8c8304b7ad
paper_authors.name: Jean-Charles Bazin
paper_authors._id: 64619c09846a6c8c8304b7ae
paper_authors.name: Yilei Li
paper_authors._id: 64619c09846a6c8c8304b7af
paper_authors.name: Zhangyang Wang
paper_authors._id: 64619c09846a6c8c8304b7b0
paper_authors.name: Rakesh Ranjan
paper_title: MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition
paper_summary: In this paper, we study a novel problem in egocentric action recognition,
  which we term as "Multimodal Generalization" (MMG). MMG aims to study how
  systems can generalize when data from certain modalities is limited or even
  completely missing. We thoroughly investigate MMG in the context of standard
  supervised action recognition and the more challenging few-shot setting for
  learning new action categories. MMG consists of two novel scenarios, designed
  to support security, and efficiency considerations in real-world applications:
  (1) missing modality generalization where some modalities that were present
  during the train time are missing during the inference time, and (2)
  cross-modal zero-shot generalization, where the modalities present during the
  inference time and the training time are disjoint. To enable this
  investigation, we construct a new dataset MMG-Ego4D containing data points with
  video, audio, and inertial motion sensor (IMU) modalities. Our dataset is
  derived from Ego4D dataset, but processed and thoroughly re-annotated by human
  experts to facilitate research in the MMG problem. We evaluate a diverse array
  of models on MMG-Ego4D and propose new methods with improved generalization
  ability. In particular, we introduce a new fusion module with modality dropout
  training, contrastive-based alignment training, and a novel cross-modal
  prototypical loss for better few-shot performance. We hope this study will
  serve as a benchmark and guide future research in multimodal generalization
  problems. The benchmark and code will be available at
  https://github.com/facebookresearch/MMG_Ego4D.
paper_ai_summary: Research investigates multimodal generalization in egocentric action recognition through two scenarios: missing and cross-modal zero-shot, using a new dataset with video, audio, and IMU data, and introduces a fusion module, contrastive alignment, and cross-modal prototypical loss for improved few-shot performance.
paper_ai_keywords: multimodal generalization egocentric action recognition missing modality generalization cross-modal zero-shot generalization MMG-Ego4D modality dropout contrastive-based alignment cross-modal prototypical loss few-shot learning