title: EvMic: Event-based Non-contact sound recovery from effective
    spatial-temporal modeling
summary: When sound waves hit an object, they induce vibrations that produce
  high-frequency and subtle visual changes, which can be used for recovering the
  sound. Early studies always encounter trade-offs related to sampling rate,
  bandwidth, field of view, and the simplicity of the optical path. Recent
  advances in event camera hardware show good potential for its application in
  visual sound recovery, because of its superior ability in capturing
  high-frequency signals. However, existing event-based vibration recovery
  methods are still sub-optimal for sound recovery. In this work, we propose a
  novel pipeline for non-contact sound recovery, fully utilizing spatial-temporal
  information from the event stream. We first generate a large training set using
  a novel simulation pipeline. Then we designed a network that leverages the
  sparsity of events to capture spatial information and uses Mamba to model
  long-term temporal information. Lastly, we train a spatial aggregation block to
  aggregate information from different locations to further improve signal
  quality. To capture event signals caused by sound waves, we also designed an
  imaging system using a laser matrix to enhance the gradient and collected
  multiple data sequences for testing. Experimental results on synthetic and
  real-world data demonstrate the effectiveness of our method.
paper_authors._id: 67f0a09a2c873f5ba90cd14a
paper_authors.name: Hao Yin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-06T08:11:07.967Z
paper_authors._id: 67ee782979018bf61e2522a4
paper_authors.avatarUrl: /avatars/819318421eb40b9ba7e4054770823a8a.svg
paper_authors.fullname: HaoYin
paper_authors.type: user
paper_authors.user: yyzqy
paper_authors._id: 67f0a09a2c873f5ba90cd14b
paper_authors.name: Shi Guo
paper_authors._id: 67f0a09a2c873f5ba90cd14c
paper_authors.name: Xu Jia
paper_authors._id: 67f0a09a2c873f5ba90cd14d
paper_authors.name: Xudong XU
paper_authors._id: 67f0a09a2c873f5ba90cd14e
paper_authors.name: Lu Zhang
paper_authors._id: 67f0a09a2c873f5ba90cd14f
paper_authors.name: Si Liu
paper_authors._id: 67f0a09a2c873f5ba90cd150
paper_authors.name: Dong Wang
paper_authors._id: 67f0a09a2c873f5ba90cd151
paper_authors.name: Huchuan Lu
paper_authors._id: 67f0a09a2c873f5ba90cd152
paper_authors.name: Tianfan Xue
paper_title: EvMic: Event-based Non-contact sound recovery from effective
    spatial-temporal modeling
paper_summary: When sound waves hit an object, they induce vibrations that produce
  high-frequency and subtle visual changes, which can be used for recovering the
  sound. Early studies always encounter trade-offs related to sampling rate,
  bandwidth, field of view, and the simplicity of the optical path. Recent
  advances in event camera hardware show good potential for its application in
  visual sound recovery, because of its superior ability in capturing
  high-frequency signals. However, existing event-based vibration recovery
  methods are still sub-optimal for sound recovery. In this work, we propose a
  novel pipeline for non-contact sound recovery, fully utilizing spatial-temporal
  information from the event stream. We first generate a large training set using
  a novel simulation pipeline. Then we designed a network that leverages the
  sparsity of events to capture spatial information and uses Mamba to model
  long-term temporal information. Lastly, we train a spatial aggregation block to
  aggregate information from different locations to further improve signal
  quality. To capture event signals caused by sound waves, we also designed an
  imaging system using a laser matrix to enhance the gradient and collected
  multiple data sequences for testing. Experimental results on synthetic and
  real-world data demonstrate the effectiveness of our method.
paper_ai_summary: A novel pipeline utilizing event cameras and a network with Mamba for long-term temporal modeling is proposed for non-contact sound recovery, demonstrating superior performance.
paper_ai_keywords: event cameras simulation pipeline Mamba spatial-temporal information spatial aggregation block event signals laser matrix