Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark
<|reference_start|>Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark: We present a new dataset called Real Acoustic Fields (RAF) that captures real acoustic room data from multiple modalities. The dataset includes high-quality and densely captured room impulse response data paired with multi-view images, and precise 6DoF pose tracking data for sound emitters and listeners in the rooms. We used this dataset to evaluate existing methods for novel-view acoustic synthesis and impulse response generation which previously relied on synthetic data. In our evaluation, we thoroughly assessed existing audio and audio-visual models against multiple criteria and proposed settings to enhance their performance on real-world data. We also conducted experiments to investigate the impact of incorporating visual data (i.e., images and depth) into neural acoustic field models. Additionally, we demonstrated the effectiveness of a simple sim2real approach, where a model is pre-trained with simulated data and fine-tuned with sparse real-world data, resulting in significant improvements in the few-shot learning approach. RAF is the first dataset to provide densely captured room acoustic data, making it an ideal resource for researchers working on audio and audio-visual neural acoustic field modeling techniques. Demos and datasets are available on our project page: https://facebookresearch.github.io/real-acoustic-fields/<|reference_end|>
@article{chen2024real,
    title={Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and
  Benchmark},
    author={Ziyang Chen, Israel D. Gebru, Christian Richardt, Anurag Kumar,
  William Laney, Andrew Owens, Alexander Richard},
    journal={arXiv preprint arXiv:2403.18821},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.18821},
    primaryClass={cs.SD cs.CV cs.MM eess.AS}
}