Self-Supervised Generation of Spatial Audio for 360 • Video
We introduce an approach to convert mono audio recorded by a 360
• video camera into spatial audio, a representation of the distribution of sound over the full viewing sphere. Spatial audio is an important component of immersive 360
• video viewing, but spatial audio microphones are still rare in current 360
• video production. Our system consists of end-to-end trainable neural networks that separate individual sound sources and localize them on the viewing sphere, conditioned on multi-modal analysis of audio and 360
• video frames. We introduce several datasets, including one filmed ourselves, and one collected in-the-wild from YouTube, consisting of 360
• videos uploaded with spatial audio. During training, ground-truth spatial audio serves as self-supervision and a mixed down mono track forms the input to our network. Using our approach, we show that it is possible to infer the spatial location of sound sources based only on 360
• video and a mono audio track.
The paper proposes an automatic method to spatialise mono-channel audio for 360degree videos, meaning that the output of the method are several audio channels (most commonly two when human-centered applications are targetted) from which one can understand where the sound source is located. The method is based on a CNN which inputs the video and the mono-channel, and that is trained with datasets for which the ambisonic sound tracks are available. The network has a specific path for sound, for RGB and for optical flow. RGB and flow are processed and the fused, further on with audio, in order to obtain an audio-visual embedding of the scene. This embedding is fed to the audio decoder, which produces a set of TF attenuation maps (one per source). The very same audio-visual embedding is used to produce the localization weights, which are combined with the STFT attenuated by the output attenuation maps. This combination allows to generate the spatialized (STFT and hence) wavemors.  I enjoyed reading the paper, the application is definitely useful, the methodology interesting, and the text reads well. However, I think the paper needs to be more sharp in some aspects that, right now, may be a bit fuzzy/confusing.  For instance, all the claims regarding the novelty of the paper should be very precise. This study is perhaps the first addressing spatialisation from video and mono-audio, but not the first work addressing spatialisation (the audio processing literature is populated with such studies).  I do not know how restrictive is to assume that "monaural audio is recorded with an omnidirectional microphone" (L122). I believe that many comercial recording devices do NOT satisfy this constraint, thus clearly limiting the applicative interest of the method. How could this be done for non-omnidirectional microphones?  The authors assume that it is required to separate $k$ sources. I think an analysis of what happens when there are more or less sources in the scene is required to understand the impact of the choice of k. Clearly, not all scenarios have the same number of sources. For instance, could (a variation of) the system be trained for k sources plus a "trash" extra source so that it can handle more than k sources? What happens when there are less sources: how do the extra separated sources sound?  There is a small notation problem that carries an underlying conceptual problem. The authors should write very clearly that the input of the network is i(t) and not \phi_w(t). I understand that under the right conditions, the former is a very good surrogate of the later, but in any case they are conceptually the same. Therefore I strongly suggest that the manuscript is revised in this direction.  When discussing the audio separation, the authors describe the inversion procedure very roughly. This deserve more details (for instance, I understand you use ISTFT and not only IFFT). Do you overlap-add? This details deserve to be mentioned. The very same happens with the envelope distance: its description is to rough not allowing for reproducibility.  Finally the reference format should be uniformized.