## INTERAURAL TIME DIFFERENCE LOSS FOR BINAURAL TARGET SOUND EXTRACTION

Carlos Hernandez-Olivan, Marc Delcroix, Tsubasa Ochiai, Naohiro Tawara, Tomohiro Nakatani, Shoko Araki

## NTT Corporation, Japan

## ABSTRACT

Binaural target sound extraction (TSE) aims to extract a desired sound from a binaural mixture of arbitrary sounds while preserving the spatial cues of the desired sound. Indeed, for many applications, the target sound signal and its spatial cues carry important information about the sound source. Binaural TSE can be realized with a neural network trained to output only the desired sound given a binaural mixture and an embedding characterizing the desired sound class as inputs. Conventional TSE systems are trained using signallevel losses, which measure the difference between the extracted and reference signals for the left and right channels. In this paper, we propose adding explicit spatial losses to better preserve the spatial cues of the target sound. In particular, we explore losses aiming at preserving the interaural level (ILD), phase (IPD), and time differences (ITD). We show experimentally that adding such spatial losses, particularly our newly proposed ITD loss, helps preserve better spatial cues while maintaining the signal-level metrics.

its binaural cues. Binaural processing can be achieved by extending a monaural TSE system to accept binaural input and output binaural signals. This idea was first introduced for speech enhancement [13] and recently applied to TSE [14]. We can train such systems using a signal-level loss on each output channel, which guides the model to output left and right signals as close as possible to the references. Such a training loss implicitly pushes the system to preserve spatial cues. However, prior works [13, 14] did not use any explicit spatial loss, which may limit the ability of TSE to recover spatial cues.

Index Terms -target sound extraction, binaural, deep learning

## 1. INTRODUCTION

Target sound extraction (TSE) aims at isolating a sound signal of interest belonging to a desired (or specified) sound class from a mixture of arbitrary sounds [1]. For example, TSE could extract the sound of a siren in a recording including also dog barking and car passing. The TSE problem can thus be seen as a generalization of the target speech extraction problem [2] to arbitrary sounds . Realizing TSE could have many practical applications, such as hearables or hearing aids that can focus on sounds of interest in everyday environments or smart audio post-production systems.

Atypical TSE system consists of a neural network that accepts a sound mixture and outputs the desired sound signal free from other sounds and noise. The network is conditioned on a clue that is used to identify the target sound in the mixture. Several types of clues can be used, such as a one-hot vector identifying the target sound class [1, 3], audio recording similar to the desired sound [3-5], video [6], onomatopoeia [7], the region where to focus [8] or multiple clues [9]. Here, we focus on TSE conditioned on the target sound class.

Most TSE frameworks focus on single-channel processing [3, 10]. However, humans rely on binaural hearing to capture spatial information about the sounds. For example, the interaural level difference (ILD) and interaural time difference (ITD) serve as essential cues to localize sound sources [11, 12]. It is thus essential to develop TSE systems that can extract a target sound while preserving

This work was completed during Carlos Hernandez-Olivan's internship at NTT Corporation. Email: carloshero@unizar.es, marc.delcroix@ntt.com. This work was supported by JST Strategic International Collaborative Research Program (SICORP), Grant Number JPMJSC2306, Japan.

This paper investigates whether adding an explicit spatial loss could further improve spatial cue preservation of binaural TSE. It is straightforward to define a loss measuring the error of ILD between the estimated and reference signals [15], as it only involves computing the difference of the ratio of the norm of the left and right signals, which is differentiable. In contrast, computing the ITD involves finding the position of the maximum in the cross-correlation between the left and right signals. This computation requires the 'argmax' operation, which is not differentiable, making it challenging to use as a loss. A recent work [15] has proposed instead a loss measuring the error of the interaural phase difference (IPD), which is related to the time difference. However, optimizing IPD errors may be challenging because of the phase wrapping problem.

We propose instead an alternative loss, which measures the errors between the cross-correlation coefficients of the estimated and the reference signals. By abuse of language, we refer to it as ITD loss. The proposed ITD loss is more directly related to the ITD computation than the IPD loss. We thus hope it will lead to better ITD cue preservation.

Note that a prior works [15, 16] have proposed using ILD and IPD losses for binaural speech enhancement. However, it is important to explore spatial losses for the TSE problem since it deals with a much larger variety of sounds than speech signals [1, 3, 5, 10]. To summarize, the contribution of this paper is twofold. First, we propose a new spatial loss that is directly related to ITD cues. Second, we compare three types of spatial losses (ITD, ILD, and the newly proposed ITD losses) for binaural TSE. We perform experiments on TSE of sound mixtures containing three to four sounds from 20 sound classes. We use a binaural TSE system as a baseline [14] and confirm experimentally that adding spatial losses can reduce ITD, ILD, and IPD errors while preserving signal-level metrics. Moreover, the newly proposed ITD loss achieves overall superior binaural TSE performance.

## 2. BINAURAL TARGET SOUND EXTRACTION

In this section, we first introduce the problem statement and describe a baseline binaural TSE model. We then discuss the signal-level and spatial losses we use to train the TSE system. In particular, we introduce our proposed ITD loss in Sec. 2.4.3.

Fig. 1 : Our proposed binaural TSE system with signal-level and spatial losses.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00344v1-figure-a09efa2d-8ba4-4a85-a757-1fc0432a3524.jpg)

## 2.1. Problem Statement

applying the decoder to the masked features as

The objective of TSE is to isolate a sound signal of interest belonging to a desired sound class from an observed signal x m consisting of a mixture of sounds defined as:

$$\mathbf x _ { m } = \sum _ { i = 1 } ^ { I } \mathbf s _ { i, m }, \quad \quad \quad ( 1 ) \quad \ \, \dot { \min } _ { \text{op} | }$$

where s i,m ∈ R T is the sound signal of the i -th source at the m -th microphone, T is the number of samples, and I is the total number of sound sources. In the following, we consider binaural recordings and define binaural mixture as x = [ x l , x r ] ∈ R T × 2 where l and r represent the index of the left and right microphones, respectively. Similarly, we denote by s tgt = [ s tgt l , s tgt r ] ∈ R T × 2 the binaural target signal, which corresponds to the desired sound class.

Binaural TSE aims to recover the binaural signal of the target by preserving the spatial information of the sound. Therefore, a TSE model follows the expression:

$$\hat { s } ^ { \text{tgf} } = T S E ( \mathbf x, \mathbf o ; \mathbf \theta ), \quad \quad \ \ ( 2 ) \quad 2. 3$$

where ˆ s tgt represents the extracted target sound signal, TSE( ) · is a neural network with parameters θ , and o = [0 , . . . , 1 , . . . 0] ⊤ is a 1-hot vector representing the target sound class, i.e., it has a value of 1 for the index of the target class and 0 otherwise. The 1-hot vector has a dimension O corresponding to the number of classes the models can handle.

## 2.2. TSE Model

Our TSE model consists of an encoder-decoder model with a mask estimation network, or extractor, as shown in Fig.1. Unlike conventional single-channel TSE models [3, 10], we use Semantic Hearing [14] as the baseline model, which is a binaural extension of the Waveformer model [10].

First, the input signal is processed by an encoder block:

$$Y = Encoder ( x ),$$

where Encoder ( ) · is the encoder layer, Y ∈ R T ′ × D is the encoded representation of the input signal, D is the feature dimension and T ′ is the number of frames. The encoded mixture representation is then passed to an extractor, which predicts a mask M ∈ R T ′ × D of the target sound to extract:

$$M = \text{Extractor} ( \boldsymbol Y, e ^ { t g t } ), \quad \quad ( 4 ) \quad \ ^ { - r r \cdot }$$

where Extractor ( ) · is an extractor neural network, and e tgt ∈ R D is an embedding vector characterizing the target sound class. The embedding vector can be simply obtained from an embedding layer, which accepts the one-hot vector characterizing the desired sound class, o , as input, i.e., e tgt = Wo , where W ∈ R D O × is an embedding matrix. Finally, the estimate of the target sound is obtained by

$$\hat { s } ^ { t g t } = \text{Decoder} ( M \odot Y ),$$

where Decoder ( ) · is the decoder layer, and ⊙ is the Hadamard product.

Training the TSE system requires the triplets of binaural sound mixtures, target sound class, and binaural target sound signals. The optimal model parameters, ˆ θ can be obtained by minimizing the training loss:

$$\hat { \theta } = \underset { \theta } { \arg \min } \, \mathcal { L } \left ( s ^ { \text{tg} }, \hat { s } ^ { \text{tg} } ( \theta ) \right ). \, \text{ \quad \ \ } \quad \ \ ( 6 )$$

Conventional TSE systems use only a signal-level training loss. Here, we investigate using a multi-task loss as:

$$\mathcal { L } = \alpha \mathcal { L } ^ { \text{signal} } + \beta \mathcal { L } ^ { \text{spatial} },$$

where α and β are weights, and L signal , L spatial are the signal-level and spatial losses, respectively, which we define below.

## 2.3. Signal-Level Losses

Conventional TSE systems are usually trained using a signal-level loss like the signal-to-noise ratio (SNR), the scale-invariant SNR (SI-SNR) [17] or a combination of both [10, 14]. We can use these losses for binaural outputs by computing a signal-level loss for each channel as proposed in [14]. For example, the SNR loss for binaural outputs is:

$$\mathcal { L } ^ { S N R } ( s ^ { t g t }, \hat { s } ^ { t g t } ) = - \left ( \frac { 1 } { 2 } S N R ( s ^ { t g t } _ { r }, \hat { s } ^ { t g t } _ { r } ) + \frac { 1 } { 2 } S N R ( s ^ { t g t } _ { l }, \hat { s } ^ { t g t } _ { l } ) \right ), \ ( 8 )$$

where SNR ( s s , ˆ) = 10log 10 ( || s || 2 2 || s - || ˆ s 2 2 ) is the SNR between the reference signal, s , and the estimated target signal, ˆ s . We can define a similar loss for the SI-SNR.

In this paper, following [14], we use a weighted sum of SNR and SI-SNR losses as the signal-level loss:

$$\mathcal { L } ^ { \text{signal} } = 0. 9 \mathcal { L } ^ { \text{SNR} } + 0. 1 \mathcal { L } ^ { \text{SI-SNR} }. \text{ \quad \ \ } ( 9 )$$

## 2.4. Spatial Losses

To help the TSE model better preserve spatial cues, we investigate adding binaural metrics to the signal-level losses. For this purpose, we employ ILD and IPD losses, which were previously proposed for speech enhancement [15], and also propose a new ITD-related loss.

## 2.4.1. Interaural Level Difference Loss

ILD aims to measure the level difference between the channels of a signal. The ILD of a binaural signal is defined as:

$$I L D = 1 0 \log _ { 1 0 } \left ( \frac { | | \mathbf s _ { l } | | _ { 2 } ^ { 2 } } { | | \mathbf s _ { r } | | _ { 2 } ^ { 2 } } \right ), \quad \quad ( 1 0 )$$

where s l ∈ R T and s r ∈ R T are left and right signals.

Then, the ILD loss can be expressed as the mean of the absolute difference of the target and predicted ILDs:

$$\mathcal { L } ^ { \text{ILD} } = \left | \text{ILD} ^ { \text{tg} } - \widehat { ILD} ^ { \text{tg} } \right |, \quad \quad ( 1 1 ) \quad \text{when} \\ \text{sigr}$$

where ILD tgt and ILD ̂ tgt are the ILD of the reference and extracted target sound signals computed with Eq. (10). Note that prior work [15] defines this loss in the Short-Time Fourier Transform (STFT) domain, whereas we define it in the waveform domain.

## 2.4.2. Interaural Phase Difference Loss

The time difference of arrival (TDOA) between microphones translates into phase differences between the received sound signals. To preserve the spatial information of the sound source, one approach is to focus on maintaining the IPD of the reference left and right signals in the extracted signals. The IPD between two signals can be calculated using the STFT of these signals. However, the phase of a complex number has inherent periodicity (i.e., 0 and 2 π are equivalent). This property, known as the circularity problem, makes direct subtraction of phases unreliable. To address this, we compute the IPD between the two channels of a signal as:

$$I P D _ { u, v } = \text{atan} \left ( \frac { \text{Im} \left ( S _ { u, v, l } S _ { u, v, r } ^ { * } \right ) } { \text{Re} \left ( S _ { u, v, l } S _ { u, v, r } ^ { * } \right ) } \right ), \quad \text{(12)} \quad \text{betw} \\ \text{as} \colon$$

where S u,v,r , S u,v,l ∈ C represent the STFT of the right and left signals, u v , are the indexes of the time and frequency bins, ∗ is the conjugate operation, Im ( ) · and Re ( ) · represent the imaginary and real parts of a complex number, and atan ( ) · denotes the arctangent function.

To leverage the IPD as a loss function, we compute the MSE between the IPD of the target and predicted signals:

$$\mathcal { L } ^ { \text{IPD} } = \frac { 1 } { U V } \sum _ { u = 1 } ^ { U } \sum _ { v = 1 } ^ { V } \left ( \text{IPD} _ { u, v } ^ { \text{tg} } - \widehat { \text{IPD} } _ { u, v } ^ { \text{tg} t } \right ) ^ { 2 }, \quad ( 1 3 )$$

where IPD tgt and IPD ̂ tgt are the IPD of the reference and extracted target sound computed with Eq. (12), and U and V are the number of time and frequency bins, respectively.

Note that a similar IPD loss was used in [15]. They computed the IPD directly as the angle of the ratio S u,v,l /S u,v,r 1 . This is mathematically equivalent to Eq. (12) but handles the phase wrapping differently as Eq. (12) defines the phase difference between -π/ 2 and π/ 2 , i.e., the smallest possible phase difference. Besides, they used a binary mask to limit the IPD computation to the regions where the source is active. In our experiments, we did not use such a mask as it led to poorer performance, probably because of the difficulty of defining a mask suitable for the diversity of sounds covered by TSE.

## 2.4.3. Proposed Interaural Time Difference Loss

Our aim is to develop a TSE system that can preserve the ITD of the target sound since it is an important spatial cue used by humans to localize sounds [11, 12]. Therefore, we propose training the TSE model to output signals with an ITD close to that of the target sound.

ITD measures the difference in time of sound arrival between the left and right microphones. It can be obtained by finding the position of the highest peak in the cross-correlation between the left and right

1 https://github.com/VikasTokala/BCCTN

signals of the target sound. We can compute the ITD as:

$$I T D = \underset { t \in [ - \tau, \tau ] } { \arg \max } \ c _ { t }, \quad \text{ \quad \ \ } ( 1 4 )$$

where c t is the cross-correlation coefficient between the left and right signals at time step t . τ is a scalar limiting the predicted delay to be in a given range, which is related to the distance between the microphones.

Typically, the cross-correlation is computed using the generalized cross-correlation phase transform (GCC-PHAT) algorithm [18] as follows:

$$\mathbf c = \mathcal { F } ^ { - 1 } \left ( \frac { \mathcal { F } ( \mathbf s _ { l } ) \odot \mathcal { F } ( \mathbf s _ { r } ) ^ { * } } { | \mathcal { F } ( \mathbf s _ { l } ) \odot \mathcal { F } ( \mathbf s _ { r } ) ^ { * } | } \right ), \quad \quad ( 1 5 )$$

where c = [ c t = -T , . . . , c t =0 , . . . , c t = T ] ∈ R 2 T +1 is the vector of the cross-correlation coefficients, F and F -1 are the Fourier transform (FT) and inverse FT (IFT), respectively. We use the entire signal (here 6 seconds in both training and evaluation) as the windows length to compute F .

A loss on the ITD should measure the difference between the ITD of the reference target sound signals, s tgt , and that of the extracted signals, ˆ s tgt . However, as seen in Eq. (14), the ITD computation involves the argmax operation, which is not differentiable. Therefore, we define the ITD loss as the mean squared error (MSE) between the cross-correlation of the reference and extracted signals as:

$$\mathcal { L } ^ { \text{ITD} } = \frac { 1 } { 2 \tau + 1 } \sum _ { t = - a u } ^ { \tau } \left ( c _ { t } ^ { \text{tgt} } - \hat { c } _ { t } ^ { \text{tgt} } \right ) ^ { 2 }, \text{ \quad \ \ } ( 1 6 )$$

where c tgt t and ˆ c tgt t are the cross-correlation between the reference and extracted signals computed with Eq. (15). Note that the proposed ITD loss is differentiable since all operations required to compute the cross-correlations, including FT and IFT, are differentiable.

## 3. EXPERIMENTS

## 3.1. Experimental Settings

## 3.1.1. Datasets

In our experiments, we used an openly available dataset of binaural sound mixtures [14]. The data consists of simulated reverberant mixtures of three to four sound events added to urban background noise. This dataset leverages 20 sound classes from FSD50K [19] (generalpurpose), ESC-50 [20] (environmental sounds), MUSDB18 [21] and noise files for the DISCO dataset [22]. The background sounds were taken from TAU Urban Acoustic Scenes 2019 [23]. Binaural mixtures were generated by convolving the sound source signals with room impulse responses (RIRs) and head-related transfer functions (HRTFs) as it is done in Semantic Hearing [14]. We used HRTFS from 43 subjects from the CIPIC corpus [24], and real and simulated RIRs from three corpora [25-27]. The sampling frequency of all signals was 44.1 kHz.

We mixed the data on the fly with Scaper [28]. From each mixture, we only extracted one of the foreground sources (the same one across the experiments) [14]. The training was done with 6-second mixtures. The number of training, validation, and testing mixtures were 100K, 1K, and 10K, respectively.

Table 1 : signal-level and spatial metrics for mixture, baseline TSE using only spatial loss and proposed systems using signal-level and spatial losses. We report ∆ ITD-GCC values (using GCC-PHAT as in Eq. (15)) and in parenthesis ∆ ITD values (using simple cross-correlation).

|        |                          |               | Spatial Metrics   | Spatial Metrics            | Spatial Metrics             |          |
|--------|--------------------------|---------------|-------------------|----------------------------|-----------------------------|----------|
| System | System                   | ↑ SI-SNR [dB] | ↑ SNR [dB]        | ↓ ∆ ILD [dB] ↓ ∆ IPD [rad] | ↓ ∆ ITD-GCC ( ∆ ITD) [ µ s] | ↓ FR [%] |
| (1)    | Mixture                  | -0.74         | -0.73             | 2.68 0.84                  | 235.7 (263.0)               | -        |
| (2)    | Baseline TSE w/ L signal | 6.50          | 7.85              | 0.84 0.88                  | 163.5 (86.3)                | 0.17     |
| (3)    | (2) + L ILD              | 6.72          | 8.10              | 0.74 0.83                  | 168.5 ( 74 . 8 )            | 0.16     |
| (4)    | (2) + L IPD              | 6.76          | 8.03 0.79         | 0.49                       | 242.9 (80.1)                | 0.16     |
| (5)    | (2) + L ITD              | 6.74          | 8.11              | 0.78 0.84                  | 137.3 (79.0)                | 0.16     |

## 3.1.2. System Configuration

We employed the same configuration for the TSE system as used in [14]. 2 The system performs online processing with a latency of 20 msec. It has 1.74M trainable parameters in total.

a low-pass filter in the loss computation, which will be part of future investigations.

The model consists of a 1-D convolution and 1-D transposed convolution for the encoder and decoder, respectively. The encoder 1-D convolution layer has a stride of L = 32 samples and a kernel size of K = 3 L . After the mask is obtained with the extractor, the 1-D transposed convolution layer returns the waveform of the target sound. The extractor consists of an encoder-decoder architecture conditioned on the target class embedding by multiplying the output of the dilated convolution with the target sound embedding, e tgt . The encoder is a stack of 1-D dilated convolution (DCC) layers followed by a Transformer-like layer. The DCC layers have a kernel size of K =3 and dilation factors set to { 2 0 , 2 1 , ..., 2 9 } . The Transformerlike layer consists of two multi-head attention (MHA) layers with 8 heads.

We trained all models for 80 epochs using a batch size of 32 and a learning rate of 5e-4. For the IPD loss, we computed the complex STFT with a Hann window of 1024, a hop length of 256, and an FFT of 1024 samples. For all models, we used the signal-level loss defined in Eq. (9), and combined it with the spatial losses according to Eq. (7) with fixed weight α = 1 . We selected the hyperparameters, including the loss weights β and the number of epochs, that achieved the best spatial metric among those that did not degrade signal-level metrics on the validation set. Accordingly, the values of β were chosen to 0.1, 1, and 1 for the ILD, IPD, and ITD losses, respectively. For ITD loss, we set τ = 1 ms.

## 3.1.3. Evaluation Metrics

Following Semantic Hearing [14], we measured the performance on both signal-level (SI-SNR and SNR), and spatial metrics by computing the difference between the reference and extracted signals, i.e., ∆ ILD, ∆ IPD, and ∆ ITD. ∆ ILD and ∆ IPD were computed with Eq. (11) and Eq. (13), respectively. ∆ ITD was obtained as the absolute difference between the ITD of the reference and extracted signals, where ITD is obtained with Eq. (14). To compute ITD, we used the GCC-PHAT as it is known to be more robust to reverberation. However, we also report results with the simple cross-correlation for comparison with prior work [14]. We name these metrics ∆ ITDGCCand ∆ ITD, respectively. Note that prior works designed spatial metrics considering human perception [11, 29], by, e.g., computing ITD only on the frequencies below 1.5 kHZ and ILD on the frequencies above 3 kHz. With such evaluation metrics, we observed a small improvement over the baseline of about 5 µ s with our proposed ITD loss. However, we expect a larger improvement if we would include

2 We used the model variant D = 256 proposed by Semantic Hearing https://github.com/vb000/SemanticHearing

Finally, we also report the failure rate (FR) to provide a rough estimate of how often TSE failed to correctly identify the target sound in the mixture. We defined FR as the percentage of test samples for which SI-SNR improvement is below 1dB [30].

## 3.2. Experimental Results

Table 1 compares the performance of the proposed losses with that of the unprocessed mixture and a baseline TSE that only uses the signal-level loss of Eq. (9) [14]. Note that the ILD, ITD and, IPD values computed on the mixture are only provided as a reference. We observe that the baseline TSE greatly reduces ILD and ITD errors compared to the mixture but not IPD. Actually, most experiments failed to improve IPD errors, which may suggest the difficulty of using IPD metrics for TSE.

Using the spatial losses in addition to the signal-level loss (systems (3)-(5)) outperforms the baseline TSE system (system (2)) for most metrics. These results show that the multi-task loss does not degrade extraction performance in terms of signal-level metrics and FR (it even slightly improves it).

We confirm that adding a spatial loss improves performance on the related spatial metric. Using the ILD loss (system (3)) performs best in terms of ∆ ILD but does not consistently improve ∆ ITD. Using the IPD loss (system (4)) greatly reduces ∆ IPD, and still slightly improves ∆ ILD. However, it does not translate into improving ITD errors, suggesting the limit of the IPD loss. In contrast, using our newly proposed ITD loss (system (5)) improves all spatial metrics compared to the baseline while achieving comparable SI-SNR, SNR, and FR values. The improvement in terms of ∆ ITD is particularly large, i.e., 26.2 µ s (more than 1 sample) or a relative improvement of 16 %. These results confirm that our proposed ITD loss contributes to improving spatial cue preservation of TSE.

## 4. CONCLUSIONS

In this paper, we introduced a new loss function based on ITD, which preserves binaural properties and improves ITD difference between left and right channels. Our proposed ITD loss is superior to a conventional IPD loss as it improves binaural cue recovery of TSE in terms of ILD, IPD, and ITD, while maintaining performance in terms of signal-level metrics.

The proposed ITD loss is general and could be used for other speech and audio processing tasks, such as binaural speech enhancement or speech separation. Future works will include such investigations.

## 5. REFERENCES

- [1] Tsubasa Ochiai, Marc Delcroix, Yuma Koizumi, Hiroaki Ito, Keisuke Kinoshita, and Shoko Araki, 'Listen to What You Want: Neural Network-Based Universal Sound Selector,' in Proc. Interspeech , 2020, pp. 1441-1445.
- [2] Katerina Zmol´ ıkov´ a, Marc Delcroix, Tsubasa Ochiai, Keisuke Kinoshita, Jan Cernock´ y, and Dong Yu, 'Neural target speech extraction: An overview,' IEEE Signal Process. Mag. , vol. 40, no. 3, pp. 8-29, 2023.
- [3] Marc Delcroix, Jorge Bennasar V´ azquez, Tsubasa Ochiai, Keisuke Kinoshita, Yasunori Ohishi, and Shoko Araki, 'Soundbeam: Target sound extraction conditioned on soundclass labels and enrollment clues for increased performance and continuous learning,' IEEE ACM Trans. Audio Speech Lang. Process. , vol. 31, pp. 121-136, 2023.
- [4] Kateˇ rina Zmol´ ıkov´, Marc Delcroix, Keisuke Kinoshita, Tsubˇ a asa Ochiai, Tomohiro Nakatani, Luk´ aˇ s Burget, and Jan ˇ ernock`, C y 'Speakerbeam: Speaker aware neural network for target speaker extraction in speech mixtures,' IEEE Journal of Selected Topics in Signal Processing , vol. 13, no. 4, pp. 800814, 2019.
- [5] Beat Gfeller, Dominik Roblek, and Marco Tagliasacchi, 'Oneshot conditional audio filtering of arbitrary sounds,' in Proc. ICASSP , 2021, pp. 501-505.
- [6] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh H. McDermott, and Antonio Torralba, 'The sound of pixels,' in Proc. Computer Vision - ECCV . 2018, vol. 11205, pp. 587-604, Springer.
- [7] Yuki Okamoto, Shota Horiguchi, Masaaki Yamamoto, Keisuke Imoto, and Yohei Kawaguchi, 'Environmental sound extraction using onomatopoeic words,' in Proc. ICASSP , 2022, pp. 221-225.
- [8] Rongzhi Gu and Yi Luo, 'Rezero: Region-customizable sound extraction,' IEEE ACM Trans. on Audio, Speech, and Lang. Proc. , vol. 32, pp. 2576-2589, 2024.
- [9] Chenda Li, Yao Qian, Zhuo Chen, Dongmei Wang, Takuya Yoshioka, Shujie Liu, Yanmin Qian, and Michael Zeng, 'Target sound extraction with variable cross-modality clues,' in Proc. ICASSP , 2023.
- [10] Bandhav Veluri, Justin Chan, Malek Itani, Tuochao Chen, Takuya Yoshioka, and Shyamnath Gollakota, 'Real-time target sound extraction,' in Proc. ICASSP .
- [11] Jens Blauert, 'Spatial hearing: The psychophysics of human sound localization,' Spatial hearing , 1997.
- [12] David R Moore, 'Anatomy and physiology of binaural hearing,' Audiology , vol. 30, no. 3, pp. 125-134, 1991.
- [13] Cong Han, Yi Luo, and Nima Mesgarani, 'Real-time binaural speech separation with preserved spatial cues,' in Proc. ICASSP , 2020, pp. 6404-6408.
- [14] Bandhav Veluri, Malek Itani, Justin Chan, Takuya Yoshioka, and Shyamnath Gollakota, 'Semantic hearing: Programming acoustic scenes with binaural hearables,' in Proc. Symposium on User Interface Software and Technology (UIST) . 2023, pp. 89:1-89:15, ACM.
- [15] Vikas Tokala, Eric Grinstein, Mike Brookes, Simon Doclo, Jesper Jensen, and Patrick A. Naylor, 'Binaural speech enhancement using deep complex convolutional transformer networks,' in Proc. ICASSP . 2024, pp. 681-685, IEEE.
- [16] T.J. Klasen, S. Doclo, T. Van den Bogaert, M. Moonen, and J. Wouters, 'Binaural multi-channel wiener filtering for hearing aids: Preserving interaural time and level differences,' in Proc. ICASSP , 2006, vol. 5, pp. V-V.
- [17] Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey, 'SDR-half-baked or well done?,' in Proc. ICASSP , 2019, pp. 626-630.
- [18] Jeffrey L. Krolik, M. Joy, Subbarayan Pasupathy, and Moshe Eizenman, 'A comparative study of the LMS adaptive filter versus generalized correlation method for time delay estimation,' in Proc. ICASSP , 1984, pp. 652-655.
- [19] Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra, 'FSD50K: an open dataset of human-labeled sound events,' IEEE ACM Trans. Audio Speech Lang. Process. , vol. 30, pp. 829-852, 2022.
- [20] Karol J. Piczak, 'ESC: dataset for environmental sound classification,' in Proc. Annual ACM Conference on Multimedia . 2015, pp. 1015-1018, ACM.
- [21] Zafar Rafii, Antoine Liutkus, Fabian-Robert St¨ oter, Stylianos Ioannis Mimilakis, and Rachel Bittner, 'The MUSDB18 corpus for music separation,' Dec. 2017.
- [22] Nicolas Furnon, Romain Serizel, Slim Essid, and Irina Illina, 'Dnn-based mask estimation for distributed speech enhancement in spatially unconstrained microphone arrays,' IEEE ACM Trans. on Audio, Speech, and Lang. Proc. , vol. 29, pp. 2310-2323, 2021.
- [23] Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen, 'A multi-device dataset for urban acoustic scene classification,' in Proc. DCASE , 2018, pp. 9-13.
- [24] V Ralph Algazi, Richard O Duda, Dennis M Thompson, and Carlos Avendano, 'The CIPIC HRTF database,' in Proc. WASPAA , 2001, pp. 99-102.
- [25] D. Satongar, Y. Lam, and C. Pike, 'The Salford BBC spatiallysampled binaural room impulse response dataset,' 2015.
- [26] IoSR-Surrey, 'Iosr-surrey/realroombrirs: Binaural impulse responses captured in real rooms,' https://github.com/ IoSR-Surrey/RealRoomBRIRs , 2016.
- [27] IoSR-Surrey, 'Simulated room impulse responses,' https: //iosr.uk/software/index.php , 2023, Accessed: April 24, 2024.
- [28] Justin Salamon, Duncan MacConnell, Mark Cartwright, Peter Li, and Juan Pablo Bello, 'Scaper: A library for soundscape synthesis and augmentation,' in Proc. WASPAA , 2017, pp. 344-348.
- [29] Mathias Dietz, Stephan D Ewert, and Volker Hohmann, 'Auditory model based direction estimation of concurrent speakers from binaural signals,' Speech Communication , vol. 53, no. 5, pp. 592-605, 2011.
- [30] Marc Delcroix, Keisuke Kinoshita, Tsubasa Ochiai, Katerina Zmolikova, Hiroshi Sato, and Tomohiro Nakatani, 'Listen only to me! How well can target speech extraction handle false alarms?,' in Proc. Interspeech , 2022, pp. 216-220.
Carlos Hernandez-Olivan Marc Delcroix Tsubasa Ochiai Naohiro Tawara Tomohiro Nakatani Shoko Araki
Interaural time difference loss for binaural target sound extraction
Binaural target sound extraction (TSE) aims to extract a desired sound from a
binaural mixture of arbitrary sounds while preserving the spatial cues of the
desired sound. Indeed, for many applications, the target sound signal and its
spatial cues carry important information about the sound source. Binaural TSE
can be realized with a neural network trained to output only the desired sound
given a binaural mixture and an embedding characterizing the desired sound
class as inputs. Conventional TSE systems are trained using signal-level
losses, which measure the difference between the extracted and reference
signals for the left and right channels. In this paper, we propose adding
explicit spatial losses to better preserve the spatial cues of the target
sound. In particular, we explore losses aiming at preserving the interaural
level (ILD), phase (IPD), and time differences (ITD). We show experimentally
that adding such spatial losses, particularly our newly proposed ITD loss,
helps preserve better spatial cues while maintaining the signal-level metrics.