## LONG-TERM CONVERSATION ANALYSIS: PRIVACY-UTILITY TRADE-OFF UNDER NOISE AND REVERBERATION

Jule Pohlhausen 1 2 , , Francesco Nespoli , Joerg Bitzer 3 1 4 ,

1 Jade University of Applied Sciences, Institute of Hearing Technology and Audiology, Oldenburg, Germany 2 Carl von Ossietzky Universit¨ at Oldenburg, Dept. Medical Physics and Acoustics, Germany 3 Microsoft, London, UK and Imperial College, Dept. Electrical and Electronic Engineering, London, UK 4 Fraunhofer IDMT Dept. HSA, Oldenburg, Germany

## ABSTRACT

Recordings in everyday life require privacy preservation of the speech content and speaker identity. This contribution explores the influence of noise and reverberation on the trade-off between privacy and utility for low-cost privacypreserving methods feasible for edge computing. These methods compromise spectral and temporal smoothing, speaker anonymization using the McAdams coefficient, sampling with a very low sampling rate, and combinations. Privacy is assessed by automatic speech and speaker recognition, while our utility considers voice activity detection and speaker diarization. Overall, our evaluation shows that additional noise degrades the performance of all models more than reverberation. This degradation corresponds to enhanced speech privacy, while utility is less deteriorated for some methods.

of speech data, as stated in the European general data protection regulation (EU GDPR). Hence, our goal is to record only acoustic features capable of analyzing conversations while protecting the two main privacy aspects of speech data: the linguistic speech content and the speaker identity [3]. These requirements define the trade-off between privacy and utility.

Index Terms -privacy, conversation analysis, speech recognition, speaker recognition, voice activity detection, speaker diarization

## 1. INTRODUCTION

The conversational behavior in our everyday life is valuable in predicting sociability and health outcomes [1]. This contribution focuses on tools for analyzing long-term recordings in everyday life, captured by portable devices over several days. In particular, the objective analysis of conversations requires a robust detection of voiced time segments (voice activity detection, V AD), which are subsequently attributed to different speakers (speaker diarization, SD). These steps reveal speaker turns and the involvement of different communication partners [2]. Since speech contains important personal identifiable information (PII) and real-life recordings involve uncontrolled environments, both in private and public, recordings in everyday life require privacy preservation and the protection

The authors would like to thank Volker Hohmann for his helpful comments on the article. This work was supported by the Graduation program of Jade University of Applied Sciences (Jade2Pro 2.0) and by the European Union Horizon 2020 program under the Marie Skłodowska-Curie grant No 95636.

Furthermore, the computational power, memory, and power consumption of portable recording devices demand features with low computational complexity. The described limitations imposed by portable devices conflict with the use of existing privacy-preserving methods like encrypting speech signals [4] or performing federated learning [5]. The conversation detection and SD system proposed by [2] is based on entropy, correlation, and energy features to preserve speech privacy. However, it requires one audio stream per speaker and a central node to combine the information from all streams, which is not feasible for unconstrained recording environments. Alternatively, the Electronically Activated Recorder (EAR) records intermittently snippets of audio, e.g., 30 s or 50 s recordings every 9 min or 12.5 min [6]. Since uninvolved parties could be recorded without their consent, this approach can not be applied in public environments. In contrast, this contribution focuses on low-cost methods such as spectral and temporal smoothing [7], speaker anonymization using the McAdams coefficient [8], and low-frequency audio [9].

The privacy evaluation considers the word error rate (WER) of an automatic speech recognition (ASR) model and the equal error rate (EER) of an automatic speaker verification (ASV) system. The utility performance is assessed with Matthews correlation coefficient (MCC) of a VAD model and the diarization error rate (DER) extracted from a SD system, which assigns unique speaker labels that are not linked to the speaker's identity. We aim to maintain utility performance while enforcing privacy, therefore minimizing DER while maximizing MCC, WER, and EER. Presently, most assessments rely on optimal testing conditions [10]. Hence, this contribution investigates the impact of noise and reverberation on the trade-off between privacy and utility.

## 2. PRIVACY-PRESERVING METHODS

Our main assumption is that all models rely on the same input features, namely log Mel filterbank energies computed on time segments of 25 ms with a hop size of L = 10 ms. This section summarizes the considered privacy-preserving methods applied during feature extraction or as pre-processing in the case of the McAdams anonymization [8].

## 2.1. Spectral smoothing

Analogously to reducing visual information content by pixelating images, mosaic speech corresponds to pixelated spectrograms that were smeared in the time-frequency domain and decrease the speech intelligibility [11]. Hence, in our case, spectral smoothing corresponds to decreasing the number of filters of the Mel filterbank while covering the same frequency range. We evaluated filterbanks with 80 Mel filters, which refers to the baseline performance, and 10 Mel filters.

## 2.2. Temporal smoothing

Similarly to mosaic speech [11], smoothed and subsampled power spectral densities (PSD) lead to degraded speech intelligibility [7]. These PSD were calculated with a window size of 25 ms and a hop size of L = 12 5 . ms. The PSD were smoothed with a first-order recursive filter with a smoothing time τ and subsampled by a factor of τ/L , which corresponds for τ = 125 ms to a subsampling factor τ/L = 10 , i.e., only every tenth smoothed PSD is stored. To restore the original time resolution of the frames, the smoothed and subsampled PSD were repeated by the subsampling factor before applying the Mel filterbank. The evaluation considers τ = 125 ms, 250 ms, 375 ms with corresponding subsampling factors.

## 2.3. McAdams speaker anonymization

The McAdams coefficient shifts the formant positions to hide speaker-related information while preserving the spoken content [8] and was the secondary baseline of the Voice Privacy Challenge (VPC) [10]. For semi-informed attackers, who have complete knowledge about the anonymization system, sampling a McAdams coefficient between 0.5 and 0.9 leads utterance-wise to strong privacy preservation in terms of EER [8], but speaker-wise only to modestly deteriorated EER [10] compared to unprotected signals. Since the McAdams coefficient was designed to preserve ASR, it deteriorated the WER only marginally [10, 8]. Therefore, to further investigate privacy preservation, we combined the McAdams coefficient with spectral and temporal smoothing.

## 2.4. Low-frequency audio

The wearable badges in [9] record low-frequency audio with a sampling rate of f s = 1250 Hz to collect privacy-preserving data of social conversations. For evaluating this method, frequencies above f / s 2 = 625 Hz were discarded.

## 3. EXPERIMENTAL SETUP

For assessing privacy and utility, this section describes the evaluation metrics, datasets, and augmentation. This contribution considers semi-informed attackers, i.e., the attacker has complete knowledge about the utilized privacy-preserving method and retrains or fine-tunes models on processed data. Following the VPC [10], the semi-informed attacker is the strongest attack paradigm. All experiments were conducted with SpeechBrain [12] using two Nvidia GeForce RTX 3090 Ti GPU and one Intel Alder Lake CPU. We report our results with confidence intervals, that were calculated with a confidence level α = 5 %using bootstrapping 1 .

## 3.1. Privacy Assessment

In this contribution, an ASR and an ASV model assess the privacy preservation of the speech content and the speaker identity, respectively. The ASR system is based on a transformer acoustic model encoder and a joint transformer decoder with connectionist temporal classification (CTC) [13], with the decoding stage integrating also CTC probabilities. The transcription performance is assessed with the WER, where a high WER is desirable in order to protect the speech content.

The ASV system uses the ECAPA-TDNN speaker encoder [14] coupled with cosine similarity scoring for the verification task. The privacy evaluation relies on the EER, which is the operating point defined by the detection threshold at which the false acceptance rate and the false rejection rate are equal. Therefore, a higher EER indicates better anonymization and an EER of 50 % reflects a random decision behavior. We report the weighted averaged EER on the VPC test subsets as in [10].

## 3.2. Utility Assessment

Our utility assessment considers a V AD and a SD model. The VAD model consists of a convolutional recurrent deep neural network [12] and performs binary classification, i.e., speech vs. non-speech, based on frame-level posterior probabilities which are processed with a sigmoid function. The VAD performance is evaluated with the MCC, which equals (-)1 for perfect (mis-)classification and 0 for random guessing.

The SD model employs the same ECAPA-TDNN embeddings as the ASV model and performs Spectral Clustering (SC) to assign a relative speaker label to each time segment [17]. For each meeting, the SD model estimates the number of speakers and utilizes oracle V AD information. We report the averaged DER with a forgiveness collar of 250 ms.

1 https://github.com/luferrer/ConfidenceIntervals

Fig. 1 . Privacy (ASR and ASV, orange) and utility (VAD and SD, blue) assessment for different SNR in dB, where the original test data is denoted by orig . The evaluation considers spectral smoothing (Mel), temporal smoothing ( τ in ms), McAdams anonymization [8], and low-frequency audio with f s = 1250 Hz [9]. The ASR is evaluated with the WER on the test-clean set of LibriSpeech [15]; the ASV is evaluated with the weighted averaged EER on the test sets of the VPC [10]; the VAD is evaluated with the MCC on the test split of LibriParty [12]; the SD estimates the number of speakers and is evaluated with the averaged DER on the eval set of the AMI Corpus [16]. The confidence intervals are given by the vertical lines.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00382v1-figure-615eb9a7-f698-43e3-9bf1-916073cb9ece.jpg)

## 3.3. Databases

tion, we randomly chose a McAdams coefficient per speaker for enrollment and trial data.

All experiments are performed on publicly available datasets sampled at 16 kHz. The ASR models were trained on the full 960 hours of LibriSpeech [15] and tested on the testclean subset. For speaker verification and diarization, the ECAPA-TDNN speaker encoder [14] was trained on VoxCeleb 2 [18]. The ASV performance was evaluated on the test splits of the VPC [10], while the SD performance was evaluated on the Augmented Multi-party Interaction (AMI) Meeting Corpus [16] with the standard Full-corpus-ASR partition using the HeadsetMix recording streams. The VAD models were trained and tested on the train and test split of LibriParty [12].

For semi-informed attackers, an anonymized version of LibriSpeech 360 h [15] is obtained by randomly sampling a McAdams coefficient from (0 5 . , 0 9) . for each utterance. This anonymized version was used to fine-tune the ASR model and the ECAPA-TDNN for further 30 epochs. Moreover, the VAD model was fully retrained on the anonymized train split of LibriParty [12], since it was the least complex model. All fine-tuned and retrained models were tested on anonymized versions of the corresponding test data. During ASV evalua-

For simulating typical environmental noise, 843 pointsource noises that are sampled from the Freesound portion of the MUSAN corpus [19, 20] were added to the test audio signals at signal-to-noise ratios (SNR) of 10 dB, 5 dB, and 0 dB, which are typical values for realistic sound environments [21].

For simulating reverberation, we convolved the test data with three different room impulse responses (RIR), which were measured in a meeting, office, and lecture room and correspond to reverberation times RT 60 of 0.21 s, 0.37 s, and 0.70 s, respectively [22].

## 4. RESULTS AND DISCUSSION

The results of our privacy and utility assessment with additional noise are presented in Figure 1 and with reverberation in Figure 2. Overall, for most methods in Section 2 both, noise and reverberation, lead to a degraded performance for all models, where the influence of SNR tends to be higher compared to RT 60 . For increasing SNR the evaluation met-

Fig. 2 . Privacy and utility assessment as in Figure 1, but for different reverberation times RT 60 in s.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00382v1-figure-13086924-c673-4cf1-9082-f967f53e013e.jpg)

rics degrade, however, this deterioration is in some cases not significant as indicated by overlapping confidence intervals.

For the ASR performance, the influence of noise and reverberation is most detrimental for methods that lead to a stronger information reduction, e.g., compare for spectral smoothing 80 and 10 Mel filters and for temporal smoothing τ of 125 ms and 250 ms. Moreover, for methods with high WER on the original (orig.) test data, the additional noise or reverberation leads to ceiling effects, e.g., compare McAdams anonymization with 80 Mel filters and τ = 375 ms or 10 Mel filters and τ = 125 ms. Contrarily to all other methods, the ASR performance for low-frequency audio with f s = 1250 Hz is more degraded by increasing RT 60 compared to increasing the SNR.

The confidence intervals of the ASV performance seem to be higher in comparison to the remaining models, however, the different ranges and scales of the evaluation metrics must be considered. In general, the confidence intervals reflect the different levels of identity privacy for different speakers. The effect of reverberation is for most methods less significant than the intervariability between the speakers.

Regarding the utility, the VAD performance is relatively robust against noise and reverberation. However, for McAdams anonymization, the additional noise leads to the strongest deterioration compared to the original test data. Since McAdams anonymization is applied to the entire signal, thus also speech pauses are processed, this seems to be more detrimental with additional noise.

Finally, the SD performance degrades with increasing SNR, except for the McAdams anonymization without additional smoothing. With reverberation, the SD performance appears to be robust for some methods compared to the original test data but degrades significantly for temporal smoothing with τ = 250 ms, McAdams anonymization with additional smoothing, and low-frequency audio.

Overall, noise and reverberation enhance speech privacy as the ASR and ASV performance degrade. Simultaneously, the utility performance of V AD and SD also degrades, but depending on the privacy-preserving method this deterioration is less severe compared to the deterioration of the ASR and ASV performance.

## 5. CONCLUSIONS

This contribution investigated the influence of noise and reverberation on the trade-off between privacy and utility for privacy-preserving features feasible for edge computing. Overall, the evaluation demonstrates that decreasing the SNR leads for most methods to stronger performance degradation compared to increasing RT 60 . The ASR performance shows the strongest deterioration with increasing SNR, while the VAD is more robust, except for the combinations with McAdams anonymization. Future work will focus on exploring the practical applicability and different input features beyond standard Mel filterbank energies to optimize both privacy and utility simultaneously.

## 6. REFERENCES

- [1] A. Milek, E. A. Butler, A. M. Tackman, D. M. Kaplan, C. L. Raison, D. A. Sbarra, S. Vazire, and M. R. Mehl, 'Eavesdropping on Happiness Revisited: A Pooled, Multisample Replication of the Association Between Life Satisfaction and Observed Daily Conversation Quantity and Quality,' Psychological Science , vol. 29, no. 9, pp. 1451-1462, 2018.
- [2] D. Wyatt, T. Choudhury, and J. A. Bilmes, 'Conversation detection and speaker segmentation in privacysensitive situated speech data.' in Proc. Interspeech , 2007, pp. 586-589.
- [3] A. Nelus, S. Gergen, J. Taghia, and R. Martin, 'Towards opaque audio features for privacy in acoustic sensor networks,' in Proc. ITG Conf. on Speech Communication, VDE , 2016, pp. 1-5.
- [4] P. Thaine and G. Penn, 'Extracting mel-frequency and bark-frequency cepstral coefficients from encrypted signals,' in Proc. Interspeech , 2019, pp. 3715-3719.
- [5] M. Hao, H. Li, G. Xu, S. Liu, and H. Yang, 'Towards efficient and privacy-preserving federated deep learning,' in Proc. IEEE Int. Conf. on Communications , 2019, pp. 1-6.
- [6] M. R. Mehl, 'The Electronically Activated Recorder (EAR): A Method for the Naturalistic Observation of Daily Social Behavior,' Current Directions in Psychological Science , vol. 26, no. 2, pp. 184-190, 2017.
- [7] J. Bitzer, S. Kissner, and I. Holube, 'Privacy-aware acoustic assessments of everyday life,' Journal of the Audio Engineering Society , vol. 64, no. 6, pp. 395-404, 2016.
- [8] J. Patino, N. Tomashenko, M. Todisco, A. Nautsch, and N. Evans, 'Speaker Anonymisation Using the McAdams Coefficient,' in Proc. Interspeech , 2021, pp. 1099-1103.
- [9] C. Raman, J. Vargas Quiros, S. Tan, A. Islam, E. Gedik, and H. Hung, 'Conflab: A data collection concept, dataset, and benchmark for machine analysis of freestanding social interactions in the wild,' Advances in Neural Information Processing Systems , vol. 35, pp. 23 701-23 715, 2022.
- [10] N. Tomashenko, X. Wang, E. Vincent, J. Patino, B. M. L. Srivastava, P.-G. No´ e, A. Nautsch, N. Evans, J. Yamagishi, B. O'Brien, A. Chanclu, J.-F. Bonastre, M. Todisco, and M. Maouche, 'The VoicePrivacy 2020 Challenge: Results and findings,' Computer, Speech and Language , vol. 74, 2022.
- [11] Y. Nakajima, M. Matsuda, K. Ueda, and G. B. Remijn, 'Temporal Resolution Needed for Auditory Communication: Measurement With Mosaic Speech,' Frontiers in Human Neuroscience , vol. 12, 2018.
- [12] M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, and et al., 'SpeechBrain: A general-purpose speech toolkit,' 2021, arXiv:2106.04624.
- [13] A. Graves, S. Fern´ andez, F. Gomez, and J. Schmidhuber, 'Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,' in Proc. Int. Conf. on Machine Learning , 2006, pp. 369376.
- [14] B. Desplanques, J. Thienpondt, and K. Demuynck, 'ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,' in Proc. Interspeech , 2020, pp. 3830-3834.
- [15] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, 'Librispeech: An ASR corpus based on public domain audio books,' in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP) , 2015, pp. 5206-5210.
- [16] J. Carletta, 'Unleashing the killer corpus: experiences in creating the multi-everything AMI meeting corpus,' Language Resources and Evaluation , vol. 41, no. 2, pp. 181-190, 2007.
- [17] N. Dawalatabad, M. Ravanelli, F. Grondin, J. Thienpondt, B. Desplanques, and H. Na, 'ECAPA-TDNN embeddings for speaker diarization,' in Proc. Interspeech , 2021, pp. 3560-3564.
- [18] J. S. Chung, A. Nagrani, and A. Zisserman, 'VoxCeleb2: Deep Speaker Recognition,' in Proc. Interspeech , 2018, pp. 1086-1090.
- [19] D. Snyder, G. Chen, and D. Povey, 'MUSAN: A music, speech, and noise corpus,' 2015, arXiv:1510.08484v1.
- [20] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, 'A study on data augmentation of reverberant speech for robust speech recognition,' in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP) , 2017, pp. 5220-5224.
- [21] K. Smeds, F. Wolters, and M. Rung, 'Estimation of Signal-to-Noise Ratios in Realistic Sound Scenarios,' Journal of the American Academy of Audiology , vol. 26, no. 2, pp. 183-196, 2015.
- [22] M. Jeub, M. Schafer, and P. Vary, 'A binaural room impulse response database for the evaluation of dereverberation algorithms,' in Proc. IEEE Int. Conf. on Digital Signal Processing , 2009, pp. 1-5.
Jule Pohlhausen Francesco Nespoli Joerg Bitzer
Long-Term Conversation Analysis: Privacy-Utility Trade-off under Noise and Reverberation
Recordings in everyday life require privacy preservation of the speech
content and speaker identity. This contribution explores the influence of noise
and reverberation on the trade-off between privacy and utility for low-cost
privacy-preserving methods feasible for edge computing. These methods
compromise spectral and temporal smoothing, speaker anonymization using the
McAdams coefficient, sampling with a very low sampling rate, and combinations.
Privacy is assessed by automatic speech and speaker recognition, while our
utility considers voice activity detection and speaker diarization. Overall,
our evaluation shows that additional noise degrades the performance of all
models more than reverberation. This degradation corresponds to enhanced speech
privacy, while utility is less deteriorated for some methods.