The PESQetarian: On the Relevance of Goodhart's Law for Speech Enhancement
<|reference_start|>The PESQetarian: On the Relevance of Goodhart's Law for Speech Enhancement: To obtain improved speech enhancement models, researchers often focus on increasing performance according to specific instrumental metrics. However, when the same metric is used in a loss function to optimize models, it may be detrimental to aspects that the given metric does not see. The goal of this paper is to illustrate the risk of overfitting a speech enhancement model to the metric used for evaluation. For this, we introduce enhancement models that exploit the widely used PESQ measure. Our "PESQetarian" model achieves 3.82 PESQ on VB-DMD while scoring very poorly in a listening experiment. While the obtained PESQ value of 3.82 would imply "state-of-the-art" PESQ-performance on the VB-DMD benchmark, our examples show that when optimizing w.r.t. a metric, an isolated evaluation on the same metric may be misleading. Instead, other metrics should be included in the evaluation and the resulting performance predictions should be confirmed by listening.<|reference_end|>
@article{de oliveira2024the,
    title={The PESQetarian: On the Relevance of Goodhart's Law for Speech
  Enhancement},
    author={Danilo de Oliveira, Simon Welker, Julius Richter, Timo Gerkmann},
    journal={arXiv preprint arXiv:2406.03460},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.03460},
    primaryClass={eess.AS cs.LG cs.SD}
}