Beyond the Calibration Point: Mechanism Comparison in Differential Privacy
<|reference_start|>Beyond the Calibration Point: Mechanism Comparison in Differential Privacy: In differentially private (DP) machine learning, the privacy guarantees of DP mechanisms are often reported and compared on the basis of a single $(\varepsilon, \delta)$-pair. This practice overlooks that DP guarantees can vary substantially even between mechanisms sharing a given $(\varepsilon, \delta)$, and potentially introduces privacy vulnerabilities which can remain undetected. This motivates the need for robust, rigorous methods for comparing DP guarantees in such cases. Here, we introduce the $\Delta$-divergence between mechanisms which quantifies the worst-case excess privacy vulnerability of choosing one mechanism over another in terms of $(\varepsilon, \delta)$, $f$-DP and in terms of a newly presented Bayesian interpretation. Moreover, as a generalisation of the Blackwell theorem, it is endowed with strong decision-theoretic foundations. Through application examples, we show that our techniques can facilitate informed decision-making and reveal gaps in the current understanding of privacy risks, as current practices in DP-SGD often result in choosing mechanisms with high excess privacy vulnerabilities.<|reference_end|>
@article{kaissis2024beyond,
    title={Beyond the Calibration Point: Mechanism Comparison in Differential
  Privacy},
    author={Georgios Kaissis and Stefan Kolek and Borja Balle and Jamie Hayes and
  Daniel Rueckert},
    journal={arXiv preprint arXiv:2406.08918},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.08918},
    primaryClass={cs.CR cs.AI cs.LG math.ST stat.ML stat.TH}
}