Making the Shoe Fit: Architectures, Initializations, and Tuning for Learning with Privacy
Because learning sometimes involves sensitive data, standard machine-learning algorithms have been extended to offer strong privacy guarantees for training data. However, in practice, this has been mostly an afterthought, with privacy-preserving models obtained by re-running training with a different optimizer, but using the same model architecture that performed well in a non-privacy-preserving setting. This approach leads to less than ideal privacy/utility tradeoffs, as we show here. Instead, we propose that model architectures and initializations are chosen and hyperparameter tuning is performed, ab initio, explicitly for privacy-preserving training. Using this paradigm, we achieve new state-of-the-art accuracy on MNIST, FashionMNIST, and CIFAR10 without any modification of the fundamental learning procedures or differential-privacy analysis.
The paper methodically analyses the settings and choices used when training neural networks (specifically CNNs) via the DP-SGD algorithm and suggests changes to the standard procedures that empirically lead to higher accuracies despite the added noise. The main statement of the paper is quite simple: optimize hyperparameters for the model that you*re training (DP-SGD) rather than the model it is inspired by. Yet, the findings an recommendations may be useful for practitioners. Nevertheless, to be more practically relevant the paper needs some modifications: The example models used in demonstrations are quite small (3 hidden layers 26,000 parameters, when, for example, a standard segmentation CNN model U-net can typically have 26,000,000, AlexNet has about 60,000,000 and so on). The results would be much more convincing if these or other models widely used in practice were used as running examples. In Figure 1 the multitude of point on the plot makes it unclear whether they represent result variability per number of filters or simply reflect variability as the number of filters grows. If it is the latter, it seems appropriate to perform a cross validation analysis and report standard deviations. Especially in the MNIST plot the values for SGD and DP-SGD are so close that they may in fact be statistically indistinguishable. Hard to tell by looking at a point estimate. The same request holds for Figure 2, where the difference may be immaterial, but as the figure currently stands it is unclear. Section 3.2 reports some numbers for test accuracy but the uncertainty of these numbers with respect to the test set changes (cross validation) is not reported and the numbers are quite close to each other. Furthermore, the dataset is not described and it is unclear what was the size of the training and the test sets. Conclusions relative Adam vs SGD seem to repeat what*s already known or been discussed about these methods outside of the DP topic. May be worth highlighting that when one knows how to set learning rates for SGD (may be via learning rate scheduler, not discussed in the paper but practically relevant) then SGD may be as good or slightly better than Adam. However note, adaptive optimizers are often preferred for their ease of use as no tweaking and searching for an optimal learning rate is required. Would not this problem be detrimental for SGD optimization affecting the privacy budget? Please add wall-clock time column to Table 5 to support the statement about 4 times gain. I think it*s more accurate to change * This confirms that earlier theoreticalanalysis (Talwar et al., 2014) also holds in the non-convex setting.* to *This suggests that earlier theoreticalanalysis (Talwar et al., 2014) also holds in the non-convex setting.*
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Making the Shoe Fit: Architectures, Initializations, and Tuning for Learning with Privacy
Because learning sometimes involves sensitive data, standard machine-learning algorithms have been extended to offer strong privacy guarantees for training data. However, in practice, this has been mostly an afterthought, with privacy-preserving models obtained by re-running training with a different optimizer, but using the same model architecture that performed well in a non-privacy-preserving setting. This approach leads to less than ideal privacy/utility tradeoffs, as we show here. Instead, we propose that model architectures and initializations are chosen and hyperparameter tuning is performed, ab initio, explicitly for privacy-preserving training. Using this paradigm, we achieve new state-of-the-art accuracy on MNIST, FashionMNIST, and CIFAR10 without any modification of the fundamental learning procedures or differential-privacy analysis.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper methodically analyses the settings and choices used when training neural networks (specifically CNNs) via the DP-SGD algorithm and suggests changes to the standard procedures that empirically lead to higher accuracies despite the added noise. The main statement of the paper is quite simple: optimize hyperparameters for the model that you*re training (DP-SGD) rather than the model it is inspired by. Yet, the findings an recommendations may be useful for practitioners. Nevertheless, to be more practically relevant the paper needs some modifications: The example models used in demonstrations are quite small (3 hidden layers 26,000 parameters, when, for example, a standard segmentation CNN model U-net can typically have 26,000,000, AlexNet has about 60,000,000 and so on). The results would be much more convincing if these or other models widely used in practice were used as running examples. In Figure 1 the multitude of point on the plot makes it unclear whether they represent result variability per number of filters or simply reflect variability as the number of filters grows. If it is the latter, it seems appropriate to perform a cross validation analysis and report standard deviations. Especially in the MNIST plot the values for SGD and DP-SGD are so close that they may in fact be statistically indistinguishable. Hard to tell by looking at a point estimate. The same request holds for Figure 2, where the difference may be immaterial, but as the figure currently stands it is unclear. Section 3.2 reports some numbers for test accuracy but the uncertainty of these numbers with respect to the test set changes (cross validation) is not reported and the numbers are quite close to each other. Furthermore, the dataset is not described and it is unclear what was the size of the training and the test sets. Conclusions relative Adam vs SGD seem to repeat what*s already known or been discussed about these methods outside of the DP topic. May be worth highlighting that when one knows how to set learning rates for SGD (may be via learning rate scheduler, not discussed in the paper but practically relevant) then SGD may be as good or slightly better than Adam. However note, adaptive optimizers are often preferred for their ease of use as no tweaking and searching for an optimal learning rate is required. Would not this problem be detrimental for SGD optimization affecting the privacy budget? Please add wall-clock time column to Table 5 to support the statement about 4 times gain. I think it*s more accurate to change * This confirms that earlier theoreticalanalysis (Talwar et al., 2014) also holds in the non-convex setting.* to *This suggests that earlier theoreticalanalysis (Talwar et al., 2014) also holds in the non-convex setting.*
assistant
The paper methodically analyses the settings and choices used when training neural networks (specifically CNNs) via the DP-SGD algorithm and suggests changes to the standard procedures that empirically lead to higher accuracies despite the added noise. The main statement of the paper is quite simple: optimize hyperparameters for the model that you*re training (DP-SGD) rather than the model it is inspired by. Yet, the findings an recommendations may be useful for practitioners. Nevertheless, to be more practically relevant the paper needs some modifications: The example models used in demonstrations are quite small (3 hidden layers 26,000 parameters, when, for example, a standard segmentation CNN model U-net can typically have 26,000,000, AlexNet has about 60,000,000 and so on). The results would be much more convincing if these or other models widely used in practice were used as running examples. In Figure 1 the multitude of point on the plot makes it unclear whether they represent result variability per number of filters or simply reflect variability as the number of filters grows. If it is the latter, it seems appropriate to perform a cross validation analysis and report standard deviations. Especially in the MNIST plot the values for SGD and DP-SGD are so close that they may in fact be statistically indistinguishable. Hard to tell by looking at a point estimate. The same request holds for Figure 2, where the difference may be immaterial, but as the figure currently stands it is unclear. Section 3.2 reports some numbers for test accuracy but the uncertainty of these numbers with respect to the test set changes (cross validation) is not reported and the numbers are quite close to each other. Furthermore, the dataset is not described and it is unclear what was the size of the training and the test sets. Conclusions relative Adam vs SGD seem to repeat what*s already known or been discussed about these methods outside of the DP topic. May be worth highlighting that when one knows how to set learning rates for SGD (may be via learning rate scheduler, not discussed in the paper but practically relevant) then SGD may be as good or slightly better than Adam. However note, adaptive optimizers are often preferred for their ease of use as no tweaking and searching for an optimal learning rate is required. Would not this problem be detrimental for SGD optimization affecting the privacy budget? Please add wall-clock time column to Table 5 to support the statement about 4 times gain. I think it*s more accurate to change * This confirms that earlier theoreticalanalysis (Talwar et al., 2014) also holds in the non-convex setting. * to *This suggests that earlier theoreticalanalysis (Talwar et al., 2014) also holds in the non-convex setting. *