Adversarial Training of Two-Layer Polynomial and ReLU Activation Networks via Convex Optimization
<|reference_start|>Adversarial Training of Two-Layer Polynomial and ReLU Activation Networks via Convex Optimization: Training neural networks which are robust to adversarial attacks remains an important problem in deep learning, especially as heavily overparameterized models are adopted in safety-critical settings. Drawing from recent work which reformulates the training problems for two-layer ReLU and polynomial activation networks as convex programs, we devise a convex semidefinite program (SDP) for adversarial training of polynomial activation networks via the S-procedure. We also derive a convex SDP to compute the minimum distance from a correctly classified example to the decision boundary of a polynomial activation network. Adversarial training for two-layer ReLU activation networks has been explored in the literature, but, in contrast to prior work, we present a scalable approach which is compatible with standard machine libraries and GPU acceleration. The adversarial training SDP for polynomial activation networks leads to large increases in robust test accuracy against $\ell^\infty$ attacks on the Breast Cancer Wisconsin dataset from the UCI Machine Learning Repository. For two-layer ReLU networks, we leverage our scalable implementation to retrain the final two fully connected layers of a Pre-Activation ResNet-18 model on the CIFAR-10 dataset. Our 'robustified' model achieves higher clean and robust test accuracies than the same architecture trained with sharpness-aware minimization.<|reference_end|>
@article{kuelbs2024adversarial,
    title={Adversarial Training of Two-Layer Polynomial and ReLU Activation
  Networks via Convex Optimization},
    author={Daniel Kuelbs, Sanjay Lall, Mert Pilanci},
    journal={arXiv preprint arXiv:2405.14033},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.14033},
    primaryClass={cs.LG math.OC}
}