  We study differentially private (DP) stochastic optimization (SO) with loss
functions whose worst-case Lipschitz parameter over all data points may be
extremely large. To date, the vast majority of work on DP SO assumes that the
loss is uniformly Lipschitz continuous over data (i.e. stochastic gradients are
uniformly bounded over all data points). While this assumption is convenient,
it often leads to pessimistic excess risk bounds. In many practical problems,
the worst-case (uniform) Lipschitz parameter of the loss over all data points
may be extremely large due to outliers. In such cases, the error bounds for DP
SO, which scale with the worst-case Lipschitz parameter of the loss, are
vacuous. To address these limitations, this work provides near-optimal excess
risk bounds that do not depend on the uniform Lipschitz parameter of the loss.
Building on a recent line of work (Wang et al., 2020; Kamath et al., 2022), we
assume that stochastic gradients have bounded $k$-th order moments for some $k
\geq 2$. Compared with works on uniformly Lipschitz DP SO, our excess risk
scales with the $k$-th moment bound instead of the uniform Lipschitz parameter
of the loss, allowing for significantly faster rates in the presence of
outliers and/or heavy-tailed data. For convex and strongly convex loss
functions, we provide the first asymptotically optimal excess risk bounds (up
to a logarithmic factor). In contrast to (Wang et al., 2020; Kamath et al.,
2022), our bounds do not require the loss function to be differentiable/smooth.
We also devise a linear-time algorithm for smooth losses that has excess risk
that is tight in certain practical parameter regimes. Additionally, our work is
the first to address non-convex non-uniformly Lipschitz loss functions
satisfying the Proximal-PL inequality; this covers some practical machine
learning models. Our Proximal-PL algorithm has near-optimal excess risk.

Machine Learning;Cryptography and Security;Optimization and Control;Machine Learning