title: Empirical Risk Landscape Analysis for Understanding Deep Neural Networks
abstract: This work aims to provide  comprehensive landscape analysis of empirical risk in deep neural networks (DNNs), including the convergence behavior of its gradient, its stationary points and the empirical risk itself to their corresponding population counterparts, which reveals how various network parameters determine the convergence performance. In particular, for an $l$-layer linear neural network consisting of $\dm_i$ neurons in the $i$-th layer, we prove the gradient of its empirical risk  uniformly converges to the one of its population risk, at the rate of $\mathcal{O}(r^{2l} \sqrt{l\sqrt{\max_i \dm_i} s\log(d/l)/n})$. Here $d$ is the total weight dimension, $s$ is the number of nonzero entries  of all the  weights and the magnitude  of weights per layer is upper bounded by $r$. Moreover, we prove the one-to-one correspondence of the non-degenerate stationary points between the empirical and population risks and provide convergence guarantee for each pair. We also establish the uniform convergence of the empirical risk to its population counterpart and further derive the stability and  generalization bounds for the empirical risk. In addition, we  analyze these properties for deep \emph{nonlinear} neural networks with sigmoid activation functions. We prove  similar results for convergence behavior of their empirical risk gradients, non-degenerate stationary points as well as the empirical risk itself.
  
  To our best knowledge, this work is the first one theoretically characterizing the uniform convergence of the gradient and stationary points of the empirical risk of DNN models, which benefits the theoretical understanding on  how  the neural network depth $l$, the layer width $\dm_i$, the network size $d$, the sparsity in weight and the parameter magnitude $r$ determine the neural network landscape.
authors: Pan Zhou Jiashi Feng
hIndex: Jiashi Feng 76