Although stochastic gradient descent (SGD) is a driving force behind the recent success of deep learning, our understanding of its dynamics in a high-dimensional parameter space is limited. In recent years, some researchers have used the stochasticity of minibatch gradients, or the signal-to-noise ratio, to better characterize the learning dynamics of SGD. Inspired by these work, we here analyze SGD from a geometrical perspective by inspecting the stochasticity of the norms and directions of minibatch gradients. We propose a model of the directional concentration for minibatch gradients through von Mises-Fisher distribution and show that the directional uniformity of minibatch gradients increases over the course of SGD. We empirically verify our result using deep convolutional networks and observe a higher correlation between the gradient stochasticity and the proposed directional uniformity than that against the gradient norm stochasticity, suggesting that the directional statistics of minibatch gradients is a major factor behind SGD.
Quality and clarity: good.
Originality and significance: This paper studies the stochasticity
of the norms and directions of the mini-batch gradients, to
understand SGD dynamics. The contributions of this paper can be
summarized as: a) This paper defines gradient norm stochasticity as
the ratio of the variance of the stochastic norm to the expectation
of the stochastic norm. It theoretically and empirically shows that
this value is reduced as the batch size increases b) This paper
empirically finds that the distribution of angles between
mini-batch gradient and a given uniformly sampled unit vector
converges to an asymptotic distribution with mean 90 degrees, which
implies a uniform distribution of the mini-batch gradients. c)	
This paper uses von Mises-Fisher Distribution to approximate the
distribution of the mini-batch gradients. By theoretically and
empirically observing that the estimated parameter \hat \kappa
decreases during training, they claim that the directional
uniformity of mini-batch gradients increases over SGD training.
The idea of measuring the uniformity of mini-batch gradients
through VMF distribution seems interesting. But it is unclear how
the study of this stochasticity dynamics of SGD can be related to
the convergence behavior of SGD for non-convex problems and/or the
generalization performance of SGD.
There are additional concerns/questions regarding both theoretical
part and empirical part:
[1] Section3.3: Assumption that p_i(w_0^0) =p_i(w_1^0) = p_i is not
reasonable when theoretically comparing \hat \kappa(w_1^0) and \hat
\kappa(w_0^0). The concentration parameter \hat \kappa(w) should be
estimated by the sum of the normalized mini-batch gradients "\hat
g_i(w)/||\hat g_i(w)||" . Instead of using mini-batch gradient,
this paper uses the sum of "p_i-w" by assuming that "p_i(w_0^0) -w"
is parallel to "\hat g_i(w)", which is ok. However, when comparing
\hat \kappa(w_0^0) and \hat \kappa(w_1^0), we say \hat
\kappa(w_0^0) = h(\sum p_i(w_0^0) - w_0^0) ) and \hat \kappa(w_1^0)
= h(\sum p_i(w_1^0) - w_1^0) ). It is not reasonable to use the
same p_i for p_i(w_0^0) and p_i(w_1^0) because p_i(w_0^0) -w_1^0 is
definitely not parallel to \hat g_i(w_1^0).
[2] Section 3.3: Assumption \hat g_i(w_t^{i-1}) \hat g_i(w_t^0) is
not convincing. With this assumption, the paper writes w_1^0 =
w_0^0 - \eta\sum_i \hat g_i(w_0^{i-1}) = w_0^0 - \eta\sum_i \hat
g_i(w_0^0) = w_0^0 - \eta \sum_i p_i-w_0^0. These equalities are
not persuasive. Because, \sum_i \hat g_i(w_0^0) is the full
gradient g(w_0^0) at w_0^0. In other words, these equalities imply
that from w_0^0 to w_1^0 (one epoch), SGD is doing a full gradient
descent: w_1^0 = w_0^0 -\eta g(w_0^0), which is not the case in
reality.
[3] Experiment: Batch size should be consistent with the given
assumption in the theoretical part. In theoretical part, \hat
\kappa(w_1^0) < \hat \kappa(w_0^0) is based on the assumption that
|\hat g_i(wt^{i-1}| \tat for all i, with *large mini-batch size*.
But in the experiment, they prove \hat \kappa(w_1^0) < \hat
\kappa(w_0^0) by using small-batch size which is 64. The authors
should either provide experiments with large batch size or try to
avoid the assumption of large batch size in theoretical part.
[4] The CNN experiment; It is better to add a discussion why the
\kappa increases in the early phase of training.
[5] The experiment results show, by the end of training, all models
FNN, DENN and CNN have very large value of \kappa which is around
10^4. This value implies that the mini-batch gradients distribution
is pretty concentrated, and it is contradictory to the statement in
the introduction which is "SGD converges or terminates when either
the norm of the minibatch gradient vanishes to zeros, or when the
angles of the mini-batch gradients are uniformly distributed and
their non-zero norms are close to each other''. It is also
contradictory to the experiment in 3.2 which implies the mini-batch
gradient are uniformly distributed after training.
[6] The notations in this paper can be improved, some notations are
using "i" for batch index, some notations are using "i" for one
data sample. Some notations in Section 3.3 and 3.1 can be moved to
Section 2 Preliminaries. It will be clearer to define all the
notations in one place.
Typos: -Section 3.1: first paragraph, E\hat g(w) -> E[\hat g(w)]; -
Paragraph before Lemma2: \hat \kappa increases -> \hat \kappa
decreases; - Paragraph after Theorem2: double the directions in "If
SGD iterations indeed drive the directions the directions of
minibatch gradients to be uniform".