character string, the goal is to predict a sequence of x and y pen co-ordinates of the corresponding
handwriting line on the whiteboard. There are, in total, 12179 handwriting line sequences. The input
string is typically more than 25 characters and the average handwriting line has a length around 700.
We used the same model architecture as in Section (5.2) of Graves [2013]. The model architecture
consists of three hidden layers of 400 LSTM cells, which produce 20 bivariate Gaussian mixture
components at the output layer, and a size 3 input layer. The character sequence was encoded with
one-hot vectors, and hence the window vectors were size 57. A mixture of 10 Gaussian functions
was used for the window parameters, requiring a size 30 parameter vector. The total number of
weights was increased to approximately 3.7M. The model is trained using mini-batches of size 8
and the Adam [Kingma and Ba, 2014] optimizer.
The combination of small mini-batch size and very long sequences makes it important to have very
stable hidden dynamics. Figure 5 shows that layer normalization converges to a comparable log
likelihood as the baseline model but is much faster.
9
0 10 20 30 40 50 60
Epoch10-710-610-510-410-310-210-1100Train NLL
BatchNorm bz128
Training state-of-the-art, deep neural networks is computationally expensive.
One way to reduce the training time is to normalize the activities of the
neurons. A recently introduced technique called batch normalization uses the
distribution of the summed input to a neuron over a mini-batch of training
cases to compute a mean and variance which are then used to normalize the
summed input to that neuron on each training case. This significantly reduces
the training time in feed-forward neural networks. However, the effect of batch
normalization is dependent on the mini-batch size and it is not obvious how to
apply it to recurrent neural networks. In this paper, we transpose batch
normalization into layer normalization by computing the mean and variance used
for normalization from all of the summed inputs to the neurons in a layer on a
single training case. Like batch normalization, we also give each neuron its
own adaptive bias and gain which are applied after the normalization but before
the non-linearity. Unlike batch normalization, layer normalization performs
exactly the same computation at training and test times. It is also
straightforward to apply to recurrent neural networks by computing the
normalization statistics separately at each time step. Layer normalization is
very effective at stabilizing the hidden state dynamics in recurrent networks.
Empirically, we show that layer normalization can substantially reduce the
training time compared with previously published techniques.
Jimmy Lei Ba Jamie Ryan Kiros Geoffrey E. Hinton