Kiros et al. [2015] showed that this model could produce generic sentence representations that
perform well on several tasks without being ﬁne-tuned. However, training this model is timeconsuming, requiring several days of training in order to produce meaningful results.
In this experiment we determine to what effect layer normalization can speed up training. Using the
publicly available code of Kiros et al. [2015]4, we train two models on the BookCorpus dataset [Zhu
et al., 2015]: one with and one without layer normalization. These experiments are performed with
Theano [Team et al., 2016]. We adhere to the experimental setup used in Kiros et al. [2015], training
a 2400-dimensional sentence encoder with the same hyperparameters. Given the size of the states
used, it is conceivable layer normalization would produce slower per-iteration updates than without.
However, we found that provided CNMeM5is used, there was no signiﬁcant difference between the
two models. We checkpoint both models after every 50,000 iterations and evaluate their performance
on ﬁve tasks: semantic-relatedness (SICK) [Marelli et al., 2014], movie review sentiment (MR)
[Pang and Lee, 2005], customer product reviews (CR) [Hu and Liu, 2004], subjectivity/objectivity
Training state-of-the-art, deep neural networks is computationally expensive.
One way to reduce the training time is to normalize the activities of the
neurons. A recently introduced technique called batch normalization uses the
distribution of the summed input to a neuron over a mini-batch of training
cases to compute a mean and variance which are then used to normalize the
summed input to that neuron on each training case. This significantly reduces
the training time in feed-forward neural networks. However, the effect of batch
normalization is dependent on the mini-batch size and it is not obvious how to
apply it to recurrent neural networks. In this paper, we transpose batch
normalization into layer normalization by computing the mean and variance used
for normalization from all of the summed inputs to the neurons in a layer on a
single training case. Like batch normalization, we also give each neuron its
own adaptive bias and gain which are applied after the normalization but before
the non-linearity. Unlike batch normalization, layer normalization performs
exactly the same computation at training and test times. It is also
straightforward to apply to recurrent neural networks by computing the
normalization statistics separately at each time step. Layer normalization is
very effective at stabilizing the hidden state dynamics in recurrent networks.
Empirically, we show that layer normalization can substantially reduce the
training time compared with previously published techniques.
Jimmy Lei Ba Jamie Ryan Kiros Geoffrey E. Hinton