8.5. Implementation of Recurrent Neural Networks from Scratch¶
In this section we implement a language model introduce in Section 8 from scratch. It is based on a character-level recurrent neural network trained on H. G. Wells’ The Time Machine. As before, we start by reading the dataset first, which is introduced in Section 8.3.
%matplotlib inline import d2l import math from mxnet import autograd, np, npx, gluon npx.set_np() tokenspx.one_hot(np.array([0, 2]), len(vocab))
array([ shape of the minibatch we sample each time is (batch size,
timestep). The
one_hot function transforms such a minibatch into a
3-D tensor with the last dimension equals to the vocabulary size. We
often transpose the input so that we will obtain a (timestep, batch
size, vocabulary size) output that fits into a sequence model easier.
X = np.arange(batch_size * num_steps).reshape(batch_size, num_steps) npx.one_hot(X.T, len(vocab)).shape
(35, 32, def normal(shape): return np.random.normal(scale=0.01, size=shape, ctx=ctx) # Hidden layer parameters W_xh = normal((num_inputs, num_hiddens)) W_hh = normal((num_hiddens, num_hiddens)) b_h = np.zeros(num_hiddens, ctx=ctx) # Output layer parameters W_hq = normal((num_hiddens, num_outputs)) b_q = np.zeros(num_outputs, ctx=ctx) # Attach gradients params = [W_xh, W_hh, b_h, W_hq, b_q] for param in params: param.attach_grad() return params
8.5.3. RNN Model¶
First, we need an
init_rnn_state function to return the hidden state
at initialization. It returns an
ndarray filled with 0 and with a
shape of (batch size, number of hidden units). Using tuples makes it
easier to handle situations where the hidden state contains multiple
variables (e.g., when combining multiple layers in an RNN where each
layer requires initializing).
def init_rnn_state(batch_size, num_hiddens, ctx): return (np.zeros(shape=(batch_size, num_hiddens), ctx=ctx), )
The following
rnn function defines how to compute the hidden state
and output in a timestep. The activation function here uses the
\(\tanh\) function. As described in Section 4.1, the mean
value of the \(\tanh\) function is 0, when the elements are evenly
distributed over the real numbers.
def rnn(inputs, state, params): # Inputs shape: (num_steps, batch_size, vocab_size) W_xh, W_hh, b_h, W_hq, b_q = params H, = state outputs = [] for X in inputs: H = np.tanh(np.dot(X, W_xh) + np.dot(H, W_hh) + b_h) Y = np.dot(H, W_hq) + b_q outputs.append(Y) return np.concatenate(outputs, axis=0), (H,)
Now we have all functions defined, next we create a class to wrap these functions and store parameters.
# Saved in the d2l package for later usepx has
((1120, 28), 1, (32, 512))
We can see that the output shape is (number steps \(\times\) batch size, vocabulary size), while the hidden.
# Saved in the d2l package for later use def predict_ch8(prefix, num_predicts, model, vocab, ctx): state = model.begin_state(batch_size=1, ctx=ctx) outputs = [vocab[prefix[0]]] def get_input(): return np))) return ''.join([vocab.idx_to_token[i] for i in outputs])
We test the
predict_rnn function first. Given that we did not train
the network it will generate nonsensical predictions. We initialize it
with the sequence
traveller and have it generate 10 additional
characters.
predict_ch8('time traveller ', 10, model, vocab, ctx)
'time traveller iiiiiiiiii'
8.5.5. Gradient Clipping¶
For a sequence of length \(T\), we compute the gradients over these \(T\) timesteps in an iteration, which results in a chain of matrix-products with length \(\mathcal{O}(T)\) during backpropagating. As mentioned in Section 4.8, it might result in numerical instability, e.g., the gradients may either explode or vanish, when \(T\) is large. Therefore, RNN models often need extra help to stabilize the training.
Recall that of making progress, whereas. Gradient clipping provides a quick fix to the gradient exploding. While it does.
# Saved in the d2l package for later use def grad_clipping(model, theta): if isinstance(model, gluon.Block): params = [p.data() for p in model.collect_params().values()] else: params = model.params norm = math.sqrt(sum((p.grad ** 2).sum() for p in params)) if norm > theta: for param in params: param.grad[:] *= theta / norm
8.5.6. Training¶
Let’s first define the function to train the model on one data epoch. It differs from the models training of Section 3.6 in three places:
Different sampling methods for sequential data (independent sampling and sequential partitioning) will result in differences in the initialization of hidden states.
We clip the gradients before updating the model parameters. This ensures that the model does not diverge even when gradients blow up at some point during the training process, and it effectively reduces the step size automatically.
We use perplexity to evaluate the model. This ensures that sequences of different length are comparable.
When the consecutive sampling is used, we initialize the hidden state at
the beginning of each epoch. Since the \(i^\mathrm{th}\) example in
the next minibatch is adjacent to the current \(i^\mathrm{th}\)
example, so the next minibatch can use the current hidden state
directly, we only detach the gradient so that we compute the gradients
within a minibatch. When using the random sampling, we need to
re-initialize the hidden state for each iteration since each example is
sampled with a random position. Same as the
train_epoch_ch3 function
in Section 3.6, we use generalized
updater,
which could be either a Gluon trainer or a scratched implementation.
# Saved in the d2l package for later use * y.size, y.size) return math.exp(metric[0]/metric[1]), metric[1]/timer.stop()
The training function again supports either we implement the model from scratch or using Gluon.
# Saved in the d2l package for later use}) def updater(batch_size): return trainer.step(batch_size) else: def updater(batch_size): return d2l.sgd(model.params, lr, batch_size) def predict(prefix): return'))
Now we can train a model. Since we only use \(10,000\) tokens in the dataset, so here the model needs more epochs to converge.
num_epochs, lr = 500, 1 train_ch8(model, train_iter, vocab, lr, num_epochs, ctx)
Perplexity 1.0, 37856 tokens/sec on gpu(0) time traveller it s against reason said filby what reason said traveller it s against reason said filby what reason said
Finally let’s check the results to use a random sampling iterator.
train_ch8(model, train_iter, vocab, lr, num_epochs, ctx, use_random_iter=True)
Perplexity 1.4, 37237 tokens/sec on gpu(0) time traveller came back andfilby s anecdote collapsed the thing traveller you can show black is white by argument said fil
While implementing the above RNN model from scratch is instructive, it is not convenient. In the next section we will see how to improve significantly on the current model and how to make it faster and easier to implement.
8.5.7. Summary¶
Sequence models need state initialization for training.
Between sequential models you need to ensure to detach the gradients, different.
Modify the predict function such as to use sampling rather than picking the most likely next character.
What happens?
Bias the model towards more likely outputs, e.g., by sampling from \(q(w_t \mid w_{t-1}, \ldots, w_1) \propto p^\alpha(w_t \mid w_{t-1}, \ldots, w_1)\) for \(\alpha > 1\).
Run the code in this section without clipping the gradient. What happens?.