content_image/302303.jpg
activation functions that form the hidden layer. 

a l j = σ$^{(}$ ∑ i w l,j i a l − 1 i + b l j ) where σ ( x ) = 1 1 + exp ( x ) 

Here, a l j is the output (”activation”) of cell j in layer l , w l,i j is the weight of the connection from cell i on layer l − 1 to cell j on layer l , a l − 1 i is the output of cell i on layer l − 1 and b l j is the ”bias” of cell j on layer l . 

A neural network is a supervised learning technique where the output of the neural net is compared to a target by means of a loss function. The type of output layer cell and the loss function are often chosen jointly for their computational properties with respect to backpropagation. A typical combination for categorical targets is a softmax layer with a cross-entropy loss function H : 

<img src='content_image/302303.jpg'>

Here, y ′ are the target values and y are the network outputs, computed in turn from the output activations a$_{i}$ of the next to last network layer. Gradients of the param- eters w l,i j , b l i with respect to the loss function are computed using backpropagation and parameters are adjusted using variants of gradient descent algorithms. 

## 3.2. Recurrent Neural Networks (RNN) 

Recurrent neural networks (RNN) are a neural network architecture popularized by work in natural language processing (NLP). NLP has moved away from explicit language models to statistical methods, specifically to RNN (for example, Mikolov et al., 2011; Sutskever et al., 2011; Graves, 2013, 2012; Zaremba et al., 2014). In a RNN, each cell’s output is not only connected to the following layer but each cell also feeds back information into itself, allowing it to maintain ”state” over time. To make this tractable within an acyclic computational graph and backpropagation, 

