Gated Recurrent Units. Gated Recurrent Units (GRU) aim to be a computationally-efficient
approximate of LSTM by using only two gates (vs three in LSTM) and a single memory cell (vs two
in LSTM). To control the flow of information over time, a GRU uses an update gate 𝑧𝑡 to decide
how much of the new input to be added to the previous hidden state and a reset gate 𝑟𝑡 to decide
how much of previous hidden state information to be forgotten.

𝑧𝑡 = 𝜎(𝑊𝑥𝑧𝑥𝑡 + 𝑊ℎ𝑧ℎ𝑡−1),
𝑟𝑡 = 𝜎(𝑊𝑥𝑟 𝑥𝑡 + 𝑊ℎ𝑟ℎ𝑡−1),
ℎ𝑡 = (1 − 𝑧𝑡) ⊙ ℎ𝑡−1 + 𝑧𝑡 ⊙ tanh(𝑊𝑥ℎ𝑥𝑡 + 𝑊𝑟ℎ(𝑟𝑡 ⊙ ℎ𝑡−1)),

(11)
(12)
(13)
AReviewofDeepLearningTechniquesforSpeechProcessing