@RoshanRane
I used your code for plotting the gradient flow (thank you!), and obtained this output:

Screen Shot 2019-05-22 at 6.54.36 PM.png822×712 67.6 KB

With the second implementation I obtain the following output:

Screen Shot 2019-07-18 at 1.11.15 PM.png884×736 57.5 KB

This is for a single layer GRU. I was surprised to see the gradient of the hidden state stay so small. The only thing I can think of as to why this would be the case is because the hidden state is re-initialized with each training example (and thus stays small), while the other gradients accumulate as a result of being connected to learned parameters. Does that seem correct? Is this what a plot of the gradient flow in a single layer GRU should typically look like?
Alternatively, for a 4 layer LSTM, I get the following output:

Screen Shot 2019-06-03 at 8.18.57 AM.png802×718 59.8 KB

Does that seem correct? Is this what a plot of the gradient flow in a multi-layer layer LSTM should typically look like? The larger gradient values are from the initial epochs. I am not sure when they are so much larger to start with. Thoughts?