For some reason, my forward pass (along with custom gradient calculation) keeps computing the gradient correctly (after feeding the inputs forward for 5 timesteps), however the backward pass works only for one epoch in the intended way - after that, the calculated gradient gets fed into the LSTM after assumingly 10 timesteps, 15 timesteps and so on, causing the gradient to vanish in an instant.
Sorry about bothering you, but is there some subtlety that I might have not considered here, especially wrt feeding in custom gradients?