Hi Richard. Yes I did that and the update is that there is definitely a problem in the LSTM.
As an experiment I froze the LSTM gradients only and directly allowed the output after the LSTM with to take gradients with the resnet and the vocalnet. And everything was normal.
So the resnet was even able to learn the initialization of the LSTM and modify it’s filters according to the output.
The gradients also didn’t decrease to the above values and my training accuracy increased every epoch . Now I’m not able to figure out what could possibly be wrong with my LSTM. It seems pretty simple.