I altered the function in order to be more readable and support batching, so I made some edits, but the issue/problem stays the same.
The input is always long enough and I did check, the forward method of the encoder Modules is being ran.

Something else you can do is iterate over the encoders and make sure that their gradients are not None and have non-zero values in them. It could be that some gradients are so small that they’re appearing to not be changing (but they are).

This is what I might try next, good point, I will come back with updates either way.
Thanks a lot for the help 
Edit: I am pretty sure I found the issue and it’s related to me using a register_forward_hook in order to extract the outputs from the encoder nn.Module instead of just taking a slice of the model and running backprop as usual. Will change and see if that works. Preemtive sorry in case this was a red hering.