You should not be flattening parameters during every forward pass: https://gist.github.com/JosephDavidsonKSWH/c50d2a4c5bd19ad5fb12602f239ce78f#file-l2l-py-L75-L77 38 . That is bad for performance and may be what is causing the issue here.