Hi,
I know this is a post long time ago and probably you are not working on it any more. But I just post my answer here in case someone else has a similar problem.
Actually, I just had one similar problem recently. I have tried anything like you did. Just before I gave up, I saw the permute function (similar as the transpose function you used) and I decided to replace it. I have almost replaced all other functions, so why not it. I deleted this function and used the batch_first option in RNN (some other operations such as the output are done accordingly). Surprisingly, it is working. I also tested the use of transpose, it is not working, either.
So I think there might be some problems about the permute or transpose function. But I haven’t written an example to prove it yet.
Try to remove these functions and try the network again. It may surprise you.