First of all, keep in mind that the optimal LR depends on the batch size (in case u increased a lot the BS).
Secondly, you are probably having troubles with view method
check this

  
    
    
    Difference between 2 reshaping operations (reshape vs permute) nlp
  
  
    if u have a cube 
c=torch.rand(3,4,5)

and you use permute 
c=torch.rand(3,4,5)
rx  = c.permute(0,2,1)
ry  = c.permute(2,1,0)
rz = c.permute(1,0,2)
print(rx.size())
print(ry.size())
print(rz.size())

torch.Size([3, 5, 4])
torch.Size([5, 4, 3])
torch.Size([4, 3, 5])

you are just rotating the tensor, but order is preserved 
On the other hand, if you reshape you can see you are modifying the ordering because this is not rotating the cube but mapping in an ordered way from right to left. It takes n…
  


it may help u