Solved by zhangboknight in post #4 
              
              
                As suggestion, I replace the Adam optimizer with AMSGrad. The problem is solved^^ It indeed comes from the stabilization issue of the Adam itself. 
In implementation, I reinstall my pytorch from source and in version 4.0, I can simply use AMSGrad with: 
optimizer = optim.Adam(model.parameters(), lr=…