As suggestion, I replace the Adam optimizer with AMSGrad. The problem is solved^^ It indeed comes from the stabilization issue of the Adam itself.
In implementation, I reinstall my pytorch from source and in version 4.0, I can simply use AMSGrad with:
optimizer = optim.Adam(model.parameters(), lr=0.001, eps=1e-3, amsgrad=True)
Thanks for your help very much!