I do use the nn.Dataparallel instead of nn.DistributedDataParallel.
I have checked the forward function of encoder, and I found that when the program went wrong, there are only 3 encode calls instead of 4. I dont know what is going on…