@mrshenli,Thanks for your general help.The details are as follows:


The version of PyTorch is 1.5.0


I use your code to test the size of my model and it printed this sentence: model_size is 9865484.So I think the size of model are 10MB…


When I scaled from 2 nodes to 4 nodes,the available memory started to decrease.The memory usage went up with the number of nodes increased.