My model is VGG16 and part of the model is put on GPU0 , the rest is put on GPU1.
Then I create three models using this model definition.
And do the followings
for i in range(10):
model_weighVersion1_train()
model_weighVersion2_train()
model_weighVersion3_train()
I find if there is only one model in training, everything will be okay. But if these three models are trained in turns, after some iterations, the loss will become NaN.
So I wonder when I create multiple identical models, will these models share memory/storage?
Or do you know what led to this NaN?