Hello, everyone. I am new to Pytorch and recently I learned to implement a CNN training for semantic segmentation. However I meet some strange GPU memory behavior and can not find the reason.
[1] Different memory consumption on different GPU platform
I have 2 GPU platforms for model training.
Same code, same ResNet50 CNN, same training data, same batchsize = 1
platform 1 [GTX 970 (4G), cuda-8.0, cudnn-5.0, Nvidia Driver-375.26] consume 3101M GPU memory
platform 2 [GTX TITAN X (12G), cuda-7.5, cudnn-5.0, Nvidia Driver-352.30] consume 1792M GPU memory
There is a big difference in GPU memory consumption, and I can not find the reason. Can anyone give some help?
[2] Strange out of memory error when training
On platform 1, when there is only one image/label pair in the training data list, the model is trained normally for 100 epochs.(batchsize = 1. thus 1 iteration per epoch) However, if I repeat the training pair thus there are 2 image/label pairs in the training data list, the training break down at the second iteration with out of memory error showed in the following image (batchsize = 1. thus 2 iteration per epoch). This error does not occur on the platform 2. So strange.
snapshot1.png1080×587 106 KB

Anyone meet the same problem ? or Can anyone give some help?
THANKS!!!