I actually trained on gpu same thing happens
Look what it does to my memory 
Screenshot from 2018-12-06 14-40-50.png1920×1080 462 KB

But I used fastai’s resnet to train on huge datasets but this does not occur on those models.
And fastai is a wrapper of pytorch.
So the problem surely is in my code.