I am training a network model but I found it’s so big such that one TITAN Xp GPU’s 12GB memory can only allow one sample to train at the same time. Now I am using one trick to mimick multi-sample training. The way is I go forward and backward the model with one sample each time and after some times I optmiizer.step() one time. I am not sure in pytorch is this way can work similiary with training with multi-samples at one time. In my network I did not use BatchNorm layer.
Thank you for your advice !