I have yet to do build the loss function, I only did forward pass yet and was afraid that backward wont work (run out of memory), because storing gradients would take as much space as storing the weights so it would double up.
Can’t reduce batchsize, I am using 3D CNNs for videos and 8 GB is a single video (65 frames) i.e. a single batch element.
And sorry I  meant require_grad = False, not reduce_grad.
And are you suggesting it may fit into 12 gigs for 8GB forward pass (will definitely try but wanted to know how things work inside).
And will look at torch.utils.checkpoint, Thanks!