SwagJ:

As you can see, I tried to delete intermediate variable right after using them. And do empty_cache(). However, that doesn’t help any bit in forward process.


You won’t be able to delete the intermediate activations, if they are needed to compute the gradients in the backward pass.
If you don’t want to store the intermediates and don’t need to backward pass, wrap the code in a with torch.no_grad() block.
Also, how large are your input tensors to the model?