I am trying to save some memory in training process to avoid OOM.
Here is the original code block:
entropy = torch.sum(-(pred1_softmax + 1e-9).log() * pred1_softmax, dim=1)   # pred1_softmax size: [1, 192, 384, 1248]

Running the line above will cause an increase of 700+ MB in memory occupancy. I guess it is because pytorch saves -(pred1_softmax + 1e-9).log(),  which is a big tensor with the same size of pred1_softmax, in order to compute grad in the backward pass.  So, I want to wrap the block with torch.no_grad. Here is the modified code:
with torch.no_grad():
    entropy = torch.sum(-(pred1_softmax + 1e-9).log() * pred1_softmax, dim=1)

However, it doesn’t work. It still occupies more than 700MB memory and I am confused.
Any help would be appreciated.