Hi, I’m having CUDA out of memory issue now, and I’m thinking to reduce some useless variables in the computation graph to save memory.
Particularly, I have a tensor sim_score which is calculated in the forward pass by some previous encoders. e.g., sim_score = tensor([[1, 2, 3, 4], [5, 6, 7, 8]]). I also have a mask tensor, e.g., mask = tensor([[0, 1, 0, 0], [1, 1, 1, 0]]). I will do something like output = torch.sum(sim_score * mask, dim=1). As you can see, 1, 3, 4, 8 in sim_score are useless in future computation. However, multiplying the mask tensor seems to keep all values in the computation graph. I wonder is there a way to don’t keep those useless values in the graph to save some gpu memory?
I’ve tried to directly assign the values in sim_score to 0 like sim_score[~mask] = 0, but I got the error: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation.
Many thanks!