Hi,
So for those of you who have similar issue in the future. I solved my problem by seeing this post:

  
    
    
    When should you save_for_backward vs. storing in ctx 
  
  
    We experienced a memory leak on a custom module using custom-make GPU kernels. Storing intermediate results in ctx.x, ctx.z etc resulted in them not being released over multiple mini-batches, quickly exhausting GPU memory. We did not experience this in pytorch 0.3.1, it sounds like it is due to some changes in pytorch 0.4. Either explicitly del ctx.x etc or adding x,z to .save_for_backward both fixed the memory leak. I would still like to get a good explanation for why you would want to use save…
  


And so indeed I manually deleted context attributes corresponding to tensors I obtained with my custom Cuda operations and it solved the memory leak.