I tried to delete the gradient tensors in my embedding_gradient list each iteration, and added a deltorch.cuda.empty_cache() afterward for both the toy example and my actual code. Sadly it didn’t solve the problem. I am not really sure where the issue is.
Yes, you are correct here. Making all the gradient low is actually my goal here. If that’s the case, then you could see that why accumulating the gradients would not be my best bet. This is because my objective is to derive all gradients closer to 0. So I need to either take the absolute value of it or square it. However, the operation becomes different if what I get is a sum of gradients, since |a|+|b| is not the same as |a+b|.
Customizing my own torch.autograd.Function seems like an interesting direction. How exactly do I go about that? Do I define something similar to torch.autograd.grad? Is there any example you can point me to? Thanks a lot for the response btw!