You can change the loop you’ve written to iterate over X only. I’m not 100% sure that this will work well with autograd, but I think it will (i.e. copies of O won’t be stored because of in-place addition, and I&W slicing overhead is constant). Well, there is an option to put this loop in autograd.Function or checkpoint… Unfortunately, I don’t know of better ways to do this, maybe addbmm.