This is what I am puzzled about:
support there is a linear layer: y=xW, the gradient of W might be computed like this: dy/dW = x.t(). This means that the backward path need the exact input value. So when I compute forward with x1 and then with x2, won’t the gradient be computed merely with x2 since x1 is covered ? Or if pytorch will store both x1 and x2, why will the memory not be used up when doing the inference (which executes merely forward path computation many times) ?