Hello everyone,
I have a -nooby- question about gradient computation. Let’s say I have an input matrix X with shape of [T, T]. I want to process this matrix in a block diagonal way, where these blocks will have a shape of [B, B] with (B << T).
For the first implementation and to check if my logic make sense, I went with a simple for loop and in each step modify only one block of X  (add a dependency factor) and nothing else. The grad_fn attribute shows (<CopySlices object at 0x7f76c424a070>, 0) and I assume that the backpropagation is working as intended.
This was extremely slow, so for the second implementation I went with a mask approach. My mask is known for a given Χ, so I precomputed it. I compute the dependency factor for each item in the matrix ([Τ, Τ]), but I want to keep only the masked factor (to be added in a block diagonal way to Χ). Finally, I compute X = X + (mask * factor)). This approach give (<MulBackward0 object at 0x7f76c4283be0>, 0)).
My question is: The two results I get from those implementations is slightly different (max absolute error 2e-5). Is that because of numeric reasons? OR Is that because on the second approach I impact every item on the [T, T] matrix, even if outside the blocks the dependency factor I add is zero?
Thanks in advance for any help and insight you can provide!