I’m astonished that gradients are summed in the batch dimension:
import torch
import torch.nn as nn

x = torch.tensor([5.0, -1.0], dtype=torch.float).cuda().view(-1, 1)

model = nn.Linear(in_features=1, out_features=1, bias=False).cuda()

model.weight.data.zero_()
model.weight.data.add_(1.0)

y = model(x)

label = torch.zeros(2, 1, dtype=torch.float).cuda()

loss = torch.sum((y - label)**2)

loss.backward()

print(model.weight.grad)

The code above prints 52.0.
I’m astonished given that the gradient is an approximation of the truth gradient, and it is supposed to be unbiased. The natural way is to take an average over the batch dimension, but pytorch should sum the grad over the batch dimension.