Hi,
I am trying to debug a network and noticed that for some reason, the outputs of a linear layer are slightly different depending on the batch size of the input tensor.
Minimal working example:
import torch
from torch import nn
torch.manual_seed(72)

ll = nn.Linear(4, 8)
data = torch.rand((16, 4))

print(ll(data)[0].mean().item())
print(ll(data[:8])[0].mean().item())
print(ll(data[:4])[0].mean().item())
print(ll(data[:2])[0].mean().item())
print(ll(data[:1])[0].mean().item())

This should give an output of
0.12711986899375916
0.12711986899375916
0.12711986899375916
0.12711986899375916
0.12711985409259796

I know the different is really small numerically, but it is strange to me that when the batch size is 1 (in the last line, the size of the input is [1, 4] whereas the top line is [16, 4]), the representation seems to be different. Why is this happening? Is it possible that this could actually affect the model performance? Specifically, I’m seeing weird discrepancies in my model during training (when the batch size is 16) vs. evaluation (when the batch size is 1), so I am worried that this is causing it.
Thanks