I train my model with batch size 128, however if I don’t use batch in the evaluation phase, the network output is wrong.
If the network’s input is in batches:
crit = nn.MSELoss(reduction='mean')
target = []
netout = []

model.eval() # To handle drop out layers and batch norm
for A, M, label in dataloaders['val']:
    with tr.set_grad_enabled(False): # We don't need gradient computation in eval mode (speed up)
        out = model(A, M)
    
        target.extend(label.data.cpu().numpy())
        netout.extend(out.data.cpu().numpy())

print(tr.Tensor(target).shape, tr.Tensor(netout).shape)
print(f'Loss: {crit(tr.Tensor(target), tr.Tensor(netout))}')

Result:

torch.Size([849]) torch.Size([849, 1])
Loss: 1.135872483253479

batch_input.png708×376 82.5 KB


If I iterate through the single elements, I attain different result:
crit = nn.MSELoss(reduction='mean')
target = []
netout = []

model.eval() # To handle drop out layers and batch norm
 for A, M, label in testset:
     with tr.set_grad_enabled(False): # We don't need gradient computation in eval mode (speed up)
         out = model(tr.unsqueeze(A, 0), tr.unsqueeze(M, 0))
        
     target.append(label.item())
     netout.append(out.item())
    

print(tr.Tensor(target).shape, tr.Tensor(netout).shape)
print(f'Loss: {crit(tr.Tensor(target), tr.Tensor(netout))}')

Result

torch.Size([849]) torch.Size([849])
Loss: 87.88565063476562
upload://d8EOWbBPkZovUeiMrUWm4aNWUze.png 1

But the result should be exactly the same, because the output is deterministic!
What could be the problem? I can overfit the model for a specific batch size, but if I evaluate on single elements, then the result is wrong.