Changing the batch size may change how computation is organized and deviations within numerical accuracy are expected.
For things like convolutions you might even have specialized “inference” kernels for batch size one, but you can even see this with much simpler operations:

  
    
    
    Precision of network gradient autograd
  
  
    It is crucial that this isn’t about the “theoretical maths” but about numerical precision. 
So this: 
a = torch.randn(5,5)
torch.zeros(10,10)
b[:5,:5] = a
a.sum() - b.sum()

will typically give something like tensor(-5.3644e-07). 
Even 
a.sum() - b[:5,:5].sum() 
will usually not give exactly zero. 
This is because the PyTorch functions make no guarantee in which order the 25 nonzero elements are summed and the sum actually depends on the order due to limited numerical precision. 
Best regards 
T…
  


If you want CPU computation you could quantize your model and see if this eliminates the effect. I would expect quantized computation to be more stable as it is essentially fixed point instead of floating point.
Best regards
Thomas