You meant average pooling or max pooling? Your script seems to be only testing average pooling no?
For average pooling, I could see different batch being averaged in a different order leading to a 1e-6 error that then gets amplified later in the network?