This is likely because of the non-deterministic CUDA kernels being fired at the backend. You can use the tensorflow-determinism tool from NVIDIA to fix this.
Here’s a guide 29 that takes a deep dive into good reproducibility practices in TensorFlow.