I assume you’ve followed all steps described in the reproducibility docs to get deterministic results?
In particular, since a convolution is used inside this layer you would have to make sure to use deterministic cuDNN algorithms.