Repro is here: https://bitbucket.org/ms_lilibeth/qat_dataparallel_bug/src/master/ 3
I’ve reproduced the bug only with my custom implementation of EfficientNet. In order to make it quantization-friendly I’ve done the following:

Multiplication and addition operations were replaced with
quantization-friendly FloatFunctional.
Other MBConvBlocks are used: they contain Conv2D instance instead of conv2d() function call
Paddings changed to be symmetrical about kernel size (3x3 kernel => 1x1 padding; 5x5 kernel => 2x2 padding). No more ZeroPad2D

Run the script multiple times: sometimes the bug appears, sometimes not. The original implementation of EfficientNet never fails