Facing the same problem with DataParallel on muti-gpu with A40 GPUS.
The code runs fine with these settings on 1080Ti, 2080 Ti and TitanXp with the identical environment setup. @semihcanturk Did you figure out a fix?