Thanks for the information.
I assume the slowdown can be observed on a single device and would thus be unrelated to SyncBatchnorm.
Could you install the latest stable PyTorch release (or the nightly binaries) with CUDA10.2 and cudnn7.6.5.32?
Note that the binaries ship with their own CUDA and cudnn rumtimes. Your local installations won’t be used so you don’t need to update the local CUDA version.