I guess the potential slowdown is not coming from distributed training (and thus NCCL) not from CUDA11, but might be coming from e.g. cudnn (which also depends on the device you are using).
Are you only seeing the slowdown using DDP or also using a single device? The latter case would point towards my assumption.
Could you give more information about your setup (GPU, model architecture) and also profile the training on a single device?