Hi, same exact same thing happened, first the NCCL warning and then everything runs slow but correctly. My batch size is[1, 3, 256, 256]  and runs 80 times for 80 files, setting batch size higher would trigger out of memory error… and I set the total epoch to 1 for testing.
I have recorded time it took to complete 1 epoch with both DataParalleled and calling only .cuda().
DataParallel : Completed 1 epoch in 106.87 seconds, NCCL Warning: Yes
.cuda() : Completed 1 epoch in 105.28 seconds, NCCL Warning: No

As you can see calling .cuda() and DataParallel has no effect for me, I did not call one model directly after the another, I made sure I logged out to clear up the memory. If I look it my evga precision X it logs the same thing for both cases as well. 99% usage on 1 card and 0 % usage on the other, however both card’s GPU clock would be maxed out, except maybe card 2 which runs slightly lower speed than card 1.
Edited: Downloaded GPUz which shows both gpu running but I am currently only using .cuda() which by default only suppose to use 1 gpu? I honestly don’t know whats going on.
Final: Edited SLOVED I feel stupid but… You need to run more than 1 batch for the other GPU to kick in. Now everything works.