Hope you can help. Basically I just need some guidance/reassurance around how batch sizes are calculated when 2 GPUs are installed but I think (!) on Windows only 1 GPU can be/is being used.
Scenario:


Remoting into PC with 2 NVIDIA GPUs running Windows 10


(someone else’s machine so no option of installing Linux)


I have the GPU enabled version of PyTorch installed


Running run_summarization.py from the transformers repo


I set the paramete per_device_train_batch_size = 4


I do a test run with 10,000 training samples


Result:


“_n_gpu=2” printed at start of run, so script has detected 2 GPUs


(I have confirmed this directly in another script using torch.cuda.device_count() )


I get the warning: “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\torch\cuda\nccl.py:15: UserWarning: PyTorch is not compiled with NCCL support”


(I believe the lack of NCCL support on Windows is the reason why multiple GPU training on Windows is not possible?)


I get 1,250 steps per epoch


Questions:


I assuming that PyTorch defaults to using just 1 GPU instead of the 2 available, hence the warning?  (it certainly runs a lot, lot quicker than just on CPU)


Given 2 GPUs installed, batch per device 4 and 1,250 seems to suggest an effective batch size of 8. So is it being automatically adjusted to 2 x 4 = 8 given only 1 GPU being used but 2 GPUs present (just checking that a batch of 4 is not being skipped for the other GPU detected but not being used?)


Many thanks!