@BramVanroy thank you very much for the reply. I took the safe (but not too sophisticated) option of simply physically removing the second GPU so that I know I have one GPU on Windows that is a setup that should work.
2 GPUs was confusing - there was the NCCL warning (I assume indirectly from PyTorch) but then both GPU had similar memory allocations and it was confusing what HF was doing in relation to samples per batch per device, where you (I !!!) don’t know if both GPUs are working or if they are simply duplicating the work.
Anyway, went for the safe option but thanks again!