hey，I have met this problem recently. And I only use one GPU to train my model. When I make sampler  as None in DataLoder, it works well. However, after I add WeightedRandomSampler, it always shows the ConnectionRefusedError when it was before the first epoch.
I have checked my dataset and divide it into two parts. Each part can go through the code.
But, when I set the num_worker as 0 , the whole dataset can also go through the code. I have doubt it was caused by the wrong  setting of num_workers.