Hi Rohan many thanks for your reply! I did not initialize init_process_group apparently this is for parallelization in CPU/GPU only? In fact I have not seen this in any XLA tutorial like:
https://www.kaggle.com/tanlikesmath/the-ultimate-pytorch-tpu-tutorial-jigsaw-xlm-r
Regarding the sampler in non distributed setting I do not use DDP but the normal Sampler like WeightedRandomSampler and work flawlessly.
Thanks!