Hi, are you initializing process groups (using init_process_group) and using DDP somewhere in your code? DistributedSampler is not intended for use outside of distributed/DDP setting. Does the training loop not hang when removing distributed components?
cc @ailzhang for XLA/TPU question