@mrshenli, I think I figured out what is going on. The DataLoader I am using has dynamic clipping and the inputs are sorted by input length. So without shuffling only small inputs are used (in the ballpark of shape(4, 100)), while with shuffling much larger inputs are being used (around shape(4,1000)). And I would guess this could also explain some of the speed difference in training.
However, I am still slightly confused why a SequentialSampler is so much faster than a DistributedSampler with num_replicates=1 and shuffling=False.
So I guess only my question about amp is left.