Ahh now I get it with regards to batch_sampler and sampler, I thought it was the other way around. This also explains why a batch_sampler=… does not speed up the data_loader.
I am in the process of benchmarking all those options for my use-case.