This is my data loader setting:
train_data_loader = DataLoader(
        train_dataset,
        batch_size=batch_size
        num_workers=os.cpu_count()
        collate_fn=model.get_feature_to_input_fn(),
        prefetch_factor=args.prefetch_factor,
    )

Inside the train_dataset iter function, I also used multiple threads to stream data from s3. The observation is, with this setup my IO is k/min, however if I create 4 data loaders with this setup, my IO is 4k/min. No matter what parameters I tune for a single data loader, the single data loader performance can not reach 4k/min.