Are you loading this large file once in your __init__?
If so, num_workers=0 should pre-load the data once and just slice it in __getitem__.
Using multiple workers might not be the best idea, as each worker will load the whole dataset, while the __getitem__ should be quite fast compared to the __init__.
Could you post the Dataset implementation?
Maybe using shared arrays 16 might work better?