The __init__ method of Dataset should not be called.
However, the data stored in the Dataset will be copied to all processes.
Are you pre-loading your data in the __init__? If so, are you seeing increased memory usage using multiple workers?
Would lazy loading not work in your case?