If your Dataset.__init__ method is slow due to some heavy data loading, you would see the slowdown in each new creation of the workers.
The recreation of the workers might yield a small slowdown, but should be negligible, if you are using lazy loading and don’t need a lot of resources in the __init__ method.
Could you check, which operations are used in the __init__?