Are you reading the data in the __init__ method of your Dataset?
If the recreation of your DataLoader takes so long it might actually point to your Dataset.
Could you post the code of your Dataset so that we can have a look?