Hi,
This is surprising indeed.
With 0 workers, does it goes through a full epoch without issues (even though it’s slow)? Could it be that one sample is problematic?
I’m afraid I don’t have a silver bullet idea to debug this.
What I would do is like an ablation study. Remove stuff in your Dataset code until it stops crashing. (you can make the training loop a no-op and just load sample and discard them to make this run faster). I would be curious to know what is the minimal change that cause this!