Thanks for following up. I’m still debugging, but it seems that the error is related to Pytorch Lightning (PL) steroids for dataloaders.

Setting to reload_dataloaders_every_epoch=True of PL.Trainer made it for some of my models (or machines?  ). The sys-admin hasn’t come back with the details of provisioning used on the machines that run with that flag. In my machine, that hack does not work.
So far, num_workers=0 is a more consistent hack, but unpractical.
Perhaps, it’s an issue related to shared memory dunno. The code uses an ancient version of PL.

I feel that it makes more sense that I upgrade the project to the last versions of Pytorch and PL. I will post an example if it keeps breaking.
After a painful debugging. The preliminary takeaways

Patience is crucial.
Whereas possible try multiple things in parallel. Namely, let the training loop run with num_workers=0 and different values for a long period of time. If the error disappears, then it might not be the logic of your custom dataset.
Cross-check the version of the packages such that you know where you stand.
Try CUDA_LAUNCH_BLOCKING=1 to isolate that it’s not a silly error.
To isolate that error is not in your custom dataset. Record the instances given to the dataset. Then, poke your dataset manually. If no error is triggered, the issue could be on the parallelism logic.