Try to narrow down the issue on a single device by checking if you are able to reproduce the issue deterministically. If so, check if any additional CUDA related work is done in this iteration, e.g. using CUDATensors in the Dataset. Also, scale down the use case as much as possible by e.g. setting the number of workers to 0 etc.