Seems like it has to do with the tensor operations in the forward pass because if you comment those out then the script will run. I’m not sure what the CUDA error is referring to.
Have you tried using DistributedDataParallel 1? It is preferred over DataParallel for multi-GPU training.