I just wanted to add that I was getting this error with torch 1.6 and cuda 10.2. I upgraded torch to 1.7 and the error changed to show me that one device was on cpu and the other was on GPU which was easily solved.
So it appears it might be happening somewhere in the error handling which would usually display a meaningful error, but some cuda/torch mismatch or bug crashes in the process.