So these CUDA errors should be fatal for the Python process they happen in but restarting the kernel should then make things work again.
The perhaps most typical way to trigger it in a working PyTorch system is to have classification targets that exceed that possible through the logit tensor size passed to CrossEntropyLoss. But something seems funny with your system, then.
At this point, maybe @ptrblck knows more…
Best regards
Thomas