Oliver:

I train a model (GAN) with a batch size larger than 4, I get this error.


Do you see the illegal memory access when running the original GAN training from a clean and working environment?
If so, could you post an executable code snippet, which would reproduce the issue and post the output of python -m torch.utils.collect_env, please?



 Oliver:

It suggested I could report an issue if the following code gets the same error, which it does:


Does this minimal conv/cuDNN code snippet also yield the illegal memory access in a clean environment or only after you’ve already hit the previous error in the GAN training?



 Oliver:

Unless I hard reset (factory reset the runtime)


Could you explain how you are performing the “factory reset”?



 tom:

So these CUDA errors should be fatal for the Python process they happen in but restarting the kernel should then make things work again.


Yes, that’s correct and if you are running into asserts (such as in the nn.CrossEntropyLoss use case) the CUDA context would be corrupted and restarting the Python kernel should work.