Are you able to reproduce the illegal memory access by using the nn.Conv2d code snippet?
If not, rerun your script with CUDA_LAUNCH_BLOCKING=1 python script.py args and post the stacktrace here, as cuDNN might just be running into an async memory violation. Also, if it’s not already the case, update to the latest stable release.