Hello every,  I always encountered this illegal memory access bug no matter what project I trained
bug description
I’ll use one project as example to describe the bug:
 1.   git clone git@github.com:weiaicunzai/pytorch-cifar100.git
 2.   cd pytorch-cifar100
 3.   python train.py -net vgg16 -gpu

Then I get:
Files already downloaded and verified
Files already downloaded and verified
/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/jit/__init__.py:1119: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
With rtol=1e-05 and atol=1e-05, found 3 element(s) (out of 100) whose difference(s) exceeded the margin of error (including 0 nan comparisons). The greatest difference was 1717986918400.0 (8.985408308668006e+16 vs. 8.985236509976166e+16), which occurred at index (0, 67).
  check_tolerance, strict, _force_outplace, True, _module_class)
/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:123: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
Training Epoch: 1 [128/50000]   Loss: 4.7142    LR: 0.000256
Training Epoch: 1 [256/50000]   Loss: 4.7673    LR: 0.000512
Training Epoch: 1 [384/50000]   Loss: 4.6759    LR: 0.000767
Training Epoch: 1 [512/50000]   Loss: 4.6476    LR: 0.001023
Training Epoch: 1 [640/50000]   Loss: 4.7125    LR: 0.001279
Training Epoch: 1 [768/50000]   Loss: 4.7280    LR: 0.001535
Training Epoch: 1 [896/50000]   Loss: 4.7359    LR: 0.001790
Training Epoch: 1 [1024/50000]  Loss: 4.7307    LR: 0.002046
Training Epoch: 1 [1152/50000]  Loss: 4.6513    LR: 0.002302
Training Epoch: 1 [1280/50000]  Loss: 4.6054    LR: 0.002558
Training Epoch: 1 [1408/50000]  Loss: 4.6228    LR: 0.002813
Training Epoch: 1 [1536/50000]  Loss: 4.6426    LR: 0.003069
Traceback (most recent call last):
  File "train.py", line 209, in <module>
    train(epoch)
  File "train.py", line 52, in train
    writer.add_scalar('LastLayerGradients/grad_norm2_weights', para.grad.norm(), n_iter)
  File "/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/utils/tensorboard/writer.py", line 344, in add_scalar
    scalar(tag, scalar_value), global_step, walltime)
  File "/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/utils/tensorboard/summary.py", line 181, in scalar
    scalar = make_np(scalar)
  File "/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/utils/tensorboard/_convert_np.py", line 28, in make_np
    return _prepare_pytorch(x)
  File "/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/utils/tensorboard/_convert_np.py", line 36, in _prepare_pytorch
    x = x.cpu().numpy()
RuntimeError: CUDA error: an illegal memory access was encountered

But the same code works fine on Google colab.Then I tried to run CUDA_LAUNCH_BLOCKING=1  python train.py -net resnet18 -gpu:
gives me this:
Files already downloaded and verified
Files already downloaded and verified
/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/jit/__init__.py:1119: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
With rtol=1e-05 and atol=1e-05, found 1 element(s) (out of 100) whose difference(s) exceeded the margin of error (including 0 nan comparisons). The greatest difference was 2779565395017728.0 (-2.68501056162248e+20 vs. -2.6850383572764302e+20), which occurred at index (0, 48).
  check_tolerance, strict, _force_outplace, True, _module_class)
/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:123: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
Training Epoch: 1 [128/50000]   Loss: 4.8053    LR: 0.000256
Training Epoch: 1 [256/50000]   Loss: 4.7550    LR: 0.000512
Training Epoch: 1 [384/50000]   Loss: 4.7402    LR: 0.000767
Training Epoch: 1 [512/50000]   Loss: 4.7109    LR: 0.001023
Training Epoch: 1 [640/50000]   Loss: 4.6995    LR: 0.001279
Training Epoch: 1 [768/50000]   Loss: 4.7558    LR: 0.001535
Training Epoch: 1 [896/50000]   Loss: 4.6715    LR: 0.001790
Training Epoch: 1 [1024/50000]  Loss: 4.7487    LR: 0.002046
Training Epoch: 1 [1152/50000]  Loss: 4.6628    LR: 0.002302
Training Epoch: 1 [1280/50000]  Loss: 4.6444    LR: 0.002558
Training Epoch: 1 [1408/50000]  Loss: 4.6350    LR: 0.002813
Training Epoch: 1 [1536/50000]  Loss: 4.5711    LR: 0.003069
Training Epoch: 1 [1664/50000]  Loss: 4.6021    LR: 0.003325
Traceback (most recent call last):
  File "train.py", line 209, in <module>
    train(epoch)
  File "train.py", line 44, in train
    loss.backward()
  File "/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/autograd/__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
Exception raised from operator() at /opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cudnn/Conv.cpp:980 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x4d (0x7f790a3ec77d in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xcad302 (0x7f790b4ff302 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0xcaf235 (0x7f790b501235 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #3: <unknown function> + 0xcaa48e (0x7f790b4fc48e in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xcac07b (0x7f790b4fe07b in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #5: at::native::cudnn_convolution_backward_input(c10::ArrayRef<long>, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool) + 0xb2 (0x7f790b4fe5d2 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xd117db (0x7f790b5637db in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xd415f8 (0x7f790b5935f8 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #8: at::cudnn_convolution_backward_input(c10::ArrayRef<long>, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool) + 0x1ad (0x7f793d7b8ced in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #9: at::native::cudnn_convolution_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool, std::array<bool, 2ul>) + 0x223 (0x7f790b4fcca3 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #10: <unknown function> + 0xd118c5 (0x7f790b5638c5 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #11: <unknown function> + 0xd41654 (0x7f790b593654 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #12: at::cudnn_convolution_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool, std::array<bool, 2ul>) + 0x1e2 (0x7f793d7c76a2 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x2c250c2 (0x7f793f48b0c2 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #14: <unknown function> + 0x2c39684 (0x7f793f49f684 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #15: at::cudnn_convolution_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool, std::array<bool, 2ul>) + 0x1e2 (0x7f793d7c76a2 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #16: torch::autograd::generated::CudnnConvolutionBackward::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) + 0x258 (0x7f793f312098 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #17: <unknown function> + 0x30d1017 (0x7f793f937017 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #18: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&, std::shared_ptr<torch::autograd::ReadyQueue> const&) + 0x1400 (0x7f793f932860 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #19: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x451 (0x7f793f933401 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #20: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x89 (0x7f793f92b579 in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #21: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x4a (0x7f7943c5a13a in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #22: <unknown function> + 0xc819d (0x7f794679019d in /home/by/miniconda3/envs/baiyu_py36/lib/python3.6/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #23: <unknown function> + 0x76db (0x7f796988a6db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #24: clone + 0x3f (0x7f79695b3a3f in /lib/x86_64-linux-gnu/libc.so.6)

Environment
Here is my current env:
Collecting environment information...
PyTorch version: 1.6.0
Is debug build: False
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce RTX 2080 Ti

Nvidia driver version: 418.152.00
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.6.0
[pip3] torchvision==0.7.0
[conda] blas                      1.0                         mkl    defaults
[conda] cudatoolkit               10.1.243             h6bb024c_0    defaults
[conda] mkl                       2020.2                      256    defaults
[conda] mkl-service               2.3.0            py36he904b0f_0    defaults
[conda] mkl_fft                   1.2.0            py36h23d657b_0    defaults
[conda] mkl_random                1.1.1            py36h0573a6f_0    defaults
[conda] numpy                     1.19.2           py36h54aff64_0    defaults
[conda] numpy-base                1.19.2           py36hfa32c7d_0    defaults
[conda] pytorch                   1.6.0           py3.6_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] torchvision               0.7.0                py36_cu101    pytorch

I’ve also posted an issue in Github repository 6, but didn’t seem to get an solution, so could some one tells me where I could possiblely  go wrong? Thanks.