I’ve got the same problem.
I never had any problems with restoring models but I recently increased the size of my model. I was able to train it for 20k iterations without any problems/errors. Now I want to resume the training process but I get a CUDA out of Memory error even though I’m resuming on the same PC and the VRAM of the GPU is empty. It seems like restoring the model requires more VRAM than simply starting a new one. It does look different than a “normal” Cuda out of memory error though. Im using Pytorch 1.6 and mixed precision. Maybe thats the reason? Here is the stacktrace:
Traceback (most recent call last):
  File "scripts/train.py", line 449, in <module>
    main(sys.argv[1:])
  File "scripts/train.py", line 230, in main
    scaled_loss.backward()
  File "/disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/autograd/__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA out of memory. Tried to allocate 110.00 MiB (GPU 0; 10.76 GiB total capacity; 9.46 GiB already allocated; 75.00 MiB free; 9.62 GiB reserved in total by PyTorch)
Exception raised from malloc at /pytorch/c10/cuda/CUDACachingAllocator.cpp:272 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f8e5447b1e2 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x1e64b (0x7f8e546d164b in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: <unknown function> + 0x1f464 (0x7f8e546d2464 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x1faa1 (0x7f8e546d2aa1 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #4: at::native::empty_cuda(c10::ArrayRef<long>, c10::TensorOptions const&, c10::optional<c10::MemoryFormat>) + 0x11e (0x7f8de3f0a90e in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0xf33949 (0x7f8de2344949 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xf4d777 (0x7f8de235e777 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0x10e9c7d (0x7f8e1d0fac7d in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0x10e9f97 (0x7f8e1d0faf97 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #9: at::empty(c10::ArrayRef<long>, c10::TensorOptions const&, c10::optional<c10::MemoryFormat>) + 0xfa (0x7f8e1d205a1a in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #10: at::native::empty_like(at::Tensor const&, c10::TensorOptions const&, c10::optional<c10::MemoryFormat>) + 0x49e (0x7f8e1ce83c3e in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x12880c1 (0x7f8e1d2990c1 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #12: <unknown function> + 0x12c3863 (0x7f8e1d2d4863 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #13: at::empty_like(at::Tensor const&, c10::TensorOptions const&, c10::optional<c10::MemoryFormat>) + 0x101 (0x7f8e1d1e8b31 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #14: at::Tensor at::native::(anonymous namespace)::host_softmax_backward<at::native::(anonymous namespace)::LogSoftMaxBackwardEpilogue, true>(at::Tensor const&, at::Tensor const&, long, bool) + 0xa7 (0x7f8de3794a97 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #15: at::native::log_softmax_backward_cuda(at::Tensor const&, at::Tensor const&, long, at::Tensor const&) + 0x65a (0x7f8de377efaa in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #16: <unknown function> + 0xf215c0 (0x7f8de23325c0 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #17: <unknown function> + 0x11141d6 (0x7f8e1d1251d6 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #18: at::_log_softmax_backward_data(at::Tensor const&, at::Tensor const&, long, at::Tensor const&) + 0x119 (0x7f8e1d1b3649 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #19: <unknown function> + 0x2ec639f (0x7f8e1eed739f in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #20: <unknown function> + 0x11141d6 (0x7f8e1d1251d6 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #21: at::_log_softmax_backward_data(at::Tensor const&, at::Tensor const&, long, at::Tensor const&) + 0x119 (0x7f8e1d1b3649 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #22: torch::autograd::generated::LogSoftmaxBackward::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) + 0x1d7 (0x7f8e1ed53057 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #23: <unknown function> + 0x3375bb7 (0x7f8e1f386bb7 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #24: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&, std::shared_ptr<torch::autograd::ReadyQueue> const&) + 0x1400 (0x7f8e1f382400 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #25: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x451 (0x7f8e1f382fa1 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #26: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x89 (0x7f8e1f37b119 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #27: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x4a (0x7f8e5522d4ba in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #28: <unknown function> + 0xbd6df (0x7f8e2c9b96df in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #29: <unknown function> + 0x76db (0x7f8e5a03f6db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #30: clone + 0x3f (0x7f8e5a378a3f in /lib/x86_64-linux-gnu/libc.so.6)


I also got a RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED the first few times when I tried to resume it. Now I only get the OOM error.
Traceback (most recent call last):
  File "scripts/train.py", line 449, in <module>
    main(sys.argv[1:])
  File "scripts/train.py", line 230, in main
    scaled_loss.backward()
  File "/disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/autograd/__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
Exception raised from createCuDNNHandle at /pytorch/aten/src/ATen/cudnn/Handle.cpp:9 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fe80cfcc1e2 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xfef088 (0x7fe799da0088 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #2: at::native::getCudnnHandle() + 0x108d (0x7fe799da196d in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #3: <unknown function> + 0xebcaec (0x7fe799c6daec in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xeb800e (0x7fe799c6900e in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0xeb9bfb (0x7fe799c6abfb in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #6: at::native::cudnn_convolution_backward_input(c10::ArrayRef<long>, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool) + 0xb2 (0x7fe799c6b152 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xf1f35b (0x7fe799cd035b in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #8: <unknown function> + 0xf4f178 (0x7fe799d00178 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #9: at::cudnn_convolution_backward_input(c10::ArrayRef<long>, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool) + 0x1ad (0x7fe7d4ba888d in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #10: at::native::cudnn_convolution_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool, std::array<bool, 2ul>) + 0x223 (0x7fe799c69823 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #11: <unknown function> + 0xf1f445 (0x7fe799cd0445 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #12: <unknown function> + 0xf4f1d4 (0x7fe799d001d4 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #13: at::cudnn_convolution_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool, std::array<bool, 2ul>) + 0x1e2 (0x7fe7d4bb7242 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #14: <unknown function> + 0x2ec9c62 (0x7fe7d687ac62 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #15: <unknown function> + 0x2ede224 (0x7fe7d688f224 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #16: at::cudnn_convolution_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool, std::array<bool, 2ul>) + 0x1e2 (0x7fe7d4bb7242 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #17: torch::autograd::generated::CudnnConvolutionBackward::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) + 0x258 (0x7fe7d6701c38 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #18: <unknown function> + 0x3375bb7 (0x7fe7d6d26bb7 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #19: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&, std::shared_ptr<torch::autograd::ReadyQueue> const&) + 0x1400 (0x7fe7d6d22400 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #20: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x451 (0x7fe7d6d22fa1 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #21: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x89 (0x7fe7d6d1b119 in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #22: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x4a (0x7fe804e604ba in /disk/no_backup/user/D4LCN/env/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #23: <unknown function> + 0xbd6df (0x7fe7e43596df in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #24: <unknown function> + 0x76db (0x7fe8119df6db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #25: clone + 0x3f (0x7fe811d18a3f in /lib/x86_64-linux-gnu/libc.so.6)