Memory is almost completely allocated and then it happens in the end of the epoch not even getting to the validation step.
Train Epoch: 1 [102/111 (91%)] Loss: 209794.437500
Train Epoch: 1 [104/111 (93%)] Loss: 188173.593750
Train Epoch: 1 [106/111 (95%)] Loss: 205403.281250
Train Epoch: 1 [108/111 (96%)] Loss: 504861.593750
Traceback (most recent call last):
  File "train.py", line 92, in <module>
    main(config, args.resume)
  File "train.py", line 61, in main
    trainer.train()
  File "source/base/base_trainer.py", line 91, in train
    result = self._train_epoch(epoch)
  File "source/trainer/trainer.py", line 77, in _train_epoch
    scaled_loss.backward()
  File "/usr/local/lib64/python3.6/site-packages/torch/tensor.py", line 102, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/usr/local/lib64/python3.6/site-packages/torch/autograd/__init__.py", line 90, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

I am not using the code from that NVIDIA workshop. I only adopted their VNET model for testing. This template is the one I am using. https://github.com/victoresque/pytorch-template 3