Hi @ptrblck,
It’s working just fine on CPU. Strangely it works with volumes of smaller size, but I cannot see any memory exhaustion when observing GPU memory. I am using a VNET implementation and input volume size of 304,304,96
Traceback (most recent call last):
  File "train.py", line 82, in <module>
    main(config, args.resume)
  File "train.py", line 51, in main
    trainer.train()
  File "/home/sources/project/source/base/base_trainer.py", line 92, in train
    result = self._train_epoch(epoch)
  File "/home/sources/project/source/trainer/trainer.py", line 71, in _train_epoch
    loss.backward()
  File "/usr/local/lib64/python3.6/site-packages/torch/tensor.py", line 102, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/usr/local/lib64/python3.6/site-packages/torch/autograd/__init__.py", line 90, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

I also moved to use half precision (NVIDIA apex) and while testing on the “large” volumes, I ran into this issue again because I had to enable CUDNN.