ezyang:

see if the problem repros (1) with cuDNN turned off (torch.backends.cudnn.enabled = False)


torch.backends.cudnn.enabled = True:

Stuck at 0.89; 80 epochs.

torch.backends.cudnn.enabled = False:

“Exception: You must be using CUDNN to use _EfficientBatchNorm”
cudnn calls are made throughout densenet_efficient.py 4


Environment:

nvidia-docker, Ubuntu 14.04, CUDA 8.0, cuDNN 6, PyTorch 0.2.0



Details:

Base environment:
docker run --runtime=nvidia --init -it --rm --ipc=host mwksmith/efficient_densenet_pytorch

Here’s the Dockerfile 1. I uploaded the Docker image to Docker Hub at mwksmith/efficient_densenet_pytorch.

Environment for cudnn = True:
Titan X Pascal, Driver Version: 384.111
Command:
CUDA_VISIBLE_DEVICES=1 python demo.py

Environment for cudnn = False:
1080 Ti, Driver Version: 387.34, different machine
I put torch.backends.cudnn.enabled = False after the import statements in demo.py.
Command:
CUDA_VISIBLE_DEVICES=0 python demo.py

@Danlu_Chan, @nikostr, @ezyang