The real stacktrace (using CUDA_LAUNCH_BLOCKING=1) is:
Loaded 100 training examples
Start training loop
Epoch: 0
Training batch: 0
Traceback (most recent call last):
  File "tmp.py", line 70, in <module>
    err.backward()
  File "/usr/local/lib/python2.7/dist-packages/torch/autograd/variable.py", line 146, in backward
    self._execution_engine.run_backward((self,), (gradient,), retain_variables)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/_functions/conv.py", line 48, in backward
    if self.needs_input_grad[0] else None)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/_functions/conv.py", line 112, in _grad_input
    cudnn.benchmark)
RuntimeError: CUDNN_STATUS_EXECUTION_FAILED

Temp fix: Adding torch.backends.cudnn.enabled = False at the beginning of your script make it work as a temporary fix.
Real Problem: Why is this cudnn call fails? I don’t know 
EDIT: this exact code that fails on a Titan X runs properly on a Titan Black.