@ptrblck Unfortunately my project is large and deeply engrained with a library I wrote, so it will take time to write a minimal reproduction. I have a stack trace I get without CUDA LAUNCH BLOCKING:
Traceback (most recent call last):
  File "train.py", line 142, in <module>
    if __name__ == "__main__": main()
  File "train.py", line 74, in main
    train_epoch(model, train_dataset, test_dataset, optimizer, epoch, eval_every=eval_every)
  File "train.py", line 113, in train_epoch
    loss.backward()
  File "C:\Users\Joe Fioti\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "C:\Users\Joe Fioti\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\autograd\__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
Exception raised from _cudnn_rnn_backward_input at ..\aten\src\ATen\native\cudnn\RNN.cpp:923 (most recent call first):
00007FF997E675A200007FF997E67540 c10.dll!c10::Error::Error [<unknown file> @ <unknown line number>]
00007FF94242529600007FF9424251E0 torch_cuda.dll!at::native::Descriptor<cudnnRNNStruct,&cudnnCreateRNNDescriptor,&cudnnDestroyRNNDescriptor>::Descriptor<cudnnRNNStruct,&cudnnCreateRNNDescriptor,&cudnnDestroyRNNDescriptor> [<unknown file> @ <unknown line number>]
00007FF94243C11B00007FF942439AD0 torch_cuda.dll!at::native::_cudnn_rnn_backward [<unknown file> @ <unknown line number>]
00007FF94243A03000007FF942439AD0 torch_cuda.dll!at::native::_cudnn_rnn_backward [<unknown file> @ <unknown line number>]
00007FF942492BA800007FF94244E400 torch_cuda.dll!at::native::set_storage_cuda_ [<unknown file> @ <unknown line number>]
00007FF9424A13DD00007FF94244E400 torch_cuda.dll!at::native::set_storage_cuda_ [<unknown file> @ <unknown line number>]
00007FF93A51BBF100007FF93A48D9D0 torch_cpu.dll!at::native::mkldnn_sigmoid_ [<unknown file> @ <unknown line number>]
00007FF93A56B9DA00007FF93A568FA0 torch_cpu.dll!at::bucketize_out [<unknown file> @ <unknown line number>]
00007FF93A552ECA00007FF93A552D40 torch_cpu.dll!at::_cudnn_rnn_backward [<unknown file> @ <unknown line number>]
00007FF93B85088900007FF93B80E010 torch_cpu.dll!torch::autograd::GraphRoot::apply [<unknown file> @ <unknown line number>]
00007FF93B85D12D00007FF93B80E010 torch_cpu.dll!torch::autograd::GraphRoot::apply [<unknown file> @ <unknown line number>]
00007FF93A51BBF100007FF93A48D9D0 torch_cpu.dll!at::native::mkldnn_sigmoid_ [<unknown file> @ <unknown line number>]
00007FF93A56B9DA00007FF93A568FA0 torch_cpu.dll!at::bucketize_out [<unknown file> @ <unknown line number>]
00007FF93A552ECA00007FF93A552D40 torch_cpu.dll!at::_cudnn_rnn_backward [<unknown file> @ <unknown line number>]
00007FF93B75C12D00007FF93B75BAF0 torch_cpu.dll!torch::autograd::generated::CudnnRnnBackward::apply [<unknown file> @ <unknown line number>]
00007FF93B747E9100007FF93B747B50 torch_cpu.dll!torch::autograd::Node::operator() [<unknown file> @ <unknown line number>]
00007FF93BCAF9BA00007FF93BCAF300 torch_cpu.dll!torch::autograd::Engine::add_thread_pool_task [<unknown file> @ <unknown line number>]
00007FF93BCB03AD00007FF93BCAFFD0 torch_cpu.dll!torch::autograd::Engine::evaluate_function [<unknown file> @ <unknown line number>]
00007FF93BCB4FE200007FF93BCB4CA0 torch_cpu.dll!torch::autograd::Engine::thread_main [<unknown file> @ <unknown line number>]
00007FF93BCB4C4100007FF93BCB4BC0 torch_cpu.dll!torch::autograd::Engine::thread_init [<unknown file> @ <unknown line number>]
00007FF97E3D08B700007FF97E3A9F90 torch_python.dll!THPShortStorage_New [<unknown file> @ <unknown line number>]
00007FF93BCABF1400007FF93BCAB780 torch_cpu.dll!torch::autograd::Engine::get_base_engine [<unknown file> @ <unknown line number>]
00007FF9C86E0E8200007FF9C86E0D40 ucrtbase.dll!beginthreadex [<unknown file> @ <unknown line number>]
00007FF9CA477BD400007FF9CA477BC0 KERNEL32.DLL!BaseThreadInitThunk [<unknown file> @ <unknown line number>]
00007FF9CA82CE5100007FF9CA82CE30 ntdll.dll!RtlUserThreadStart [<unknown file> @ <unknown line number>]

This refers to a line in the RNN.cpp file with this:
auto datatype = getCudnnDataType(input);

Not sure if this helps or not.