Sorry for the delay. I am unfortunately still getting the error, and sometimes I get CUDNN_STATUS_EXECUTION_FAILED.
I created two pytorch environments, the first was using the exact command related to the pytorch nightly build. The second was cloning the pytorch nightly branch and compiling from source against cudatoolkit 10.2, and cuDNN 7.6.5. Pytorch takes awhile to build and the windows instructions on building was non-intuitive in certain areas, hence the delay in my response.
Both environments result in similar error profiles.
Here is a gist for the errors from the pytorch nightly environment: pytorch-nightly-cuDNN-GRU-errors 4