This doesn’t seem to be a cudnn issue, if disabling it breaks the output or am I misunderstanding the issue?
Could you update PyTorch to 1.7.1 as well as the nightly release and test your model again, as there might have been a known issue in the native conv implementation (which would be used in cudnn is disabled), which might have been fixed already?