For anyone stumbling upon this:
Turns out it isn’t a bug per se but it is a corner case of ReLU’s gradient at the (non-differentiable) point 0 - CuDNN says it’s one (the limit from the right), “native” PyTorch has been switched to zero (the limit from the left).
Demonstration code is in the bug report. 6
Now, with a ReLU-RNN I would expect that corner case to be somewhat relevant and it makes me wonder if that makes ReLU-RNN actually a bit of a bad idea.
Best regards
Thomas