I think the deterministic behavior for each script run is expected, but there is no guarantee to get deterministic output inside the script by rerunning the code.
For the cudnn RNN implementations you can see that the dropout state is initialized once in these lines of code 2, which explains why you would see the same outcome when rerunning the entire script.