Thanks for the code!
Could you try to pass some cuDNN flags in case you are using the GPU as described in the Reproducibility docs 5 and see if the error lowers?
The initial error suggests floating point precision issues.
Could you just for debugging purposes also try to cast all parameters and the data input to .double() and check the error again?