If you use CuDNN, deterministic should work (but you need nightlies or CuDNN < 7.5(!)).
You need to follow the input conventions and probably check for the grad_fn to be the CuDNN one.
Or you could write the deterministic native backward.
Best regards
Thomas