Could you compare the number of parameters for both implementations?
Your TF reference uses placeholders, so that it’s not easy to compare the layer setup.
Also, you might want to remove cudnn.deterministic = True as this might pick a deterministic but slow algorithm.