Additionally to setting the seeds you should also disable cudnn.benchmark and enable the cudnn deterministic behavior, if you are using a GPU. Have a look at the Reproducibility docs 38 for more information.
Also, how large are the differences? As explained in the docs, some atomic operations might yield non-determinisitc behavior.