The updates should be equal, as I also get exactly the same gradients for all parameters in these models.
How reproducible is the accuracy difference? I.e. are you always seeing this gap for e.g. 10 runs with different seeds?