The problem is that with float numbers, if you compute two things in two different way, you can end up with slightly different values. When you do gradient descent, these small differences are amplified and you get completely different networks.
You should try different random seeds to see what happens.