Hi,
So yes this is expected. The very small difference from each iteration is increased by the gradient descent and the two version will converge to different set of weights.
You can see this as using different random seeds for your initialization: you get a different set of weights but the final performance will be very close.

Also, by running cuda-memcheck , I got many errors, but I found no explicit errors after looking very carefully the logic of my code. Would you please tell me what is my problem ?

It is hard to say. I am by no mean a cuda expert. What kind of errors do you see? Do they point to your code directly?