@ptrblck thank you for the reply.
Here is the gradient for the first FC layer for Caffe and PyTorch:
Top: Caffe gradient Bottom: PyTorch grad . As I can see the gradients are rounded off.

Screenshot from 2020-03-18 12-43-26781×666 73.5 KB

Yes, I also couldn’t make sense of why loss stays the same for quite a while and changes weirdly in some of the iterations. One of my guesses is since its only FC layers that are updated, loss = sum(w.x + b), which might lead to that error. But still strange behaviour.