That’s some great debugging.
Based on the output it seems the losses are quite shaky around ~200.
I guess the difference might be due to accumulated rounding errors in both approaches.
You’ve mentioned that the final result is worse in PyTorch, right?
Is this reproducible using different seeds for each framework, i.e. does Caffe always converge to a better accuracy?