Hi @ptrblck,

With different SEED, Caffe still converges better and gives a much better model than PyTorch
I also modified the data loader to have similar to Caffe (not worrying much about the discrepancy of the updates). Still the performance of the end model is not any better.

Questions

Do you think I should worry about how PyTorch and Caffe updates are happening ?
so far my idea was if the Caffe and PyTorch updates are very similar over good number of iterations, then I can say problem lies somewhere else.!

Do you have any suggestions for further debugging.?