Hi @ptrblck, in order to verify the forward, backward pass. I kept the input same for 100 iterations for both caffe and pytorch implementation.
I see the loss of Caffe completely different from PyTorch after few iterations (observe at the end)
PyTorch           | Caffe
-------------------------
757.2705688476562   757.271
942.8200073242188   942.82
799.31884765625     799.319
812.5123291015625   812.512
667.9791259765625   667.979
413.325439453125    413.325
562.3018188476562   562.302
350.0400390625      350.04
297.4466552734375   297.447
343.02001953125     343.02
394.5355224609375   394.536
228.69955444335938  228.7
251.35780334472656  251.358
307.5043640136719   307.504
223.70187377929688  223.702
233.28732299804688  233.288
315.7559814453125   315.756
226.81146240234375  226.811
181.53457641601562  181.533
206.6217041015625   206.625
210.15628051757812  210.157
212.7760009765625   212.775
209.96849060058594  209.974
214.49813842773438  214.493
183.6826629638672   183.672
207.22445678710938  207.245
168.16375732421875  168.184
201.4786376953125   201.458
141.69427490234375  141.673
255.06842041015625  255.063
277.982421875       278.047
228.63650512695312  228.61
265.3466796875      265.355
245.4405059814453   245.387
217.6405792236328   217.715
243.95590209960938  244.007
225.05352783203125  225.017
202.35430908203125  202.491
242.1893768310547   242.393
196.67547607421875  197.126
221.29019165039062  221.316
229.14529418945312  229.14
208.05859375        207.983
201.7965087890625   201.89
204.59408569335938  204.142
222.6988525390625   222.059
212.80035400390625  213.449
262.5123291015625   262.53
153.87637329101562  155.095
238.25704956054688  239.609
284.07769775390625  284.95
194.6011962890625   197.989
171.880126953125    171.607
264.2562255859375   267.292
202.17262268066406  201.321
149.88726806640625  146.92
213.33978271484375  219.864
294.94012451171875  298.808
206.38232421875     201.994
184.72096252441406  194.253
255.96823120117188  237.341
211.44276428222656  194.038
151.47923278808594  149.074
172.3814697265625   154.429
179.71731567382812  183.083
151.15765380859375  164.78
157.54025268554688  161.271
191.8701629638672   195.957
166.78912353515625  162.826
166.08004760742188  164.285
177.39981079101562  184.853
170.86624145507812  160.088
165.30255126953125  169.323
145.57489013671875  156.784
194.71859741210938  151.346
238.37454223632812  264.286
175.21041870117188  192.347
136.05355834960938  162.889
172.6317138671875   155.631
208.10299682617188  185.586
221.0286865234375   217.085
173.4173583984375   155.914
253.78131103515625  244.531
248.7365264892578   249.555
194.50607299804688  183.464
212.595947265625    227.499
252.08670043945312  224.977
243.98223876953125  214.467
175.81005859375     139.689
200.88902282714844  170.642
168.1161346435547   189.527
210.65390014648438  243.979
246.09844970703125  232.952
207.2693634033203   178.198
124.55677795410156  182.764
248.9339599609375   264.27
230.8294677734375   216.812
159.265625          130.281
178.865966796875    229.354
157.16854858398438  207.201
224.73690795898438  204.391

Do you have an idea or suggestion on why this could be?
@smth Any suggestions or idea ?