Thank you for the reply @ptrblck.
Just to be more transparent to you on the complete setup and also to see if I missed my attention to any of the details.  I’m sharing more info to narrow down the problem:

The network is trained with 5-conv layer (weights freezed) and  4-FC layers (trained). The network architecture is shown below. Here I have kept dropout ratio = 0.0 for both Caffe and PyTorch, to observe if the outputs and updates are the same. I made weight decay = 0 as well in both.

        self._net = CaffeNet(pretrained_model_path=pretrained_model)
        dropout_ratio = 0.0
        self._classifier = nn.Sequential(nn.Linear(256 * 6 * 6 * 2, 4096),
                                         nn.ReLU(inplace=True),
                                         nn.Dropout(dropout_ratio),
                                         nn.Linear(4096, 4096),
                                         nn.ReLU(inplace=True),
                                         nn.Dropout(dropout_ratio),
                                         nn.Linear(4096, 4096),
                                         nn.ReLU(inplace=True),
                                         nn.Dropout(dropout_ratio),
                                         nn.Linear(4096, num_output))



At the 100th iteration, I observed the output of conv-5 layer is the same, both in Caffe and PyTorch. This concludes that my inputs are the same and no errors made in this.


L1loss is Caffe is implemented as below:
Power layer implements -1 * gt. Eltwise layer does the element-wise sum (pred - 1*gt) and the Reduction layer does the summation to scalar loss.


layer {
  name: "neg"
  bottom: "bbox"
  top: "bbox_neg"
  type: "Power"
  power_param {
    power: 1
    scale: -1
    shift: 0
  }
}
layer {
  name: "flatten"
  type: "Flatten"
  bottom: "bbox_neg"
  top: "bbox_neg_flat"
}

layer {
  name: "subtract"
  type: "Eltwise"
  bottom: "fc8"
  bottom: "bbox_neg_flat"
  top: "out_diff"
}
layer {
  name: "abssum"
  type: "Reduction"
  bottom: "out_diff"
  top: "loss"
  loss_weight: 1
  reduction_param {
    operation: 2
  }
}

and for PyTorch I use.
loss = torch.nn.L1Loss(size_average=False)(pred_bb, gt_bb.float())

Conclusion and Observations:

As a whole, whole learning(forward/backward/update) is happening in FC layers. These updates are different in Caffe and PyTorch. To be debugged.

Training with different SEED

I haven’t tried with different seed in Caffe. I will take a day to retrain with different seed. I will do that and share.