I just want to add a note since there’s others with similar experience
Here it’s also found that caffe version is better for finetuning for object detection task:

  
      
      GitHub
  
  
    

jwyang/faster-rcnn.pytorch 17

A faster pytorch implementation of faster r-cnn. Contribute to jwyang/faster-rcnn.pytorch development by creating an account on GitHub.


  
  
    
    
  
  


You can get the caffe pretrained version there, or another version is at: https://github.com/Cadene/pretrained-models.pytorch 36