Okay. Can reproduce the issue simply by changing the POOLING_MODE in pytorch-faster-rcnn/experiments/cfgs/res101.yml from crop to roi. This means the issue is somewhere within the roi pooling module.
To reproduce the error, please checkout my fork (master):
git clone https://github.com/Feynman27/pytorch-faster-rcnn.git

After checking out the repo, you’ll have to download pascal voc 2007:
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCdevkit_08-Jun-2007.tar

In the repo’s data directory, create a softlink to VOCdevkit called VOCdevkit2007.
You’ll also need pretrained resnet101 weights from  here. Place this in data/pretrained_models/res101.pth.
After this, build the nms and roi pooling modules by running ./make_modules.sh and  run the training script using ./experiments/scripts/run_resnet.sh.