I am using https://github.com/DebasmitaGhose/Multimodal_Influenza_Detection/tree/master/faster-rcnn.pytorch.
In order to remove the 4th max pooling layer I modified the lib/model/vgg16.py as follows
    list1 = list(vgg.classifier._modules.values())[:23]
    list1.extend(list(vgg.classifier._modules.values())[24:-1])

    list2 = list(vgg.features._modules.values())[:23]
    list2.extend(list(vgg.features._modules.values())[24:-1])
    vgg.classifier = nn.Sequential(*list1)
    self.RCNN_base = nn.Sequential(*list2)

Moreover in https://github.com/DebasmitaGhose/Multimodal_Influenza_Detection/blob/master/faster-rcnn.pytorch/lib/model/faster_rcnn/faster_rcnn.py, I modified the following lines
self.RCNN_roi_pool = _RoIPooling(cfg.POOLING_SIZE, cfg.POOLING_SIZE, 1.0/8.0)
self.RCNN_roi_align = RoIAlignAvg(cfg.POOLING_SIZE, cfg.POOLING_SIZE, 1.0/8.0)

In the lib/model/utils/config.py I made the following modifications
__C.DEDUP_BOXES = 1. / 16.
__C.FEAT_STRIDE = [16, ]

to
__C.DEDUP_BOXES = 1. / 8.
__C.FEAT_STRIDE = [8, ]