once you loaded the pretrained resnet-101, make sure you set it in eval() mode. Otherwise BatchNorm effects might be weird (your inference performance will depend on batch size).
model = torchvision.models.resnet101(pretrained=True)
model.eval()