Even when using model.eval() I get different predictions when changing the batch size. I’ve found this issue when working on a project with Faster R-CNN and my own data, but I can replicate it in the tutorial “TorchVision Object Detection Finetuning Tutorial” (TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 1.9.0+cu102 documentation 4), which uses Mask R-CNN.
Steps to replicate the issue:

Open collab version: Google Colaboratory 1

Run all cells
Insert a new cell at the bottom with the code below and run it:

def get_device():
    if torch.cuda.is_available():
        return torch.device('cuda')  
    else:
        return torch.device('cpu')

def predict(model, image_tensors):
    """
    Generate model's prediction (bounding boxes, scores and labels) for a batch 
    of image tensors
    """
    model.eval()
    with torch.no_grad():
        predictions = model([x.to(get_device()) for x in image_tensors])
    return predictions

def generate_preds(model, batch_size):
  """
  Create dataloader for test dataset with configurable batch size.
  Generate predictions. Return a list of predictions per sample.
  """
  dataloader = torch.utils.data.DataLoader(
    dataset_test, batch_size=batch_size, shuffle=False, num_workers=2,
    collate_fn=utils.collate_fn)
  all_pred = []
  for batch in dataloader: 
    image_tensors, targets = batch
    predictions = predict(model, image_tensors)
    all_pred += predictions
  return all_pred

# Generate two sets of predictions, only change is batch size
preds1 = generate_preds(model, 1)
preds8 = generate_preds(model, 8)
assert len(preds1) == len(preds8)

# Investigate first five samples:
for x in range(5):
  print(f"\nSample {x}:")
  print("-Boxes")
  print(preds1[x]["boxes"])
  print(preds8[x]["boxes"])
  print("-Scores")
  print(preds1[x]["scores"])
  print(preds8[x]["scores"])
  print("-Labels")
  print(preds1[x]["labels"])
  print(preds8[x]["labels"])

The code above generates two sets of predictions for the test set. The first one is generated with a batch size 1 and the second with a batch size 8. The output that I get when I run that cell:
Sample 0:
-Boxes
tensor([[ 61.2343,  37.6461, 197.8525, 325.6508],
        [276.4769,  23.9664, 290.8987,  73.1913]], device='cuda:0')
tensor([[ 59.1616,  36.3829, 201.7858, 331.4406],
        [276.4261,  23.7988, 290.8489,  72.8123],
        [ 81.2091,  37.6342, 192.8113, 217.8009]], device='cuda:0')
-Scores
tensor([0.9989, 0.5048], device='cuda:0')
tensor([0.9988, 0.6410, 0.1294], device='cuda:0')
-Labels
tensor([1, 1], device='cuda:0')
tensor([1, 1, 1], device='cuda:0')

Sample 1:
-Boxes
tensor([[ 90.7305,  60.1291, 232.4859, 341.7854],
        [245.7694,  56.3715, 305.2585, 349.5301],
        [243.0723,  16.5198, 360.2888, 351.5983]], device='cuda:0')
tensor([[ 91.1201,  59.8146, 233.0968, 342.2685],
        [245.7369,  56.6024, 305.2173, 349.3939],
        [241.1119,  32.6983, 362.4162, 346.0358]], device='cuda:0')
-Scores
tensor([0.9976, 0.9119, 0.1945], device='cuda:0')
tensor([0.9975, 0.9128, 0.1207], device='cuda:0')
-Labels
tensor([1, 1, 1], device='cuda:0')
tensor([1, 1, 1], device='cuda:0')

Sample 2:
-Boxes
tensor([[281.1774,  53.5141, 428.7436, 330.3915],
        [139.6456,  23.7953, 264.7703, 330.2114]], device='cuda:0')
tensor([[281.7463,  53.2942, 429.3290, 327.9640],
        [138.7147,  23.8612, 264.6823, 332.3202]], device='cuda:0')
-Scores
tensor([0.9969, 0.9947], device='cuda:0')
tensor([0.9968, 0.9945], device='cuda:0')
-Labels
tensor([1, 1], device='cuda:0')
tensor([1, 1], device='cuda:0')

Sample 3:
-Boxes
tensor([[175.3683,  34.3320, 289.3029, 306.8307],
        [ 76.7871,  15.4444, 187.0855, 299.1662],
        [  0.0000,  45.9045,  51.3796, 222.0583],
        [319.1224,  53.0593, 377.1693, 232.7251],
        [260.2587,  55.8976, 309.0191, 229.4261],
        [ 70.2029,  27.2173, 126.4584, 234.3767],
        [ 38.0638,  55.5370,  65.4132, 164.1965],
        [ 98.7189,  91.5356, 172.5915, 295.5404],
        [ 70.1933,  56.1804, 103.6161, 218.4743]], device='cuda:0')
tensor([[175.1848,  36.0377, 288.8358, 305.3505],
        [ 76.8171,  15.7485, 187.4645, 299.5779],
        [  0.0000,  45.9045,  51.3796, 222.0582],
        [319.1060,  53.0140, 377.3391, 232.7926],
        [260.2587,  55.8976, 309.0191, 229.4261],
        [ 70.2030,  27.2173, 126.4584, 234.3767],
        [ 38.0638,  55.5370,  65.4132, 164.1965],
        [ 70.1933,  56.1804, 103.6161, 218.4743]], device='cuda:0')
-Scores
tensor([0.9968, 0.9959, 0.9942, 0.9937, 0.9271, 0.8133, 0.4273, 0.1163, 0.0884],
       device='cuda:0')
tensor([0.9974, 0.9965, 0.9942, 0.9937, 0.9271, 0.8133, 0.4273, 0.0884],
       device='cuda:0')
-Labels
tensor([1, 1, 1, 1, 1, 1, 1, 1, 1], device='cuda:0')
tensor([1, 1, 1, 1, 1, 1, 1, 1], device='cuda:0')

Sample 4:
-Boxes
tensor([[318.0241,  60.4089, 450.3268, 348.4254],
        [167.0622,  27.6761, 242.5035, 316.6244],
        [221.8452,  26.9947, 310.0547, 291.2983],
        [295.6860,  23.4690, 379.8831, 260.1526],
        [140.3205,  44.4713, 223.6427, 281.9173],
        [141.0462,  24.9851, 313.7406, 301.5022],
        [252.8210,  28.4908, 358.8223, 261.0169]], device='cuda:0')
tensor([[317.8378,  63.2861, 450.5063, 350.6856],
        [167.0629,  27.6768, 242.5045, 316.6241],
        [221.8452,  26.9948, 310.0548, 291.2983],
        [295.6860,  23.4690, 379.8831, 260.1525],
        [142.1777,  24.9079, 313.1906, 302.9822],
        [140.3205,  44.4713, 223.6428, 281.9174],
        [252.8209,  28.4907, 358.8222, 261.0172]], device='cuda:0')
-Scores
tensor([0.9969, 0.9948, 0.9910, 0.9733, 0.1821, 0.1696, 0.0668],
       device='cuda:0')
tensor([0.9968, 0.9948, 0.9910, 0.9733, 0.1832, 0.1821, 0.0668],
       device='cuda:0')
-Labels
tensor([1, 1, 1, 1, 1, 1, 1], device='cuda:0')
tensor([1, 1, 1, 1, 1, 1, 1], device='cuda:0')

As far as I know, the prediction output for each sample for the batch size 1 and batch size 8 should be the same. However, there are differences in scores, bounding boxes, number of proposals…
Any help will be appreciated