Both my dataloaders have a batch size of 1
    data_loader_train = torch.utils.data.DataLoader(
        train,
        batch_size=1, 
        shuffle=True, 
        num_workers=1,
        collate_fn=utils.collate_fn
    )

    data_loader_test = torch.utils.data.DataLoader(
        test, 
        batch_size=1, 
        shuffle=False, 
        num_workers=1,
        collate_fn=utils.collate_fn
    )

Also the evaluation function has the @torch.no_grad() decorator. As per the torchvision detection engine code.