For anyone interested I’ve found a workaround taken from this answer here http://5.9.10.113/64957944/error-when-trying-to-train-fasterrcnn-with-custom-backbone-on-grayscale-images 3
Basically, we need to redefine the GeneralizedRCNNTransform and attach it to your model
grcnn = torchvision.models.detection.transform.GeneralizedRCNNTransform(min_size=800, max_size=1333, image_mean=[0.485], image_std=[0.229])
model.transform = grcnn
model.to(device)