Hello!
Finally I was able to solve the problem.
My boxes where in the format [xmin, ymin, w, h] whilst they should be as [xmin, ymin, xmax, ymax]
Moreover, the image values to feed the network were between (0, 255) and they should be between (-1, 1) (isn’t it?)
Also, when I create the model I should pass the mean and std of the images set:
"""#Finetuning the model"""

num_classes  = 2
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True, image_mean = mean, image_std = std)

Otherwise, the fasterrcnn_resnet50_fpn would use some default mean and std (from ImageNet maybe?) -see part of fasterrcnn_resnet50_fpn  code below-
if image_mean is None:
            image_mean = [0.485, 0.456, 0.406]
        if image_std is None:
            image_std = [0.229, 0.224, 0.225]
        transform = GeneralizedRCNNTransform(min_size, max_size, image_mean, image_std)

Thankyou so much for your time!!