Your code looks generally fine.
I’m not sure, which model you are passing to your custom model, as the torchvision resnet50 doesn’t have a body and fpn module.
Could you explain the last issue a bit more?
Your model should return the features tensor regardless of the used batch size.