Thank you for your reply. I am using the Faster R-CNN 1 example from the pytorch website, which is why there is body and fpn modules.
The issue is, that when I train the model without the fusion layers i.e.
z = F.relu(torch.cat((x, y), dim=1))
z = self.conv2(z)

the model trains fine, however, when I use the fusion layers, the model seems to be training with the loss reducing, however, the predictions are very bad.
Your model should return the features tensor regardless of the used batch size.

Sorry I should have been more clear, I meant to say that when I have a low number of samples in my trainloader, the model does not produce any predictions.
I hope I have answered your questions.