Oh ok so you could just create a different sequential model and pass the second inputs into that. Then you would have to do the
model.backbone

and the other layers to make the input go through each individual part of the model. For example
x = model.transform(x)
x = model.backbone(x)
x1 = OTHERMODEL(x1)
x = cat(x,x1)

then pass the x into the fpn, rpn, roi, roi heads, and box predictors