Ya so just do what I said above. Just pass one input into the fasterrcnn backbone and then have another sequential model and pass an input into that and concatenate them together and then pass them through the next layers.