I’m not really familiar with the architecture of YOLO and am nut sure, how exactly you would like to add skip connections between your two models.
Anyway, based on your example model, you could just add layers to your model (e.g. from YOLO) and manipulate the forward method as you wish.