I want to implement a model similar to the one described in the picture below taken from https://datascience.stackexchange.com/questions/26103/merging-two-different-models-in-keras 49

mergedmodels.jpg851×897 98 KB

I have implementations of ModelA and ModelB that work fine when I train them separately. I am thinking of creating a class that will merge both of them inspired by this: Combining Trained Models in PyTorch 128.
My questions would be:

How do I handle the parameters? I want to pass ModelA and ModelB parameters to the same optimizer object. Can they be concatenated easily with some pytorch funcion?.
Should the loss function be the sum of lossA and lossB?. That question comes because even though the data set is the same for both models, ModelA's loss “units” are different from those of ModelB's. For example: lossA are in units of length and   lossB are in units of force or energy.

I would appreciate any advice with this and even some code  I am working on this and when coming to a solution I will post it here for reference.