There are many possible ways of structuring your model, and it will probably depend on your application.
One common approach is to return the results in a list (one for each branch), and have the loss accept the list.
But your might need some experimentation in order to find what works best for you.