I have read a paper about a fancy multitask model, and I am trying to rebuild it through pytorch. While I have encountered with several questions. In the model(shown below), the multitask model has two part of input, and they share an encoder. As a freshman, I am a little confused about how to build the model.
I have some thoughts, but I don’t whether they would work.
Should I input all the inputs into one model but separate them into different parts?
But if I do this, how can i solve the problem that the two datasets have different size?
Thank you very much!!!
the model:

image.png1270×896 65.3 KB