Really appreciate your suggestions.
The reason why I am looking for a single model that can conquer all the problems mentioned above is that I have to utilize the model in a different domain (i.e in clinical). And   I will further pre-train the model with domain-specific data. As you know this pre-training process is very expensive, I am looking for a single model for all the problems. Also, I have already worked with bert by pre-training on a domain-specific dataset but the result was not promising.
Will look into the encoder-decoder models that you are suggesting.