What kind of CNNs are you using at the moment and how do these architectures relate to your encoder-decoder model?
Do your current models share some layers (e.g. is the base of the model “similar”)?
If so, you could try to retrain a new model by keeping the common architecture and use two different heads for each task.