Methods and systems for training a neural network model include training (404) a modular neural network model (100), which has a shared encoder and one or more task-specific decoders, including training one or more policy networks that control connections between the shared encoder and the one or more task-specific decoders in accordance with multiple tasks. A multitask neural network model (120) is trained (404) for the multiple tasks, with an output of the modular neural network model and the multitask neural network model being combined to form a final output.
MODULAR NETWORKS WITH DYNAMIC ROUTING FOR MULTI-TASK RECURRENT MODULES