Solved by InnovArul in post #4 
              
              
                Yes this would work. For example, the following work seems to follow this paradigm. 
ensemblenet: end-to-end optimization of multi-headed models 

This might help.