Solved by ptrblck in post #4 
              
              
                Your use case sounds really interesting.  
Additionally to the mentioned approaches, you might also want to checkout differentiable architecture search approaches, such as DARTS. (I don’t know, if it’s the state-of-the art or what is used nowadays  ).