Solved by ptrblck in post #2 
              
              
                The optimizer would need the references to all parameters it should update. In the usual case you would either create two separate optimizers (one for each model) or pass the parameters of both models to a single optimizer. 
In your current setup you would also train mlp_model and cnn_model would be…