Solved by ptrblck in post #4 
              
              
                If I’m not mistaken, you are passing the inputs first to the Autoencoder and pass this output to the CNN. 
The code for this looks generally alright and you wouldn’t need to use two losses, if your complete model only returns a single output. 
Note that the last linear layers are used without any no…