Hello @MrRobot, I am also facing a similar problem to train a network with multiple different branches. Did you find any solution to your problem?
Especially, I am interested to know, how to train a learning network multiple branch network with a shared network and different types of losses (MSE, CrossEntropyLoss, and L2) as well as optimizers for different branches
@ptrblck If you have time could you please take look? It will be really helpful, I am stuck on this issue for several weeks. Thanks