Shameless plug: I wrote a little helper library that makes it a little easier to do multi-task learning: torchMTL 153. It should work for your example and makes it easy to combine the losses while keeping control over the training loop. I thought it might be of interest for people who are running into similar issues.