Solved by RupertP in post #3 
              
              
                I fixed it by changing how the task weights are stored. 
I created a new nn.Module with torch.Parameters(torch.tensor(1,1,1)). They require grads and the loss weighting happens in the feedforward function.