Let’s say following multi-task learning setting like Fig:1, where we have two related datasets (dataset 1 and dataset 2). For each iteration creat a batch of 64 (32+32) that goes through some layers. Each dataset has different classifiers. Next to those common middle layers, there are two classifiers for each dataset. Both losses affect on middle layers.
Now, let’s say another experimental situation (fig: 2), where I don’t calculate loss 2 and its gradients, But both datasets go through the middle layers. Here, I only calculate loss 1 and it optimizes using this loss.
My question is, should not the gradient calculation and optimization be the same of if I don’t use the second dataset at all like fig:3?  Because, I think, the computation graph of loss 1 (fig:2) does not contain any information about the second dataset.
But, this is not happening in my case. Fig:3 model optimizes much better and give better performance than fig: 2
Screen Shot 2020-10-08 at 6.34.23 PM780×444 25.3 KB