Did you mean that you take the mean of the 3 sets of gradients before applying the update?
If you meant to say that you take the mean of the 3 model outputs, then I am fairly certain that won’t work because PyTorch is smart enough to know where the 3 parts came from and backpropagate accordingly. In other words, if model1 produces a large loss on sample1, and model2 produces a small loss on sample2, then model1 will get larger gradients even if you average the losses before backpropagating.