I think what you suggest might be wrong. These two optimizers are scoped with different parts of the network. Yes for the main optimizer, it is given the whole network parameters but the loss is computed over the main output only. Therefore, when I backward() the aux output layer is not updated. So I believe they do not overwrite each other. At least this is what I imagine here.