Assuming that your sharedNet can deal with batches, you can do something like

def forward(self, input1, input2):
    out = self.sharedNet(torch.cat([input1, input2], 0))
    b = input1.size(0)
    if b == 1:
        out = out.view(1, -1, out.size(2), out.size(3))  # change if 1d/3d input
    else:
        out = torch.cat((out[:b], out[b:]), dim=1)
    out = self.header(out)
    return out


The set of params each optimizer updates is unrelated with how you structure your code. It is entirely decided by the collection of params you gave at constructing the optimizer, e.g. Adam(itertools.chain(net1.parameters(), net2.parameters(), ...). And all optimizers do is to update parameters basing on the computed .grad gradients. You won’t be able to easily update multiple times even if you want to.