Hi,
I’m wondering if there’s a simple way to make a single optimizer product out of several optimizers (over disjoint parameters), such that the resulting optimizer product is a well formed instance of torch.optim.Optimizer.
The straightforward answer 9 works well in simple cases, but fails when the optimizer product needs to be a well formed instance of torch.optim.Optimizer (e.g. when checkpointing or using frameworks such as pytorch lightning).