Is it possible to use multiple optimizers to train a model?
For example, I would like to start training with the Adam optimizer and then at a certain point continue training with SGD. How would I correctly do this in pytorch?
Would something like this work?
optimizer = torch.optim.Adam(model.parameters(), lr=0.0004)

for e in range(epochs):
    for i, data in enumerate(dataloader()):
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, label)
        loss.backward()
        optimizer.step()

        if e == 100:
             optimizer = torch.optim.SGD(model.parameters(), lr=0.002, momentum=0.9)