Hi there,
I was training a model with SGD and decided to move to Adam.
I was using batch size 20 for SGD, however max BS i can use with Adam is 2.
    optimizer = torch.optim.SGD([{'params': model.unet_model.parameters()},{'params': model.audio_s.parameters()}, {'params': model.drn_model.parameters(), 'lr': args.DRNlr},
                                 ], lr=LR,
                                weight_decay=WEIGTH_DECAY)

All I did is changing this line
    optimizer = torch.optim.Adam([{'params': model.unet_model.parameters()},{'params': model.audio_s.parameters()}, {'params': model.drn_model.parameters(), 'lr': args.DRNlr},
                                 ], lr=LR,
                                weight_decay=WEIGTH_DECAY)

is there any memory usage comparison among all the optimizers? or is that memory usage normal?