@mMagmer I see. It works once I changed lr to 0.0001.
I also tried to manually adjust lr when then batch size is low: optimizer.param_groups[0]['lr'] *= 0.1, and then reset it after calling the step() function. This seems like a hack, not sure about if there’s any side effect.
Thank you so much for your suggestion.