I am training a seq2seq model using SGD and I get decent results. My batch size is 2, and I don’t average the loss over the number of steps.
I am using PyTorch this way:
optimizer = torch.optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-3)

SGD works fine, I can observe losses decreasing slowly, and the final accuracy is pretty good.
Obviously, I wanted to try Adam optimizer to check if I can get the results any faster.
optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-3)

But Adam doesn’t work at all. None of the training/dev/test losses decreases (only noise) and they have far too big values.
Do you have any idea what may cause such behaviour?