As @jubick said the first thing that comes to mind is lowering the learning rate. What is your current learning rate?  Adam doesn’t adjust the learning rate, for that you would need to use a learning rate scheduler. Example of how to set the learning rate for Adam:
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

Share a code snippet and it’ll help debugging what could be wrong in your code