If you have that few parameters, you could try LBFGS. I use that e.g. for Gaussian Processes.
Note that it necessarily needs a closure (re-) evaluating the model.
Here is a snippet that I use with GPs (which are a bit special in that they take the data and loss into the model, you’d have to feed data into m and probably also go from predictions to loss - called objective here), which should give you the idea:
opt = torch.optim.LBFGS(m.parameters(), lr=1e-2, max_iter=40)
def eval_model():
    obj = m()
    opt.zero_grad()
    obj.backward()
    return obj

for i in range(50):
    obj = m()
    opt.zero_grad()
    obj.backward()
    opt.step(eval_model)
    if i % 5==0:
        print(i,':',obj.item())


Best regards
Thomas