The issue was with the values that I was providing both Adam and L-BFGS:
  optimizer = optim.LBFGS([img], optim_state)
  optimizer = optim.Adam([img], optim_state)

optim_state = None
if params.optimizer == 'lbfgs':
  optim_state = {
    "max_iter": params.num_iterations,
    "tolerance_change": -1,
    "tolerance_grad": -1,
  }
elif params.optimizer == 'adam':
  optim_state = {
    "lr": 1,
  }

Though I’m not sure how PyTorch’s optimizers expect the inputs to be setup as?