Hi,
The optimizer step done with pytorch optimizers are not differentiable I’m afraid. So the gradient won’t be able to flow all the way back. You can check the higher library here 2 to see how they do this using their custom differentiable optimizers.
Also you should not use .data. If you want to update the poison_image without the gradients being tracked, you should do:
with torch.no_grad():
  poison_image += 0.1*p_grad[0].sign()