Hi,
Your gradient update is not done properly I’m afraid:

You should not need retain_graph=True. This is only needed here because the gradient update is wrong, you can remove it.
To update a weight without the autograd tracking the update, you should use torch.no_grad():

with torch.no_grad():
  self.fully_connected_layer_1_weight -= self.fully_connected_layer_1_weight.grad*learning_rate