Hey @diegoalejogm, I was trying to implement the DQN algorithm by myself and had faced the same issue.

image1285×292 52.8 KB

If I understand right, it is mentioned to clip the update of the difference, which probably means the gradient. so this becomes another case of Gradient Clipping and not the Loss clipping directly.
The official DQN code in the pytorch website does gradient clipping as well.
You can find the code here - Reinforcement Learning (DQN) Tutorial — PyTorch Tutorials 1.9.0+cu102 documentation 11

#Optimize the model

optimizer.zero_grad()
loss.backward()
for param in policy_net.parameters():
    param,grad.data.clamp_(-1,1)
optimizer.step()