I found that the usage of smooth l1 loss (Huber) always led to divergence on the cart pole environment (somebody else also had that problem I’ll add the link later)
It worked for me when using l2 (mse) loss
Further investigation showed that the loss explodedn with the Huber loss, (I ll need to check what happened to the gradient)
I found this very strange considering that the Huber loss is specifically designed to be stable
If anyone has any idea why this happens please let me know
Tldr if you usw smoothl1 loss, just try mse