Hello everybody,
the authors (Bjorck et al. 2021) of the paper “Towards Deeper Deep Reinforcement Learning 1” share a plot (Figure 3) showing the gradient of the actor and the critic loss (SAC).

plot1404×465 83.2 KB

I’ve got no idea how this plots was achieved, because the gradient of the loss is 1.0 after calling backward() (e.g. loss.grad). Do you guys have any idea on what’s behind this plot? Is it simply tracking the magnitude of the last layer? Or is it likely that the mean of the magnitude of all layers are visualized?
Unfortunately I could not find a proper way to contact the author himself. However, I created an issue of an unrelated repository of the first author.