Solved by ptrblck in post #2 
              
              
                I don’t think these plots show the loss gradient, but might show the (accumulated) gradients of all parameters. 
From the paper: 

Results averaged over five runs are shownin Figure 3. The losses initially start on the same scale, but the modern network later has losses thatincrease dramatically — s…