you can get parameters gradients using param.grad,
gradients_norms = []
for param in model.parameters():
    gradien_norms.append(param.grad.abs().sum())

Then you can plot these gradient norms.
In case of calling optimizer.zero_grad() after optimizer.step() the graph should decrease, since we are converging the loss functions.
But in case of calling optimizer.step() only, the graph should not decrease, in fact it should increase since we were adding the gradients for each update.