Thank you @SimonW, I totally  forgot that the forward step usually reconstructs the whole graph from scratch, and that this is what usually happens with the canned models!
So, the graph is re-built, but the variables the represent the model parameters are re-used, so the gradients for those parameters need to get set to zero in order to avoid accumulating them.