In pytorch, if you want to get just one gradient, you should be calling zero_grad() before calling backward, in that case, the two things are the same since the previous gradBias and gradWeight are 0.