The .grad attribute is None only at creation, then after it’s been used once, it will be full of zeros, this is done when you call optimizer.zero_grad() usually. So an unused element will either have None or only zeros (if you zeroed them out just before the backward, otherwise it will be whatever what there before + the gradient you just propagated).