momentum would be such an internal state, which uses the previous gradients to update the parameters even with a zero gradients.
Here is a small example showing this behavior:
# Setup
model = nn.Conv2d(1, 10, 3, 1, 1)
weight_reference = model.weight.clone()
optimizer = torch.optim.SGD(model.parameters(), lr=1., momentum=0.0)

# Should fail
model(torch.randn(1, 1, 3, 3)).mean().backward()
optimizer.step()
print((model.weight == weight_reference).all())

# Should work with momentum = 0.0
optimizer.zero_grad()
weight_reference = model.weight.clone()

model.weight.register_hook(lambda grad: grad * 0)
model(torch.randn(1, 1, 3, 3)).mean().backward()
optimizer.step()

print((model.weight == weight_reference).all())

The parameters should not be updated using mementum=0.0, while any other value for momentum will still update the parameters.