I am curious whether zeroing the gradients speeds up the operation, does the  Pytorch implementation of .backward() support sparse weight update ?
If there is no speed up, what stops you from zeroing the weights (instead of the gradients) after taking the optimization step ? in this case, the optimizer class doesn’t matter, unless you are trying to avoid accumulating meta information in the optimizer such as momentum.
You can prune the gradients by changing the forward pass as follows.
If you have W = Variable(torch.randn(10), requires_grad=True) and you would like the gradients of the first 5 coordinates only, maybe you can do something like this in the forward pass,
loss = (torch.mv(X[:, :5], W[:5]) +
           torch.mv(X[:, 5:], Variable(W[5:].data)) - y)**2
loss = loss.sum()

# Compute grad for W[:5] only
loss.backward()

which computes the gradient for W[:5] only.