That would be trickier, I don’t think our optimizer API supports removing parameters.
That being said, if the .grad field of that Tensor is None. Then the optimizer will just ignore it. And because it is not in the network anymore, it won’t be updated. So you can leave it in the optimizer.