Now I can see the error clearly. When the optimizer starts consuming the filter iterator, the membership test id(p) not in encode_params throws an error.
This snippet reproduces the error:
ls = [torch.randn(3), torch.randn(2, 3)]
ls[0] in ls # returns True
torch.randn(4) in ls # error
id(torch.randn(3)) in ls # error again

You can think of the membership test looking like this:
if obj is value:
   return True
else:
   return bool(obj == value)

This explains why the first example returns True and the other two throw an error.
And this is why you should use sets instead of lists to solve this issue.