albanD:

Then you can set them back (as leafs) when you want to make a new iteration. So maybe extract them as a state_dict and restore them with load_state_dict.


I don’t think we should do that, if we do they would be used as non-leafs in the next iteration and won’t appear in the computation graph properly. It should be a chain of caused by the iterative use of wt.
Let me think about it…
I also thought we could have some special string like the word param in the field when we set it and thus we could loop through all the fields of the object that contain that special string.