yes, it is not auto-moved, as it is not considered a part of the model, just because you’ve assigned it as an object attribute (without nn.Parameter wrapper). adam probably mitigates this error by having more robust code.