I see. Yeah, I saw your reply elsewhere: the “edge” case when having weight decay, if we append all parameters into the optimizer. But not sure if certain parameters are not included in the optimizer at all, would the “edge” case still happen. Thanks