You’ll need need to hack it by updating parameters for each evaluation point and then restoring to original values. It’s kind of inherent to structure of torch.nn – since parameters are a state you need to modify state. Parameter=state structure makes it easier for a common cases like SGD, but awkward when you need to try several parameter values per step (ie, line-search or bayesian optimization)