I’m not sure how your new optimizer is defined, but e.g. LBFGS (which also requires a closure) can be memory intensive as stated in the docs:

This is a very memory intensive optimizer (it requires additional  param_bytes * (history_size + 1)  bytes). If it doesn’t fit in memory try reducing the history size, or use a different algorithm.


If your optimizer’s step method expects the output and loss as a return value, then the closure looks correct. You could compare it to this example 2.
Might be and it depends, what optimizer.step is doing with the closure. If you are storing the history of the losses and outputs, then an increased memory usage is expected.