Ok had an idea and modified one more thing – switched to using tfp.math.minimize instead of manual optimization. This has some niceties like automatically using a tf.while_loop and tf.function compiling things. It returns losses by default, but I wrote a custom trace_fn to trace all quantities (vars, grads, etc) just to demonstrate how that could be used. You can write whatever you want in trace_fn and it’ll preserve those values and return them, per-step, at the end of optimization.