you’re suggesting that I append all frames of a clip from each iteration then compute them in a single batch for the clip level optimization.  At the same time, I’d be doing per-frame optimization for each iteration.
I think this would work, but it would be slow since we’re doing a forward pass twice for the same frames.