Hi,
I am not very familiar with this kind of tasks and why they might not converge. But GitHub - facebookresearch/higher: higher is a pytorch library allowing users to obtain higher order gradients over losses spanning training loops rather than individual training steps. 4 seems very close to what you’re trying to do no?