I don’t think it’s a tuning issue. The trpo update just won’t be correct until you’re able to take the gradient of the gradients in g and k. Without create_graph=True, g[p].grad_fn and k[p].grad_fn will just return None instead of keeping track of the gradient with respect to the gradient.