To update, I had added retain_graph=True to all backwards except the final one (which should never need to retrain the graph. This is the backward used in the gradient penalty after the torch.autograd.grad. Notably, this is also the backward where the error occurs. However, adding retain_graph=True to this backward fixes the problem. This seems strange to me, since it means the same backward which is trying to use the freed graph is also the backward which is doing the undesired freeing. Does anyone know why this might be happening? Again, it is only occurring when using InstanceNorm to replace BatchNorm. This suggests to me that InstanceNorm is doing something strange when used in a second order gradient.