I get the same behavior after upgrading to Pytorch 1.8.0. I’ve also observed AMP having a higher memory cost in non meta-learning settings but I think that was due to not setting the entire forward pass in autocast. Could that be the case here somehow?
It’s disappointing because meta-learning is expensive in memory and AMP would help a lot of folks in the field. Although I suspect that even after this memory issue is sorted, the (outer) gradient calculation will still be inaccurate in this case, because the graph is “chaotic” and gradient degradation is often an issue even in FP32.