Wow. Thank you very much! Who would have thought? I just googled “Adam optimizer, Pytorch vs Tensorflow” and found this. Turns out I made the same mistake as well (a different application but I also need to set creat_graph=True). I guess it is kinda a universal problem that are easy to miss. If I understand Pytorch more thoroughly I would have known but there is no way I can catch this problem in a short period of time without reading your post. Well guess we all need to calculate high-order derivatives sometimes