Thank you for a valuable advice on saving the state of AdamW. But that is not the root problem here. I don’t run train after save/load and it’s performing way worse right after loading.
Davide Libenzi is trying hard to help me with the issue here https://github.com/pytorch/xla/issues/1245 18