Managed to load and save optimizer and scheduler state with my original Pytorch XLA script!
This confirms some issue with accelerate
Let me know if you want to follow up sgugger i can send you both of my XLA and accelerate script! Thanks for your help!