I had the exactly same problem. The training loop just stuck after first step. Not loading new data. I could monitor that the memory was keeping increasing until leaking. I am using pytorch-xla 1.9.
@gabrielwong1991 Have you resolved this problem?