Make sure you have a source install of Accelerate as there was a bug in the optimizer state reloading fixed last week. Then can you try to load the state dictionary on the CPU? That should avoid the OOM:
optimizer.load_state_dict(torch.load("optimizer.pth.tar", map_location="cpu"))

You should load the state in all processes as there is nothing that will synchronize them otherwise.
We also have utilities to make checkpointing easier on the roadmap, so it will soon get easier hopefully!