There is no common way to expose optimizer state AFAIK. If you know how you can access the state of your optimizer then you’ll be able to synchronize it by using torch.distributed collectives directly, e.g. broadcast.