Thanks I will try it.
Actually I have another question about v1.1.0 DDP.
I tried to inference the model with syncbatchnorm layer ( Actually, it becomes batchnorm layer after load from checkpoint ). The results turned to be different between:

Only turn on evaluate mode.

model.eval()
# inference...


Manually set track_running_stats of each BN layer to False after model.eval().

model.eval()
set_BN_track_running_stats(model, False)
# do inference..

It is strange that the second one is much better than first one on early epochs. Is this also a version problem?
P.S. However, after more epochs training, results of two inference method are similiar but still has small differences.
Below is sample code of set_BN_track_running_stats():
def set_BN_track_running_stats(module, ifTrack=True):
    if isinstance(module, torch.nn.modules.batchnorm._BatchNorm):
        module.track_running_stats = ifTrack
    for child in module.children():
        set_BN_track_running_stats(module = child, ifTrack=ifTrack)