Hey @Erica_Zheng, are you using DataParallel, DistributedDataParallel or torch.distributed? And can you include a min repro? Looks like the above example does not include the forward() function or how the model forward pass was launched?