Yes, I have followed your instruction to reinstall Apex. My problem is strange. I printed the abnormal value  during the distributed training:
First, my model has various scales of feature map predicted （cascaded CNN and in each stage/stack has 5 scale output)
This is normal for some batches, and the element-wise output should in the range of [0,1] (gaussian heatmap regression)
heatmap L2 loss per stack.........   [ 69848.01   59730.246 223546.12   60869.35 ]
 heatmap L2 loss per stack.........   [15058.608 13271.5   13770.041 13684.25 ]
 heatmap L2 loss per stack.........   [3515.2559 3062.7026 2899.563  2879.3105]
 heatmap L2 loss per stack.........   [ 84485.94  76283.11 219553.47  77723.48]
 heatmap L2 loss per stack.........   [ 70769.086  63346.633 209632.16   64268.496]
 heatmap L2 loss per stack.........   [18312.457 17451.66  17986.875 17935.975]

However, the loss is abnormal suddenly , and the elements of the output become very large such as 223, and then grow rapidly and resulting in gradient explosion.
Dangerous! Check pred, gt, mask_miss: ======>  tensor(223.1250, device='cuda:2', dtype=torch.float16, grad_fn=<MaxBackward1>) tensor(1., device='cuda:2') tensor(1., device='cuda:2')
 heatmap L2 loss per stack.........   [0. 0. 0. 0.]
Dangerous! Check pred, gt, mask_miss: ======>  tensor(223.1250, device='cuda:1', dtype=torch.float16, grad_fn=<MaxBackward1>) tensor(1., device='cuda:1') tensor(1., device='cuda:1')
Dangerous! Check pred, gt, mask_miss: ======>  tensor(222.7500, device='cuda:3', dtype=torch.float16, grad_fn=<MaxBackward1>) tensor(1., device='cuda:3') tensor(1., device='cuda:3')
Dangerous! Check pred, gt, mask_miss: ======>  tensor(222.7500, device='cuda:0', dtype=torch.float16, grad_fn=<MaxBackward1>) tensor(1., device='cuda:0') tensor(1., device='cuda:0')