I guess the issue happens between the dataParallel wrapper and the half precision. I have tried apex/amp and the same issue exists except the O1 optimization. The issue may be reproduced simply with a linear projection. The error’s position is not fixed, sometimes happens at the very first linear layer, I guess you can replicate it with simply a linear layer, > 1 GPUs and DataParallel wrapper.
model, optimizer = amp.initialize(model, optimizer,
                                  opt_level='O2'
                                  )
model = torch.nn.DataParallel(model, device_ids=[0,1,2])