I found when I run the training with batchsize 1024 on 8 1080 ti gpu cards,  it will report the error. When I run the code with batchsize 512 on  8 gpu cards, the error disappeared.  when I run the code with smaller batchsize on a single gpu card, the error can’t be reproduced. So it might be hard to reproduce the error on cpu.