@apaszke  I tried your patch with mvapich-2.2.2-cuda8.0, the time per iteration for resnet50-batchsize32 is reduced from 0.45s to 0.4s(on 8 GPUs). Unfortunately the processes stuck when I  try to use more than one node, need further debugging. Anyway it brings some improvement!