I suspect this is due to either BatchNorm or Dropout. Although, I suspect BatchNorm is more likely. Maybe it takes a very long time to compute the running stats, or maybe the layers are changing so quickly.