Hi, thanks for replying. If that is possible (DataParallel on some part of model), then your suggestion is to calculate the scalar using a single GPU? I will go through the Imagenet example now to look into what is being done there.