I was wondering if gradients are scaled from fp16 to fp32  before all reducing as in Apex Distributed data parallel default flag allreduce_always_fp32 .
Also what is the equivalent of delay_allreduce in torch.nn.distributed.DistributedDataParallel ?