What are the loss values you get for mean and sum reductions?
@ptrblck Does it make sense to use a higher learning rate incase of mean reduction since it could be the case where loss & gradients are small? or use a constant multiplier for loss to magnify it?