shivammehta007:

But my model is on multiple GPUs
So when my loss is output from the model how do I scale it?


You mean your gradients are on multiple GPUs, so that you cannot directly pass it to the same operator? @ptrblck @albanD does this mean the gradients need to be move to the same device first, and then calculate the scaling ratio? Or is the recommended way to do per device/model-shard scaling?