You could skip it by adding your loss directly into your model and calculate the losses on each replica.

ohhh, I see now. This is basically because the dataset (labels) sit on the first device. It’s not a design choice but rather a consequence of where I put the loss function (or rather the inputs for the loss function in terms of the labels). Makes so much sense now. Thanks!