IMHO, adding trainable parameters to the loss function makes it part of the network to be trained. We need to think out of the box a bit here. So what I reckon that you could do is to wrap the criterion into your network. This is going to require a bit of change to how people usually write foward() method. Here is an example.
def forward(self, x, y=None):
    # Regular forward pass
    output = self.model(x)
    # Insert your criterion here
    if self.training:
        assert y is not None, "Target should be passed during training."
        loss = self.criterion(output, y)
        return loss
    return output


Basically, the code snippet above includes the computation of loss as part of model’s forward pass. And because your criterion is part of your network, you don’t have to explicitly add its parameters to the optimiser anymore. And torch.nn.parallel.DistributedDataParallel will make sure even the parameters of the criterion are synced across GPUs during forward pass.
Hope this helps.