A weight parameter could be seen as an internal state and would yield a device mismatch error.
Of course you might define the weight parameter as a CUDATensor, but you could also move the criterion to the device:
output = torch.randn(10, 10, requires_grad=True, device='cuda')
target = torch.randint(0, 10, (10,), device='cuda')

weight = torch.empty(10).uniform_(0, 1)
criterion = nn.CrossEntropyLoss(weight=weight)

loss = criterion(output, target) # error
> RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'weight' in call to _thnn_nll_loss_forward

criterion.cuda()
loss = criterion(output, target) # works