How to combine multiple criterions to a loss function?
<SCODE>def loss_calc(data,targets):
  data = Variable(torch.FloatTensor(data)).cuda()
  targets = Variable(torch.LongTensor(targets)).cuda()
  output= model(data)
  final = output[-1,:,:]
  loss = criterion(final,targets)
  return loss
<ECODE> <SCODE>def loss_calc(data,targets):
  data = Variable(torch.FloatTensor(data)).cuda()
  targets = Variable(torch.LongTensor(targets)).cuda()
  output= model(data)
  final = output[-1,:,:]
  loss = 0.0
  for b in range(batch_size):
    loss += criterion(final[b], targets[b])
  return loss
<ECODE> I doesn;t return errors, but the network simply won;t train. This alternative results an error <SCODE>def loss_calc(data,targets):
  data = Variable(torch.FloatTensor(data)).cuda()
  targets = Variable(torch.LongTensor(targets)).cuda()   
  output= model(data)
  final = output[-1,:,:]
  loss = []
  for b in range(batch_size):
    loss.append(criterion(final[b], targets[b]))
  loss = torch.sum(loss)
  return loss
<ECODE> Note, this is a dummy example. If I understand how to fix this, I can apply that to the recursive neural nets. For the final project, the sequence of losses have arbitrary length. For example, sometimes it’s adding 4 losses, other times 6. So it’s not an option to pre-allocate a Tensor and save the losses to the Tensor
robromijnders
apaszke
Thank you. That helped a lot.
robromijnders
@apaszke For eg. I have a network with two outputs and corresponding losses - crossentropy2d and crossentropy. I tried the following but it throws an error about incompatible types. Is there a way around? <SCODE>b = cross_entropy2d(output_x, x_labels)
a = nn.CrossEntropyLoss(output_y, y_labels)
loss = a + b
<ECODE>
meetshah1995
jekbradbury
@jekbradbury Thanks a lot.
meetshah1995
<SCODE>b = nn.MSELoss(output_x, x_labels)
a = nn.CrossEntropyLoss(output_y, y_labels)
loss = a + b<ECODE>
Bixqu
Doing that is fine, it would be: <SCODE>b = nn.MSELoss()(output_x, x_labels)
a = nn.CrossEntropyLoss()(output_y, y_labels)
loss = a + b

loss.backward()
<ECODE> Note the additional parentheses, as James mentioned above. This is equivalent to: <SCODE>b = nn.MSELoss()
a = nn.CrossEntropyLoss()

loss_a = a(output_x, x_labels)
loss_b = b(output_y, y_labels)

loss = loss_a + loss_b

loss.backward()<ECODE>
Jordan_Campbell
which way should I take? Thanks.
D-X-Y
both give you same result. I’d say (1) is simpler.
smth
I did this but got the following error : add received an invalid combination of arguments - got (torch.FloatTensor), but expected one of: (float value)
didn’t match because some of the arguments have invalid types: (torch.FloatTensor) (torch.cuda.FloatTensor other)
didn’t match because some of the arguments have invalid types: (torch.FloatTensor) (torch.cuda.sparse.FloatTensor other)
didn’t match because some of the arguments have invalid types: (torch.FloatTensor) (float value, torch.cuda.FloatTensor other) (float value, torch.cuda.sparse.FloatTensor other) Both my loss are of the same type - nn.CrossEntropyLoss() <class ‘torch.autograd.variable.Variable’> Any suggestion on how to resolve this please?
BikashgG
If you stare down the error message, you see that you have one cuda and one not. Best regards Thomas
tom
BikashgG
Is it possible to weight the losses when multiple losses are used and how? Is it correct by using the following code? <SCODE>    mse_loss = nn.MSELoss(size_average=True)
    a = weight1 * mse_loss(inp, target1)
    b = weight2 * mse_loss(inp, target2)
    loss = a + b

    loss.backward()
<ECODE>
platero
Yup, you can certainly weigh the losses however you see fit. What you’re doing in your example should be fine.
achaiah
what if my second loss function requires some computed value from first loss (or even the grad of first loss?) in that case I can’t add two loss together; they must be gradients respectively; and retain_graph=True gives wrong results as well as the intermediate grads not correct
ElleryL
thank you very much Jordan! your answer help me to combine my custom loss function with nn.Loss.
PeterXiaoGuo
Hi BikashgG, I think you could use .type() to change tensor’s type from torch.FloatTensor to torch.LongTensor. I face a similar problem when I use this CrossEntropyLoss(). Check the official document on this loss function, there is a requirement on the type of feed-in tensor. Hope it helps, Peter
PeterXiaoGuo
What if the losses are computed over different parts of the network, say loss1 is for first 3 layers and loss2 is for first 7 layers (incl. the first 3)? Wouldn’t the sum of losses method also backprop loss1 through layers 4-7? Would calling loss1.backward() and loss2.backward() separately be recommended in that case?
tarun
Hope it helps!
sid87