I am trying to code a test case for multi task learning, using my own loss function. The idea is that the output layer is 3-dimensional,  the first output is used for 1D regression, the last two are used for 2-class classification. So, my combined loss function is a weighted sum of L1 loss and CELoss.
However, pytorch is complaining about my datatypes?
class my_loss:

    def __init__(self,weights):
      self.weights=weights

      #age loss:
      self.L1=nn.SmoothL1Loss(reduction='mean',beta=0.05)

      #gender loss:
      self.CE=nn.CrossEntropyLoss(reduction='mean')

    def __call__(self,output,target):
      loss = self.weights[0]*self.L1(output[:,0],target[:,0])+self.weights[1]*self.CE(output[:,1:3],target[:,1])

      return loss

#testing:
it=iter(dataloaders_dict['train'])
X,y=it.next()

criterion = my_loss(weights=(1,1))

out=model_ft(X)
loss=criterion(out,y)
print(y.dtype)
print(X.dtype)
print(out.dtype)
print(loss.dtype)

loss.backward()

This returns:
torch.int64
torch.float32
torch.float32
torch.float32
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-88-fc96d4e3072b> in <module>()
     31 print(loss.dtype)
     32 
---> 33 loss.backward()

1 frames
/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    145     Variable._execution_engine.run_backward(
    146         tensors, grad_tensors_, retain_graph, create_graph, inputs,
--> 147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
    148 
    149 

RuntimeError: Found dtype Long but expected Float

If only my targets are long, how come loss.backward is finding long’s, and how do I fix it?