Based on the code it seems you are using nn.CrossEntropyLoss and LongTensors for the mask and depth. While this approach seems to be fine for the mask target, I’m not sure it is right for the depth estimation, which seems to be a regression task.
I would recommend to try to keep depth as a FloatTensor and use nn.MSELoss for the loss_depth calculation.