Thank you very much for your insight @ptrblck.
Yes, you are exactly right, I am able to predict the lanes but not of each lane separately.
A post processing step should be OK for my benchmark evaluation. I did not think about that! I think I will do that first.
However I would still like to use the multi-label approach as well to see if there are any differences in performance. I have actually tried passing in a [16, 5, 368, 720] tensor into my model but received an error.
RuntimeError: 1only batches of spatial targets supported (3D tensors) but got targets of size: : [16, 5, 368, 640]
I believe I have to change my loss criterion?
class MixSoftmaxCrossEntropyLoss(nn.CrossEntropyLoss):
    def __init__(self, aux=True, aux_weight=0.2, ignore_index=-1, **kwargs):
        super(MixSoftmaxCrossEntropyLoss, self).__init__(ignore_index=ignore_index)
        self.aux = aux
        self.aux_weight = aux_weight

    def _aux_forward(self, *inputs, **kwargs):
        *preds, target = tuple(inputs)

        loss = super(MixSoftmaxCrossEntropyLoss, self).forward(preds[0], target)
        for i in range(1, len(preds)):
            aux_loss = super(MixSoftmaxCrossEntropyLoss, self).forward(preds[i], target)
            loss += self.aux_weight * aux_loss
        return loss

    def _multiple_forward(self, *inputs):
        *preds, target = tuple(inputs)
        loss = super(MixSoftmaxCrossEntropyLoss, self).forward(preds[0], target)
        for i in range(1, len(preds)):
            loss += super(MixSoftmaxCrossEntropyLoss, self).forward(preds[i], target)
        return loss

    def forward(self, *inputs, **kwargs):
        preds, target = tuple(inputs)
        inputs = tuple(list(preds) + [target])
        if self.aux:
            return dict(loss=self._aux_forward(*inputs))
        elif len(preds) > 1:
            return dict(loss=self._multiple_forward(*inputs))
        else:
            return dict(loss=super(MixSoftmaxCrossEntropyLoss, self).forward(*inputs))

The function that got called for the error is loss = super(MixSoftmaxCrossEntropyLoss, self).forward(preds[0], target[0]) as above.
I hope I have given enough information such that it is not confusing.
Thank you again for your help, I really appreciate it.