Thanks for your response @ptrblck; I re-encoded on the fly using a transform (see below).
In addition to what you mentioned (having irrelevant classes), my initial worry was that the class loss would be somehow “diluted” if I would set the number of output activations to a higher value than the actual number of classes.
The transform
class EncodeClasses(object):
    """Re-encode classes

    Useful to limit the number of output activations to the actual number of
    classes when initial class codes do not follow a simple sequence ``[1:len(classes)]``
    """
    def __init__(self, encoding):
        """
        Args:
            encoding (dict): Dict of old code, new code mapping
        """
        self.encoding = encoding

    def __call__(self, image, target):
        labels = target['labels']
        for k,v in self.encoding.items():
            labels[labels == k] = v
        target['labels'] = labels
        return image, target