It sounds like you are describing a multilabel objective. One possible loss function for something like this would be multilabel softmarginloss: MultiLabelSoftMarginLoss — PyTorch 1.9.0 documentation 1