I’m currently implementing pseudo labeling, where I create the labels for the unlabeled part of the datset by simply running the samples trough the model and using the prediction as ground truth. I’m only using the prediction for a sample as ground truth, however, if its confidence surpasses a given threshold.
To implement this, I tried using two approaches:
conf, pseudo_label = F.softmax(out, dim=1).max(axis=1)
mask = conf > threshold

# Option 1
loss = F.cross_entropy(out[mask], pseudo_label[mask])

# Option 2
loss = (F.cross_entropy(out, pseudo_label, reduction='none') * mask).mean()

Which of them is preferrable? Option 1 produces nan values when the mask is False for every item, but option 2 is way more computationally expensive since it always runs the whole dataset trough the loss function.