That’s a tricky question, as your training and testing routine are different, i.e. during training your model architecture was changing based on the target.
How did you implement the validation step or did you just train the model?
Once approach would be to use a small hold-out set to train another classifier on top of your current model to use the 4 predictions and output only the two classes.