yeah, that’s good advice @lewtun, I’m actually playing with this approach now.
The key challenge is to get good training data for the artificial “Other” class: few false positives, good balance of semantic (dis)similarity to the main class etc. If you have any hints for literature (for keywords to google), that would be very helpful. Sorting out irrelevant sentences/inputs seems to be a key issue for real world applications for transformers, but I didn’t find that much guidance on it online. (good topics seem to be “PU learning” or one-class classification)