Hi there,
I’m currently training a classifier to classify news sentences in 20 classes of policy measures against COVID-19 like “Lockdown”, “Curfew” etc.  It will be part of a system that identifies new policy measure announcements in news articles. (An older development version of the model is here on the model hub)
The issue: In reality, 99% of sentences in news on covid have nothing to do with new policy announcements. So my task is (1) to identify the 1% of sentences which announce a new policy and (2) to classify these sentences. Step (2) works fine, but step (1) is quite difficult.
My current approach: train a classifier for 21 classes:  20 for the 20 policy types and 1 “Other/garbage” class. I’m creating the data for the “Other” class by extracting sentences from news which are semantically very different from the policy announcements. I get decent accuracy on the 20 policy types, but the big issue is the “Other” class, which either creates too many false positives or false negatives.
My question: What are best practices for eliminating “Other” sentences which are not relevant for the classification task? I feel like this must be a very common problem for real-world classification tasks (e.g. same issue for sentiment classifiers, which are only trained to classify as positive/negative - but in reality 99% sentences in e.g. news are neutral). But I couldn’t find literature on the issue.
Would be thankful for advice or hints for literature!
Best,
Moritz