Ah framing the problem as an “anomaly detection” task is a nice idea! (The idea being that the 1% of the sentences you care about are “anomalous” in some sense).
At least for time series, one interesting paper I saw last year was based on using the “concentration phenomenon” to detect anomalies in high-dimensional datasets: http://proceedings.mlr.press/v119/sipple20a/sipple20a.pdf 1
I wonder whether this idea could be extended to the embedding space of your transformer models? The paper also refers to previous work on anomaly detectors, including one-class SVMs that might also be applicable to your use case.
Another idea off the top of my head is whether you could use the Local Outlier Factor 2 method to the embedding space (or some dimensional projection theoreof)?