Have you looked at sentence-transformers? That’s what they do, outputting a single embedding for a single sentence. You can then use that with your own downstream classifier. They do not provide many different models though, so that can be an issue for you. But again, this shouldn’t be so hard to implement yourself.
I am not sure what you mean with “taking into account attention masks”, why do you need those?