SOTA applications typically finetune full models on their downstream task.

We’re in agreement here.

Averaging token embeddings is as simple as torch.mean if that is what you want

It’s not so simple when taking into account attention masks. That’s why I was hoping the HF library could provide pooling functionality appropriate for every model.

otherwise use the sentence-transformers library or similar which provides finetuned models to get better sentence representations.

Again, I’m not interested in pretrained parameters for a classifier. I just want to generate sentence embeddings so I can fine tune the parameters on my own downstream task. My observations are that:

BERT has a pooled_output.
XLNet does not have a pooled_output but instead uses SequenceSummarizer.
sgugger says that SequenceSummarizer will be removed in the future, and there is no plan to have XLNet provide its own pooled_output.
Folks like me doing NLU need to produce a sentence embedding so we can fine-tune a downstream classifier.