Is it possible to get document embeddings using GPT-2? If so, how?
# ❓ Questions & Help

I posted a question here on SO: https://stackoverflow.com/questions/61641257/how-to-get-document-embeddings-using-gpt-2?noredirect=1#comment109035633_61641257

But unfortunately was informed that "Please notice that SO is not a tutorial service, and also that recommendation requests for external resources are explicitly off-topic."

## Details
I'm curious if using GPT-2 might yield a higher accuracy for document vectors (with greatly varying length) or not (would it surpass the state of the art?)

Really I'm most interested in document embeddings that are as accurate as possible. I'm wondering if using GPT-2 will get results that are more accurate than Paragraph Vectors for example.

I heard that in order to get vectors from GPT-2 "you can use a weighted sum and/or concatenation of vector outputs at its hidden layers (typically the last few hidden layers) as a representation of its corresponding words or even "meaning" of the entire text, although for this role BERT is used more often as it is bi-directional and takes into account of both forward and backward contexts."

As a machine learning and NLP beginner, I'd love to know how to go about this, or to be pointed in the right direction to learn more about how to attempt this in Python.

I've tried fine-tuning GPT-2 before but I have no idea how to extract vectors from it for text.

**A link to original question on Stack Overflow**: https://stackoverflow.com/questions/61641257/how-to-get-document-embeddings-using-gpt-2?noredirect=1#comment109035633_61641257

Not from Huggingface, but for this, I would use something like SentenceBERT. As far as I know.<|||||>@moinnadeem Thank you for pointing me in that direction! But would sentencebert really work for documents?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
