How to input word2vec embeddings to gpt2 model?
Hi,

I am working on the huggingface gpt2 model. I have a word2vec model trained on a dataset (dimensions similar to gpt2, 768). Now I want to input these embeddings to gpt2. I understand I must use inputs_embeds to input the embeddings but I am little unclear about how exactly to do it. Any source or help would be appreciated.
Hi @tejaravi675 👋  Yes, you can pass embeddings to the model as you described (through `inputs_embeds`), but the embeddings will be unknown to the model, as it was not trained with them. In essence, you will have to build a script to finetune the model using your embeddings (for GPT-2, with the causal language modeling task). You can see a few examples [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling) (pytorch) and [here](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling) (tensorflow).

As per our [issues guidelines](https://github.com/huggingface/transformers/blob/main/ISSUES.md), we reserve GitHub issues for bugs in the repository and/or feature requests. For any other requests, we'd like to invite you to use our [forum](https://discuss.huggingface.co/) 🤗  I'm closing this issue, but feel free to reopen with queries that fit the criteria I described.<|||||>Hi, I understand that I have to finetune the model with my embeddings. My problem is I am unable to understand how exactly to code (Not very handy with complex coding). So I wanted to know if there were any examples which I can refer to to understand the finetuning part and understand how to add my word embeddings to gpt2 model.<|||||>The examples I linked above are the closest we have to the task you are describing, but they require some modification to run your use case :) Sadly, we don't have the capacity to further help you with your task -- try in the forums, maybe some other user tried to do a similar thing.