Word Embedding input to GPT-2
# ❓ Questions & Help


<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->
Is there any way in which we can give custom word embeddings as input to GPT-2 instead of tokenized words?


You can pass embeddings to the model by using the `input_embeds` keyword argument:
```
result = model(input_embeds=...)
```
They should be if shape `batch_size, sequence_length, hidden_size)`.<|||||>Hi @sgugger 
I tried the following code snippet but it is giving an error
<pre>
    from transformers import GPT2Tokenizer, GPT2LMHeadModel
    modelGPT = GPT2LMHeadModel.from_pretrained('gpt2')
    modelGPT(input_embeds=out[2][-1]) #where out[2][-1] is of shape [64, 60, 768] --> [batch_size, sequence_length, hidden_size]
</pre>

Error:
<pre>
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-81-40a402b51d0d> in <module>()
----> 3 modelGPT(input_embeds=out[2][-1])

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

TypeError: forward() got an unexpected keyword argument 'input_embeds'
</pre><|||||>Sorry I made a typo, I meant `inputs_embeds` (with an s).<|||||>Worked like a charm! Thanks