```input_embeds``` keyword not working properly (GPT2)
```transfomer-cli env``` command was not recognized so I do not know which environment I have.

- `transformers` version: latest
- Platform: windows 10
- Python version: 3.8.5
- PyTorch version (GPU?): CPU
- Tensorflow version (GPU?): None
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten, @LysandreJik

## Information

Model I am using (GPT2):

When I want to provide my own embeded representation instead of input indices to a GPT2 that I want to train from scratch, I get the following error:
```Python
Traceback (most recent call last):

  File "<ipython-input-18-d0df910b9d57>", line 10, in <module>
    model(inputs_embeds=inputs_embeds)  # runs without error

  File "C:\Users\cnelias\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)

TypeError: forward() got an unexpected keyword argument 'inputs_embeds'
```
It does not come from my implementation since the following code also throws the same error:
```
from transformers import GPT2Model, GPT2Tokenizer
import torch

model = GPT2Model.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2', pad_token='<PAD>')

input_ids = tokenizer.encode("Hello, how are you?", return_tensors='pt')
inputs_embeds = model.wte(input_ids)

model(inputs_embeds=inputs_embeds) 
```

I installed pytorch via conda, this could be the cause.

Hi,

I cannot reproduce the error. What versions of `PyTorch` and `transformers` are you using?
Do you mind installing everything through pip again and checking if the problem still persists?<|||||>I can't install pytorch through pip, I don't have a cuda compatible GPU and get the following error ```ERROR: Could not find a version that satisfies the requirement torchaudio (from versions: none)
ERROR: No matching distribution found for torchaudio```. Apparently, this is because torchaudio does not support windows yet.

The versions I am using:
- ```torch``` : 1.9.1
-  ```transformers``` : 2.1.1<|||||>The problem is the outdated `transformers` version, `GPT2Model.forward` in `v2.1.1` does not support `input_embeds` argument. Please upgrade `transformers` to the latest (4.11.3) and see if the problem is solved.<|||||>Oh. The doc on ```GPT2Model``` still mentions ```input_embeds```:

class transformers.GPT2Model
```Python
forward(input_ids=None, past_key_values=None, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, encoder_hidden_states=None, encoder_attention_mask=None, use_cache=None, output_attentions=None, output_hidden_states=None, return_dict=None)
```
- inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) –

  Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.

Thanks for pointing this out, I will update ```transformers``` and try it out.
<|||||>I think you probably checked a wrong docs version, the GPT2 in [v2.1.1 docs](https://huggingface.co/transformers/v2.1.1/model_doc/gpt2.html) was still under construction at that moment.<|||||>This is from the [v4.11.3](https://huggingface.co/transformers/model_doc/gpt2.html).
Also, do you know why running ```conda update transformers``` also installs the 2.1.1 version?  Same thing with pip ... <|||||>Sorry I'm not using conda to manage my packages. Would you mind creating a new conda virtual environment and then using `pip` to install all required packages?

Note that when you install PyTorch related packages, please add this arg: `-f https://download.pytorch.org/whl/torch_stable.html` as not all PyTorch packages seem to exist on the official pypi repository.

E.g.
```
pip install torch torchvision torchaudio -f https://download.pytorch.org/whl/torch_stable.html
```

<|||||>This does not work, I get this error 
```
ERROR: Could not find a version that satisfies the requirement torchaudio (from versions: none)
ERROR: No matching distribution found for torchaudio
```
This issue has been reported by other users on the torchaudio github page, and it seems to be due to the fact that windows in not supported yet.<|||||>I'm sorry to hear that. I can install all three packages on windows without any error, therefore, I'm pretty sure that `torchaudio` is supported. 

Maybe this is an environment problem on your system, you can try directly downloading whl packages from the index https://download.pytorch.org/whl/torch_stable.html and then install them manually. <|||||>I'll try that. I do have one last question: you mention the fact that in the latest version, ```input_embeds``` is not used anymore. Yet, when I look at the latest [doc](https://huggingface.co/transformers/model_doc/gpt2.html), I see that it is still an argument from the ```forward``` method. If the argument doesnt exist anymore, what is the current name?<|||||>> I think you probably checked a wrong docs version, the GPT2 in [v2.1.1 docs](https://huggingface.co/transformers/v2.1.1/model_doc/gpt2.html) was still under construction at that moment.

You might misunderstand my point.  Here I mentioned above was **not** the latest docs, it was the version `v2.1.1` you were previously using. At that moment (v2.1.1), the docs was still under construction which didn't indicate whether `input_embeds` is usable or not.

So I checked out the codebase, the conclusion is:
The `input_embeds` argument is **not supported** in `v2.1.1` but is **supported** in the latest version `v4.x`.

Hope this clarification is helpful.

<|||||>Understood, thanks for the clarification.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.