Getting IndexError: index out of range in self while finetuning GPTNeo on Text Classification
## Environment info]
- `transformers` version: current master branch
- Platform: Colab
- Python version: 3.7.10
- PyTorch version (GPU?):  1.8.1+cu101 
- Tensorflow version (GPU?): Not using
- Using GPU in script?: With GPU also has issue 
- Using distributed or parallel set-up in script?: yes I am using trainer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below) mention in below colab

The tasks I am working on is:
* [x] an official GLUE/SQUaD task:  emotion
* [ ] my own task or dataset: (give details below) 

## To reproduce

Steps to reproduce the behavior:
Run this [colab](https://github.com/bhadreshpsavani/ExploringSentimentalAnalysis/blob/main/SentimentalAnalysisWithGPTNeo.ipynb)

## Error:
```
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-13-5845bc74dd04> in <module>()
      5                   train_dataset=emotions_encoded["train"],
      6                   eval_dataset=emotions_encoded["validation"])
----> 7 trainer.train();

/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, **kwargs)
   1261                         tr_loss += self.training_step(model, inputs)
   1262                 else:
-> 1263                     tr_loss += self.training_step(model, inputs)
   1264                 self.current_flos += float(self.floating_point_ops(inputs))
   1265 

/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in training_step(self, model, inputs)
   1739                 loss = self.compute_loss(model, inputs)
   1740         else:
-> 1741             loss = self.compute_loss(model, inputs)
   1742 
   1743         if self.args.n_gpu > 1:

/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in compute_loss(self, model, inputs, return_outputs)
   1771         else:
   1772             labels = None
-> 1773         outputs = model(**inputs)
   1774         # Save past state if it exists
   1775         # TODO: this needs to be fixed and made cleaner later.

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/transformers/models/gpt_neo/modeling_gpt_neo.py in forward(self, input_ids, past_key_values, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict)
   1098             output_attentions=output_attentions,
   1099             output_hidden_states=output_hidden_states,
-> 1100             return_dict=return_dict,
   1101         )
   1102         hidden_states = transformer_outputs[0]

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/transformers/models/gpt_neo/modeling_gpt_neo.py in forward(self, input_ids, past_key_values, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)
    811 
    812         if inputs_embeds is None:
--> 813             inputs_embeds = self.wte(input_ids)
    814         position_embeds = self.wpe(position_ids)
    815         hidden_states = inputs_embeds + position_embeds

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/sparse.py in forward(self, input)
    156         return F.embedding(
    157             input, self.weight, self.padding_idx, self.max_norm,
--> 158             self.norm_type, self.scale_grad_by_freq, self.sparse)
    159 
    160     def extra_repr(self) -> str:

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1914         # remove once script supports set_grad_enabled
   1915         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1916     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1917 
   1918 

IndexError: index out of range in self
```

Note: Issue comes while using it with CPU. In case of GPU also Issue is coming but different error

### Who can help
@patil-suraj  @sgugger
This is because you are using a pretrained model but add new tokens without changing the weights. There is thus a problem when you pass that new token as input ID, it doesn't have a matching embedding. You can resize the embedding matrix but you will then lose all the weights of that embedding matrix, so you won't be properly applying transfer learning.<|||||>Thanks, @sgugger,
If I train the model with 1 batch size and no new padding token it works as you said that the exact reason.
So to add any new token in the tokenizer I need to train the model again for language modeling?<|||||>Not necessarily for language modeling, but you will need more training since all the embeddings will be randomly initialized. The only way I see around it is to manually create the embeddings weights to use the pretrained weights for all the tokens except the padding token and then put a line of 0s for the padding token. That should work. <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.