using TFOpenAIGPTLMHeadModel load pytorch model doesn't work well
## Environment info
- `transformers` version: 4.9.2
- Platform: Darwin-20.5.0-x86_64-i386-64bit
- Python version: 3.7.11
- PyTorch version (GPU?): 1.9.0 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
@patrickvonplaten, @LysandreJik

## Information
Model I am using (OpenAIGPT):
I download file from [CDial-GPT_LCCC-large](https://huggingface.co/thu-coai/CDial-GPT_LCCC-large/tree/main), and loaded as follow way:
```
tokenizer = BertTokenizer.from_pretrained(……,do_lower_case=True)
model_pt = OpenAIGPTLMHeadModel.from_pretrained(……)
model_tf = TFOpenAIGPTLMHeadModel.from_pretrained(……,from_pt=True)
```
it's fine when loading `OpenAIGPTLMHeadModel`, but it encountered some problems as loading `TFOpenAIGPTLMHeadModel`：
```
Some weights of the PyTorch model were not used when initializing the TF 2.0 model TFOpenAIGPTLMHeadModel: ['transformer.h.2.attn.bias', 'transformer.h.5.attn.bias', 'transformer.h.9.attn.bias', 'transformer.h.6.attn.bias', 'transformer.h.0.attn.bias', 'lm_head.weight', 'transformer.h.8.attn.bias', 'transformer.h.3.attn.bias', 'transformer.h.1.attn.bias', 'transformer.h.10.attn.bias', 'transformer.h.4.attn.bias', 'transformer.h.11.attn.bias', 'transformer.h.7.attn.bias']
- This IS expected if you are initializing TFOpenAIGPTLMHeadModel from a PyTorch model trained on another task or with another architecture (e.g. initializing a TFBertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFOpenAIGPTLMHeadModel from a PyTorch model that you expect to be exactly identical (e.g. initializing a TFBertForSequenceClassification model from a BertForSequenceClassification model).
All the weights of TFOpenAIGPTLMHeadModel were initialized from the PyTorch model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFOpenAIGPTLMHeadModel for predictions without further training.
```
their logits are quite different, how can i solve this？
Hello! Do you have a reproducible code example that exposes the differences between the logits? Thank you<|||||>> > Hello! Do you have a reproducible code example that exposes the differences between the logits? Thank you

@LysandreJik this is my full code refered to [CDial-GPT/interact.py](https://github.com/thu-coai/CDial-GPT/blob/master/interact.py):
```
texts = ["别爱我没结果","没结果的爱是多么的痛","但是爱过就可以了"]
SPECIAL_TOKENS = ["[CLS]", "[SEP]", "[PAD]", "[speaker1]", "[speaker2]"]
bos, eos, pad, speaker1, speaker2 = tokenizer.convert_tokens_to_ids(SPECIAL_TOKENS)
texts_ids=[tokenizer.convert_tokens_to_ids(tokenizer.tokenize(text)) for text in texts]
sequence=[[speaker2 if i % 2 else speaker1] + s for i, s in enumerate(texts_ids)]
input_ids=[bos]
for i in sequence:
    input_ids+=i
token_type_ids=[bos] + [speaker2 if i % 2 else speaker1 for i, s in enumerate(sequence) for _ in s]    
special_tokens_ids = tokenizer.convert_tokens_to_ids(SPECIAL_TOKENS)
max_length=40

#pytorch
input_ids_pt = torch.tensor(input_ids+[speaker2], dtype=torch.long).unsqueeze(0)
token_type_ids_pt = torch.tensor(token_type_ids+[speaker2], dtype=torch.long).unsqueeze(0)
response_ids_pt=[]
for _ in range(max_length):
    outputs_pt = model_pt(input_ids_pt, token_type_ids=token_type_ids_pt)
    logits_pt = outputs_pt.logits
    next_token_logits_pt = logits_pt[0, -1, :]
    next_token_pt = torch.multinomial(F.softmax(next_token_logits_pt, dim=-1), num_samples=1)
    if next_token_pt.item() in special_tokens_ids:
        break
    response_ids_pt.append(next_token_pt.item())
    input_ids_pt = torch.cat((input_ids_pt, next_token_pt.unsqueeze(0)), dim=1)
    token_type_ids_pt = torch.cat((token_type_ids_pt,torch.tensor([[speaker2]])),dim=1) 
response_pt=tokenizer.decode(response_ids_pt, skip_special_tokens=True)
    
#tensorflow
input_ids_tf = tf.expand_dims(tf.constant(input_ids+[speaker2]),axis=0)
token_type_ids_tf = tf.expand_dims(tf.constant(token_type_ids+[speaker2]),axis=0)
response_ids_tf=[]
for _ in range(max_length):
    outputs_tf = model_tf(input_ids_tf, token_type_ids=token_type_ids_tf)
    logits_tf = outputs_tf.logits
    next_token_logits_tf = tf.expand_dims(logits_tf[0, -1, :], axis=0)
    next_token_tf = tf.random.categorical(tf.nn.softmax(next_token_logits_tf,axis=-1), num_samples=1)
    next_token_tf = tf.cast(next_token_tf, dtype=tf.int32)
    if next_token_tf in special_tokens_ids:
         break
    response_ids_tf.append(next_token_tf.numpy().item())
    input_ids_tf = tf.concat((input_ids_tf, next_token_tf), axis=1)
    token_type_ids_tf=tf.concat((token_type_ids_tf,tf.constant(speaker2,shape=(1,1))),axis=1)  
response_tf=tokenizer.decode(response_ids_tf, skip_special_tokens=True)
```
when i generated `response_ids_pt`, the loop can meeted `special_tokens_ids` and broken, the `response_ids_tf` generative loop can't broken untill `max_length`.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.