Inconsistent loss computation?
`transformers` version: 4.3.2, Python version: 3.7, Tensorflow version (GPU?): 2.3.1, Using GPU in script?: No, Using distributed or parallel set-up in script?: No

## To reproduce

Steps to reproduce the behavior:

``` python
import tensorflow as tf
from transformers import TFGPT2LMHeadModel
from transformers.modeling_tf_utils import TFCausalLanguageModelingLoss

import copy 
import numpy as np

model = TFGPT2LMHeadModel.from_pretrained('gpt2')


one_line_dset = (np.random.rand(1, 1024)>.5)*1
input_ids = one_line_dset
target_ids = one_line_dset

# explicit loss calculation
prediction = model.predict(input_ids).logits
prediction = tf.convert_to_tensor(prediction)
l = TFCausalLanguageModelingLoss().compute_loss(target_ids, prediction)

# internal loss calculation
outputs = model(input_ids, labels=target_ids)

print(tf.math.reduce_mean(l), tf.math.reduce_mean(outputs[0]))
print(l.shape, outputs[0].shape)
print('How many are the same? ', np.mean(outputs.loss==l[:-1]))

print('Are they equal? ', tf.math.reduce_mean(l) == tf.math.reduce_mean(outputs[0]))


```

I'm trying to understand how the loss is computed inside the model when the argument labels is provided, but what I am trying above doesn't seem to work. What am I doing wrong?

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

I would expect `tf.math.reduce_mean(l) == tf.math.reduce_mean(outputs[0])` to be True.
Hello! If you look inside the `TFGPT2LMHeadModel`, you'll see it automatically shifts the labels for you.

The model generates tokens given a past. It then compares the generated token to the "true" token contained in the labels you passed to it. If you shift the tokens as it is done in the model, you should get identical results:

```py
import tensorflow as tf
from transformers import TFGPT2LMHeadModel
from transformers.modeling_tf_utils import TFCausalLanguageModelingLoss
import numpy as np

model = TFGPT2LMHeadModel.from_pretrained('gpt2')


one_line_dset = (np.random.rand(1, 1024)>.5)*1
input_ids = one_line_dset
target_ids = one_line_dset

# explicit loss calculation
prediction = model.predict(input_ids).logits
prediction = tf.convert_to_tensor(prediction)

# internal loss calculation
outputs = model(input_ids, labels=target_ids, training=False)

target_ids = target_ids[:, 1:]
prediction = prediction[:, :-1]
l = TFCausalLanguageModelingLoss().compute_loss(target_ids, prediction)

print(tf.math.reduce_mean(l), tf.math.reduce_mean(outputs[0]))
```

You'll see that the two results are very close to being equal. Note the very close, as they are not exactly equal. @jplu can chime in here but I believe this has to do with the switch to graph mode happening with the call to `.predict`.<|||||>I entirely second what @LysandreJik said! Nice and clear explanation 👍 <|||||>@LysandreJik @jplu you are super fast guys! Thanks! a lot!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.