How to calculate GPT-2 sentence loss (for each sentence) if batch has 2 or more sentences?
I need to calculate loss for batch of sentence but when i do this I get only average loss for all the sentences not all the loses.   
 
    from transformers import GPT2LMHeadModel, GPT2Tokenizer
    model = GPT2LMHeadModel.from_pretrained('distilgpt2')
   loss_list = model(input_ids, labels=input_ids)

Is it possible to somehow calculate the loss of each sentence separatelly in one batch or I have to use only batch-size=1 which will be much slower.   

  
Hello,

[Not an official answer, I'm not part of the the team]
The `GPT2LMHeadModel` uses the `CrossEntropyLoss` with its default parameters. The default for `reduction` is 'mean', so the loss is averaged. One thing that you can do is to define your custom `GPT2LMHeadModel` and replace this:

```
loss_fct = CrossEntropyLoss()
loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
```

with:

```
loss_fct = CrossEntropyLoss(reduction='none')
loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
```

this gives you the loss for every token. If you want the average by sequence, you need to undo the flatten first:
```
loss_by_sequence = loss.view(logits.size(0), -1).mean(dim=1)
```

Be aware that I say "by sequence" and not "by sentence" because that depends on how you structure the inputs and nothing prevents a sequence from containing more than 1 sentence.<|||||>Awesome. Thanks very much. This should speed up things considerably.

I need to change this in file: transformers/models/gpt2/modeling_gpt2.py and I get loss for every token. So if i have 2 sequences Do I get this [ [loss_word1a,loss_word2a ...], [loss_word1b,loss_word2b,...] ?




<|||||>Yes, that's what you'd get in `loss`.

In `loss_by_sequence` you'd get [loss_seq1, loss_seq2, ...]<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.