GPT2 Sentence Probability: Necessary to Prepend "<|endoftext|>"?
When computing sentence probability, do we need to prepend the sentence with a dummy start token (e.g. <|endoftext|>) to get the full sentence probability? I am currently using the following implemention (from https://github.com/huggingface/pytorch-transformers/issues/473):

```
model = GPT2LMHeadModel.from_pretrained("gpt2")
model.eval()
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

def score(sentence):
    tokenize_input = tokenizer.tokenize(sentence)
    tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
    loss=model(tensor_input, lm_labels=tensor_input)
    return -loss[0] * len(tokenize_input)

a=['there is a book on the desk',
                'there is a plane on the desk',
                        'there is a book in the desk']
print([score(i) for i in a])
```

With this implementation, say for the sentence "there is a book on the desk", is it taking into consideration all the words when computing the full sentence probability (i.e. it's computing P(there|<|endoftext|>) \* P(is|there,<|endoftext|>) \* ... * P(desk|the,...))? If not, what's the right way to prepend the dummy start token?
Dig into this a little, and it looks like the answer is yes:

```
text = "the book is on the desk."
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')
input_ids = torch.tensor(tokenizer.encode(text)).unsqueeze(0)  # Batch size 1
tokenize_input = tokenizer.tokenize(text)
#50256 is the token_id for <|endoftext|>
tensor_input = torch.tensor([ [50256]  +  tokenizer.convert_tokens_to_ids(tokenize_input)])
with torch.no_grad():
    outputs = model(tensor_input, labels=tensor_input)
    loss, logits = outputs[:2]
print("a=", loss*len(tokenize_input))

lp = 0.0
for i in range(len(tokenize_input)):
    masked_index = i
    predicted_score = logits[0, masked_index]
    predicted_prob = softmax(np.array(predicted_score))
    lp += np.log(predicted_prob[tokenizer.convert_tokens_to_ids([tokenize_input[i]])[0]])

print("b=", lp)
```
produces:
a= tensor(32.5258)
b= -32.52579879760742

Without prepending [50256]:
a= tensor(30.4421)
b= -59.90513229370117
<|||||>@jhlau hello, out of curiosity, why are you multiplying the loss with length of tokenize_input? <|||||>The loss returned is the average loss (i.e. it is already divided by the length); since I am interested in getting the sentence probability, I need to revert that.<|||||>Instead of hard-coding `50256` better to use:

```
tokenizer.convert_tokens_to_ids(tokenizer.special_tokens_map['eos_token'])
```
<|||||>You can also use `tokenizer. eos_token_id` ([doc](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.eos_token_id))<|||||>Hope this question is simple to answer: How can I run the probability calculation entirely on gpu? When I start with numpy in the for loop I am supposed to put my data back on cpu right? I'd like to avoid that as long as possible.<|||||>@jhlau your code does not seem to be correct to me. Refer to [this](https://github.com/simonepri/lm-scorer/blob/master/lm_scorer/models/gpt2.py#L20-L60) or #2026 for a (hopefully) correct implementation.

You can also try [`lm-scorer`](https://github.com/simonepri/lm-scorer), a tiny wrapper around `transformers` I wrote that allows you to get sentences probabilities using models that support it (only GPT2 models are implemented at the time of writing).

> I included this here because this issue is still the first result when searching from GitHub/Google about using transformers' models to get sentences probabilities and I think it might be useful to many.
<|||||>I see. So I should be using self.tokenizer.bos_token and self.tokenizer.eos_token to start and end a sentence properly (instead of the hardcoded 50526 |endoftext| token). I'll give it a run and see if I find much difference.<|||||>> The loss returned is the average loss (i.e. it is already divided by the length); since I am interested in getting the sentence probability, I need to revert that.

I think this is incorrect. If you multiply by length, you will get higher probability for long sentences even if they make no sense. The average aims to normalize so that the probability is independent of the number of tokens. Does that make sense?<|||||>I understand that of course. I need the full sentence probability because I intend to do other types of normalisation myself (e.g. based unigram frequencies). I am not saying returning the average loss is wrong - I was just clarifying to another user why I multiplied the average loss with length (because I need the full sentence probability).<|||||>> I understand that of course. I need the full sentence probability because I intend to do other types of normalisation myself (e.g. based unigram frequencies). I am not saying returning the average loss is wrong - I was just clarifying to another user why I multiplied the average loss with length (because I need the full sentence probability).

AAAAh I see. Thanks<|||||>> When computing sentence probability, do we need to prepend the sentence with a dummy start token (e.g. <|endoftext|>) to get the full sentence probability? I am currently using the following implemention (from #473):
> 
> ```
> model = GPT2LMHeadModel.from_pretrained("gpt2")
> model.eval()
> tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
> 
> def score(sentence):
>     tokenize_input = tokenizer.tokenize(sentence)
>     tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
>     loss=model(tensor_input, lm_labels=tensor_input)
>     return -loss[0] * len(tokenize_input)
> 
> a=['there is a book on the desk',
>                 'there is a plane on the desk',
>                         'there is a book in the desk']
> print([score(i) for i in a])
> ```
> 
> With this implementation, say for the sentence "there is a book on the desk", is it taking into consideration all the words when computing the full sentence probability (i.e. it's computing P(there|<|endoftext|>) * P(is|there,<|endoftext|>) * ... * P(desk|the,...))? If not, what's the right way to prepend the dummy start token?

```sent_probability = math.exp(-1.0 * loss * (num_of_word_piece - 1))```

num_of_word_piece is the num of encoded ids by the tokenizer.
When calculating sent probability, it is appropriate to prepend "<|endoftext|>" in front of the sent text. 
tokenizer will tokenize the "<|endoftext|>" into one token_id, which is tokenizer.eos_token_id.

The loss is calculated from the cross-entropy of `shift_logits` and `shift_labels`. By default, cross_entropy gives the mean reduction.  And in this case, it is the mean reduction of `num_of_word_piece - 1` word_pieces. 



<|||||>For anyone who's interested in **batching** the above process, here's the code:
```python
lines = [tokenizer.eos_token + line for line in lines]

tok_res = tokenizer.batch_encode_plus(lines, return_tensors='pt', pad_to_max_length=True)
input_ids = tok_res['input_ids']
attention_mask = tok_res['attention_mask']
lines_len = torch.sum(tok_res['attention_mask'], dim=1)

outputs = gpt2_model(input_ids=input_ids, attention_mask=attention_mask, labels=input_ids)
loss, logits = outputs[:2]

for line_ind in range(len(lines)):
    line_log_prob = 0.0
    for token_ind in range(lines_len[line_ind] - 1):
        token_prob = F.softmax(logits[line_ind, token_ind], dim=0)
        token_id = input_ids[line_ind, token_ind + 1]
        line_log_prob += torch.log(token_prob[token_id])
    print(f'line_log_prob:{line_log_prob}')
```
A caveat was that `token_type_ids` from `tokenizer.batch_encode_plus` should not be passed to the `gpt2_model` in order to obtain the same results as the line-by-line inference.<|||||>I think there's a mistake in the approach taken here.

It seems like the OP concluded that you can score the whole sentence including the first word, by appending a `bos_token` (`<|endoftext|>`) at the beginning of the string.

From what I understand, though, this is probably not a good idea, since it is __unlike training__, as mentioned by @thomwolf in another thread (https://github.com/huggingface/transformers/issues/473#issuecomment-482280934) (emphasis mine):

> Unfortunately, given __the way the model is trained (without using a token indicating the beginning of a sentence)__, I would say it does not make sense to try to get a score for a sentence with only one word.

So, the right way to get a sentence's probability would be 

In [1]:
```python
import torch
import torch.nn.functional as F
import numpy as np
from tqdm import tqdm
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModelForMaskedLM
from transformers import logging

model_spec = 'gpt2'
model = AutoModelForCausalLM.from_pretrained(model_spec)
tokenizer = AutoTokenizer.from_pretrained(model_spec)

def score(sentence):
    ids = tokenizer(sentence, return_tensors="pt").input_ids[0]
    with torch.no_grad():
        outs = model(input_ids=ids, labels=ids)
    return -outs.loss * (len(ids) - 1) # the first word is not predicted
text = "the book is on the table."
print("sentence score = ", score(text).item())
```
Out [1]:

>     sentence score =  -23.651351928710938

We can verify where this score comes from. 
In the spirit of the OP, I'll print each word's logprob and then sum
In [2]:
```python
ids = tokenizer(text, return_tensors="pt").input_ids[0]
with torch.no_grad():
    outs = model(input_ids=ids, labels=ids)
logits = outs.logits
logprob = 0.0
print("", "id", "token", "logprob", sep='\t')
for i in range(len(ids)-1):
    predicted_logprob = torch.log_softmax(logits[i], dim=-1)
    logprob_i = predicted_logprob[ids[i+1]]
    print(i, ids[i+1].item(), tokenizer.decode(ids[i+1]), logprob_i.item(), sep='\t')
    logprob += logprob_i
print("total logprob = ", logprob.item(), sep = "\t")
```
Out [2]:

    	id	token	logprob
    0	1492	book	-7.818896770477295
    1	318	is	-1.9839171171188354
    2	319	on	-4.946821212768555
    3	262	the	-1.473121166229248
    4	3084	table	-4.56355619430542
    5	13	.	-2.865037441253662
    total logprob = 	-23.651350021362305


Basically, I think we shouldn't prepend anything, if it wasn't like that in training, and so we shouldn't include the first word's score when we score a sentence from GPT2. Am I wrong?