Does GPT2LMHeadModel need <|startoftext|> and <|endoftext|> tokens?
## ❓ Does GPT2LMHeadModel need <|startoftext|> and <|endoftext|> tokens?

Hey!

I'm using GPT2LMHeadModel to get a good representation of a Language Model - I want to get probabilities for each word. The problem is - the **model predicts probabilities very well for all tokens except for the first one**. The first's token probability is often very small no matter what word I choose. 
I've read that there is "<|startoftext|>" token, but have not found information on how to use it. It also doesn't exist in GPT2Tokenizer.vocabulary. Do we have to use it?

### Example code:

```
import torch
from pytorch_transformers import *

pretrained_weights='gpt2'

tokenizer = GPT2Tokenizer.from_pretrained(pretrained_weights)
model = GPT2LMHeadModel.from_pretrained(pretrained_weights)
model.eval()

def show_probabilities(INPUT_TEXT):
    input_ids = torch.tensor([tokenizer.encode(INPUT_TEXT)])
    
    with torch.no_grad():
        index=0
        outputs = model(input_ids=input_ids)
        logits = outputs[0][0]
        probs = torch.softmax(logits, 1)
        for index in range(0, len(input_ids[0])):
            token_id = input_ids[0][index]
            probability = probs[index - 1][token_id].item()
            print(f"Probability for the token \"{tokenizer.decode(token_id.item())}\" is {probability}")
    print("\n")



show_probabilities('To be or not to be <|endoftext|>')
show_probabilities('<|startoftext|> To be or not to be <|endoftext|>')
show_probabilities('<|endoftext|> To be or not to be <|endoftext|>')
show_probabilities('Hello world is so wierd?')

```

### Output:
###### (so that you dont have to run it)
```
Probability for the token " To" is 6.045737421800368e-08
Probability for the token " be" is 0.01369183138012886
Probability for the token " or" is 0.0001948970602825284
Probability for the token " not" is 0.7490634322166443
Probability for the token " to" is 0.5098284482955933
Probability for the token " be" is 0.9639962911605835
Probability for the token "<|endoftext|>" is 0.00017062896222341806


Probability for the token " <" is 1.5030431086415774e-06
Probability for the token "|" is 0.0006586791132576764
Probability for the token "start" is 7.143173570511863e-05
Probability for the token "of" is 0.0012107481015846133
Probability for the token "text" is 0.0007207148591987789
Probability for the token "|" is 0.4524894058704376
Probability for the token ">" is 0.027218399569392204
Probability for the token " To" is 0.0003593114379327744
Probability for the token " be" is 0.015610950998961926
Probability for the token " or" is 0.0021431492641568184
Probability for the token " not" is 0.46310704946517944
Probability for the token " to" is 0.8615797162055969
Probability for the token " be" is 0.9770862460136414
Probability for the token "<|endoftext|>" is 0.0008418861543759704


Probability for the token "<|endoftext|>" is 3.0863736810715636e-06
Probability for the token " To" is 3.549279790604487e-05
Probability for the token " be" is 0.04548846557736397
Probability for the token " or" is 0.0003993543505202979
Probability for the token " not" is 0.8718274831771851
Probability for the token " to" is 0.9372356534004211
Probability for the token " be" is 0.9853253960609436
Probability for the token "<|endoftext|>" is 0.0009108908707275987


Probability for the token " Hello" is 0.00041539999074302614
Probability for the token " world" is 0.00014912338519934565
Probability for the token " is" is 0.029302824288606644
Probability for the token " so" is 0.01128558162599802
Probability for the token " w" is 0.00020273651171009988
Probability for the token "ier" is 0.008098911494016647
Probability for the token "d" is 0.8924543857574463
Probability for the token "?" is 0.0036612364929169416
```


Huggingface GPT2's default beggining of sentence token is `<|endoftext|>`, not `<|startoftext|>` as mentioned [here](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2tokenizer). So either just use `<|endoftext|>` or replace tokenizer's default `bos` attribute with `<|startoftext|>`. Or you may add `<|startoftext|>` as `additional_speacial_token` (read more [here](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.add_special_tokens)). As seen in the part you provide above, GPT2 tokenizer splits `<|startoftext|>` to byte-pairs. So you need to specify it as either one of special tokens or as additional special token.<|||||>Thank you, it really helps! 
To give more insight for future dwellers:

I also found the [code](https://github.com/huggingface/transformers/blob/fbaf05bd92249b6dd961f5f8d60eb0892c541ac8/transformers/tokenization_gpt2.py#L119-L121) for tokenization_gpt2 that uses bos/eos/unk tokens and an [example](https://github.com/huggingface/transformers/blob/fbaf05bd92249b6dd961f5f8d60eb0892c541ac8/transformers/tokenization_utils.py#L577-L589) of using `<CLS>` token. 

I've run tests with adding `<CLS>` token, `<|startoftext|>` token and `<|endoftext|>` token.
While adding `<CLS>`  or `<|startoftext|>` at the beginning of the sentence raises the probability of the first token 10<sup>3</sup> times greater, results differ a little bit ("To" was 6.04e-8, now is 9.51e-5 or 7,67e-5). But it means we can just use `<|endofsentence|>` at the beginning and it will work too.
 Adding `<|endoftext|>` token at the end in GPT2LMHeadModel doesn't change the resulting probabilities, but I haven't checked how it influences text prediction.
 

<|||||>> The problem is - the model predicts probabilities very well for all tokens except for the first one.

I think you should start the for loop from `1` instead of `0` otherwise you will access `probs[-1]` which is not correct. If you add the `bos` token, intuitively this means that you don't consider the probability of the `bos` token in your summation (which you can't have anyway).

I published a (hopefully) corrected and vectorized version of your code together with [`lm-scorer`](https://github.com/simonepri/lm-scorer).


> Adding <|endoftext|> token at the end in GPT2LMHeadModel doesn't change the resulting probabilities, but I haven't checked how it influences text prediction.

I actually observed the opposite.
In the following example, you can see that the sentence without the dot at the end of the sentence has a lower probability than the (correct) one with the correct punctuation. Without the `eos` the incorrect one would have higher probability instead.

```bash
$ lm-scorer -t - <<< """I like it.                                
I like it"""

I    0.018321
Ġlike    0.0066431
Ġit    0.042104
.    0.23876
<|endoftext|>    0.0067232

I    0.018321
Ġlike    0.0066431
Ġit    0.042104
<|endoftext|>    0.00023855

$lm-scorer - <<< """I like it. 
I like it"""

I like it.    8.2257e-09
I like it    1.2224e-09
```

More tests [here](https://github.com/simonepri/lm-scorer/blob/master/tests/models/test_gpt2.py#L32-L239). 