is there any <SOS> or <EOS> token in reformer-enwik8?
## Environment info
None

### Who can help
@patrickvonplaten 

Models:
reformer-enwik8

## Information

Model I am using reformer-enwik8:

The problem arises when using:

```
def encode(list_of_strings, pad_token_id=0):
    max_length = max([len(string) for string in list_of_strings])

    # create emtpy tensors
    attention_masks = torch.zeros((len(list_of_strings), max_length), dtype=torch.long)
    input_ids = torch.full((len(list_of_strings), max_length), pad_token_id, dtype=torch.long)

    for idx, string in enumerate(list_of_strings):
        # make sure string is in byte format
        if not isinstance(string, bytes):
            string = str.encode(string)

        input_ids[idx, :len(string)] = torch.tensor([x + 2 for x in string])
        attention_masks[idx, :len(string)] = 1

    return input_ids, attention_masks
model = ReformerModelWithLMHead.from_pretrained("google/reformer-enwik8")
ids, masks = encode(["I COULD LABEL THIS ON THE INGREDIENTS AS MEAT".capitalize()])
logits = model(input_ids=ids, attention_mask=masks)["logits"]
```

The tasks I am working on is:
try to get LM prob of one certain sequence

## Expected behavior
I expect 0 and 1 represent `<SOS>` and `<EOS>` respectively, but I don't know if it is correct
I think `reformer-enwik8` was not trained using a `<SOS> or <EOS>` token - it's just the model to evaluate Reformer's compression capabilities on enwik8, see paper: https://arxiv.org/abs/2001.04451<|||||>> I think `reformer-enwik8` was not trained using a or token - it's just the model to evaluate Reformer's compression capabilities on enwik8, see paper: https://arxiv.org/abs/2001.04451

Thanks a lot for the quick reply.
btw, is there any other pretrained character-level language model provide by huggingface now?<|||||>To add a related question: Is there any way of knowing which characters are part of the vocabulary of the pre-trained enwik-8 model? To my knowledge there only exists information on the `vocab_size` which is set to 258, but no information on which characters are part of the vocabulary of the pre-trained model.<|||||>Reformer simple uses Python's `chr()` and `ord()` methods to tokenize and decode. See: https://huggingface.co/google/reformer-enwik8#reformer-language-model-on-character-level-and-trained-on-enwik8 <|||||>This is true. But those methods work with the Unicode character set (i.e. up to 1,114,111) which does not correspond to a `vocab_size` of 258. This can also be seen with the shape of `outputs.scores` which is (1, 258). It seems to me that the vocab_size is just the first 258 characters of the unicode standard, i.e. Basic Latin & Latin-1 Supplement. <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.