Llama: Generating text token by token removes whitespaces 
### System Info

- `transformers` version: 4.28.0.dev0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.10
- Huggingface_hub version: 0.13.4
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help?

 @ArthurZucker @sgugger 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run the following script

```python
import torch
from transformers import (
    LlamaForCausalLM,
    LlamaTokenizer,
    LogitsProcessorList,
    RepetitionPenaltyLogitsProcessor,
    TemperatureLogitsWarper,
    TopPLogitsWarper,
)

device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = LlamaTokenizer.from_pretrained("./path/to/llama")
model = LlamaForCausalLM.from_pretrained(
    "./models/llama-7b-hf", torch_dtype=torch.float16
).to(device)


@torch.no_grad()
def stream_generate(
    prompt: str,
    temperature=1.0,
    max_new_tokens=512,
    top_p=1.0,
    repetition_penalty=1.0,
):
    global tokenizer, model
    if tokenizer is None or model is None:
        return {"error": "Model not loaded."}

    input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)

    original_size = len(input_ids[0])

    logits_processor = LogitsProcessorList(
        [
            TemperatureLogitsWarper(temperature=temperature),
            RepetitionPenaltyLogitsProcessor(penalty=repetition_penalty),
            TopPLogitsWarper(top_p=top_p),
        ]
    )
    while True:
        # Get logits for the next token
        logits = model(input_ids).logits[:, -1, :]

        logits = logits_processor(input_ids, logits)

        probs = torch.nn.functional.softmax(logits, dim=-1)
        next_token_id = torch.multinomial(probs, num_samples=1)

        next_token = tokenizer.decode(next_token_id[0], skip_special_tokens=True)
        print(next_token, end="", flush=True)

        input_ids = torch.cat((input_ids, next_token_id), dim=1)

        if len(input_ids[0]) >= original_size + max_new_tokens:
            break


stream_generate("In a shocking finding, ")
```

### Expected behavior

Text should be printed in a streaming manner, similar to OpenAI's playground, this behaviour properly happens with models like GPT-2 or GPT-J, however, with LLaMA, there are no whitespaces inbetween words.

I believe this is related to this, mentionned in the [documentation](https://huggingface.co/docs/transformers/main/en/model_doc/llama) 

> The LLaMA tokenizer is based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. “Banana”), the tokenizer does not prepend the prefix space to the string. To have the tokenizer output the prefix space, set decode_with_prefix_space=True in the LlamaTokenizer object or in the tokenizer configuration.

However, it seems that `decode_with_prefix_space` has been removed
cc @gante 
Note that the streaming API is new and still experimental :-)<|||||>Also note that we aim to match the original tokenizer 1 to 1, while trying not to add super model specific pieces of code in the more general api 😉 <|||||>> Note that the streaming API is new and still experimental :-)

I may be wrong as I'm not sure what you are mentionning, but I don't believe I am using the streaming API
The example code is something I wrote based on the generate() function in transformers, It's an old piece of code that I attempted to run with LLaMa and that seems to pose issues

<|||||>Oh my bad, I read to fast. I don't know what this `decode_with_prefix_space` mentioned in the documentation is. It does not exist in the codebase at all (not for LLaMA and not for any other sentencepiece model either).<|||||>> I don't know what this decode_with_prefix_space mentioned in the documentation is. It does not exist in the codebase at all (not for LLaMA and not for any other sentencepiece model either).

I haven't had much time to check why , but i noticed that it was removed in the latest commit to LLaMA's tokenizer
https://github.com/huggingface/transformers/commit/c0f99b4d2ec73090595914dde4c16da207e21d73

> Also note that we aim to match the original tokenizer 1 to 1, while trying not to add super model specific pieces of code in the more general api 

I understand, is it then normal for the tokenizer to do this as it is based on sentencepiece? <|||||>Not sure I understand your question. Yes it is normal, if you use the sentencepiece tokenizer, you will get the same results when decoding. 
```python 
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained('huggyllama/llama-30b')
>>> tokenizer.batch_decode(tokenizer.encode("Hey how are you doing?"), skip_special_tokens = True)
['', 'Hey', 'how', 'are', 'you', 'doing', '?']
``` 
whether you are using the fast or slow tokenizer, both will output the same. 
If you use `tokenizer.sp_model.decode` (which is basically the sentencpiece model that the original llama uses) then you have no additional prefix space (see the original codebase [here](https://github.com/facebookresearch/llama/blob/main/llama/tokenizer.py)). However, the doc should be updated to remove the `decode_with_prefix_space` thanks for catching this. <|||||>> Not sure I understand your question. Yes it is normal, if you use the sentencepiece tokenizer, you will get the same results when decoding.

I apologize, my question was meant to ask if it is normal for the tokenizer to not have an option to `decode_with_prefix_space`
As it is the case, I will close this issue, I will find another way to fix my piece of code <|||||>This feature was added on my request so I do not think it is justified that this got closed.
decode_with_prefix_space was meant as a workaround for the fact that the tokenizer is unsuitable for downstream tasks like the one mentioned in this issue. The default behavior was off, but it allowed implementations to turn it on and get proper generations.
I do not know if it ever worked right, but without this my users keep complaining that every generation lacks the space. If this is not added back I will have to do this myself in my own code, but that would be unfortunate for the rest of the ecosystem.<|||||>@henk717 While I can't help with the removal of `decode_with_prefix_space` from the library, here is how I dealt with it, hope this helps

```python
    while True:
        # Get logits for the next token
        logits = model(input_ids).logits[:, -1, :]
        # Apply logits processor
        next_tokens_scores = logits_processor(input_ids, logits)

        probs = torch.nn.functional.softmax(next_tokens_scores, dim=-1)
        next_token_id = torch.multinomial(probs, num_samples=1)

        # Note: This is done to handle Sentencepiece based tokenizers,
        # as they don't preprend the prefix space to the start of a word
        tokens_previous = tokenizer.decode(input_ids[0], skip_special_tokens=True)
        input_ids = torch.cat((input_ids, next_token_id), dim=1)
        tokens = tokenizer.decode(input_ids[0], skip_special_tokens=True)

        new_tokens = tokens[len(tokens_previous) :]
```
This is a fairly unclean way to fix the issue, but it works.
<|||||>> @henk717 While I can't help with the removal of `decode_with_prefix_space` from the library, here is how I dealt with it, hope this helps
> 
> ```python
>     while True:
>         # Get logits for the next token
>         logits = model(input_ids).logits[:, -1, :]
>         # Apply logits processor
>         next_tokens_scores = logits_processor(input_ids, logits)
> 
>         probs = torch.nn.functional.softmax(next_tokens_scores, dim=-1)
>         next_token_id = torch.multinomial(probs, num_samples=1)
> 
>         # Note: This is done to handle Sentencepiece based tokenizers,
>         # as they don't preprend the prefix space to the start of a word
>         tokens_previous = tokenizer.decode(input_ids[0], skip_special_tokens=True)
>         input_ids = torch.cat((input_ids, next_token_id), dim=1)
>         tokens = tokenizer.decode(input_ids[0], skip_special_tokens=True)
> 
>         new_tokens = tokens[len(tokens_previous) :]
> ```
> 
> This is a fairly unclean way to fix the issue, but it works.

it works for me, thank you!