How to detokenize a BertTokenizer output? 
For example, let's tokenize a sentece "why isn't Alex' text tokenizing":

    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
    tokens = tokenizer.tokenize("why isn't Alex' text tokenizing")

We are getting the next output:

    ['why', 'isn', "'", 't', 'Alex', "'", 'text', 'token', '##izing']

I want to convert it to:

    why isn't Alex' text tokenizing


It seems the tokenizer doesn't do its job in the best way. If after tokenization we would have something like that:

    ['why', 'isn', "##'", '##t', 'Alex', "##'", 'text', 'token', '##izing']

It would be easy to convert:

    tokens = ['why', 'isn', "##'", '##t', 'Alex', "##'", 'text', 'token', '##izing']
    restored_text = [None] * len(tokens)
    tokens.extend(['#', '#'])
    for i in range(len(tokens )-2):
        if re.findall("#{2}", tokens[i+1]):
          restored_text [i] = tokens[i] + tokens[i+1].replace('##', '')
          if re.findall("#{2}", tokens [i+2]):
            restored_text[i] = restored_text[i] + tokens[i+2].replace('##', '')
        else:
          restored_text[i] = tokens[i]
    
    restored_text_without_masks = []
    for i in range(len(restored_text)):
        if not restored_text[i].startswith('#'):
            restored_text_without_masks.append(restored_text[i])
I came across the same problem some days ago. I like your code, however it can be faster:

```python

def is_subtoken(word):
    if word[:2] == "##":
        return True
    else:
        return False

tokens = ['why', 'isn', "##'", '##t', 'Alex', "##'", 'text', 'token', '##izing']
restored_text = []
for i in range(len(tokens)):
    if not is_subtoken(tokens[i]) and (i+1)<len(tokens) and is_subtoken(tokens[i+1]):
        restored_text.append(tokens[i] + tokens[i+1][2:])
        if (i+2)<len(tokens) and is_subtoken(tokens[i+2]):
            restored_text[-1] = restored_text[-1] + tokens[i+2][2:]
    elif not is_subtoken(tokens[i]):
        restored_text.append(tokens[i])
```<|||||>@GuillemGSubies Did you solve this problem in your task?<|||||>> @GuillemGSubies Did you solve this problem in your task?

I used a modification of the code I posted above (in my use case I needed to interact with some NER labels also). However if you want to detokenize I think your code works perfectly.<|||||>@GuillemGSubies 
I just want to clarify something in my question.

Let's consider two sentences:

    "why isn't Alex's text tokenizing? The house on the left is the Smiths' house"

Now let's tokenize and decode:

    from transformers import BertTokenizer
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
    tokenizer.decode(tokenizer.convert_tokens_to_ids(tokenizer.tokenize("why isn't Alex's text tokenizing? The house on the left is the Smiths' house")))

We get:

    "why isn't alex's text tokenizing? the house on the left is the smiths'house"



**My question is how dealing with missing space in some possessives like *smiths'house*?**


For me, it seems that the process of tokenization in Transformers is done not right. Let's consider output of

    tokenizer.tokenize("why isn't Alex's text tokenizing? The house on the left is the Smiths' house")

we get:

    ['why', 'isn', "'", 't', 'alex', "'", 's', 'text', 'token', '##izing', '?', 'the', 'house', 'on', 'the', 'left', 'is', 'the', 'smith', '##s', "'", 'house']

So in this step, we already have lost important information about the last apostrophe. It would be much better if tokenization was done in the another way:

    ['why', 'isn', "##'", '##t', 'alex', "##'", '##s', 'text', 'token', '##izing', '?', 'the', 'house', 'on', 'the', 'left', 'is', 'the', 'smith', '##s', "##'", 'house']

In this way, tokenization keeps all information about apostrophes, and we will not have problems with possessives.
<|||||>This goes beyond of my understanding of the library, I am not a dev, sorry<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
