Transformers and BERT: dealing with possessives and apostrophes when encode
Let's consider two sentences:

    "why isn't Alex's text tokenizing? The house on the left is the Smiths' house"

Now let's tokenize and decode:

    from transformers import BertTokenizer
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
    tokenizer.decode(tokenizer.convert_tokens_to_ids(tokenizer.tokenize("why isn't Alex's text tokenizing? The house on the left is the Smiths' house")))

We get:

    "why isn't alex's text tokenizing? the house on the left is the smiths'house"



**My question is how dealing with missing space in some possessives like *smiths'house*?**


For me, it seems that the process of tokenization in Transformers is done not right. Let's consider output of

    tokenizer.tokenize("why isn't Alex's text tokenizing? The house on the left is the Smiths' house")

we get:

    ['why', 'isn', "'", 't', 'alex', "'", 's', 'text', 'token', '##izing', '?', 'the', 'house', 'on', 'the', 'left', 'is', 'the', 'smith', '##s', "'", 'house']

So in this step, we already have lost important information about the last apostrophe. It would be much better if tokenization was done in the another way:

    ['why', 'isn', "##'", '##t', 'alex', "##'", '##s', 'text', 'token', '##izing', '?', 'the', 'house', 'on', 'the', 'left', 'is', 'the', 'smith', '##s', "##'", 'house']

In this way, tokenization keeps all information about apostrophes, and we will not have problems with possessives.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Any idea about how to solve this apostrophe seperating word token with different word id in bert rokenizer ?