BertTokenizer never_split issue
Hi,

I'm using the BertTokenizer to tokenize a piece of text where I use some entity markers to mark the beginning and end of entities, e.g.:

> This was among a batch of paperback [E1] Oxford World [/E1] ' s Classics

I've manually added such entity markers to _vocab file_ and the _never_split_ tuple in _BertTokenizer_. My purpose is to retain such markers as a token, not to be split into wordpieces.

However, when I test the code from command line in linux terminal, the _never_split_ does not work. Entity markers are split into wordpieces. Here is the printout: 
```
06/28/2019 11:41:20 - INFO - __main__ -   Writing example 0 of 1000
['In', '1983', ',', 'a', 'year', 'after', 'the', 'rally', ',', '[', 'E', '##1', ']', 'For', '##sberg', '[', '/', 'E', '##1', ']', 'received', 'the', 'so', '-', 'called', '`', '`', 'genius', 'award', "'", "'", 'from', 'the', '[', 'E', '##2', ']', 'John', 'D', '.', '[', '/', 'E', '##2', ']', 'and', 'Catherine', 'T', '.', 'MacArthur', 'Foundation', '.']
```

The strange thing is that, the _never_split_ works pretty fine when I test it in PyCharm, and I get my desired output:

```
06/28/2019 11:53:01 - INFO - __main__ -   Writing example 0 of 1000
06/28/2019 11:53:01 - INFO - __main__ -   *** Example ***
06/28/2019 11:53:01 - INFO - __main__ -   guid: train-61b3a65fb9b7111c4ca4
06/28/2019 11:53:01 - INFO - __main__ -   tokens: [CLS] In 1983 , a year after the rally , [E1] For ##sberg [/E1] received the so - called ` ` genius award ' ' from the [E2] John D . [/E2] and Catherine T . MacArthur Foundation . [SEP]
06/28/2019 11:53:01 - INFO - __main__ -   input_ids: 101 1130 2278 117 170 1214 1170 1103 11158 117 20 1370 19945 21 1460 1103 1177 118 1270 169 169 13533 2574 112 112 1121 1103 22 1287 141 119 23 1105 6017 157 119 21045 2974 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
``` 
The tests are done on the same machine (my local PC) under same virtual environment. The code, pre-trained bert model and parameters used in Pycharm and terminal are the **same**. 

Since I need to migrate the code to a server for model training, I really need to resolve this issue. I spent some time debugging but have no idea what could be the cause. Could anyone please provide some hints?

Thanks in advance.


 
which version of python do you use in these environments?<|||||>> which version of python do you use in these environments?

Hi Thomwolf,

I'm using Python 3.6.8 for all these environments.

<|||||>In case it's helpful, I create a gist to include some details of this issue: https://gist.github.com/ardellelee/4d80ee7a07166bb6d1a203fdd4d7cc07<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
