BertWordPieceTokenizer cannot be pickled
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
**Bert**

Language I am using the model on (English, Chinese ...):
**English**

The problem arises when using:
* [X] my own modified scripts: 


The tasks I am working on is:
* [X] my own task or dataset: 

## To reproduce



``` python
import torch
import tokenizers
import pandas as pd
from torch.utils import data

class config:
    MAX_LEN = 128
    TRAIN_BATCH_SIZE = 64
    VALID_BATCH_SIZE = 16
    EPOCHS = 5
    BERT_PATH = "../input/bert-base-uncased/"
    MODEL_PATH = "model.bin"
    TRAINING_FILE = "../input/tweet-sentiment-extraction/train_folds.csv"
    TOKENIZER = tokenizers.BertWordPieceTokenizer(
        f"{BERT_PATH}/vocab.txt", 
        lowercase=True
    )

def process_data(tweet, selected_text, sentiment, tokenizer, max_len):
    len_st = len(selected_text)
    idx0 = -1
    idx1 = -1
    for ind in (i for i, e in enumerate(tweet) if e == selected_text[0]):
        if tweet[ind: ind+len_st] == selected_text:
            idx0 = ind
            idx1 = ind + len_st - 1
            break

    char_targets = [0] * len(tweet)
    if idx0 != -1 and idx1 != -1 :
        for ct in range(idx0, idx1 + 1):
            char_targets[ct] = 1
    
    tok_tweet = tokenizer.encode(tweet)
    input_ids_orig = tok_tweet.ids[1:-1]
    tweet_offsets = tok_tweet.offsets[1:-1]
    
    target_idx = []
    for j, (offset1, offset2) in enumerate(tweet_offsets):
        if sum(char_targets[offset1: offset2]) > 0:
            target_idx.append(j)
    
    targets_start = target_idx[0]
    targets_end = target_idx[-1]

    sentiment_id = {
        'positive': 3893,
        'negative': 4997,
        'neutral': 8699
    }
    
    input_ids = [101] + [sentiment_id[sentiment]] + [102] + input_ids_orig + [102]
    token_type_ids = [0, 0, 0] + [1] * (len(input_ids_orig) + 1)
    mask = [1] * len(token_type_ids)
    tweet_offsets = [(0, 0)] * 3 + tweet_offsets + [(0, 0)]
    targets_start += 3
    targets_end += 3

    padding_length = max_len - len(input_ids)
    if padding_length > 0:
        input_ids = input_ids + ([0] * padding_length)
        mask = mask + ([0] * padding_length)
        token_type_ids = token_type_ids + ([0] * padding_length)
        tweet_offsets = tweet_offsets + ([(0, 0)] * padding_length)
    
    return {
        'ids': input_ids,
        'mask': mask,
        'token_type_ids': token_type_ids,
        'targets_start': targets_start,
        'targets_end': targets_end,
        'orig_tweet': tweet,
        'orig_selected': selected_text,
        'sentiment': sentiment,
        'offsets': tweet_offsets
    }

class TweetDataset(data.Dataset):
    def __init__(self, tweet, sentiment, selected_text):
        self.tweet = tweet
        self.sentiment = sentiment
        self.selected_text = selected_text
        self.tokenizer = config.TOKENIZER
        self.max_len = config.MAX_LEN
    
    def __len__(self):
        return len(self.tweet)

    def __getitem__(self, item):
        data = process_data(
            self.tweet[item], 
            self.selected_text[item], 
            self.sentiment[item],
            self.tokenizer,
            self.max_len
        )

        return {
            'ids': torch.tensor(data["ids"], dtype=torch.long),
            'mask': torch.tensor(data["mask"], dtype=torch.long),
            'token_type_ids': torch.tensor(data["token_type_ids"], dtype=torch.long),
            'targets_start': torch.tensor(data["targets_start"], dtype=torch.long),
            'targets_end': torch.tensor(data["targets_end"], dtype=torch.long),
            'orig_tweet': data["orig_tweet"],
            'orig_selected': data["orig_selected"],
            'sentiment': data["sentiment"],
            'offsets': torch.tensor(data["offsets"], dtype=torch.long)
        }

dfx = pd.read_csv(config.TRAINING_FILE)
fold = 4
df_train = dfx[dfx.kfold != fold].reset_index(drop=True)
df_valid = dfx[dfx.kfold == fold].reset_index(drop=True)

train_dataset = TweetDataset(
        tweet=dfx.text.values,
        sentiment=dfx.sentiment.values,
        selected_text=dfx.selected_text.values
    )

train_data_loader = torch.utils.data.DataLoader(
        train_dataset,
        batch_size=config.TRAIN_BATCH_SIZE,
        num_workers=1
    )

if __name__ =='__main__':
    a = enumerate(train_data_loader)
```


## Expected behavior

The enumerate should return the iterable.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
Output of `transformers-cli env`

transformers version: 2.9.1
Platform: Windows-10-10.0.18362-SP0
Python version: 3.8.2
PyTorch version (GPU?): 1.5.0 (True)
Tensorflow version (GPU?): not installed (NA)
Using GPU in script?: Yes
Using distributed or parallel set-up in script?: No
If needed I can even provide the dataset (Did not want to clutter): 
the error stacktrace : 
```
Traceback (most recent call last):
  File "error.py", line 129, in <module>
    a = enumerate(train_data_loader)
  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\site-packages\torch\utils\data\dataloader.py", line 279, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\site-packages\torch\utils\data\dataloader.py", line 719, in __init__
    w.start()
  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\multiprocessing\context.py", line 224, in _Popen
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\multiprocessing\context.py", line 326, in _Popen
  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\multiprocessing\spawn.py", line 116, in spawn_main
    return Popen(process_obj)
exitcode = _main(fd, parent_sentinel)  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__

  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\multiprocessing\spawn.py", line 126, in _main
    reduction.dump(process_obj, to_child)
self = reduction.pickle.load(from_parent)  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\multiprocessing\reduction.py", line 60, in dump

EOFError: Ran out of input
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'Tokenizer' object
```
But when run in kaggle notebook this works perfectly well. (The same script having same tokenizer and transformers version)
@julien-c @sshleifer any help here?<|||||>~Yes, this is fixed by PR #4389 , so you could `pip install -e .` off of that branch.~<|||||>Hi @sshleifer I did these steps : 
1. git fetch origin pull/4389/head:temp_fix
2. git checkout temp_fix
3. pip install -e  .
still the above fix doesn't seem to work. 

By looking into the PR , I am guessing that it is fixed for `MarianTokenizer ` and not for `BertWordPieceTokenizer` that I am using in the above script.
 <|||||>Tested with another environment with python = 3.7.7, same issue is observed.<|||||>Just wanted to mention that providing `num_workers = 0` bypasses the problem. So it only fails when multiprocessing is involved. This issue is not only in `BertWordPieceTokenizer`, It also fails with `ByteLevelBPETokenizer` .<|||||>And this probably should be moved to the tokenizer repo @sshleifer to confirm.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
