Add German Bert model to code, update readme
We have been training a German BERT model from scratch on some 12 GB of clean text. It outperforms the multilingual BERT (cased + uncased) in 4 out of 5 German NLP tasks.
![deepset_performance](https://user-images.githubusercontent.com/3264870/59521557-d9cbde80-8ecc-11e9-87cf-8af9f141201c.png)

Furthermore we evaluated the pre-training not just by observing the train loss, but continuous downstream task checks.
For more details on our experiments you can check our official post here: https://deepset.ai/german-bert

Code-wise we only had to do very few adoptions to the dictionaries for model, vocab and sequence length. We also included our evaluation post into the README, because we believe it might be interesting for the community.
This looks great @Timoeller – do you have an estimate for the compute power you used to train your model?

UPDATE. Ok the answer is in the blogpost: https://deepset.ai/german-bert
> We trained using Google's Tensorflow code on a single cloud TPU v2 with standard settings. 
> We trained 840k steps with a batch size of 1024 for sequence length 128. Training took about 9 days. 
<|||||>Sorry, I just realized we made a wrong oversimplification. We of course trained in the end for 30k steps on a longer batch size.
I updated the article accordingly: We trained 810k steps with a batch size of 1024 for sequence length 128 and 30k steps with sequence length 512. Training took about 9 days. <|||||>Looks great, thanks a lot @Timoeller!<|||||>@Timoeller (and @tholor also I guess): in the coming release 0.6.3, I'm switching to a split file format for Bert (like already done in GPT/GPT-2/Transformer-XL) in which we separately store config and weights files on the S3 to avoid having to untar an archive at each instantiation of the model.

In the short term I'll be storing your model's files on our s3 but you can also split the archive yourself and I can switch back to your s3 if you would like to.<|||||>Correctly guessed. @tholor and me are working together. I also created another PR for the updated file locations.<|||||>Hello, could you please share how did you generate the vocab list? <|||||>This is the code how we did it. There was a special "," symbol at index 0, which got used unintentionally as padding token. So we swapped the first two strings in the vocab.txt and created a "[unused3001]". See also the discussion in: https://github.com/huggingface/pytorch-transformers/issues/778

Hope that helps.

```

spm.SentencePieceTrainer.Train(
    f'--input={INPUT_FILE} --model_prefix={TEMP_FILE} --vocab_size={VOCAB_SIZE} --character_coverage=1.0 --model_type=bpe')

df = pd.read_csv(TEMP_FILE + ".vocab", sep="\t",  # use a char for separation that cannot be inside vocab
                 header=None, names=["vocab", "unk"],
                 encoding='utf-8', dtype=str, quotechar="\r",  # use a char for quoting that cannot be inside vocab
                 engine='python')
vocab = df.vocab.values
print(vocab.shape)

print(len(vocab))
for i, current in enumerate(vocab):
    current = str(current)
    if current.startswith("▁"):
        vocab[i] = current[1:]
    else:
        vocab[i] = "##" + current

unused = []

for i in range(1, UNUSED_TOKENS + 1):
    unused.append("[unused%i]" % i)
toadd = np.array(["[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"])
vocab = np.concatenate((toadd, vocab[3:], unused), axis=0)
```<|||||>@Timoeller thanks for the open sourcing the model :+1: and the great [FARM](https://github.com/deepset-ai/FARM) library.

I trained a German BERT model from scratch a while ago (16GB of text, incl. some WMT monolingual data for German) and here are some preliminary results:

| Task | Result
| --------- | -------
| CoNLL-2003 | 85.49
| GermEval | 84.38
| GermEval18Coarse | 74.60 (reproduced result for German BERT was 74.06)

So on average the model is +0.48% better. My question to @Timoeller and @thomwolf does it make sense to include another cased German BERT model 🤔
<|||||>Hey @stefan-it 
Thanks for linking our library and also sharing your results. We have made the experience that multiple downstream runs vary in performance by some degree. We are also in contact with people applying German Bert for germeval19, using an ensemble of multiple downstream runs with quite good results (the official ranking isnt out yet though). 

Concerning the performance of your Bert model: Your NER results seem to be consistently better than with our German Bert. Maybe it lacks behind in the other tasks? 

How about we both have a call this week to see where the differences of our Berts come from and if it makes sense to include your model as well? Just pm me. <|||||>Awesome, I just send you an email :)