Order of tokens in vocabulary of German model
The vocabulary for the German model ('bert-base-german-cased') has the token '[unused3001]' at position 0 (and the '[PAD]' token at position 1). However, the BertEmbedding has padding_idx=0 as usual.

Is this behaviour intended and if so would it be possible to  get some insight into the rationale behind it?

@tholor and @timoeller may have some insights on these<|||||>Hey Sebastian, thanks for using the German Bert and digging into its details. The mysterious [unused3001] token was actually a special comma symbol to get rid of [UNK] tokens in some of our training texts. But we covered it up later on in the process + didn't anticipate it would be coming back to us : ) 

So agreed, it is unwanted behaviour. 

Though TL;DR, we don't believe it is impacting either pretraining or downstream task training. 



Apparently the token at index 0 (= [unused3001]) is used as padding token in TF Bert and pytorch Bert and the implementations do not really care if it is called [unused3001] [PAD] or [something].

To be a bit more intuitive we now swapped [unused3001] and [PAD] in the vocab files (pytorch and TF) only. 
Might be that future code somehow substitutes "[PAD]" input strings, which could cause problems.

The only thing that seems worrisome to us is that the embedding values for this padding token are non-zero (and change over the course of training) for our German Bert but also for Googles open sourced models. 

I tried to check how the padding embedding is handled in TF but am not familiar with debugging there...

Maybe you want to dig more into it and raise an issue in the original TF Bert repro? Maybe this closed issue could be related to a rather unwanted padding embedding handling: https://github.com/google-research/bert/issues/113

Hope that helps, good luck!<|||||>Thanks a lot for the input @Timoeller (not quite sure who Christian is, though ;) ). I also got the feeling that it doesn't really impact downstream applications (NER in this case). At least not heavily.

I'll do some more experiments and raise an issue with the original repo if it feel it is warranted. 

Thanks again and all the best,
Sebastian

<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
