Solved by sgugger in post #2 
              
              
                You can use the base BERT tokenizer I would say (since it’s a BERT model). Just make sure the pad token is compatible with what the model expects.