pie3636:

transformers/modeling_bert.py


This is surprising, can you provide a smaller repro so that we can investigate this further, something like this snippet alone:
   if inputs_embeds is None:
        inputs_embeds = self.word_embeddings(input_ids)
    token_type_embeddings = self.token_type_embeddings(token_type_ids)

    embeddings = inputs_embeds + token_type_embeddings