Solved by nielsr in post #2 
              
              
                Hi, 
As you can see here, if you provide inputs_embeds yourself, they will only be used to replace the token embeddings. The token type and position embeddings will be added separately.