When I try to load the model, which was trained with an older version of Transformers, now in my env with Transformers 4.1, I get the following error:
'BertEmbeddings' object has no attribute 'position_embedding_type'
I had similar issues when we upgraded from Transformers 3 to 4, attributes that were missing.