@sshleifer Thanks for the insights!!
I am now able to clear up most of the naming mismatches.
The last issue is that DPREncoder has a variable from BertModel class.
(which I already solved but seeking confirmation, see below)
And there is one important different implementation between
Pytorch’s BertModel vs. TFBertModel
While in TFBertModel we have one more nested class TFBertMainLayer (which initialize with name='bert' ), in Pytorch’s BertModel we have none of this MainLayer class, so the naming is inherently different.
In Pytorch name is like
ctx_encoder.bert_model.encoder.layer.3.attention.self.key.weight
and TF name has ‘extra’ bert inside due to the TFBertMainLayer class:
ctx_encoder.bert_model.bert.encoder.layer.3.attention.self.key.weight
===========
Solution
So the solution I made is to simply change from TFBertModel to TFBertMainLayer, and now all pytorch weights are correctly loading for ctx_encoder , question_encoder and reader . To progress in this TFDPR, I will open issue in Github seeking for PR suggestion .
Thank you again!