MRPC hyperparameters question
When describing how you reproduced the MRPC results, you say:
"Our test ran on a few seeds with the original implementation hyper-parameters gave evaluation results between 82 and 87."
and you link to the SQuAD hyperparameters (https://github.com/google-research/bert#squad).

Is the link a mistake? Or did you use the SQuAD hyperparameters for tuning on MRPC? More generally, I'm wondering if there's a reason the MRPC dev set accuracy is slightly lower (in [82, 87] vs. [84, 88] reported by Google)
Hi Ethan,
Thanks we used the MRPC hyper-parameters indeed, I corrected the README.
Regarding the dev set accuracy, I am not really surprised there is a slightly lower accuracy with the PyTorch version (even though the variance is high so it's hard to get something significant). That is something that is generally observed (see for example [the work of Remi Cadene](https://github.com/Cadene/pretrained-models.pytorch)) and we also experienced that with [our TF->PT port of the OpenAI GPT model](https://github.com/huggingface/pytorch-openai-transformer-lm).
My personal feeling is that there are slight differences in the way the backends of TensorFlow and PyTorch handle the operations and these differences make the pre-trained weights sub-optimal for PyTorch. <|||||>Great, thanks for clarifying that. Regarding the slightly lower accuracy, that makes sense. Thanks for your help and for releasing this!<|||||>Maybe it would help to train the Tensorflow pre-trained weights for e.g. one epoch in PyTorch (using the MLM and next-sentence objective)? That may help transfer to other tasks, depending on what the issue is<|||||>Hi @ethanjperez, actually the weight initialization fix (`tf. truncated_normal_initializer(stddev=0.02)` was translated in `weight.data.normal_(0.02)` instead of `weight.data.normal_(mean=0.0, std=0.02)` fixed in 2a97fe22) has brought us back to the TensorFlow results on MRPC (between 84 and 88%).
I am closing this issue.<|||||>@thomwolf Great to hear - thanks for working to fix it!