Performance dramatically drops down after replacing pytorch-pretrained-bert with pytorch-transformers
I am trying to run a baseline model, whose encoder is the pretrained BERT ('bert-base-uncased'). I have tried both versions of this package and found that the performance of the pytorch-transformers-BERT is much worse than pytorch-pretrained-bert-BERT, i.e. the BLeU-4 has dropped from 8. to 2.
Below is my codes, I wanna see if there is some important difference between the two versions that will lead to the drop, or it's the wrong way to call the functions in my codes that causes the bad performance.

(1) pytorch-pretrained-bert:
```python
from pytorch_pretrained_bert import BertModel

pretrained = BertModel.from_pretrained('bert-base-uncased')

enc_outputs, *_ = pretrained(src_seq, token_type_ids=src_sep, output_all_encoded_layers=True)
enc_output = enc_outputs[-1]
```

(2) pytorch-transformers
```python
from pytorch_transformers import BertModel, BertConfig

config = BertConfig.from_pretrained('bert-base-uncased')
config.output_hidden_states = True
pretrained = BertModel(config)

enc_outputs = pretrained(src_seq, token_type_ids=src_sep)
enc_output = enc_outputs[0]
enc_outputs = enc_outputs[2][1:]
```
Same here. I am finetuning language models on new dataset. Once I change from `pytorch-pretrained-bert` to `pytorch-transformers`, generation quality dramatically drops.  <|||||>I  have the same problem. I refer to a example of Named Entity Recognition which used pytorch-pretrained-bert. I changed it to pytorch-transformers, but I got a bad F1 score. It's suppsed to be  0.78, I got 0.41.  <|||||>I'm also seeing similar problems after the refactoring related to BertForMultipleChoice models (issue here: https://github.com/huggingface/pytorch-transformers/issues/931) <|||||>Similar issue here. Working on custom adaptation of BERT for STS benchmark dataset. Spearman correlation drops by about 2 points (.78 -> .76) after refactoring 0.6.1 to 1.0.0, even though all parameters are the same. If I reload my old models, I still get the old (higher) scores.

I suspect that this might be due to a different linear warmup function used in 0.6.1 (compared to 0.6.2 and 1.0.0), that returns smaller learning rates.<|||||>I think these differences originate from different modifications so it's not really possible to have all of them in one issue like here with no specific description of the setup and condition of each of you.

I've set up templates for the issues to incite people to give more information.

Please re-open separate issues with more details on each setup.

In particular, there is a template called "MIGRATION" which is specifically concerned with giving information on migration issues from pytorch-pretrained-bert.

In the meantime, I will close this issue.<|||||>@YuxiXie @dykang @teng1996 Any updates on this? 