Not getting expected results for freshly-pretrained BERT 
# ❓ Questions & Help

I have a freshly trained BERT (using the original BERT) which doesn't seem to function as expected after moving it to PyTorch/Transformers. I don't have any syntax errors anywhere. So I am not sure how to go about understanding the bug. Wondering if anyone has suggestions for me. 
Below I am providing details on every single step of my training process.

Here is a comment from my training: 
```
2020-05-14 08:36:41,144 :  Saving checkpoints for 1875000 into gs:// .... /perbert_L-12_H-768_A-12/model.ckpt.
2020-05-14 08:37:07,703 :  loss = 1.7460283, step = 1875000 (1664.300 sec)
2020-05-14 08:37:07,704 :  global_step/sec: 15.0213
2020-05-14 08:37:07,704 :  examples/sec: 1922.73
2020-05-14 08:37:07,705 :  Enqueue next (25000) batch(es) of data to infeed.
2020-05-14 08:37:07,705 :  Dequeue next (25000) batch(es) of data from outfeed.
```
As you can see, the loss function is pretty low after being trained for about 180k steps. 

Now I had it converted with `convert_tf_checkpoint_to_pytorch`: 
```python
from pytorch_pretrained_bert.convert_tf_checkpoint_to_pytorch import convert_tf_checkpoint_to_pytorch

model = "perbert_L-12_H-768_A-12"
checkpoint = "1875000"

convert_tf_checkpoint_to_pytorch(
    f'/models/{model}/model.ckpt-{checkpoint}.index',
    bert_config_file=f'models/{model}/bert_config.json',
    pytorch_dump_path=f"models/{model}/saved_model_{checkpoint}")
```

Then I load the results in masked language model to see whether it does something reasonable or not: 
```python 
import torch
from transformers import BertConfig, BertTokenizer, BertForMaskedLM

dir = "/Users/danielk/ideaProjects/farsi-language-models/src/models/perbert_L-12_H-768_A-12/"
checkpoint = 1875000
tokenizer = BertTokenizer.from_pretrained(dir)
config = BertConfig.from_json_file(dir + '/bert_config.json')
model = BertForMaskedLM.from_pretrained(dir + '/saved_model_' + str(checkpoint), config=config)

string = "بزرگترین اقتصاد دنیا در حال فروپاشی موقت است. این امر مساله ای کاملا طبیعی است"
tokens = tokenizer.tokenize(string)
print("tokens: " + tokens)
print("indices: " +  tokenizer.encode(string))
input_ids = torch.tensor([tokenizer.encode(string)])
outputs = model(input_ids)

predictions = outputs[0]
for masked_index, _ in enumerate(tokens):
    print(" * input string: " + tokens[masked_index])
    predictions_numpy = predictions[0, masked_index].cpu().data.numpy()
    predictions_with_index = list(enumerate(predictions_numpy))
    predictions_with_index = sorted(predictions_with_index, key=lambda x: -x[1])
    top_indices = [x for x in predictions_with_index[:10]]
    predicted_tokens = tokenizer.convert_ids_to_tokens([x[0] for x in top_indices])
    print(predicted_tokens)
    print("...")
```
The output of the tokenization is correct; the tokens are reasonable and they're mapped to the right indices: 
```
tokens: ['بزرگترین', 'اقتصاد', 'دنیا', 'در', 'حال', 'فروپاشی', 'موقت', 'است', '[UNK]', 'این', 'امر', 'مساله', 'ای', 'کاملا', 'طبیعی', 'است']
indices: [2, 989, 499, 655, 6, 166, 16459, 3756, 13, 1, 11, 1312, 3703, 36, 1089, 1070, 13, 3]
```

However, the output of the masking is not as expected: 
```
 * input string: بزرگترین
['Plugin', '##AN', '##AZ', 'ورزشکاران', 'خانوارها', 'items', 'Moha', 'بکنید', '##Style', 'سهیلی']
...
 * input string: اقتصاد
['استان', 'آنها', 'نماد', 'বন', 'ویتنام', 'محكوم', '##جاری', '##sc', 'public', 'آلبرت']
...
 * input string: دنیا
['##Lang', 'public', 'آی', 'یافتهاند', 'check', 'یادتون', 'ضریب', 'رصدخانه', 'تزئینات', 'رف']
...
 * input string: در
['##Lang', 'Plugin', 'বন', '##BX', 'گرن', 'مغز', 'منجمد', 'فضل', 'سرمایشی', 'public']
...
 * input string: حال
['##Lang', 's', 'تمهیدات', 'خخخ', 'Product', 'حلزون', '##ten', '##تیپ', '##sc', 'آی']
...
 * input string: فروپاشی
['##Pack', 'বন', 'باشد', 'عجایب', '##File', '##Lang', '##column', 's', '##BG', '334']
...
 * input string: موقت
['بن', 'عاشورایی', 'تایباد', 'Vivo', 'بلوغ', '##dani', 'اشتیاق', '1351', '[UNUSED_194]', '##أفادت']
...
 * input string: است
['85', 'مشار', '##گرایان', '##جاری', 'while', '##cause', 'ژیل', 'مولد', 'نقلیه', 'رحم']
...
 * input string: [UNK]
['آنها', 'چسبندگی', 'سرمایشی', '##یس', 'کلی', 'آی', 'آزمایشی', '##گ', 'مولد', 'سینماهای']
...
 * input string: این
['اسپانیایی', '##زآپ', 'سهیلی', 'رسیدند', 'استان', 'بن', 'کریر', '##پد', 'تایباد', '##ث']
...
 * input string: امر
['ش', 'বন', '##آيند', 'اعتباری', 'نامرئی', '##ملی', 'public', 'اومد', 'یافتهاند', 'موعظه']
...
 * input string: مساله
['إيران', 'বন', 'public', '##سمح', 'wp', 'نماد', 'شئون', 'website', '##تیپ', '##اعتراض']
...
 * input string: ای
['مردمان', '[UNUSED_203]', 'خورده', 'public', 'لهم', 'آنها', 'کعبه', 'الهی', 'رصدخانه', 'سرمایشی']
...
 * input string: کاملا
['public', '77', 'مروج', 'تایباد', '##aghi', 'منصب', '##Lang', '[UNUSED_203]', '##198', 'مغز']
...
 * input string: طبیعی
['##Lang', 'Plugin', '##den', '##مفید', 'وزنه', 'دلهره', 'check', 'Vivo', 'Biz', 'خواهان']
...
 * input string: است
['ju', '##Lang', 'عقربه', 'استان', 'اثرانگشت', '##aghi', 'انزلی', 'چسبندگی', '##tis', 'فرماید']
...
```
The above output that the masked language model has returned is quite random. Clearly, something is off. 

I briefly inspected one of my tfrecord files, but they also seem to be reasonable; here is one example block: 
```json
{
  "features": {
    "feature": {
      "masked_lm_positions": {
        "int64List": {
          "value": [
            "2",
            "3",
            "5",
            "15",
            "23",
            "27",
            "33",
            "41",
            "55",
            "60",
            "72",
            "74",
            "87",
            "97",
            "103",
            "104",
            "106",
            "112",
            "0",
            "0"
          ]
        }
      },
      "masked_lm_ids": {
        "int64List": {
          "value": [
            "5601",
            "33",
            "4125",
            "12",
            "6",
            "11635",
            "9808",
            "178",
            "8",
            "287",
            "6",
            "3213",
            "6",
            "23",
            "8",
            "814",
            "613",
            "12466",
            "0",
            "0"
          ]
        }
      },
      "masked_lm_weights": {
        "floatList": {
          "value": [
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            0.0,
            0.0
          ]
        }
      },
      "segment_ids": {
        "int64List": {
          "value": [
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0"
          ]
        }
      },
      "input_ids": {
        "int64List": {
          "value": [
            "2",
            "2824",
            "4",
            "4",
            "6",
            "4",
            "599",
            "5388",
            "16",
            "16598",
            "4513",
            "5",
            "2824",
            "2219",
            "217",
            "5891",
            "7",
            "3117",
            "4710",
            "24069",
            "16",
            "69",
            "5",
            "4",
            "127",
            "166",
            "3386",
            "4",
            "178",
            "13",
            "1",
            "278",
            "26085",
            "5791",
            "5",
            "1401",
            "1801",
            "5",
            "2348",
            "127",
            "905",
            "4",
            "13",
            "1",
            "2824",
            "1127",
            "8447",
            "7144",
            "18646",
            "5",
            "168",
            "14763",
            "9",
            "6",
            "7103",
            "4",
            "3878",
            "1978",
            "5",
            "430",
            "287",
            "1",
            "3",
            "12466",
            "23",
            "814",
            "6",
            "14259",
            "6",
            "4426",
            "46",
            "453",
            "4",
            "707",
            "4",
            "46",
            "6",
            "4426",
            "365",
            "338",
            "7935",
            "9",
            "6",
            "748",
            "6",
            "4426",
            "44",
            "4",
            "476",
            "1",
            "1",
            "7",
            "12466",
            "10",
            "12053",
            "1070",
            "6",
            "4",
            "1832",
            "44",
            "8012",
            "7968",
            "46",
            "4",
            "4",
            "49",
            "4",
            "287",
            "14",
            "6",
            "14259",
            "707",
            "4",
            "1",
            "1348",
            "44",
            "6849",
            "62",
            "6931",
            "31",
            "1",
            "3",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0"
          ]
        }
      },
      "input_mask": {
        "int64List": {
          "value": [
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0"
          ]
        }
      },
      "next_sentence_labels": {
        "int64List": {
          "value": [
            "1"
          ]
        }
      }
    }
  }
}
```

Here are some thoughts: 
 - Could the model conversion somehow go wrong without raising any errors? 
 - Persian is a right-to-left language; could that be a factor? In theory, it should not be an issue, since from the perspective of the either code (TF and PyTorch) these are just strings/byres sequenced after each other. 




This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
