'LayerNorm' object has no attribute 'cls'
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I try to use `load_tf_weights_in_bert` to convert my fine-tuned tf classification model in Pytorch. I original trained the model by tensorflow BERT.
I used this code:
```import torch
from transformers.modeling_bert import BertConfig, BertForPreTraining, load_tf_weights_in_bert, BertForSequenceClassification

tf_checkpoint_path="./model.ckpt-98400"
bert_config_file = "./bert_config.json"
pytorch_dump_path="pytorch_bert"

config = BertConfig.from_json_file(bert_config_file)
config.num_labels = 21

print("Building PyTorch model from configuration: {}".format(str(config)))
model = BertForSequenceClassification(config)

# Load weights from tf checkpoint
load_tf_weights_in_bert(model, config, tf_checkpoint_path)

# Save pytorch-model
print("Save PyTorch model to {}".format(pytorch_dump_path))
torch.save(model.state_dict(), pytorch_dump_path)
```
I noticed  this method to solve my problem:  https://github.com/huggingface/transformers/issues/580#issuecomment-489519231

 I add to line in `modeling_bert.py`.

```          elif l[0] == 'output_bias' or l[0] == 'beta':
                pointer = getattr(pointer, 'cls')
                pointer = getattr(pointer, 'bias')
            elif l[0] == 'output_weights':
                pointer = getattr(pointer, 'cls')
                pointer = getattr(pointer, 'weight')
```
But I still get error `AttributeError: 'LayerNorm' object has no attribute 'cls'`

```
Building PyTorch model from configuration: {
  "attention_probs_dropout_prob": 0.1,
  "directionality": "bidi",
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_labels": 21,
  "output_attentions": false,
  "output_hidden_states": false,
  "pooler_fc_size": 768,
  "pooler_num_attention_heads": 12,
  "pooler_num_fc_layers": 3,
  "pooler_size_per_head": 128,
  "pooler_type": "first_token_transform",
  "pruned_heads": {},
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 119547
}

I1009 15:47:03.337315 47631408520768 modeling_bert.py:65] Converting TensorFlow checkpoint from ./model.ckpt-98400
I1009 15:47:03.344796 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/LayerNorm/beta with shape [768]
I1009 15:47:03.350771 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:03.356130 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:03.361214 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/LayerNorm/gamma with shape [768]
I1009 15:47:03.366278 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:03.371291 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:03.376359 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/position_embeddings with shape [512, 768]
I1009 15:47:03.383015 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/position_embeddings/adam_m with shape [512, 768]
I1009 15:47:03.388718 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/position_embeddings/adam_v with shape [512, 768]
I1009 15:47:03.394378 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/token_type_embeddings with shape [2, 768]
I1009 15:47:03.400012 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/token_type_embeddings/adam_m with shape [2, 768]
I1009 15:47:03.405249 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/token_type_embeddings/adam_v with shape [2, 768]
I1009 15:47:03.410350 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/word_embeddings with shape [119547, 768]
I1009 15:47:03.575059 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/word_embeddings/adam_m with shape [119547, 768]
I1009 15:47:03.743357 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/word_embeddings/adam_v with shape [119547, 768]
I1009 15:47:03.908991 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:03.915453 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:03.921177 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:03.926633 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:03.932333 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:03.937757 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:03.942878 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/dense/bias with shape [768]
I1009 15:47:03.947972 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:03.953052 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:03.958150 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:03.964268 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:03.970259 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:03.976348 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/key/bias with shape [768]
I1009 15:47:03.981755 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:03.986970 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:03.992337 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/key/kernel with shape [768, 768]
I1009 15:47:03.998313 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:04.004308 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:04.010322 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/query/bias with shape [768]
I1009 15:47:04.015577 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:04.020884 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:04.026228 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/query/kernel with shape [768, 768]
I1009 15:47:04.032151 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:04.038157 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:04.044193 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/value/bias with shape [768]
I1009 15:47:04.049786 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:04.055586 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:04.060960 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/value/kernel with shape [768, 768]
I1009 15:47:04.067193 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:04.073462 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:04.079773 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/intermediate/dense/bias with shape [3072]
I1009 15:47:04.084890 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:04.090381 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:04.096321 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:04.105394 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:04.114429 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:04.123474 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta with shape [768]
I1009 15:47:04.128776 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:04.133818 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:04.138939 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma with shape [768]
I1009 15:47:04.144054 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:04.149456 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:04.154736 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/dense/bias with shape [768]
I1009 15:47:04.159813 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/dense/bias/adam_m with shape [768]
I1009 15:47:04.165272 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/dense/bias/adam_v with shape [768]
I1009 15:47:04.170530 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/dense/kernel with shape [3072, 768]
I1009 15:47:04.179566 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:04.188571 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:04.197654 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:04.203165 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:04.208374 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:04.213510 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:04.218636 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:04.223767 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:04.229122 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/dense/bias with shape [768]
I1009 15:47:04.234088 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:04.239236 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:04.244547 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:04.250508 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:04.256635 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:04.262910 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/key/bias with shape [768]
I1009 15:47:04.268387 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:04.273391 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:04.278625 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/key/kernel with shape [768, 768]
I1009 15:47:04.284610 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:04.290718 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:04.296881 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/query/bias with shape [768]
I1009 15:47:04.302387 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:04.307479 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:04.312391 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/query/kernel with shape [768, 768]
I1009 15:47:04.318128 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:04.324467 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:04.330887 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/value/bias with shape [768]
I1009 15:47:04.336219 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:04.341366 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:04.346387 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/value/kernel with shape [768, 768]
I1009 15:47:04.352337 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:04.358580 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:04.365348 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/intermediate/dense/bias with shape [3072]
I1009 15:47:04.371170 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:04.376696 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:04.381763 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:04.390794 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:04.400085 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:04.409393 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta with shape [768]
I1009 15:47:04.414902 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:04.420410 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:04.425521 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma with shape [768]
I1009 15:47:04.430895 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:04.436064 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:04.441035 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/dense/bias with shape [768]
I1009 15:47:04.446047 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/dense/bias/adam_m with shape [768]
I1009 15:47:04.451446 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/dense/bias/adam_v with shape [768]
I1009 15:47:04.456856 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/dense/kernel with shape [3072, 768]
I1009 15:47:04.465820 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:04.474748 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:04.484099 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:04.489739 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:04.495579 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:04.500854 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:04.506203 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:04.511337 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:04.516572 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/dense/bias with shape [768]
I1009 15:47:04.521539 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:04.526908 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:04.532043 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:04.538394 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:04.544577 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:04.550608 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/key/bias with shape [768]
I1009 15:47:04.555929 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:04.561278 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:04.566563 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/key/kernel with shape [768, 768]
I1009 15:47:04.573011 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:04.579270 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:04.585390 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/query/bias with shape [768]
I1009 15:47:04.590508 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:04.595518 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:04.601018 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/query/kernel with shape [768, 768]
I1009 15:47:04.607660 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:04.614161 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:04.620493 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/value/bias with shape [768]
I1009 15:47:04.625949 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:04.631571 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:04.637210 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/value/kernel with shape [768, 768]
I1009 15:47:04.643343 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:04.649336 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:04.655313 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/intermediate/dense/bias with shape [3072]
I1009 15:47:04.660685 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:04.666079 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:04.671396 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:04.680429 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:04.689695 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:04.698902 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta with shape [768]
I1009 15:47:04.704335 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:04.710071 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:04.715506 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma with shape [768]
I1009 15:47:04.720938 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:04.726950 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:04.732291 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/dense/bias with shape [768]
I1009 15:47:04.737487 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/dense/bias/adam_m with shape [768]
I1009 15:47:04.742859 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/dense/bias/adam_v with shape [768]
I1009 15:47:04.748318 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/dense/kernel with shape [3072, 768]
I1009 15:47:04.757482 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:04.766298 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:04.775268 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:04.780487 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:04.785785 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:04.791079 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:04.796396 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:04.801618 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:04.806764 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/dense/bias with shape [768]
I1009 15:47:04.811792 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:04.817137 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:04.822446 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:04.828487 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:04.834693 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:04.840850 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/key/bias with shape [768]
I1009 15:47:04.846439 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:04.851520 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:04.856792 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/key/kernel with shape [768, 768]
I1009 15:47:04.862816 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:04.869044 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:04.875606 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/query/bias with shape [768]
I1009 15:47:04.880790 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:04.886349 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:04.892000 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/query/kernel with shape [768, 768]
I1009 15:47:04.898570 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:04.904909 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:04.911062 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/value/bias with shape [768]
I1009 15:47:04.916296 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:04.921895 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:04.927209 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/value/kernel with shape [768, 768]
I1009 15:47:04.933332 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:04.939111 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:04.945370 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/intermediate/dense/bias with shape [3072]
I1009 15:47:04.950793 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:04.955806 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:04.961072 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:04.970463 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:04.979833 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:04.989530 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta with shape [768]
I1009 15:47:04.995094 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:05.000446 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:05.005793 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma with shape [768]
I1009 15:47:05.011140 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:05.016781 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:05.022187 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/dense/bias with shape [768]
I1009 15:47:05.027360 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/dense/bias/adam_m with shape [768]
I1009 15:47:05.032415 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/dense/bias/adam_v with shape [768]
I1009 15:47:05.037586 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/dense/kernel with shape [3072, 768]
I1009 15:47:05.046856 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:05.055976 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:05.065228 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:05.070782 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:05.075919 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:05.081003 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:05.086196 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:05.091385 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:05.096379 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/dense/bias with shape [768]
I1009 15:47:05.101588 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:05.106899 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:05.112151 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:05.117895 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:05.123686 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:05.129775 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/key/bias with shape [768]
I1009 15:47:05.135063 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:05.140272 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:05.145637 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/key/kernel with shape [768, 768]
I1009 15:47:05.151918 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:05.158202 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:05.164404 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/query/bias with shape [768]
I1009 15:47:05.170278 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:05.175527 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:05.180814 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/query/kernel with shape [768, 768]
I1009 15:47:05.187093 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:05.193276 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:05.199446 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/value/bias with shape [768]
I1009 15:47:05.204873 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:05.210621 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:05.215833 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/value/kernel with shape [768, 768]
I1009 15:47:05.221964 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:05.228211 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:05.234716 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/intermediate/dense/bias with shape [3072]
I1009 15:47:05.240120 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:05.245725 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:05.250889 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:05.260031 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:05.269109 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:05.278731 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta with shape [768]
I1009 15:47:05.283982 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:05.289493 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:05.294659 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma with shape [768]
I1009 15:47:05.299949 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:05.305051 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:05.310529 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/dense/bias with shape [768]
I1009 15:47:05.315993 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/dense/bias/adam_m with shape [768]
I1009 15:47:05.321487 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/dense/bias/adam_v with shape [768]
I1009 15:47:05.326727 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/dense/kernel with shape [3072, 768]
I1009 15:47:05.335873 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:05.345036 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:05.354362 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:05.359932 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:05.365148 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:05.370391 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:05.375550 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:05.380681 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:05.385793 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/dense/bias with shape [768]
I1009 15:47:05.390934 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:05.396157 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:05.401309 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:05.407191 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:05.413287 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:05.419596 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/key/bias with shape [768]
I1009 15:47:05.424950 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:05.430454 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:05.435939 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/key/kernel with shape [768, 768]
I1009 15:47:05.441898 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:05.448148 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:05.454164 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/query/bias with shape [768]
I1009 15:47:05.459583 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:05.465055 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:05.470114 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/query/kernel with shape [768, 768]
I1009 15:47:05.476166 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:05.482553 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:05.489023 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/value/bias with shape [768]
I1009 15:47:05.494502 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:05.500063 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:05.505194 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/value/kernel with shape [768, 768]
I1009 15:47:05.511651 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:05.517767 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:05.524090 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/intermediate/dense/bias with shape [3072]
I1009 15:47:05.529507 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:05.534897 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:05.540130 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:05.549204 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:05.558589 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:05.568243 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta with shape [768]
I1009 15:47:05.573777 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:05.578975 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:05.584223 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma with shape [768]
I1009 15:47:05.589388 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:05.594596 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:05.600200 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/dense/bias with shape [768]
I1009 15:47:05.605475 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/dense/bias/adam_m with shape [768]
I1009 15:47:05.610627 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/dense/bias/adam_v with shape [768]
I1009 15:47:05.616336 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/dense/kernel with shape [3072, 768]
I1009 15:47:05.625247 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:05.634833 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:05.644434 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:05.649833 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:05.655650 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:05.660920 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:05.666323 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:05.671495 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:05.676601 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/dense/bias with shape [768]
I1009 15:47:05.681863 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:05.686859 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:05.692172 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:05.698084 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:05.703886 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:05.709926 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/key/bias with shape [768]
I1009 15:47:05.715487 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:05.720532 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:05.725626 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/key/kernel with shape [768, 768]
I1009 15:47:05.731551 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:05.737704 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:05.743711 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/query/bias with shape [768]
I1009 15:47:05.748904 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:05.754198 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:05.759485 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/query/kernel with shape [768, 768]
I1009 15:47:05.765789 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:05.772267 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:05.778261 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/value/bias with shape [768]
I1009 15:47:05.783635 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:05.788740 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:05.794108 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/value/kernel with shape [768, 768]
I1009 15:47:05.800311 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:05.806456 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:05.812658 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/intermediate/dense/bias with shape [3072]
I1009 15:47:05.818162 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:05.823700 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:05.828876 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:05.837978 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:05.847428 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:05.856863 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta with shape [768]
I1009 15:47:05.862351 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:05.867667 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:05.873092 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma with shape [768]
I1009 15:47:05.878434 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:05.883496 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:05.888758 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/dense/bias with shape [768]
I1009 15:47:05.894265 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/dense/bias/adam_m with shape [768]
I1009 15:47:05.899498 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/dense/bias/adam_v with shape [768]
I1009 15:47:05.904521 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/dense/kernel with shape [3072, 768]
I1009 15:47:05.913688 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:05.922709 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:05.931876 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:05.937582 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:05.942691 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:05.947911 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:05.953302 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:05.958757 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:05.963871 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/dense/bias with shape [768]
I1009 15:47:05.969243 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:05.974361 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:05.979577 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:05.985836 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:05.991879 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:05.998053 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/key/bias with shape [768]
I1009 15:47:06.003427 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:06.008820 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:06.014085 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/key/kernel with shape [768, 768]
I1009 15:47:06.020233 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:06.026285 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:06.032388 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/query/bias with shape [768]
I1009 15:47:06.038056 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:06.043298 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:06.048563 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/query/kernel with shape [768, 768]
I1009 15:47:06.054869 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:06.061104 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:06.067418 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/value/bias with shape [768]
I1009 15:47:06.072948 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:06.078219 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:06.083603 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/value/kernel with shape [768, 768]
I1009 15:47:06.090325 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:06.096712 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:06.102761 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/intermediate/dense/bias with shape [3072]
I1009 15:47:06.108000 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:06.113360 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:06.118700 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:06.128049 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:06.137369 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:06.146910 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta with shape [768]
I1009 15:47:06.152923 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:06.158300 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:06.163956 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma with shape [768]
I1009 15:47:06.169229 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:06.174710 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:06.179936 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/dense/bias with shape [768]
I1009 15:47:06.185377 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/dense/bias/adam_m with shape [768]
I1009 15:47:06.190515 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/dense/bias/adam_v with shape [768]
I1009 15:47:06.196292 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/dense/kernel with shape [3072, 768]
I1009 15:47:06.205745 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:06.215335 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:06.224854 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:06.230671 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:06.235839 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:06.241382 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:06.246639 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:06.251890 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:06.257052 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/dense/bias with shape [768]
I1009 15:47:06.262314 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:06.267627 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:06.272980 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:06.279123 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:06.285258 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:06.291564 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/key/bias with shape [768]
I1009 15:47:06.296821 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:06.302075 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:06.307455 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/key/kernel with shape [768, 768]
I1009 15:47:06.313520 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:06.319566 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:06.325647 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/query/bias with shape [768]
I1009 15:47:06.331008 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:06.336525 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:06.342104 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/query/kernel with shape [768, 768]
I1009 15:47:06.348503 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:06.354544 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:06.361086 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/value/bias with shape [768]
I1009 15:47:06.366425 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:06.371979 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:06.377128 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/value/kernel with shape [768, 768]
I1009 15:47:06.383063 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:06.389229 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:06.395555 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/intermediate/dense/bias with shape [3072]
I1009 15:47:06.401404 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:06.406746 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:06.411887 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:06.420855 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:06.430287 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:06.439881 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta with shape [768]
I1009 15:47:06.445644 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:06.450891 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:06.456028 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma with shape [768]
I1009 15:47:06.461563 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:06.467115 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:06.472524 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/dense/bias with shape [768]
I1009 15:47:06.478100 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/dense/bias/adam_m with shape [768]
I1009 15:47:06.483274 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/dense/bias/adam_v with shape [768]
I1009 15:47:06.488587 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/dense/kernel with shape [3072, 768]
I1009 15:47:06.497782 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:06.506931 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:06.516527 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:06.522197 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:06.527507 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:06.532787 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:06.538027 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:06.543310 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:06.548686 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/dense/bias with shape [768]
I1009 15:47:06.553850 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:06.559104 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:06.564384 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:06.570770 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:06.576959 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:06.583284 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/key/bias with shape [768]
I1009 15:47:06.588658 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:06.593865 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:06.598938 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/key/kernel with shape [768, 768]
I1009 15:47:06.604933 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:06.611174 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:06.617308 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/query/bias with shape [768]
I1009 15:47:06.622585 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:06.627687 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:06.633019 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/query/kernel with shape [768, 768]
I1009 15:47:06.639252 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:06.645524 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:06.651858 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/value/bias with shape [768]
I1009 15:47:06.657348 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:06.662594 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:06.667656 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/value/kernel with shape [768, 768]
I1009 15:47:06.673909 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:06.680046 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:06.686088 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/intermediate/dense/bias with shape [3072]
I1009 15:47:06.691427 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:06.696527 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:06.701802 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:06.710916 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:06.719931 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:06.729498 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta with shape [768]
I1009 15:47:06.734817 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:06.739971 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:06.745015 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma with shape [768]
I1009 15:47:06.750005 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:06.755349 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:06.760453 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/dense/bias with shape [768]
I1009 15:47:06.765552 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/dense/bias/adam_m with shape [768]
I1009 15:47:06.770809 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/dense/bias/adam_v with shape [768]
I1009 15:47:06.775943 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/dense/kernel with shape [3072, 768]
I1009 15:47:06.785551 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:06.795017 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:06.804197 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:06.809925 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:06.815610 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:06.820969 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:06.825976 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:06.831141 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:06.836268 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/dense/bias with shape [768]
I1009 15:47:06.841736 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:06.847084 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:06.852262 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:06.858251 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:06.864098 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:06.870030 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/key/bias with shape [768]
I1009 15:47:06.875577 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:06.881125 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:06.886339 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/key/kernel with shape [768, 768]
I1009 15:47:06.892580 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:06.898903 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:06.905303 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/query/bias with shape [768]
I1009 15:47:06.910978 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:06.916463 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:06.921678 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/query/kernel with shape [768, 768]
I1009 15:47:06.927710 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:06.933935 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:06.940702 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/value/bias with shape [768]
I1009 15:47:06.946018 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:06.951382 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:06.957077 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/value/kernel with shape [768, 768]
I1009 15:47:06.963493 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:06.969939 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:06.976311 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/intermediate/dense/bias with shape [3072]
I1009 15:47:06.982092 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:06.987428 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:06.992400 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:07.001886 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:07.011502 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:07.020718 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta with shape [768]
I1009 15:47:07.026179 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:07.031653 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:07.037088 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma with shape [768]
I1009 15:47:07.042681 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:07.047634 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:07.052863 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/dense/bias with shape [768]
I1009 15:47:07.058085 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/dense/bias/adam_m with shape [768]
I1009 15:47:07.063718 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/dense/bias/adam_v with shape [768]
I1009 15:47:07.069069 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/dense/kernel with shape [3072, 768]
I1009 15:47:07.078701 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:07.088213 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:07.097591 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:07.103389 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:07.109018 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:07.114469 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:07.119768 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:07.125015 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:07.130143 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/dense/bias with shape [768]
I1009 15:47:07.135096 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:07.140460 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:07.145726 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:07.151650 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:07.157860 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:07.163643 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/key/bias with shape [768]
I1009 15:47:07.168712 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:07.174322 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:07.179837 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/key/kernel with shape [768, 768]
I1009 15:47:07.185978 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:07.192014 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:07.198294 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/query/bias with shape [768]
I1009 15:47:07.203473 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:07.209064 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:07.214386 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/query/kernel with shape [768, 768]
I1009 15:47:07.220609 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:07.226944 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:07.233198 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/value/bias with shape [768]
I1009 15:47:07.238353 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:07.243831 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:07.249445 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/value/kernel with shape [768, 768]
I1009 15:47:07.255527 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:07.261869 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:07.268050 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/intermediate/dense/bias with shape [3072]
I1009 15:47:07.273308 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:07.278274 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:07.283823 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:07.293444 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:07.302655 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:07.312628 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta with shape [768]
I1009 15:47:07.318495 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:07.323935 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:07.329243 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma with shape [768]
I1009 15:47:07.334521 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:07.339932 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:07.345828 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/dense/bias with shape [768]
I1009 15:47:07.351069 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/dense/bias/adam_m with shape [768]
I1009 15:47:07.356699 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/dense/bias/adam_v with shape [768]
I1009 15:47:07.362353 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/dense/kernel with shape [3072, 768]
I1009 15:47:07.371929 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:07.381705 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:07.390963 47631408520768 modeling_bert.py:71] Loading TF weight bert/pooler/dense/bias with shape [768]
I1009 15:47:07.396645 47631408520768 modeling_bert.py:71] Loading TF weight bert/pooler/dense/bias/adam_m with shape [768]
I1009 15:47:07.401854 47631408520768 modeling_bert.py:71] Loading TF weight bert/pooler/dense/bias/adam_v with shape [768]
I1009 15:47:07.406987 47631408520768 modeling_bert.py:71] Loading TF weight bert/pooler/dense/kernel with shape [768, 768]
I1009 15:47:07.412913 47631408520768 modeling_bert.py:71] Loading TF weight bert/pooler/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:07.419250 47631408520768 modeling_bert.py:71] Loading TF weight bert/pooler/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:07.425417 47631408520768 modeling_bert.py:71] Loading TF weight global_step with shape []
I1009 15:47:07.430413 47631408520768 modeling_bert.py:71] Loading TF weight output_bias with shape [21]
I1009 15:47:07.435464 47631408520768 modeling_bert.py:71] Loading TF weight output_bias/adam_m with shape [21]
I1009 15:47:07.440396 47631408520768 modeling_bert.py:71] Loading TF weight output_bias/adam_v with shape [21]
I1009 15:47:07.445353 47631408520768 modeling_bert.py:71] Loading TF weight output_weights with shape [21, 768]
I1009 15:47:07.450436 47631408520768 modeling_bert.py:71] Loading TF weight output_weights/adam_m with shape [21, 768]
I1009 15:47:07.455591 47631408520768 modeling_bert.py:71] Loading TF weight output_weights/adam_v with shape [21, 768]
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-34-7cd6093dcd13> in <module>
     15 
     16 # Load weights from tf checkpoint
---> 17 load_tf_weights_in_bert(model, config, tf_checkpoint_path)
     18 
     19 # Save pytorch-model

~/py3.6/lib/python3.6/site-packages/transformers/modeling_bert.py in load_tf_weights_in_bert(model, config, tf_checkpoint_path)

~/py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    589                 return modules[name]
    590         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 591             type(self).__name__, name))
    592 
    593     def __setattr__(self, name, value):

AttributeError: 'LayerNorm' object has no attribute 'cls'
```
Hi @chiyuzhang94 . What do you get if you remove everything related to `cls` in

```
                pointer = getattr(pointer, 'cls')
                pointer = getattr(pointer, 'bias')
            elif l[0] == 'output_weights':
                pointer = getattr(pointer, 'cls')
                pointer = getattr(pointer, 'weight')
```

?<|||||>> Hi @chiyuzhang94 . What do you get if you remove everything related to `cls` in
> 
> ```
>                 pointer = getattr(pointer, 'cls')
>                 pointer = getattr(pointer, 'bias')
>             elif l[0] == 'output_weights':
>                 pointer = getattr(pointer, 'cls')
>                 pointer = getattr(pointer, 'weight')
> ```
> 
> ?

<|||||>Hi @rlouf 
When I removed `cls` parts, I got the following error: 
```
I1009 17:18:05.159223 47655893610048 modeling_bert.py:81] Skipping bert/encoder/layer_9/output/dense/bias/adam_m
I1009 17:18:05.159870 47655893610048 modeling_bert.py:81] Skipping bert/encoder/layer_9/output/dense/bias/adam_v
I1009 17:18:05.160566 47655893610048 modeling_bert.py:115] Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'dense', 'kernel']
I1009 17:18:05.161365 47655893610048 modeling_bert.py:81] Skipping bert/encoder/layer_9/output/dense/kernel/adam_m
I1009 17:18:05.162042 47655893610048 modeling_bert.py:81] Skipping bert/encoder/layer_9/output/dense/kernel/adam_v
I1009 17:18:05.162718 47655893610048 modeling_bert.py:115] Initialize PyTorch weight ['bert', 'pooler', 'dense', 'bias']
I1009 17:18:05.163373 47655893610048 modeling_bert.py:81] Skipping bert/pooler/dense/bias/adam_m
I1009 17:18:05.164020 47655893610048 modeling_bert.py:81] Skipping bert/pooler/dense/bias/adam_v
I1009 17:18:05.164690 47655893610048 modeling_bert.py:115] Initialize PyTorch weight ['bert', 'pooler', 'dense', 'kernel']
I1009 17:18:05.165374 47655893610048 modeling_bert.py:81] Skipping bert/pooler/dense/kernel/adam_m
I1009 17:18:05.166048 47655893610048 modeling_bert.py:81] Skipping bert/pooler/dense/kernel/adam_v
I1009 17:18:05.166695 47655893610048 modeling_bert.py:81] Skipping global_step
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-1-a2cbceaf2173> in <module>
     15 
     16 # Load weights from tf checkpoint
---> 17 load_tf_weights_in_bert(model, config, tf_checkpoint_path)
     18 
     19 # Save pytorch-model

~/py3.6/lib/python3.6/site-packages/transformers-2.1.0-py3.6.egg/transformers/modeling_bert.py in load_tf_weights_in_bert(model, config, tf_checkpoint_path)
     90                 pointer = getattr(pointer, 'weight')
     91             elif l[0] == 'output_bias' or l[0] == 'beta':
---> 92                 pointer = getattr(pointer, 'bias')
     93             elif l[0] == 'output_weights':
     94                 pointer = getattr(pointer, 'weight')

~/py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    589                 return modules[name]
    590         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 591             type(self).__name__, name))
    592 
    593     def __setattr__(self, name, value):
```<|||||>Hi @chiyuzhang94, as a side question which TensorFlow version did you use to train your bert model ?

Do you observe the same behavior by loading the .index file directly using:

```python
config = BertConfig.from_json_file('your/tf_model/config.json')
model = BertForSequenceClassification.from_pretrained('your/tf_model/xxxx.ckpt.index', from_tf=True, config=config)
```<|||||>Hi @mfuntowicz ,
I trained the model with tensorflow 1.12.0.

My am currently using tensorflow 1.13.1  and torch 1.2.0 for this converting task. 

If I use your suggestion code, it is also same issue:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-35a5c9b35e78> in <module>
     12 
     13 print("Building PyTorch model from configuration: {}".format(str(config)))
---> 14 model = BertForSequenceClassification.from_pretrained(tf_checkpoint_path, from_tf=True, config=config)
     15 # Load weights from tf checkpoint
     16 # load_tf_weights_in_bert(model, config, tf_checkpoint_path)

~/py3.6/lib/python3.6/site-packages/transformers-2.1.0-py3.6.egg/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    352             if resolved_archive_file.endswith('.index'):
    353                 # Load from a TensorFlow 1.X checkpoint - provided by original authors
--> 354                 model = cls.load_tf_weights(model, config, resolved_archive_file[:-6])  # Remove the '.index'
    355             else:
    356                 # Load from our TensorFlow 2.0 checkpoints

~/py3.6/lib/python3.6/site-packages/transformers-2.1.0-py3.6.egg/transformers/modeling_bert.py in load_tf_weights_in_bert(model, config, tf_checkpoint_path)
     90                 pointer = getattr(pointer, 'weight')
     91             elif l[0] == 'output_bias' or l[0] == 'beta':
---> 92                 pointer = getattr(pointer, 'bias')
     93             elif l[0] == 'output_weights':
     94                 pointer = getattr(pointer, 'weight')

~/py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    589                 return modules[name]
    590         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 591             type(self).__name__, name))
    592 
    593     def __setattr__(self, name, value):

AttributeError: 'BertForSequenceClassification' object has no attribute 'bias'
```<|||||>This happens because you are trying to load weights the functions wasn't designed for. Unfortunately we cannot support every possible file. You will have to modify `modeling_bert.py` manually to support your file. The part you need to modify is:

```
    for name, array in zip(names, arrays):
        name = name.split('/')
        # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
        # which are not required for using pretrained model
        if any(n in ["adam_v", "adam_m", "global_step"] for n in name):
            logger.info("Skipping {}".format("/".join(name)))
            continue
        pointer = model
        for m_name in name:
            if re.fullmatch(r'[A-Za-z]+_\d+', m_name):
                l = re.split(r'_(\d+)', m_name)
            else:
                l = [m_name]
            if l[0] == 'kernel' or l[0] == 'gamma':
                pointer = getattr(pointer, 'weight')
            elif l[0] == 'output_bias' or l[0] == 'beta':
                pointer = getattr(pointer, 'bias')
            elif l[0] == 'output_weights':
                pointer = getattr(pointer, 'weight')
            elif l[0] == 'squad':
                pointer = getattr(pointer, 'classifier')
            else:
                try:
                    pointer = getattr(pointer, l[0])
                except AttributeError:
                    logger.info("Skipping {}".format("/".join(name)))
                    continue
            if len(l) >= 2:
                num = int(l[1])
                pointer = pointer[num]
        if m_name[-11:] == '_embeddings':
            pointer = getattr(pointer, 'weight')
        elif m_name == 'kernel':
            array = np.transpose(array)
        try:
            assert pointer.shape == array.shape
        except AssertionError as e:
            e.args += (pointer.shape, array.shape)
            raise
        logger.info("Initialize PyTorch weight {}".format(name))
        pointer.data = torch.from_numpy(array)
```<|||||>Hi @rlouf ,
Thanks for your answer.  I think adding `pointer = getattr(pointer, 'cls')` to the two if-so section make sense. But I am wondering how I can deal with the question of 'LayerNorm' object has no attribute 'cls'. Could you please provide me any hint?<|||||>Separating out the conditional statement:
`elif l[0] == 'output_bias' or l[0] == 'beta':`
into two, while maintaining the original functionality in the beta conditional should work?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
