Answers.Body: <p>change</p>
  <pre><code>bert_output = bert_model.get_pooled_output()
  </code></pre>
  <p>to</p>
  <pre><code>bert_output = tf.reduce_mean(bert_model.get_sequence_output()[:,1:,:],1)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-11T09:47:58.747Z
Answers.LastActivityDate: 2020-09-11T09:47:58.747Z
Answers.UserDisplayName: CoderOnly
Body: <p>I am using pretraining code from <a href="https://github.com/NVIDIA/DeepLearningExamples" rel="nofollow noreferrer">https://github.com/NVIDIA/DeepLearningExamples</a></p>
  <p>Pretrain parameters:</p>
  <pre><code> 15:47:02,534: INFO tensorflow 140678508230464   init_checkpoint: bertbase3layer-extract-from-google
   15:47:02,534: INFO tensorflow 140678508230464   optimizer_type: lamb
   15:47:02,534: INFO tensorflow 140678508230464   max_seq_length: 64
   15:47:02,534: INFO tensorflow 140678508230464   max_predictions_per_seq: 5
   15:47:02,534: INFO tensorflow 140678508230464   do_train: True
   15:47:02,535: INFO tensorflow 140678508230464   do_eval: False
   15:47:02,535: INFO tensorflow 140678508230464   train_batch_size: 32
   15:47:02,535: INFO tensorflow 140678508230464   eval_batch_size: 8
   15:47:02,535: INFO tensorflow 140678508230464   learning_rate: 5e-05
   15:47:02,535: INFO tensorflow 140678508230464   num_train_steps: 10000000
   15:47:02,535: INFO tensorflow 140678508230464   num_warmup_steps: 10000
   15:47:02,535: INFO tensorflow 140678508230464   save_checkpoints_steps: 1000
   15:47:02,535: INFO tensorflow 140678508230464   display_loss_steps: 10
   15:47:02,535: INFO tensorflow 140678508230464   iterations_per_loop: 1000
   15:47:02,535: INFO tensorflow 140678508230464   max_eval_steps: 100
   15:47:02,535: INFO tensorflow 140678508230464   num_accumulation_steps: 1
   15:47:02,535: INFO tensorflow 140678508230464   allreduce_post_accumulation: False
   15:47:02,535: INFO tensorflow 140678508230464   verbose_logging: False
   15:47:02,535: INFO tensorflow 140678508230464   horovod: True
   15:47:02,536: INFO tensorflow 140678508230464   report_loss: True
   15:47:02,536: INFO tensorflow 140678508230464   manual_fp16: False
   15:47:02,536: INFO tensorflow 140678508230464   amp: False
   15:47:02,536: INFO tensorflow 140678508230464   use_xla: True
   15:47:02,536: INFO tensorflow 140678508230464   init_loss_scale: 4294967296
   15:47:02,536: INFO tensorflow 140678508230464   ?: False
   15:47:02,536: INFO tensorflow 140678508230464   help: False
   15:47:02,536: INFO tensorflow 140678508230464   helpshort: False
   15:47:02,536: INFO tensorflow 140678508230464   helpfull: False
   15:47:02,536: INFO tensorflow 140678508230464   helpxml: False
   15:47:02,536: INFO tensorflow 140678508230464 **************************
  </code></pre>
  <p>Pretrain loss: (I remove nsp_loss)</p>
  <pre><code>{'throughput_train': 1196.9646684552622, 'mlm_loss': 0.9837073683738708, 'nsp_loss': 0.0, 'total_loss': 0.9837073683738708, 'avg_loss_step': 1.200513333082199, 'learning_rate': '0.00038143058'}
  {'throughput_train': 1230.5063662500734, 'mlm_loss': 1.3001925945281982, 'nsp_loss': 0.0, 'total_loss': 1.3001925945281982, 'avg_loss_step': 1.299936044216156, 'learning_rate': '0.00038143038'}
  {'throughput_train': 1236.4348949169155, 'mlm_loss': 1.473339319229126, 'nsp_loss': 0.0, 'total_loss': 1.473339319229126, 'avg_loss_step': 1.2444063007831574, 'learning_rate': '0.00038143017'}
  {'throughput_train': 1221.2668264552692, 'mlm_loss': 0.9924975633621216, 'nsp_loss': 0.0, 'total_loss': 0.9924975633621216, 'avg_loss_step': 1.1603020071983337, 'learning_rate': '0.00038142994'}
  </code></pre>
  <p>Fine-tune code:</p>
  <pre><code>self.train_op = tf.train.AdamOptimizer(0.00001).minimize(self.loss, global_step=self.global_step)
  </code></pre>
  <p>Fine-tune accuracy: (restore from my ckpt pretrained from <a href="https://github.com/NVIDIA/DeepLearningExamples" rel="nofollow noreferrer">https://github.com/NVIDIA/DeepLearningExamples</a>)</p>
  <pre><code>epoch 1:
  training step 895429, loss 4.98, acc 0.079
  dev loss 4.853, acc 0.092
  
  epoch 2:
  training step 895429, loss 4.97, acc 0.080
  dev loss 4.823, acc 0.092
  
  epoch 3:
  training step 895429, loss 4.96, acc 0.081
  dev loss 4.849, acc 0.092
  
  epoch 4:
  training step 895429, loss 4.95, acc 0.082
  dev loss 4.843, acc 0.092
  </code></pre>
  <p>Without restore the pretrained ckpt:</p>
  <pre><code>epoch 1:
  training step 10429, loss 2.48, acc 0.606
  dev loss 1.604, acc 0.8036
  </code></pre>
  <p>Restore the google's BERT-Base pretrained ckpt. Or restore from a pretrained ckpt pretrained from <a href="https://github.com/guotong1988/BERT-GPU" rel="nofollow noreferrer">https://github.com/guotong1988/BERT-GPU</a></p>
  <pre><code>epoch 1:
  training loss 1.89, acc 0.761
  dev loss 1.351, acc 0.869
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-11T09:52:38.987Z
Comments.Text: Hello. Welcome to AI SE! "fine-tune not work" is not a question, so please edit your post to explicitly ask a specific question. Moreover, note that programming questions are generally off-topic here. Here we focus on theoretical, social and philosophical aspects of AI. Your question seems to be a programming issue, so it's off-topic here.  See https://ai.stackexchange.com/help/on-topic for more info.
Comments.UserDisplayName: nbro
Tags: <transformer><bert><transfer-learning><pretrained-models><fine-tuning>
Title: BERT: After pretraining 880000 step, why fine-tune not work?
UserDisplayName: CoderOnly
ThreadText: # BERT: After pretraining 880000 step, why fine-tune not work?
  
  (Asked by: username_0 on 2020-09-11)
  
  I am using pretraining code from <https://github.com/NVIDIA/DeepLearningExamples>
  
  Pretrain parameters:
  
  ```
   15:47:02,534: INFO tensorflow 140678508230464   init_checkpoint: bertbase3layer-extract-from-google
   15:47:02,534: INFO tensorflow 140678508230464   optimizer_type: lamb
   15:47:02,534: INFO tensorflow 140678508230464   max_seq_length: 64
   15:47:02,534: INFO tensorflow 140678508230464   max_predictions_per_seq: 5
   15:47:02,534: INFO tensorflow 140678508230464   do_train: True
   15:47:02,535: INFO tensorflow 140678508230464   do_eval: False
   15:47:02,535: INFO tensorflow 140678508230464   train_batch_size: 32
   15:47:02,535: INFO tensorflow 140678508230464   eval_batch_size: 8
   15:47:02,535: INFO tensorflow 140678508230464   learning_rate: 5e-05
   15:47:02,535: INFO tensorflow 140678508230464   num_train_steps: 10000000
   15:47:02,535: INFO tensorflow 140678508230464   num_warmup_steps: 10000
   15:47:02,535: INFO tensorflow 140678508230464   save_checkpoints_steps: 1000
   15:47:02,535: INFO tensorflow 140678508230464   display_loss_steps: 10
   15:47:02,535: INFO tensorflow 140678508230464   iterations_per_loop: 1000
   15:47:02,535: INFO tensorflow 140678508230464   max_eval_steps: 100
   15:47:02,535: INFO tensorflow 140678508230464   num_accumulation_steps: 1
   15:47:02,535: INFO tensorflow 140678508230464   allreduce_post_accumulation: False
   15:47:02,535: INFO tensorflow 140678508230464   verbose_logging: False
   15:47:02,535: INFO tensorflow 140678508230464   horovod: True
   15:47:02,536: INFO tensorflow 140678508230464   report_loss: True
   15:47:02,536: INFO tensorflow 140678508230464   manual_fp16: False
   15:47:02,536: INFO tensorflow 140678508230464   amp: False
   15:47:02,536: INFO tensorflow 140678508230464   use_xla: True
   15:47:02,536: INFO tensorflow 140678508230464   init_loss_scale: 4294967296
   15:47:02,536: INFO tensorflow 140678508230464   ?: False
   15:47:02,536: INFO tensorflow 140678508230464   help: False
   15:47:02,536: INFO tensorflow 140678508230464   helpshort: False
   15:47:02,536: INFO tensorflow 140678508230464   helpfull: False
   15:47:02,536: INFO tensorflow 140678508230464   helpxml: False
   15:47:02,536: INFO tensorflow 140678508230464 **************************
  
  ```
  
  Pretrain loss: (I remove nsp\_loss)
  
  ```
  {'throughput_train': 1196.9646684552622, 'mlm_loss': 0.9837073683738708, 'nsp_loss': 0.0, 'total_loss': 0.9837073683738708, 'avg_loss_step': 1.200513333082199, 'learning_rate': '0.00038143058'}
  {'throughput_train': 1230.5063662500734, 'mlm_loss': 1.3001925945281982, 'nsp_loss': 0.0, 'total_loss': 1.3001925945281982, 'avg_loss_step': 1.299936044216156, 'learning_rate': '0.00038143038'}
  {'throughput_train': 1236.4348949169155, 'mlm_loss': 1.473339319229126, 'nsp_loss': 0.0, 'total_loss': 1.473339319229126, 'avg_loss_step': 1.2444063007831574, 'learning_rate': '0.00038143017'}
  {'throughput_train': 1221.2668264552692, 'mlm_loss': 0.9924975633621216, 'nsp_loss': 0.0, 'total_loss': 0.9924975633621216, 'avg_loss_step': 1.1603020071983337, 'learning_rate': '0.00038142994'}
  
  ```
  
  Fine-tune code:
  
  ```
  self.train_op = tf.train.AdamOptimizer(0.00001).minimize(self.loss, global_step=self.global_step)
  
  ```
  
  Fine-tune accuracy: (restore from my ckpt pretrained from <https://github.com/NVIDIA/DeepLearningExamples>)
  
  ```
  epoch 1:
  training step 895429, loss 4.98, acc 0.079
  dev loss 4.853, acc 0.092
  
  epoch 2:
  training step 895429, loss 4.97, acc 0.080
  dev loss 4.823, acc 0.092
  
  epoch 3:
  training step 895429, loss 4.96, acc 0.081
  dev loss 4.849, acc 0.092
  
  epoch 4:
  training step 895429, loss 4.95, acc 0.082
  dev loss 4.843, acc 0.092
  
  ```
  
  Without restore the pretrained ckpt:
  
  ```
  epoch 1:
  training step 10429, loss 2.48, acc 0.606
  dev loss 1.604, acc 0.8036
  
  ```
  
  Restore the google's BERT-Base pretrained ckpt. Or restore from a pretrained ckpt pretrained from <https://github.com/guotong1988/BERT-GPU>
  
  ```
  epoch 1:
  training loss 1.89, acc 0.761
  dev loss 1.351, acc 0.869
  
  ```
  
  **Tags:** transformer, bert, transfer-learning, pretrained-models, fine-tuning
  
  > **Comments:**
  > **username_1**: Hello. Welcome to AI SE! "fine-tune not work" is not a question, so please edit your post to explicitly ask a specific question. Moreover, note that programming questions are generally off-topic here. Here we focus on theoretical, social and philosophical aspects of AI. Your question seems to be a programming issue, so it's off-topic here.  See https://ai.stackexchange.com/help/on-topic for more info.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-09-11. Score: 0 | Accepted answer
  
  change
  
  ```
  bert_output = bert_model.get_pooled_output()
  
  ```
  
  to
  
  ```
  bert_output = tf.reduce_mean(bert_model.get_sequence_output()[:,1:,:],1)
  
  ```
  
  
  ---
  
  