Incrementally Train BERT with minimum QnA records - to get improved results
This question is posted in stackexchange too, but is pointing to BERT group:
https://datascience.stackexchange.com/questions/47406/incrementally-train-bert-with-minimum-qna-records    
Question is after training on my data on some new questions and answers, new checkpoints are generated. With new checkpoints, when asked same question the answer is not correct, and is wrong again. Why is training not helping making answer right?  
  
Though the questions are point to tensorflow version, but same is tried in pytorch version too and results are same. Some experts on BERT or transformers or neural network probably can better point the issue. 

Details:
We are using Google BERT for Question and Answering. We have fine tuned BERT with SQUAD QnA release train data set (https://github.com/google-research/bert , https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json)

It generated new checkpoints and BERT is giving good answers for most of questions we asked on our text documents. However, there are some questions which it is answering wrong, so we are trying to further fine tune with our Question and known answer on our text document. We further trained based on last generated checkpoint and got new checkpoint.

With new checkpoint when we are asking the same question, the answer did not got corrected! Previously BERT was giving wrong answer with 99% confidence and now also giving same wrong answer with 95% confidence.

Can someone suggest, if they have same or similar experience, and suggest please.
Following are questions in BERT github Issues, and are unanswered for quite some time:

BERT accuracy reduced after providing custom training..The answer is also not correct : https://github.com/google-research/bert/issues/492
Unable to incrementally train BERT with custom training: https://github.com/google-research/bert/issues/482
Little training has no impact: https://github.com/google-research/bert/issues/481
Custom Domain Training: https://github.com/google-research/bert/issues/498
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Wondering Noone is replying

On Sat, 25 May, 2019, 4:09 PM stale[bot], <notifications@github.com> wrote:

> Closed #384
> <https://github.com/huggingface/pytorch-pretrained-BERT/issues/384>.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-pretrained-BERT/issues/384?email_source=notifications&email_token=AHRBKICW2VUPHXBBGSQD5O3PXEJN5A5CNFSM4G666UB2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGORUM5MZY#event-2367280743>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AHRBKIG2BUNIOBDUV6QGHQDPXEJN5ANCNFSM4G666UBQ>
> .
>
