Answers.Body: <p>The answer is yes but 'lightweight' will require a 'lightweight' model.</p>
  <p>Your application for 'domain one' is called open domain question answering (ODQA).  Here is a demonstration of ODQA using BERT: <a href="https://www.pragnakalp.com/demos/BERT-NLP-QnA-Demo/" rel="nofollow noreferrer">https://www.pragnakalp.com/demos/BERT-NLP-QnA-Demo/</a></p>
  <p>Your application for 'domain two' is a little different.  It is about learning sequences from sequences.  More specifically these are called sequence to sequence models.  <a href="https://mccormickml.com/2020/03/10/question-answering-with-a-fine-tuned-BERT/" rel="nofollow noreferrer">Here</a> is an example using a pre-trained BERT model fine-tuned on the Stanford Question Answering (SQuAD) dataset.</p>
  <p><em><a href="https://rajpurkar.github.io/SQuAD-explorer/" rel="nofollow noreferrer">Stanford Question Answering Dataset (SQuAD)</a> is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.</em></p>
  <p>In both applications, the resources required are going to rely on the performance you require.  There are many sizes of BERT models.  Generally, the larger the model, the larger the GPU memory requirements, and the higher the performance (i.e. accuracy, precision, recall, F1-score, etc.).  For example, I can run BERT Base on a particular data set on a GTX 1080Ti and a RTX 2080Ti but not BERT Large.</p>
  <p>This article, <a href="https://blog.exxactcorp.com/nvidia-quadro-rtx-6000-bert-large-fine-tune-benchmarks-with-squad-dataset/" rel="nofollow noreferrer">NVIDIA Quadro RTX 6000 BERT Large Fine-tune Benchmarks with SQuAD Dataset</a> shows performance for BERT using TensorFlow on four NVIDIA Quadro RTX 6000 GPUs.</p>
  <p>There is a 'mobile' version of BERT called MobileBERT for running on small devices like smartphones.  Here is an article on using that with SQuAD: <a href="https://www.tensorflow.org/lite/models/bert_qa/overview" rel="nofollow noreferrer">https://www.tensorflow.org/lite/models/bert_qa/overview</a></p>
  <p>cdQA-suite is a good package.  The following should help in fine-tuning on your own corpus:</p>
  <ul>
  <li><a href="https://towardsdatascience.com/how-to-create-your-own-question-answering-system-easily-with-python-2ef8abc8eb5" rel="nofollow noreferrer">How to create your own Question-Answering system easily with python</a></li>
  <li><a href="https://github.com/cdqa-suite/cdQA/tree/master/examples" rel="nofollow noreferrer">cdQA examples</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T13:22:11.047Z
Answers.Text: 1) For Domain 1, I have a list of articles from which I want the model to answer questions, so it is Closed Domain, or rather I want it that way.
  2)  For Domain 2, yes I'm up to date with BERT and the memory issues, what I want to know specifically, is whether just a text corpus can be used to fine-tune a model.
Answers.UserDisplayName: Anirban Saha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T13:32:55.297Z
Answers.Text: Yes, you only need a text corpus.
Answers.UserDisplayName: Brian O'Donnell
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T13:47:27.913Z
Answers.Text: Can you point me to a resource, using which I can fine-tune a model only using a text-corpus and then use this model for Question-Answering.
Answers.UserDisplayName: Anirban Saha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T14:08:50.893Z
Answers.Text: I added some information on the cdQA Suite which should help.
Answers.UserDisplayName: Brian O'Donnell
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T14:14:53.123Z
Answers.Text: I've also used cdQA extensively, the problem with the same is the cdQA-annotator tool depends on us manually highlighting answer passages, which would be difficult to achieve when we have close to 5000 question-answer pairs.
Answers.UserDisplayName: Anirban Saha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T14:26:40.367Z
Answers.Text: That is not how I understand their architecture.  For example, their PDF tutorial shows that their data frame requires a title of the document and the 'paragraphs' field is the content of the entire document.  There is no training on discretely defined questions and answer pairs.  https://github.com/cdqa-suite/cdQA/blob/master/examples/tutorial-use-pdf-converter.ipynb
Answers.UserDisplayName: Brian O'Donnell
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T14:28:59.010Z
Answers.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/110276/discussion-between-brian-odonnell-and-anirban-saha).
Answers.UserDisplayName: Brian O'Donnell
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T12:10:24.743Z
Answers.LastActivityDate: 2020-07-06T14:07:59.307Z
Answers.LastEditDate: 2020-07-06T14:07:59.307Z
Answers.UserDisplayName: Brian O'Donnell
Body: <p>I wish to train two domain-specific models:</p>
  <ul>
  <li>Domain 1: Constitution and related Legal Documents</li>
  <li>Domain 2: Technical and related documents.</li>
  </ul>
  <p>For Domain 1, I've access to a text-corpus with texts from the constitution and no question-context-answer tuples. For Domain 2, I've access to Question-Answer pairs.</p>
  <p>Is it possible to fine-tune a light-weight BERT model for Question-Answering using just the data mentioned above?</p>
  <p>If yes, what are the resources to achieve this task?</p>
  <p>Some examples, from the huggingface/models library would be mrm8488/bert-tiny-5-finetuned-squadv2, sshleifer/tiny-distilbert-base-cased-distilled-squad, /twmkn9/albert-base-v2-squad2.</p>
  
Tags: <natural-language-processing><bert><fine-tuning><question-answering>
Title: How to fine tune BERT for question answering?
UserDisplayName: Anirban Saha
ThreadText: # How to fine tune BERT for question answering?
  
  (Asked by: username_0 on 2020-07-06)
  
  I wish to train two domain-specific models:
  
  * Domain 1: Constitution and related Legal Documents
  * Domain 2: Technical and related documents.
  
  For Domain 1, I've access to a text-corpus with texts from the constitution and no question-context-answer tuples. For Domain 2, I've access to Question-Answer pairs.
  
  Is it possible to fine-tune a light-weight BERT model for Question-Answering using just the data mentioned above?
  
  If yes, what are the resources to achieve this task?
  
  Some examples, from the huggingface/models library would be mrm8488/bert-tiny-5-finetuned-squadv2, sshleifer/tiny-distilbert-base-cased-distilled-squad, /twmkn9/albert-base-v2-squad2.
  
  **Tags:** natural-language-processing, bert, fine-tuning, question-answering
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-06. Score: 1
  
  The answer is yes but 'lightweight' will require a 'lightweight' model.
  
  Your application for 'domain one' is called open domain question answering (ODQA). Here is a demonstration of ODQA using BERT: <https://www.pragnakalp.com/demos/BERT-NLP-QnA-Demo/>
  
  Your application for 'domain two' is a little different. It is about learning sequences from sequences. More specifically these are called sequence to sequence models. [Here](https://mccormickml.com/2020/03/10/question-answering-with-a-fine-tuned-BERT/) is an example using a pre-trained BERT model fine-tuned on the Stanford Question Answering (SQuAD) dataset.
  
  *[Stanford Question Answering Dataset (SQuAD)](https://rajpurkar.github.io/SQuAD-explorer/) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.*
  
  In both applications, the resources required are going to rely on the performance you require. There are many sizes of BERT models. Generally, the larger the model, the larger the GPU memory requirements, and the higher the performance (i.e. accuracy, precision, recall, F1-score, etc.). For example, I can run BERT Base on a particular data set on a GTX 1080Ti and a RTX 2080Ti but not BERT Large.
  
  This article, [NVIDIA Quadro RTX 6000 BERT Large Fine-tune Benchmarks with SQuAD Dataset](https://blog.exxactcorp.com/nvidia-quadro-rtx-6000-bert-large-fine-tune-benchmarks-with-squad-dataset/) shows performance for BERT using TensorFlow on four NVIDIA Quadro RTX 6000 GPUs.
  
  There is a 'mobile' version of BERT called MobileBERT for running on small devices like smartphones. Here is an article on using that with SQuAD: <https://www.tensorflow.org/lite/models/bert_qa/overview>
  
  cdQA-suite is a good package. The following should help in fine-tuning on your own corpus:
  
  * [How to create your own Question-Answering system easily with python](https://towardsdatascience.com/how-to-create-your-own-question-answering-system-easily-with-python-2ef8abc8eb5)
  * [cdQA examples](https://github.com/cdqa-suite/cdQA/tree/master/examples)
  
  > **Comments:**
  > **username_0**: 1) For Domain 1, I have a list of articles from which I want the model to answer questions, so it is Closed Domain, or rather I want it that way. 2)  For Domain 2, yes I'm up to date with BERT and the memory issues, what I want to know specifically, is whether just a text corpus can be used to fine-tune a model.  
  > **username_1**: Yes, you only need a text corpus.  
  > **username_0**: Can you point me to a resource, using which I can fine-tune a model only using a text-corpus and then use this model for Question-Answering.  
  > **username_1**: I added some information on the cdQA Suite which should help.  
  > **username_0**: I've also used cdQA extensively, the problem with the same is the cdQA-annotator tool depends on us manually highlighting answer passages, which would be difficult to achieve when we have close to 5000 question-answer pairs.  
  > **username_1**: That is not how I understand their architecture.  For example, their PDF tutorial shows that their data frame requires a title of the document and the 'paragraphs' field is the content of the entire document.  There is no training on discretely defined questions and answer pairs.  https://github.com/cdqa-suite/cdQA/blob/master/examples/tutorial-use-pdf-converter.ipynb  
  > **username_1**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/110276/discussion-between-username_1-odonnell-and-username_0-username_0).  
  
  ---
  
  