Answers.Body: <p>There are multiple approaches to implementing the task you've described. I'll outline one of the most effective methods given the current tools available.</p>
  <p>You're looking to implement a <a href="https://huggingface.co/tasks/question-answering" rel="nofollow noreferrer">Q/A prediction system</a>. This involves searching for an answer to a given question within a context file and then performing a classification task to compare the retrieved answer with a student’s response. Fortunately, there is a machine learning architecture ideally suited for this, known as <a href="https://en.wikipedia.org/wiki/BERT_(language_model)" rel="nofollow noreferrer">BERT</a>. In a nutshell, BERT consists of stacked encoders from the Transformer architecture, and during pre-training, it has two main objectives: Token Masking and Next Sentence Prediction. The latter is particularly relevant for your task.</p>
  <p>BERT can be fine-tuned for Q/A extraction, which aligns well with your requirement. This fine-tuning involves training the model to predict the start and end positions of the answer within the context. The pipeline would involve using a BERT model to identify the answer in the context by providing it with a question and context, resulting in start and end positions. Subsequently, the same BERT model can be adapted to perform binary classification between the student's answer and the identified answer. A high probability score indicates similarity between the two answers, while a low score suggests a lack of correlation.</p>
  <p>One challenge with using Large Language Models (LLMs) like BERT is the limitation on context length. Most LLMs cannot process book-sized texts, although research, such as <a href="https://arxiv.org/abs/2307.02486" rel="nofollow noreferrer">Microsoft’s LongNet</a>, aims to overcome this limitation. A potential solution is to employ <a href="https://youtu.be/klTvEwg3oJ4" rel="nofollow noreferrer">vectorial databases</a>, which involve dividing the full text into chunks manageable for the model and assigning each a vector in an embedding space. The database can then intelligently query these vectors to retrieve the most relevant chunk for Q/A prediction. A practical example of a vectorial database is <a href="https://weaviate.io/" rel="nofollow noreferrer">Weaviate</a>, which offers a <a href="https://weaviate.io/developers/weaviate/modules/reader-generator-modules/qna-transformers" rel="nofollow noreferrer">Q/A module</a> for extracting answers from texts.</p>
  <p>This approach leverages another BERT model to store predictions of information, enabling the semantic search capabilities of LLMs. It provides a scalable long-term memory for the application but does come with the downside of treating data objects as disconnected, which can be problematic if an answer relies on multiple texts.</p>
  <p>An alternative solution is to use generative models like GPT, which have demonstrated strong reasoning skills. However, The stochastic nature of GPT models means that the model may generate different answers for the same prompt, and that it may need a chain of prompts to generate a good response,something that could be leveraged with tools like <a href="https://www.langchain.com/" rel="nofollow noreferrer">Langchain</a>. It's worth noting that this approach could potentially be more computationally expensive than the BERT approach.</p>
  <p>In conclusion, question answering is a natural application for LLMs, with the main limitation being context size. Currently, vectorial databases offer a viable solution, but future developments may see the advent of models capable of handling longer contexts.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-26T20:16:09.127Z
Answers.Text: Thank you. To clarify: In my case the question its answer are „hard coded“ - so this is solely about checking whether a second answer (e. g. by a student) is correct or not. Would BERT still be the way to go? I mean two people could answer the same question in very different ways (short/lengthy with unneccessary details), but in both cases correct.
Answers.UserDisplayName: florian norbert bepunkt
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-28T00:23:20.900Z
Answers.Text: Sure. You would only had to remove the Q/A extraction part. An example of a BERT model is the OpenAI Embeddings API, which maps a text to a vector and then you can perform cosine similarity to check if the texts are similar. Remember that the pre-training of BERT consists of sentence matching, checking if a sentence is a continuation of the other. So a raw inference will not check if both answers are responding to the same thing, but if they have semantic proximity. So maybe you would like to fine-tune it with an objective based on a context, to check if both answers refer to the same thing.
Answers.UserDisplayName: Cesar Ruiz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-24T01:21:03.653Z
Answers.LastActivityDate: 2023-09-24T01:21:03.653Z
Answers.UserDisplayName: Cesar Ruiz
Body: <p>I'm exploring LLMs for educative purposes an came across the topic question answering, e. g. building a system that ingests documents like PDFs and is able to answer questions about its content.</p>
  <p>My question: Are there models / keywords / tutorials when it comes to inverting this process? Let's say you would ingest the PDF of a Sherlock Holmes story and the define hard coded questions like &quot;Who was the perpetrator?&quot; &quot;What was the motive?&quot;. People could write descriptive answers in their own words to these questions and the LLM would return a probability, whether the given answer is correct or not.</p>
  
Tags: <natural-language-processing>
Title: AI: Inverse questions answering - quiz style: Verify descriptive answers to a static question
UserDisplayName: florian norbert bepunkt
ThreadText: # AI: Inverse questions answering - quiz style: Verify descriptive answers to a static question
  
  (Asked by: username_0 on 2023-09-16)
  
  I'm exploring LLMs for educative purposes an came across the topic question answering, e. g. building a system that ingests documents like PDFs and is able to answer questions about its content.
  
  My question: Are there models / keywords / tutorials when it comes to inverting this process? Let's say you would ingest the PDF of a Sherlock Holmes story and the define hard coded questions like "Who was the perpetrator?" "What was the motive?". People could write descriptive answers in their own words to these questions and the LLM would return a probability, whether the given answer is correct or not.
  
  **Tags:** natural-language-processing
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-09-24. Score: 0 | Accepted answer
  
  There are multiple approaches to implementing the task you've described. I'll outline one of the most effective methods given the current tools available.
  
  You're looking to implement a [Q/A prediction system](https://huggingface.co/tasks/question-answering). This involves searching for an answer to a given question within a context file and then performing a classification task to compare the retrieved answer with a student’s response. Fortunately, there is a machine learning architecture ideally suited for this, known as [BERT](https://en.wikipedia.org/wiki/BERT_(language_model)). In a nutshell, BERT consists of stacked encoders from the Transformer architecture, and during pre-training, it has two main objectives: Token Masking and Next Sentence Prediction. The latter is particularly relevant for your task.
  
  BERT can be fine-tuned for Q/A extraction, which aligns well with your requirement. This fine-tuning involves training the model to predict the start and end positions of the answer within the context. The pipeline would involve using a BERT model to identify the answer in the context by providing it with a question and context, resulting in start and end positions. Subsequently, the same BERT model can be adapted to perform binary classification between the student's answer and the identified answer. A high probability score indicates similarity between the two answers, while a low score suggests a lack of correlation.
  
  One challenge with using Large Language Models (LLMs) like BERT is the limitation on context length. Most LLMs cannot process book-sized texts, although research, such as [Microsoft’s LongNet](https://arxiv.org/abs/2307.02486), aims to overcome this limitation. A potential solution is to employ [vectorial databases](https://youtu.be/klTvEwg3oJ4), which involve dividing the full text into chunks manageable for the model and assigning each a vector in an embedding space. The database can then intelligently query these vectors to retrieve the most relevant chunk for Q/A prediction. A practical example of a vectorial database is [Weaviate](https://weaviate.io/), which offers a [Q/A module](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/qna-transformers) for extracting answers from texts.
  
  This approach leverages another BERT model to store predictions of information, enabling the semantic search capabilities of LLMs. It provides a scalable long-term memory for the application but does come with the downside of treating data objects as disconnected, which can be problematic if an answer relies on multiple texts.
  
  An alternative solution is to use generative models like GPT, which have demonstrated strong reasoning skills. However, The stochastic nature of GPT models means that the model may generate different answers for the same prompt, and that it may need a chain of prompts to generate a good response,something that could be leveraged with tools like [Langchain](https://www.langchain.com/). It's worth noting that this approach could potentially be more computationally expensive than the BERT approach.
  
  In conclusion, question answering is a natural application for LLMs, with the main limitation being context size. Currently, vectorial databases offer a viable solution, but future developments may see the advent of models capable of handling longer contexts.
  
  > **Comments:**
  > **username_0**: Thank you. To clarify: In my case the question its answer are „hard coded“ - so this is solely about checking whether a second answer (e. g. by a student) is correct or not. Would BERT still be the way to go? I mean two people could answer the same question in very different ways (short/lengthy with unneccessary details), but in both cases correct.  
  > **username_1**: Sure. You would only had to remove the Q/A extraction part. An example of a BERT model is the OpenAI Embeddings API, which maps a text to a vector and then you can perform cosine similarity to check if the texts are similar. Remember that the pre-training of BERT consists of sentence matching, checking if a sentence is a continuation of the other. So a raw inference will not check if both answers are responding to the same thing, but if they have semantic proximity. So maybe you would like to fine-tune it with an objective based on a context, to check if both answers refer to the same thing.  
  
  ---
  
  