Answers.Body: <p><strong>Context and Model Overview:</strong></p>
  <p><strong>Model Capabilities:</strong> The <code>timpal0l/mdeberta-v3-base-squad2</code> model, seem to be based on <a href="https://github.com/microsoft/DeBERTa" rel="nofollow noreferrer">DeBERTa V3</a>, utilizes relative position embeddings and is capable of processing very large sequences. While it doesn't have a strict token limit, practical considerations such as memory and processing time can impact its performance with extensive texts. Refer this issue on <a href="https://github.com/huggingface/transformers/issues/16998#issuecomment-1183941501" rel="nofollow noreferrer">GitHub</a>.</p>
  <p><strong>Processing Time:</strong> Longer contexts lead to increased response times due to tokenization and embedding overhead. Character-based tokenizers contribute more to this overhead compared to efficient word/subword-based tokenizers, which handle semantic units more effectively.</p>
  <p><strong>Strategies for efficiently handling large context:</strong></p>
  <ul>
  <li><p><strong>RAG Approach:</strong> Implement Retrieval-Augmented Generation <a href="https://aws.amazon.com/what-is/retrieval-augmented-generation/" rel="nofollow noreferrer">(RAG)</a> by segmenting your large text into smaller, retrievable passages. Use a retrieval-based method to select relevant passages based on the question and apply the QA model specifically to these smaller chunks.</p>
  </li>
  <li><p><strong>Compression and Optimization:</strong> Where feasible, compress or summarize the context without losing crucial information. Techniques like paraphrasing can help maintain context relevance while reducing computational load.</p>
  </li>
  <li><p><strong>Context Management:</strong> Preprocess and store the tokenized representation representations of your large context to avoid repeated tokenization. This preprocessing step can significantly reduce latency for frequently asked questions.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-25T11:56:52.967Z
Answers.Text: Thanks. Seems to me RAG is the choice now I can understand it a little more. But I'm facing a dilemma about how to choose the best chunk size. If it is too big, I have performance problems. If it is too small, I loose semantics ( one chunk of text may need informations from another chunk ). So I'm learning how to partition my text efficiently
Answers.UserDisplayName: Magno C
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-28T17:49:53.730Z
Answers.Text: I would use overlap chunking. Try various combinations of `chunk_size` and `overlap_chunk_size`.
Answers.UserDisplayName: Kulin Patel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-18T06:37:21.093Z
Answers.LastActivityDate: 2024-06-18T07:02:27.597Z
Answers.LastEditDate: 2024-06-18T07:02:27.597Z
Answers.UserDisplayName: Kulin Patel
Answers.Body: <p>The Attention computation is contrains by square of the contexte. Tokenize is mandatory and avoid to tokenize / detokenize too many time can help. Your RAM or VRam is alsoo limitation as memory size is... square of the context.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-23T11:50:05.287Z
Answers.Text: Thanks, but my question was edited by @nbro and lost the meaning. I do not care how big the context can be. My question was about how can I work with big contexts without need to wait a life for the answer. The key was given by "Revolucion for Monica" with the word "preprocess"
Answers.UserDisplayName: Magno C
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-19T06:24:05.550Z
Answers.LastActivityDate: 2024-01-19T06:24:05.550Z
Answers.UserDisplayName: Luc BOLOGNA
Body: <p>I'm new on AI, Neural Networks, ChatBots and all this ecosystem. I'm trying to use a classical example of pre-trained models, more specifically <code>timpal0l/mdeberta-v3-base-squad2</code>.</p>
  <p>As I could see in the examples, it is necessary to provide a small text (context) and a question, which the model will respond to by extracting information from the context and using training on the structure of the language. This is my working code so far:</p>
  <pre><code>from transformers import AutoTokenizer, AutoModelForQuestionAnswering, AutoConfig, DefaultDataCollator
  import torch
  
  tokenizer = AutoTokenizer.from_pretrained(&quot;timpal0l/mdeberta-v3-base-squad2&quot;)
  model = AutoModelForQuestionAnswering.from_pretrained(&quot;timpal0l/mdeberta-v3-base-squad2&quot;, return_dict=False )
  
  # From portuguese: My cat is called Helena. She is fat and has spots all over her body.
  little_text = &quot;minha gata se chama helena. Ela é gorda e tem manchas pelo corpo.&quot;
  
  # From portuguese: who is helena?
  question = &quot;quem é helena?&quot;
  
  inputs = tokenizer.encode_plus(question, little_text, add_special_tokens=False, return_tensors=&quot;pt&quot;)
  
  input_ids = inputs[&quot;input_ids&quot;].tolist()[0]
  
  text_tokens = tokenizer.convert_ids_to_tokens(input_ids)
  answer_start_scores, answer_end_scores = model(**inputs)
  
  answer_start = torch.argmax(
      answer_start_scores
  )  
  answer_end = torch.argmax(answer_end_scores) + 1
  
  answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))
  
  print(f&quot;Question: {question}&quot;)
  print(f&quot;Answer: {answer}\n&quot;)
  
  ------------------------------------------------
  Question: quem é helena?
  Answer: minha gata # From portuguese: My cat 
  </code></pre>
  <p>Everything went well. The model is able to perfectly answer questions about the small text provided as context.</p>
  <p>What I couldn't understand is: when I provide a much larger text to serve as context (from a file, for example) the time for the response increases considerably. So every time I go to ask a question about the given text I have to wait a long time to get the answer.</p>
  <p>So my question is: Am I doing this correctly? My intention is to create a chatbot to help answer questions about a specific system (a kind of help bot), so I simply copied all the text from the online help and pasted it into a single file but I believe that this method only works for small texts because at every question I must <code>tokenizer.encode_plus(question, little_text ... </code></p>
  <p>How big the context can be?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-08T13:55:05.100Z
Comments.Text: When working with large texts as context, the time for the response can indeed increase significantly. This is because the model needs to process and understand the entire context before generating an answer. The larger the context, the more time it will take for the model to encode and analyze it.
  ▰
  To avoid reprocessing the entire context for each question, please preprocess the context separately and encode it once, storing the representation for later. Then, for each question, you can encode only the question and pass it along with the preprocessed context to the model for inference.
Comments.UserDisplayName: Revolucion for Monica
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-08T20:01:09.187Z
Comments.Text: Can you please show where I must change my code to achieve this? Asked to GPT but the answer was a terrible mess.
Comments.UserDisplayName: Magno C
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-09T08:57:56.307Z
Comments.Text: @MagnoC Don't rely on ChatGPT for this kind of questions. ChatGPT is only relatively good at generating new random text that is in a way related to some text that you pass to it. ChatGPT is can be bad at providing info about facts and especially specific facts (remember that it only produces the most likely tokens and it doesn't really reason like us). Having said that, models usually have a fixed context, including ChatGPT (which should have like 4096 context length). How long is your text?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-09T08:59:51.590Z
Comments.Text: Anyway, this question seems to be specific to a specific pre-trained model, so it seems to be off-topic. See our on-topic page: https://ai.stackexchange.com/help/on-topic. We focus on the theory.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-10T15:21:13.217Z
Comments.Text: Ok. Thanks anyway. But @RevolucionforMonica gives me a solution that is generic for this KIND of network and not for this specific model because any model can be used with this code ( I think )
Comments.UserDisplayName: Magno C
Tags: <natural-language-processing><pytorch><bert><language-model><question-answering>
Title: How big the context can be using HuggingFace models?
UserDisplayName: Magno C
ThreadText: # How big the context can be using HuggingFace models?
  
  (Asked by: username_0 on 2023-06-07)
  
  I'm new on AI, Neural Networks, ChatBots and all this ecosystem. I'm trying to use a classical example of pre-trained models, more specifically `timpal0l/mdeberta-v3-base-squad2`.
  
  As I could see in the examples, it is necessary to provide a small text (context) and a question, which the model will respond to by extracting information from the context and using training on the structure of the language. This is my working code so far:
  
  ```
  from transformers import AutoTokenizer, AutoModelForQuestionAnswering, AutoConfig, DefaultDataCollator
  import torch
  
  tokenizer = AutoTokenizer.from_pretrained("timpal0l/mdeberta-v3-base-squad2")
  model = AutoModelForQuestionAnswering.from_pretrained("timpal0l/mdeberta-v3-base-squad2", return_dict=False )
  
  # From portuguese: My cat is called Helena. She is fat and has spots all over her body.
  little_text = "minha gata se chama helena. Ela é gorda e tem manchas pelo corpo."
  
  # From portuguese: who is helena?
  question = "quem é helena?"
  
  inputs = tokenizer.encode_plus(question, little_text, add_special_tokens=False, return_tensors="pt")
  
  input_ids = inputs["input_ids"].tolist()[0]
  
  text_tokens = tokenizer.convert_ids_to_tokens(input_ids)
  answer_start_scores, answer_end_scores = model(**inputs)
  
  answer_start = torch.argmax(
      answer_start_scores
  )  
  answer_end = torch.argmax(answer_end_scores) + 1
  
  answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))
  
  print(f"Question: {question}")
  print(f"Answer: {answer}\n")
  
  ------------------------------------------------
  Question: quem é helena?
  Answer: minha gata # From portuguese: My cat 
  
  ```
  
  Everything went well. The model is able to perfectly answer questions about the small text provided as context.
  
  What I couldn't understand is: when I provide a much larger text to serve as context (from a file, for example) the time for the response increases considerably. So every time I go to ask a question about the given text I have to wait a long time to get the answer.
  
  So my question is: Am I doing this correctly? My intention is to create a chatbot to help answer questions about a specific system (a kind of help bot), so I simply copied all the text from the online help and pasted it into a single file but I believe that this method only works for small texts because at every question I must `tokenizer.encode_plus(question, little_text ...`
  
  How big the context can be?
  
  **Tags:** natural-language-processing, pytorch, bert, language-model, question-answering
  
  > **Comments:**
  > **username_1**: When working with large texts as context, the time for the response can indeed increase significantly. This is because the model needs to process and understand the entire context before generating an answer. The larger the context, the more time it will take for the model to encode and analyze it. ▰ To avoid reprocessing the entire context for each question, please preprocess the context separately and encode it once, storing the representation for later. Then, for each question, you can encode only the question and pass it along with the preprocessed context to the model for inference.  
  > **username_0**: Can you please show where I must change my code to achieve this? Asked to GPT but the answer was a terrible mess.  
  > **username_2**: @username_0 Don't rely on ChatGPT for this kind of questions. ChatGPT is only relatively good at generating new random text that is in a way related to some text that you pass to it. ChatGPT is can be bad at providing info about facts and especially specific facts (remember that it only produces the most likely tokens and it doesn't really reason like us). Having said that, models usually have a fixed context, including ChatGPT (which should have like 4096 context length). How long is your text?  
  > **username_2**: Anyway, this question seems to be specific to a specific pre-trained model, so it seems to be off-topic. See our on-topic page: https://ai.stackexchange.com/help/on-topic. We focus on the theory.  
  > **username_0**: Ok. Thanks anyway. But @username_1 gives me a solution that is generic for this KIND of network and not for this specific model because any model can be used with this code ( I think )  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2024-06-18. Score: 1 | Accepted answer
  
  **Context and Model Overview:**
  
  **Model Capabilities:** The `timpal0l/mdeberta-v3-base-squad2` model, seem to be based on [DeBERTa V3](https://github.com/microsoft/DeBERTa), utilizes relative position embeddings and is capable of processing very large sequences. While it doesn't have a strict token limit, practical considerations such as memory and processing time can impact its performance with extensive texts. Refer this issue on [GitHub](https://github.com/huggingface/transformers/issues/16998#issuecomment-1183941501).
  
  **Processing Time:** Longer contexts lead to increased response times due to tokenization and embedding overhead. Character-based tokenizers contribute more to this overhead compared to efficient word/subword-based tokenizers, which handle semantic units more effectively.
  
  **Strategies for efficiently handling large context:**
  
  * **RAG Approach:** Implement Retrieval-Augmented Generation [(RAG)](https://aws.amazon.com/what-is/retrieval-augmented-generation/) by segmenting your large text into smaller, retrievable passages. Use a retrieval-based method to select relevant passages based on the question and apply the QA model specifically to these smaller chunks.
  * **Compression and Optimization:** Where feasible, compress or summarize the context without losing crucial information. Techniques like paraphrasing can help maintain context relevance while reducing computational load.
  * **Context Management:** Preprocess and store the tokenized representation representations of your large context to avoid repeated tokenization. This preprocessing step can significantly reduce latency for frequently asked questions.
  
  > **Comments:**
  > **username_0**: Thanks. Seems to me RAG is the choice now I can understand it a little more. But I'm facing a dilemma about how to choose the best chunk size. If it is too big, I have performance problems. If it is too small, I loose semantics ( one chunk of text may need informations from another chunk ). So I'm learning how to partition my text efficiently  
  > **username_3**: I would use overlap chunking. Try various combinations of `chunk_size` and `overlap_chunk_size`.  
  
  ---
  
  ### Answer by username_4 on 2024-01-19. Score: 0
  
  The Attention computation is contrains by square of the contexte. Tokenize is mandatory and avoid to tokenize / detokenize too many time can help. Your RAM or VRam is alsoo limitation as memory size is... square of the context.
  
  > **Comments:**
  > **username_0**: Thanks, but my question was edited by @username_2 and lost the meaning. I do not care how big the context can be. My question was about how can I work with big contexts without need to wait a life for the answer. The key was given by "username_1 for username_1" with the word "preprocess"  
  
  ---
  
  