Answers.Body: <p>It sounds like you need a method to determine what subset of your data is necessary for a given query. There's a lot of different methods for document retrieval. I'd start with something simple like BM25 or Tf-IDF, the <a href="https://www.sbert.net/examples/applications/retrieve_rerank/README.html" rel="nofollow noreferrer">sentence-transformers</a> library also has some models you can try for semantic search.</p>
  <p>Depending on how complex your question is, you can also have the LLM generate sub-queries and retrieve documents based on those as well.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-16T05:12:53.683Z
Answers.LastActivityDate: 2023-07-16T05:12:53.683Z
Answers.UserDisplayName: Alexander Wan
Body: <p>I have an issue where I'm trying to use the openAI API to input a very large custom knowledge base (exceeding 1GB) that allows the user to ask questions based on that base to receive intelligent answers. However, the openAI API is very restraining in the tokens and requests that can be inputted, and I was wondering if there is another tool that is more suited for my problem. I figured it would have to utilize an LLM to understand and parse the info.</p>
  <p>I tried different measures such as smartly looking up specific sections of info then inputting a smaller chunk it into GPT, but I'm still largely restricted by the API's restraints.</p>
  <p>I'm quite new to this, so any ideas and suggestions would be appreciated</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-04T12:12:32.820Z
Comments.Text: Perhaps this is a good case for training your own model on your knowledge base?
Comments.UserDisplayName: Bruce Adams
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-05T03:38:02.487Z
Comments.Text: The model needs to be able to make inferences between different documents in the knowledge base. My goal is to have something with the intelligence of an LLM like ChatGPT, and I don't think training my own model to do that is feasible. I'd like to know if something like that could be done though.
Comments.UserDisplayName: Hou Wan
Tags: <natural-language-processing><chatgpt>
Title: What is an appropriate tool to use that takes in a large knowledge base in string form and can answer questions based on the knowledge base?
UserDisplayName: Hou Wan
ThreadText: # What is an appropriate tool to use that takes in a large knowledge base in string form and can answer questions based on the knowledge base?
  
  (Asked by: username_0 on 2023-07-04)
  
  I have an issue where I'm trying to use the openAI API to input a very large custom knowledge base (exceeding 1GB) that allows the user to ask questions based on that base to receive intelligent answers. However, the openAI API is very restraining in the tokens and requests that can be inputted, and I was wondering if there is another tool that is more suited for my problem. I figured it would have to utilize an LLM to understand and parse the info.
  
  I tried different measures such as smartly looking up specific sections of info then inputting a smaller chunk it into GPT, but I'm still largely restricted by the API's restraints.
  
  I'm quite new to this, so any ideas and suggestions would be appreciated
  
  **Tags:** natural-language-processing, chatgpt
  
  > **Comments:**
  > **username_1**: Perhaps this is a good case for training your own model on your knowledge base?  
  > **username_0**: The model needs to be able to make inferences between different documents in the knowledge base. My goal is to have something with the intelligence of an LLM like ChatGPT, and I don't think training my own model to do that is feasible. I'd like to know if something like that could be done though.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-07-16. Score: 1
  
  It sounds like you need a method to determine what subset of your data is necessary for a given query. There's a lot of different methods for document retrieval. I'd start with something simple like BM25 or Tf-IDF, the [sentence-transformers](https://www.sbert.net/examples/applications/retrieve_rerank/README.html) library also has some models you can try for semantic search.
  
  Depending on how complex your question is, you can also have the LLM generate sub-queries and retrieve documents based on those as well.
  
  
  ---
  
  