Answers.Body: <p>Two approaches that I am aware of:</p>
  <ul>
  <li><p><a href="https://github.com/hwchase17/chat-your-data" rel="nofollow noreferrer">Chat your data</a><br />
  This GitHub repository is accompanied by a <a href="https://blog.langchain.dev/tutorial-chatgpt-over-your-data/" rel="nofollow noreferrer">blog post</a> on how it works schematically. The overall approach is based on the <a href="https://github.com/hwchase17/langchain" rel="nofollow noreferrer">LangChain library</a>.</p>
  </li>
  <li><p><a href="https://github.com/Azure-Samples/azure-search-openai-demo" rel="nofollow noreferrer">Azure Search OpenAI demo</a>
  This approach goes over your own data using the Retrieval Augmented Generation pattern. It uses Azure OpenAI Service to access the ChatGPT model (gpt-35-turbo), and Azure Cognitive Search for data indexing and retrieval. Careful, the Azure costs might be substantial for 2TB of data.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-28T19:50:44.240Z
Answers.LastActivityDate: 2023-04-28T19:56:04.683Z
Answers.LastEditDate: 2023-04-28T19:56:04.683Z
Answers.UserDisplayName: Hans-Peter Schrei
Body: <p>I'm still very new to this stuff. I have close to 2TB worth of data hoarded from IRC chats to everyday chats with friends and family.</p>
  <p>But is there a way to pass in this much data into GPT to ask questions about it? Or would I require something else?</p>
  <p><strong>For example:</strong></p>
  <p>&quot;When did Bob tell Jane about the legos he had in school when they were at home?&quot;</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-29T06:17:35.213Z
Comments.Text: Please also consider the privacy implication of this huge stash of personal data. It is not only your personal data but also those of your friends and family. Do you want to only use to learn about data-processing and natural language models or do you want to do something with the results, maybe even publish them in some way?
Comments.UserDisplayName: quarague
Tags: <natural-language-processing><data-preprocessing><chatgpt><gpt>
Title: Process 2TB worth of conversational data hoarded over 40 years. How can I pass this into GPT to ask questions about it?
UserDisplayName: Patoshi パトシ
ThreadText: # Process 2TB worth of conversational data hoarded over 40 years. How can I pass this into GPT to ask questions about it?
  
  (Asked by: username_0 on 2023-04-28)
  
  I'm still very new to this stuff. I have close to 2TB worth of data hoarded from IRC chats to everyday chats with friends and family.
  
  But is there a way to pass in this much data into GPT to ask questions about it? Or would I require something else?
  
  **For example:**
  
  "When did Bob tell Jane about the legos he had in school when they were at home?"
  
  **Tags:** natural-language-processing, data-preprocessing, chatgpt, gpt
  > **Comments:**
  > **username_1**: Please also consider the privacy implication of this huge stash of personal data. It is not only your personal data but also those of your friends and family. Do you want to only use to learn about data-processing and natural language models or do you want to do something with the results, maybe even publish them in some way?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-04-28. Score: 3
  
  Two approaches that I am aware of:
  
  * [Chat your data](https://github.com/hwchase17/chat-your-data)  
    This GitHub repository is accompanied by a [blog post](https://blog.langchain.dev/tutorial-chatgpt-over-your-data/) on how it works schematically. The overall approach is based on the [LangChain library](https://github.com/hwchase17/langchain).
  * [Azure Search OpenAI demo](https://github.com/Azure-Samples/azure-search-openai-demo)
    This approach goes over your own data using the Retrieval Augmented Generation pattern. It uses Azure OpenAI Service to access the ChatGPT model (gpt-35-turbo), and Azure Cognitive Search for data indexing and retrieval. Careful, the Azure costs might be substantial for 2TB of data.
  
  
  ---
  
  