Answers.Body: <p>There are two approaches that I am aware of:</p>
  <ul>
  <li><p><a href="https://github.com/hwchase17/chat-your-data" rel="nofollow noreferrer">Chat your data</a><br />
  This GitHub repository is accompanied by a <a href="https://blog.langchain.dev/tutorial-chatgpt-over-your-data/" rel="nofollow noreferrer">blog post</a> on how it works schematically. The overall approach is based on the <a href="https://github.com/hwchase17/langchain" rel="nofollow noreferrer">LangChain library</a>.</p>
  </li>
  <li><p><a href="https://github.com/Azure-Samples/azure-search-openai-demo" rel="nofollow noreferrer">Azure Search OpenAI demo</a>
  This approach goes over your own data using the Retrieval Augmented Generation pattern. It uses Azure OpenAI Service to access the ChatGPT model (gpt-35-turbo), and Azure Cognitive Search for data indexing and retrieval. Careful, the Azure costs might be substantial depending on the size of your data.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-29T09:42:13.957Z
Answers.LastActivityDate: 2023-04-29T09:47:18.523Z
Answers.LastEditDate: 2023-04-29T09:47:18.523Z
Answers.UserDisplayName: Hans-Peter Schrei
Body: <h3>Short version</h3>
  <p>I have a computer which doesn't work properly but I can't figure out what's wrong. I've tried to install other operating systems, run some system tests and look at the system logs, but couldn't figure out what's wrong.</p>
  <p>The log files I have are tens of thousand of lines long and around the megabyte, as an order of magnitude. That's way too big for me to carefully look at everything and understand which warnings/errors are the important ones and which ones are unrelated.</p>
  <p>Is there any AI tool which I could use to look at my logs and tell me which lines are the most suspicious and should be investigated further?</p>
  <p>I believe LLMs like ChatGPT might be able to do this, but I don't see how I could use them to process 1MB of data.</p>
  <hr />
  <h3>Long version with specific details</h3>
  <p>The computer I mentioned is an Android phone. It's stuck displaying the boot animation; sometimes it displays the real screen for a few seconds, but then it goes back to the boot animation; all this, without ever rebooting. I factory-resetted it and installed other ROMs, but the problem persists: sometimes it shows the welcome screen, which seems to work too, but it doesn't last long.</p>
  <p>I can connect to that phone via ADB and managed to look at the logs. I saved the output of <code>dmesg</code> (5809 lines, 539KB) and <code>logcat</code> (24964 lines, 2.9MB), but they're too big for me to look manually, and they contain way too many errors, most of which are unrelated to the real problem.</p>
  <p>I tried to write some scripts to help me process those files (e.g. collapse the duplicate lines, group the lines according to some regexes etc), but that's still not enough.</p>
  <p>I ran out of ideas and the only remaining solution I can think of is to use some AI to go through those logs and spot the lines which are the most worrying. I'm unsure whether any AI exists which could help me with this though.</p>
  
Tags: <natural-language-processing><data-preprocessing>
Title: AI to understand what problem my computer has, for instance by looking at the logs which are too big for me
UserDisplayName: peoro
ThreadText: # AI to understand what problem my computer has, for instance by looking at the logs which are too big for me
  
  (Asked by: username_0 on 2023-04-29)
  
  ### Short version
  
  I have a computer which doesn't work properly but I can't figure out what's wrong. I've tried to install other operating systems, run some system tests and look at the system logs, but couldn't figure out what's wrong.
  
  The log files I have are tens of thousand of lines long and around the megabyte, as an order of magnitude. That's way too big for me to carefully look at everything and understand which warnings/errors are the important ones and which ones are unrelated.
  
  Is there any AI tool which I could use to look at my logs and tell me which lines are the most suspicious and should be investigated further?
  
  I believe LLMs like ChatGPT might be able to do this, but I don't see how I could use them to process 1MB of data.
  
  ---
  
  ### Long version with specific details
  
  The computer I mentioned is an Android phone. It's stuck displaying the boot animation; sometimes it displays the real screen for a few seconds, but then it goes back to the boot animation; all this, without ever rebooting. I factory-resetted it and installed other ROMs, but the problem persists: sometimes it shows the welcome screen, which seems to work too, but it doesn't last long.
  
  I can connect to that phone via ADB and managed to look at the logs. I saved the output of `dmesg` (5809 lines, 539KB) and `logcat` (24964 lines, 2.9MB), but they're too big for me to look manually, and they contain way too many errors, most of which are unrelated to the real problem.
  
  I tried to write some scripts to help me process those files (e.g. collapse the duplicate lines, group the lines according to some regexes etc), but that's still not enough.
  
  I ran out of ideas and the only remaining solution I can think of is to use some AI to go through those logs and spot the lines which are the most worrying. I'm unsure whether any AI exists which could help me with this though.
  
  **Tags:** natural-language-processing, data-preprocessing
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-29. Score: 1
  
  There are two approaches that I am aware of:
  
  * [Chat your data](https://github.com/hwchase17/chat-your-data)  
    This GitHub repository is accompanied by a [blog post](https://blog.langchain.dev/tutorial-chatgpt-over-your-data/) on how it works schematically. The overall approach is based on the [LangChain library](https://github.com/hwchase17/langchain).
  * [Azure Search OpenAI demo](https://github.com/Azure-Samples/azure-search-openai-demo)
    This approach goes over your own data using the Retrieval Augmented Generation pattern. It uses Azure OpenAI Service to access the ChatGPT model (gpt-35-turbo), and Azure Cognitive Search for data indexing and retrieval. Careful, the Azure costs might be substantial depending on the size of your data.
  
  
  ---
  
  