Answers.Body: <p>There's been a couple ideas thrown around wrt the task of &quot;prompt compression&quot;. For example, the <a href="https://github.com/vlad-ds/gptrim/tree/main" rel="nofollow noreferrer">gpttrim</a> algorithm, applies some basic NLP preprocessing to reduce unnecessary tokens (e.g., removing stopwords). <a href="https://arxiv.org/pdf/2304.12512.pdf" rel="nofollow noreferrer">Others have tried prompting ChatGPT to compress text.</a></p>
  <p>There's a couple things to be careful about with these techniques, though:</p>
  <ol>
  <li>Make sure the number of <em>tokens</em> is decreasing, not the number of characters. You're charged per token.</li>
  <li>Such compression is usually lossy, which for your use case of category labels may not be ideal.</li>
  </ol>
  <p>A better approach may be to reconsider your classification pipeline, keeping in mind the accuracy and cost tradeoff of different algorithms. For example: 700 categories is a lot, and I'm sure there's a significant portion that are &quot;obviously wrong&quot; for a given article. Going off of this, you could try some cheap, classical NLP techniques (e.g., TF-IDF + logistic regression) to narrow down the set of possible predictions first, then use the more expensive API.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-24T08:52:13.380Z
Answers.Text: Thank Alexander, can you give me an example of TF-IDF ?
Answers.UserDisplayName: Advanced
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-24T08:57:03.300Z
Answers.Text: Here's an example using scikit-learn: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html
Answers.UserDisplayName: Alexander Wan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-22T07:36:53.203Z
Answers.LastActivityDate: 2023-07-22T07:36:53.203Z
Answers.UserDisplayName: Alexander Wan
Body: <p>I have a collection of 700 categories, all potential classifications for articles. My current need is to create a system that can dynamically categorize short texts or articles according to these 700 categories.</p>
  <p>I've been experimenting with a rudimentary approach using chatGPT to read the categories from a PDF via a plugin. The process is quite straightforward - I input the title and the first two lines of an article, and chatGPT does a fairly decent job of predicting the most fitting category.</p>
  <p>The downside? I'm concerned about its scalability and economic viability. The current method might not work so well when we're talking about classifying a significant number of articles.</p>
  <p>My question to you, my fellow AI enthusiasts: How would you approach designing a system, via an API, capable of doing this quickly and on a large scale?</p>
  <p><strong>Is there a feature that allows the Language Learning Model (LLM) to retain the list of 700 categories in its memory so that I don't have to pass it every time?</strong> I'm aware that the billing structure is token-based, so it would be ideal to submit the categories once (or as few times as possible) and then pose a simple query like:</p>
  <blockquote>
  <p>&quot;Categorize this article based on the categories I previously gave you. Article title: 'Barbie vs Oppenheimer: Which Movie Will Garner Greater Success?'&quot;</p>
  </blockquote>
  <p>Ideally, I'd want this system to be persistently active and capable of processing countless queries over an extended period, say a month or a year.</p>
  <p>So, any ideas on how to design such a system? There are undoubtedly numerous routes to take. I'm really just seeking some initial direction so that I can dive deeper into research on my own.</p>
  
Tags: <classification><open-ai><chatgpt><text-classification>
Title: Noob crafting a simple "Zero-Shot Classifier" Using an API . How can I avoid passing the categories every single request?
UserDisplayName: Advanced
ThreadText: # Noob crafting a simple "Zero-Shot Classifier" Using an API . How can I avoid passing the categories every single request?
  
  (Asked by: username_0 on 2023-07-22)
  
  I have a collection of 700 categories, all potential classifications for articles. My current need is to create a system that can dynamically categorize short texts or articles according to these 700 categories.
  
  I've been experimenting with a rudimentary approach using chatGPT to read the categories from a PDF via a plugin. The process is quite straightforward - I input the title and the first two lines of an article, and chatGPT does a fairly decent job of predicting the most fitting category.
  
  The downside? I'm concerned about its scalability and economic viability. The current method might not work so well when we're talking about classifying a significant number of articles.
  
  My question to you, my fellow AI enthusiasts: How would you approach designing a system, via an API, capable of doing this quickly and on a large scale?
  
  **Is there a feature that allows the Language Learning Model (LLM) to retain the list of 700 categories in its memory so that I don't have to pass it every time?** I'm aware that the billing structure is token-based, so it would be ideal to submit the categories once (or as few times as possible) and then pose a simple query like:
  
  > "Categorize this article based on the categories I previously gave you. Article title: 'Barbie vs Oppenheimer: Which Movie Will Garner Greater Success?'"
  
  Ideally, I'd want this system to be persistently active and capable of processing countless queries over an extended period, say a month or a year.
  
  So, any ideas on how to design such a system? There are undoubtedly numerous routes to take. I'm really just seeking some initial direction so that I can dive deeper into research on my own.
  
  **Tags:** classification, open-ai, chatgpt, text-classification
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-22. Score: 2
  
  There's been a couple ideas thrown around wrt the task of "prompt compression". For example, the [gpttrim](https://github.com/vlad-ds/gptrim/tree/main) algorithm, applies some basic NLP preprocessing to reduce unnecessary tokens (e.g., removing stopwords). [Others have tried prompting ChatGPT to compress text.](https://arxiv.org/pdf/2304.12512.pdf)
  
  There's a couple things to be careful about with these techniques, though:
  
  1. Make sure the number of *tokens* is decreasing, not the number of characters. You're charged per token.
  2. Such compression is usually lossy, which for your use case of category labels may not be ideal.
  
  A better approach may be to reconsider your classification pipeline, keeping in mind the accuracy and cost tradeoff of different algorithms. For example: 700 categories is a lot, and I'm sure there's a significant portion that are "obviously wrong" for a given article. Going off of this, you could try some cheap, classical NLP techniques (e.g., TF-IDF + logistic regression) to narrow down the set of possible predictions first, then use the more expensive API.
  
  > **Comments:**
  > **username_0**: Thank username_1, can you give me an example of TF-IDF ?  
  > **username_1**: Here's an example using scikit-learn: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html  
  
  ---
  
  