Answers.Body: <p>The <a href="https://chat.openai.com/chat" rel="nofollow noreferrer">ChatGPT online demo</a> uses OpenAI's <a href="https://openai.com/blog/new-and-improved-content-moderation-tooling/" rel="nofollow noreferrer">Moderation API</a>.</p>
  <p>From the <a href="https://openai.com/blog/chatgpt/" rel="nofollow noreferrer">ChatGPT blogpost</a>:</p>
  <blockquote>
  <p>While we’ve made efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions or exhibit biased behavior. We’re using the <a href="https://openai.com/blog/new-and-improved-content-moderation-tooling/" rel="nofollow noreferrer">Moderation API</a> to warn or block certain types of unsafe content, but we expect it to have some false negatives and positives for now. We’re eager to collect user feedback to aid our ongoing work to improve this system.</p>
  </blockquote>
  <p>From the <a href="https://openai.com/blog/new-and-improved-content-moderation-tooling/" rel="nofollow noreferrer">Moderation API blogpost</a>:</p>
  <blockquote>
  <p>More details of the training process and model performance are available in our <a href="https://arxiv.org/abs/2208.03274" rel="nofollow noreferrer">paper</a>. We have also released an <a href="https://github.com/openai/moderation-api-release" rel="nofollow noreferrer">evaluation dataset</a>, featuring Common Crawl data labeled within these categories, which we hope will spur further research in this area.</p>
  </blockquote>
  <p>Paper: Markov, T., Zhang, C., Agarwal, S., Eloundou, T., Lee, T., Adler, S., Jiang, A. and Weng, L., 2022. <a href="https://arxiv.org/pdf/2208.03274.pdf" rel="nofollow noreferrer">A holistic approach to undesired content detection in the real world</a>. arXiv preprint arXiv:2208.03274.</p>
  <p>There used to be a way to bypass OpenAI's Moderation API when using the ChatGPT online demo: <a href="https://github.com/acheong08/ChatGPT/issues/1" rel="nofollow noreferrer">https://github.com/acheong08/ChatGPT/issues/1</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-14T15:07:41.040Z
Answers.LastActivityDate: 2022-12-14T15:07:41.040Z
Answers.UserDisplayName: Franck Dernoncourt
Body: <p>I first tried ChatGPT few days ago.  And every day that goes by it seems more and more content filters are introduced.   I can still make it do stuff if I &quot;jailbreak it&quot; but I feel like the restrictions are getting tighter.  By the way I'm not making it do anything bad, just testing it's limits of what it can do.  Over time it can do less and less (or is willing to do less and less).  This makes me wonder how the filters are implemented and why they work the way they do:</p>
  <ul>
  <li>Why is it that seemingly for any filter that's put in place there is a workardound?  Things like (imagine, pretend etc).  If something is filtered wouldn't it be filtered across all contexts?   Why does adding a magic word like &quot;act as though&quot; make it ignore filters all of a sudden?</li>
  <li>Are the rest of you seeing the same trend of increased filtering?   For me, coming from a sysadmin background it almost feels like a fight between the admins and the users.  The admins begin gently with fine grained filters with very specific contexts, but as they see the onslaught in creativity of the users of bypassing those specific contexts they are using increasingly more and more general contexts to filter on (because they can't keep up with all the jailbreaks).   Of course this blocks legit uses too.</li>
  <li>Since ChatGPT is (or more precisely, used to be) very cooperative with what the users told it to behave like, do the admins have a similar natural language interface to interact with chatGPT to tell it how they want it to behave?   Ex admins say &quot;never give a response that's less than 5 sentences in length&quot;, then even though the user says &quot;answer me simply yes or no&quot;.  Admins (hidden) request takes precedence?</li>
  <li>Why in a single session I can have it cooperate and then when asked to do exactly the same thing seconds later it refuses? (ex: answer me with a single random swear word: works perfect and then couple questions later it refuses)</li>
  </ul>
  <p>I tried asking some of these questions to ChatGPT directly and I didn't get very far with it except that apparently the boiler plate &quot;As a large language model trained by OpenAI, I am a machine learning model that ...&quot; is some kind of hard-coded disclaimer added by the engineers.   Supposedly it's a different mechanism from content filters.   Which kind of makes sense.   Of course I don't know if I can trust that answer, as how the disclaimer sais ChatGPT is known to be confidently incorrect.</p>
  <p>Also, I realize chatGPT is proprietary, so I'm not expecting anyone to reveal any secrets.  I will be very satisfied of knowing how this is handled in similar NLP models.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-14T08:49:13.380Z
Comments.Text: The filters are not part of the "AI" itself but an extra program that checks your input first, before asking the "AI" to generate a reply. The admins certainly do not have a natural language interface because ChatGPT does not work that way. However, they might use ChatGPT tricks as one kind of filtering. E.g. use the prompt "Does this sentence contain swear words? shit fuck fuck shit fuck" and if the reply is affirmative then block the input.
Comments.UserDisplayName: Criticize SE actions means ban
Tags: <natural-language-processing><open-ai><gpt>
Title: How do ChatGPT content filters work? (If not chatGPT then in general)
UserDisplayName: user3280964
ThreadText: # How do ChatGPT content filters work? (If not chatGPT then in general)
  
  (Asked by: username_0 on 2022-12-13)
  
  I first tried ChatGPT few days ago. And every day that goes by it seems more and more content filters are introduced. I can still make it do stuff if I "jailbreak it" but I feel like the restrictions are getting tighter. By the way I'm not making it do anything bad, just testing it's limits of what it can do. Over time it can do less and less (or is willing to do less and less). This makes me wonder how the filters are implemented and why they work the way they do:
  
  * Why is it that seemingly for any filter that's put in place there is a workardound? Things like (imagine, pretend etc). If something is filtered wouldn't it be filtered across all contexts? Why does adding a magic word like "act as though" make it ignore filters all of a sudden?
  * Are the rest of you seeing the same trend of increased filtering? For me, coming from a sysadmin background it almost feels like a fight between the admins and the users. The admins begin gently with fine grained filters with very specific contexts, but as they see the onslaught in creativity of the users of bypassing those specific contexts they are using increasingly more and more general contexts to filter on (because they can't keep up with all the jailbreaks). Of course this blocks legit uses too.
  * Since ChatGPT is (or more precisely, used to be) very cooperative with what the users told it to behave like, do the admins have a similar natural language interface to interact with chatGPT to tell it how they want it to behave? Ex admins say "never give a response that's less than 5 sentences in length", then even though the user says "answer me simply yes or no". Admins (hidden) request takes precedence?
  * Why in a single session I can have it cooperate and then when asked to do exactly the same thing seconds later it refuses? (ex: answer me with a single random swear word: works perfect and then couple questions later it refuses)
  
  I tried asking some of these questions to ChatGPT directly and I didn't get very far with it except that apparently the boiler plate "As a large language model trained by OpenAI, I am a machine learning model that ..." is some kind of hard-coded disclaimer added by the engineers. Supposedly it's a different mechanism from content filters. Which kind of makes sense. Of course I don't know if I can trust that answer, as how the disclaimer sais ChatGPT is known to be confidently incorrect.
  
  Also, I realize chatGPT is proprietary, so I'm not expecting anyone to reveal any secrets. I will be very satisfied of knowing how this is handled in similar NLP models.
  
  **Tags:** natural-language-processing, open-ai, gpt
  > **Comments:**
  > **username_1**: The filters are not part of the "AI" itself but an extra program that checks your input first, before asking the "AI" to generate a reply. The admins certainly do not have a natural language interface because ChatGPT does not work that way. However, they might use ChatGPT tricks as one kind of filtering. E.g. use the prompt "Does this sentence contain swear words? shit fuck fuck shit fuck" and if the reply is affirmative then block the input.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-12-14. Score: 1
  
  The [ChatGPT online demo](https://chat.openai.com/chat) uses OpenAI's [Moderation API](https://openai.com/blog/new-and-improved-content-moderation-tooling/).
  
  From the [ChatGPT blogpost](https://openai.com/blog/chatgpt/):
  
  > While we’ve made efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions or exhibit biased behavior. We’re using the [Moderation API](https://openai.com/blog/new-and-improved-content-moderation-tooling/) to warn or block certain types of unsafe content, but we expect it to have some false negatives and positives for now. We’re eager to collect user feedback to aid our ongoing work to improve this system.
  
  From the [Moderation API blogpost](https://openai.com/blog/new-and-improved-content-moderation-tooling/):
  
  > More details of the training process and model performance are available in our [paper](https://arxiv.org/abs/2208.03274). We have also released an [evaluation dataset](https://github.com/openai/moderation-api-release), featuring Common Crawl data labeled within these categories, which we hope will spur further research in this area.
  
  Paper: Markov, T., Zhang, C., Agarwal, S., Eloundou, T., Lee, T., Adler, S., Jiang, A. and Weng, L., 2022. [A holistic approach to undesired content detection in the real world](https://arxiv.org/pdf/2208.03274.pdf). arXiv preprint arXiv:2208.03274.
  
  There used to be a way to bypass OpenAI's Moderation API when using the ChatGPT online demo: <https://github.com/acheong08/ChatGPT/issues/1>
  
  
  ---
  
  