Answers.Body: <p>The better formulation of this question is <em>“How do LLMs get aligned with human values?</em>”. Which humans and what values is an interesting societal question. Let’s say, for the purpose of this conversation, it’s with a set of humans who don’t like to hear curse words.</p>
  <p>There are 4 main forms of achieving alignment.  The layer at which each is implemented is shown in [brackets].</p>
  <ol>
  <li>[model] <a href="https://stats.stackexchange.com/questions/193082/what-is-pre-training-a-neural-network"><strong>Pre-train</strong></a> your model on the data set you want to learn from. If you don’t want curse words, strip all the training data of curse words laden content, and the model is better aligned with your “no curse words” values.</li>
  <li>[model] <strong>Fine-tuning</strong> &amp; reinforcement learning w/ human feedback (<a href="https://huggingface.co/blog/rlhf" rel="nofollow noreferrer"><strong>RLHF</strong></a>).  If you have a base model that is good in performance but needs alignment, you can take a smaller set of data than what was used to pre-train, and do a post-train step.  This fine-tuning data is often very carefully curated to achieve specific outcomes or match a specific data distribution.  Fine-tuning on that data often “re-wires” the neurons (via updating the weights of the neurons) to reduce the curse words.  Reinforcement learning can further achieve alignment using a more complicated process of creating a reward model that the base model is further tuned on.</li>
  <li>[application layer] At the prompt layer, <a href="https://openai.com/research/language-models-are-few-shot-learners" rel="nofollow noreferrer"><strong>few-shot learning</strong></a> and <a href="https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api" rel="nofollow noreferrer"><strong>system prompts</strong></a> can be used to prevent the model from using certain words, by way of giving examples.  This is also called “in-context learning” and can be used for many outcomes, alignment is one of them.</li>
  <li>Post-generation of the model, the final result can be passed through another model or ensemble of models. These are typical classifier models that are looking for specific things. You could put any kind of “model” in this step from something naive as keyword filtering, to more sophisticated models like “hate speech”, “violence” or “sexually explicit material’ detection. The best practice is to pass prompt inputs (from users) through <a href="https://platform.openai.com/docs/guides/moderation" rel="nofollow noreferrer"><strong>moderation systems</strong></a>, but the outputs from the LLM can also be checked.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-22T02:15:57.790Z
Answers.LastActivityDate: 2023-07-22T02:15:57.790Z
Answers.UserDisplayName: sghael
Answers.Body: <p><strong>TL;DR; There is no model inherently censored, as big models have big training data which can't be all modified to not include explicit content. Also, we don't want to not-include the explicit content lest we want to take away the abstractions that the NN might learn from that explicit content also. There are multiple layers to causal reasoning involved to design commercially available products like OpenAI's models.</strong></p>
  <p>That's an interesting one. Let's first differentiate between different terms being used. A foundational model is a model that is pre-trained on a set of training data and can be fine-tuned to perform downstream tasks, whereas consequentially a fine-tuned model is one that can perform specific tasks like summarization, topic extraction, NER, etc.</p>
  <p>Now, let's talk about the GPT-3 family for a bit which houses both the Davinci model and ChatGPT (GPT-3.5) model. If you read about it a little bit, OpenAI lets you fine-tune Davinci but not ChatGPT. Davinci was fine-tuned to perform better generations/completions. On the other hand, ChatGPT was fine-tuned to perform humanesque chat conversations and consequentially started outputting more uncensored content (because of-course human texts tend to get witty).</p>
  <p>Then comes the concept of causal reasoning. You can understand it most easily as a conditional statement which is an added layer over the output of the language model. Essentially it forces the model to take certain actions given certain scenarios. One example might be a prompt saying to use a different selection strategy <code>(it becomes greedy at temperature=0)</code> if the next generated token leads to a vulgar word/slang. Another might be pre-added instructions to model input requesting it to return <code>'I can't respond to this query as it pertains to copyright information'</code>. It's basically an elaborate <code>if-else</code> statement that works with language models.</p>
  <p>Coming back to our example of OpenAI's models, while not completely uncensored, DaVinci is susceptible to producing more uncensored content than ChatGPT as evidenced in their blog asking users to use a layer of causal reasoning to best meet their market requirements and that OpenAI will not be held liable to any explicit content generated.</p>
  <p>UPDATE (21 July 2023) - According to <a href="https://decrypt.co/149272/chatgpts-performance-is-slipping-new-study-says" rel="nofollow noreferrer">this</a> UC Berkley study, this causal reasoning might be actually reducing the effectiveness of ChatGPT due to increasing limitations on the content that it can produce.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-21T23:58:40.793Z
Answers.Text: Are you saying it is not doable to prevent sensitive training data, using amount of data as an excuse, from entering the learning process?
Answers.UserDisplayName: harism
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-22T00:23:20.740Z
Answers.Text: I think it's more practicality than the amount of data that I mean here. Of course, you can engineer methods to reduce bias (which of course is necessary), but it shouldn't come at a cost of decreased associations which might in itself be a bias. Ultimately, sensitive information like vulgar words to socially unacceptable ideas should be included in the training data and the model should later be fine-tuned to behave a certain way. We want a foundational model to be able to understand as much as it can.
Answers.UserDisplayName: Chinmay
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-22T00:38:19.433Z
Answers.Text: It is a practicality for sure, and I can only image how much training data there is available and used, when it comes to more larger LLM. Still I wonder maybe social media companies like Facebook have means to recognise offensive language from the stream of text they keep receiving. And if this is doable, why not filter offensive language from LLM training similarly too, just before it enters the training loop.
Answers.UserDisplayName: harism
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-18T16:52:06.453Z
Answers.LastActivityDate: 2023-07-21T23:21:14.173Z
Answers.LastEditDate: 2023-07-21T23:21:14.173Z
Answers.UserDisplayName: Chinmay
Body: <p>I'm looking for references (articles) about how LLM generated content is moderated.</p>
  <p>From a technical point of view, what makes the difference between a so-called &quot;uncensored&quot; LLM, such as Pygmalion 7B, and a &quot;censored&quot; one?</p>
  <p>Does a LLM always generate text that gets moderated later, or is it pre-trained / fine-tuned to generate moderated content by default?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-21T21:52:25.357Z
Comments.Text: Is LLM like ChatGPT able to generate content that needs to be moderated? I am thinking about this from training point of view, is it possible that LLM generates sensitive or illegal content from training set not containing such?
Comments.UserDisplayName: harism
Tags: <reference-request><large-language-models><generative-ai>
Title: How is LLM generated content moderated?
UserDisplayName: StashOfCode
ThreadText: # How is LLM generated content moderated?
  
  (Asked by: username_0 on 2023-07-18)
  
  I'm looking for references (articles) about how LLM generated content is moderated.
  
  From a technical point of view, what makes the difference between a so-called "uncensored" LLM, such as Pygmalion 7B, and a "censored" one?
  
  Does a LLM always generate text that gets moderated later, or is it pre-trained / fine-tuned to generate moderated content by default?
  
  **Tags:** reference-request, large-language-models, generative-ai
  
  > **Comments:**
  > **username_1**: Is LLM like ChatGPT able to generate content that needs to be moderated? I am thinking about this from training point of view, is it possible that LLM generates sensitive or illegal content from training set not containing such?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-07-22. Score: 2
  
  The better formulation of this question is *“How do LLMs get aligned with human values?*”. Which humans and what values is an interesting societal question. Let’s say, for the purpose of this conversation, it’s with a set of humans who don’t like to hear curse words.
  
  There are 4 main forms of achieving alignment. The layer at which each is implemented is shown in [brackets].
  
  1. [model] [**Pre-train**](https://stats.stackexchange.com/questions/193082/what-is-pre-training-a-neural-network) your model on the data set you want to learn from. If you don’t want curse words, strip all the training data of curse words laden content, and the model is better aligned with your “no curse words” values.
  2. [model] **Fine-tuning** & reinforcement learning w/ human feedback ([**RLHF**](https://huggingface.co/blog/rlhf)). If you have a base model that is good in performance but needs alignment, you can take a smaller set of data than what was used to pre-train, and do a post-train step. This fine-tuning data is often very carefully curated to achieve specific outcomes or match a specific data distribution. Fine-tuning on that data often “re-wires” the neurons (via updating the weights of the neurons) to reduce the curse words. Reinforcement learning can further achieve alignment using a more complicated process of creating a reward model that the base model is further tuned on.
  3. [application layer] At the prompt layer, [**few-shot learning**](https://openai.com/research/language-models-are-few-shot-learners) and [**system prompts**](https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api) can be used to prevent the model from using certain words, by way of giving examples. This is also called “in-context learning” and can be used for many outcomes, alignment is one of them.
  4. Post-generation of the model, the final result can be passed through another model or ensemble of models. These are typical classifier models that are looking for specific things. You could put any kind of “model” in this step from something naive as keyword filtering, to more sophisticated models like “hate speech”, “violence” or “sexually explicit material’ detection. The best practice is to pass prompt inputs (from users) through [**moderation systems**](https://platform.openai.com/docs/guides/moderation), but the outputs from the LLM can also be checked.
  
  
  ---
  
  ### Answer by username_3 on 2023-07-18. Score: 2
  
  **TL;DR; There is no model inherently censored, as big models have big training data which can't be all modified to not include explicit content. Also, we don't want to not-include the explicit content lest we want to take away the abstractions that the NN might learn from that explicit content also. There are multiple layers to causal reasoning involved to design commercially available products like OpenAI's models.**
  
  That's an interesting one. Let's first differentiate between different terms being used. A foundational model is a model that is pre-trained on a set of training data and can be fine-tuned to perform downstream tasks, whereas consequentially a fine-tuned model is one that can perform specific tasks like summarization, topic extraction, NER, etc.
  
  Now, let's talk about the GPT-3 family for a bit which houses both the Davinci model and ChatGPT (GPT-3.5) model. If you read about it a little bit, OpenAI lets you fine-tune Davinci but not ChatGPT. Davinci was fine-tuned to perform better generations/completions. On the other hand, ChatGPT was fine-tuned to perform humanesque chat conversations and consequentially started outputting more uncensored content (because of-course human texts tend to get witty).
  
  Then comes the concept of causal reasoning. You can understand it most easily as a conditional statement which is an added layer over the output of the language model. Essentially it forces the model to take certain actions given certain scenarios. One example might be a prompt saying to use a different selection strategy `(it becomes greedy at temperature=0)` if the next generated token leads to a vulgar word/slang. Another might be pre-added instructions to model input requesting it to return `'I can't respond to this query as it pertains to copyright information'`. It's basically an elaborate `if-else` statement that works with language models.
  
  Coming back to our example of OpenAI's models, while not completely uncensored, DaVinci is susceptible to producing more uncensored content than ChatGPT as evidenced in their blog asking users to use a layer of causal reasoning to best meet their market requirements and that OpenAI will not be held liable to any explicit content generated.
  
  UPDATE (21 July 2023) - According to [this](https://decrypt.co/149272/chatgpts-performance-is-slipping-new-study-says) UC Berkley study, this causal reasoning might be actually reducing the effectiveness of ChatGPT due to increasing limitations on the content that it can produce.
  
  > **Comments:**
  > **username_1**: Are you saying it is not doable to prevent sensitive training data, using amount of data as an excuse, from entering the learning process?  
  > **username_3**: I think it's more practicality than the amount of data that I mean here. Of course, you can engineer methods to reduce bias (which of course is necessary), but it shouldn't come at a cost of decreased associations which might in itself be a bias. Ultimately, sensitive information like vulgar words to socially unacceptable ideas should be included in the training data and the model should later be fine-tuned to behave a certain way. We want a foundational model to be able to understand as much as it can.  
  > **username_1**: It is a practicality for sure, and I can only image how much training data there is available and used, when it comes to more larger LLM. Still I wonder maybe social media companies like Facebook have means to recognise offensive language from the stream of text they keep receiving. And if this is doable, why not filter offensive language from LLM training similarly too, just before it enters the training loop.  
  
  ---
  
  