Answers.Body: <p>You are very vague in the description, and I don't know anything about GLSL shader, which means I have no idea how articulate is this.</p>
  <p>However, consider this:</p>
  <ul>
  <li>if your set of actions (possible combinations of your values) is countable, and in the order of the hundreds, which I don't think it's the case as it would be too simple, don't use models</li>
  <li>if it's not countable, but still relatively small, consider simple function approximators, such as linear regression and use some coding, like tile coding or coarse coding (see for example here <a href="https://cseweb.ucsd.edu//%7Egary/190-RL/Lecture_Ch_8-2013.pdf" rel="nofollow noreferrer">https://cseweb.ucsd.edu//~gary/190-RL/Lecture_Ch_8-2013.pdf</a>)</li>
  <li>if it's not that small, then pass to DeepRL</li>
  </ul>
  <p>Now, onto the implementation... You have a problem, which is that you are dealing with only one state in your MDP, and a ton of actions. RL would learn a single good configuration of those parameters. To overcome this, you have to use the trick of GANs, which is to use random noise as input</p>
  <p>RLHF in your case needs only 2 things, a policy and a reward function</p>
  <ul>
  <li>Policy: create a NN that takes noise and outputs values (now if they are discrete, use output layers with softmax activation function, if they are continuous, use linear or some bounded activation that fits your domain)</li>
  <li>Reward: a neural network that takes some configuration of your parameters, and will output a score  (better this model to be simple, so it's easier and quicker to learn)</li>
  </ul>
  <p>At this point, start training:</p>
  <ol>
  <li>train reward function with some samples from your &quot;human&quot; with supervised learning</li>
  <li>train the policy with PPO or REINFORCE or whatever actor policy optimization you prefer</li>
  <li>repeat</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-06T10:41:19.140Z
Answers.Text: The GLSL shader has 10 parameters, each can have a value between 0.00 and 5.00, so many combinations are possible. If I understand your suggestion correctly the reward functions will learn a representation of the users preferences from the user directly, so that later I can take the human out of the loop so that the policy network can learn on its own. My case a bit special because there always will be a "human in the loop", the network should not learn on its own, I know it will be slow but thats how I want it. So isnt it enough to pass the reward from the user directly to the PPO?
Answers.UserDisplayName: fiatmoney
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-06T10:43:23.183Z
Answers.Text: Also I didnt mention this but I want to pretrain the policy network with a GAN so it produces outputs that are similar to some examples, so its not completely random. After that this network will be "fitted" to the user trough RL. So maybe the suggestion by @foreverska is valid. I pretrain the network as a regular GAN and then replace the discriminator by the user. Maybe that would also be a solution?
Answers.UserDisplayName: fiatmoney
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-06T10:46:22.907Z
Answers.Text: @fiatmoney the problem with such solution is that GANs need a ton of data to converge, otherwise the discriminator can easily learn by heart which are the exact samples in the actual dataset, and then the only way the generator can fool the discriminator, is to actually predict the exact same samples you have in your dataset, thus you won't get generalization... maybe a pretraining with a VAE might be slightly better
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-04T22:38:59.530Z
Answers.LastActivityDate: 2023-08-04T22:38:59.530Z
Answers.UserDisplayName: Alberto
Body: <p>I'm trying to build a simple reinforcement learning model that will output a set of parameters that will be passed to a GLSL shader. The human user will rate this visual output, for example &quot;good&quot;, &quot;neutral&quot; or &quot;bad&quot; (1, 0, -1). I want to use this as a reward for a neural net, so that I can optimize it. The expected result is that over time the model will produce outputs that the user will like more.</p>
  <p>I'm new to reinforcement learning and a bit confused about what type of model to use for this problem. So far I looked at <a href="https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html" rel="nofollow noreferrer">DQN</a> and <a href="https://pytorch.org/tutorials/intermediate/reinforcement_ppo.html" rel="nofollow noreferrer">PPO</a> examples from the official Pytorch documentation. As far as I can tell these apply to more &quot;game-like&quot; scenarios where you pass the neural net the current state of the environment and the output is the probability of the action that the agent should take in the next step. My situation is quiet different, because instead of the state I will pass the network a random input tensor (for now, maybe it wont be random in the end) and the output will be a one dimensional tensor of a certain length that I will pass to the GLSL shader. If the user will like the output then the whole output is &quot;correct&quot; and is getting a reward. While for example DQN is more like a classification problem where only one &quot;action&quot; is correct.</p>
  <p>My intuition is telling me that I probably don't need something as complicated as DQN or PPO. Instead I think I should use a regular fully connected linear feed-forward model with some activation functions and just construct a loss function that will somehow take the reward and backpropagate through it to optimize the model. But I'm not sure if this would be the correct approach or how to do it.</p>
  <p>If anyone could suggest a reinforcement learning model that would suitable for this problem or a loss function that would allow me to optimize a simple feed-forward network using a reward. Thank you</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-04T20:33:23.323Z
Comments.Text: As much as I too love jumping to RL for a lot of things I don't think this is a fit.  RL often tries to solve the problem of requiring many steps to reach a reward.  It sound like in this scheme the algorithm is rewarded or not at every step with no continuity in actions.  This feels a bit closer to a GAN with a human as the discriminator.  If one was simply looking for a singular pattern someone might like, it probably could be accomplished in a "20 questions"-esque tree search manner.
Comments.UserDisplayName: foreverska
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-04T22:28:20.827Z
Comments.Text: @foreverska I don't agree with you, this is definitely an RL problem, as you don't have any way to backpropagate the error, since there is no such thing as a correct answer/target
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-05T16:39:19.577Z
Comments.Text: There are correct answers, they are the ones the user evaluates as pleasing.  A GAN can be trained if the discriminator is a predictor for human preference.  Probably should start by pre-training the discriminator on example settings which do something (because there are likely many settings which do little to nothing).  Show generator outputs that the discriminator thinks are in the positive class.  If a human interacts with a given visualization (positively or negatively) add it to the discriminator training set.  Train the GAN as normal.
Comments.UserDisplayName: foreverska
Tags: <reinforcement-learning><loss>
Title: Simple reinforcement learning with human feedback to generate "pleasurable" visual output
UserDisplayName: fiatmoney
ThreadText: # Simple reinforcement learning with human feedback to generate "pleasurable" visual output
  
  (Asked by: username_0 on 2023-08-04)
  
  I'm trying to build a simple reinforcement learning model that will output a set of parameters that will be passed to a GLSL shader. The human user will rate this visual output, for example "good", "neutral" or "bad" (1, 0, -1). I want to use this as a reward for a neural net, so that I can optimize it. The expected result is that over time the model will produce outputs that the user will like more.
  
  I'm new to reinforcement learning and a bit confused about what type of model to use for this problem. So far I looked at [DQN](https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html) and [PPO](https://pytorch.org/tutorials/intermediate/reinforcement_ppo.html) examples from the official Pytorch documentation. As far as I can tell these apply to more "game-like" scenarios where you pass the neural net the current state of the environment and the output is the probability of the action that the agent should take in the next step. My situation is quiet different, because instead of the state I will pass the network a random input tensor (for now, maybe it wont be random in the end) and the output will be a one dimensional tensor of a certain length that I will pass to the GLSL shader. If the user will like the output then the whole output is "correct" and is getting a reward. While for example DQN is more like a classification problem where only one "action" is correct.
  
  My intuition is telling me that I probably don't need something as complicated as DQN or PPO. Instead I think I should use a regular fully connected linear feed-forward model with some activation functions and just construct a loss function that will somehow take the reward and backpropagate through it to optimize the model. But I'm not sure if this would be the correct approach or how to do it.
  
  If anyone could suggest a reinforcement learning model that would suitable for this problem or a loss function that would allow me to optimize a simple feed-forward network using a reward. Thank you
  
  **Tags:** reinforcement-learning, loss
  > **Comments:**
  > **username_1**: As much as I too love jumping to RL for a lot of things I don't think this is a fit.  RL often tries to solve the problem of requiring many steps to reach a reward.  It sound like in this scheme the algorithm is rewarded or not at every step with no continuity in actions.  This feels a bit closer to a GAN with a human as the discriminator.  If one was simply looking for a singular pattern someone might like, it probably could be accomplished in a "20 questions"-esque tree search manner.  
  > **username_2**: @foreverska I don't agree with you, this is definitely an RL problem, as you don't have any way to backpropagate the error, since there is no such thing as a correct answer/target  
  > **username_1**: There are correct answers, they are the ones the user evaluates as pleasing.  A GAN can be trained if the discriminator is a predictor for human preference.  Probably should start by pre-training the discriminator on example settings which do something (because there are likely many settings which do little to nothing).  Show generator outputs that the discriminator thinks are in the positive class.  If a human interacts with a given visualization (positively or negatively) add it to the discriminator training set.  Train the GAN as normal.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-08-04. Score: 1
  
  You are very vague in the description, and I don't know anything about GLSL shader, which means I have no idea how articulate is this.
  
  However, consider this:
  
  * if your set of actions (possible combinations of your values) is countable, and in the order of the hundreds, which I don't think it's the case as it would be too simple, don't use models
  * if it's not countable, but still relatively small, consider simple function approximators, such as linear regression and use some coding, like tile coding or coarse coding (see for example here [https://cseweb.ucsd.edu//~gary/190-RL/Lecture\_Ch\_8-2013.pdf](https://cseweb.ucsd.edu//%7Egary/190-RL/Lecture_Ch_8-2013.pdf))
  * if it's not that small, then pass to DeepRL
  
  Now, onto the implementation... You have a problem, which is that you are dealing with only one state in your MDP, and a ton of actions. RL would learn a single good configuration of those parameters. To overcome this, you have to use the trick of GANs, which is to use random noise as input
  
  RLHF in your case needs only 2 things, a policy and a reward function
  
  * Policy: create a NN that takes noise and outputs values (now if they are discrete, use output layers with softmax activation function, if they are continuous, use linear or some bounded activation that fits your domain)
  * Reward: a neural network that takes some configuration of your parameters, and will output a score (better this model to be simple, so it's easier and quicker to learn)
  
  At this point, start training:
  
  1. train reward function with some samples from your "human" with supervised learning
  2. train the policy with PPO or REINFORCE or whatever actor policy optimization you prefer
  3. repeat
  
  > **Comments:**
  > **username_0**: The GLSL shader has 10 parameters, each can have a value between 0.00 and 5.00, so many combinations are possible. If I understand your suggestion correctly the reward functions will learn a representation of the users preferences from the user directly, so that later I can take the human out of the loop so that the policy network can learn on its own. My case a bit special because there always will be a "human in the loop", the network should not learn on its own, I know it will be slow but thats how I want it. So isnt it enough to pass the reward from the user directly to the PPO?  
  > **username_0**: Also I didnt mention this but I want to pretrain the policy network with a GAN so it produces outputs that are similar to some examples, so its not completely random. After that this network will be "fitted" to the user trough RL. So maybe the suggestion by @foreverska is valid. I pretrain the network as a regular GAN and then replace the discriminator by the user. Maybe that would also be a solution?  
  > **username_2**: @fiatmoney the problem with such solution is that GANs need a ton of data to converge, otherwise the discriminator can easily learn by heart which are the exact samples in the actual dataset, and then the only way the generator can fool the discriminator, is to actually predict the exact same samples you have in your dataset, thus you won't get generalization... maybe a pretraining with a VAE might be slightly better  
  
  ---
  
  