Answers.Body: <p>RL is a learning algorithm, just like evolutionary strategies or SGD. ANNs can be used as the value function in RL, which is learned. The classic example is deep RL with AlphaGo. However, there are many RL methods which don’t use NNs, like Q learning as you mentioned.</p>
  <p>A clarification by Sutton and Barto which may clear up the area of confusion: RL is both a general field of models/techniques as well as a set of learning algorithms. It depends on the context, but it seems like what you are referring to is RL as a method.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-04T10:56:10.200Z
Answers.Text: RL is not **a** learning algorithm. It's a field/area of study. But there are RL algorithms. I think it's important to use the right terminology to avoid further confusion.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-01T19:19:51.630Z
Answers.LastActivityDate: 2024-07-04T19:40:04.540Z
Answers.LastEditDate: 2024-07-04T19:40:04.540Z
Answers.UserDisplayName: lnx
Answers.Body: <p>So now that I inquired further about the question, here are some answers.</p>
  <p>Generally speaking, when we use reinforcement learning (RL) and artificial neural networks (ANNs) in the same sentence, it's about using ANNs to approximate RL algorithms. In this case, we train the ANNs as we would normally do (Stochastic gradient descent...).</p>
  <p>Here, the ANNs serve as the RL algorithms agents.</p>
  <p>However, we can also use RL for :</p>
  <ul>
  <li>Hyperparameter optimization and Neural architecture search (NAS), where ANNs are treated as the environment. We use RL algorithms to find the best architecture (number for layers...) or hyperparameters (activation function...). See Zoph, B., &amp; Le, Q.V. (2017) &quot;Neural Architecture Search with Reinforcement Learning&quot; as an example.</li>
  <li>Weights optimization (i.e. RL algorithms replacing the classical training algorithms), which is a very rare use case as our traditional optimizers prove to be already pretty efficient. See Bello, I., Zoph, B., Vasudevan, V., &amp; Le, Q.V. (2017) &quot;Neural Optimizer Search with Reinforcement Learning&quot; as an example.</li>
  </ul>
  <p>In those cases, the RL algorithms are the agent, and the ANN is the subject being optimised (environment).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-30T15:28:08.823Z
Answers.LastActivityDate: 2024-06-30T15:29:37.670Z
Answers.LastEditDate: 2024-06-30T15:29:37.670Z
Answers.UserDisplayName: Esteban
Answers.Body: <p>I would say that this is just be the semantics of the sentence you read. I would understand the sentence 'RL can be used as a framework to train an ANN' to mean that we are using an RL algorithm to learn the weights of an ANN -- an example would be to learn the parameters of a deep Q-network, which is used to approximate Q-values.</p>
  <p>What's important to remember is that, as @Alberto points out in the comments, RL is generally separate to ANNs. RL is a branch of machine learning that <em>can</em> use ANNs as function approximators for the Q-function/policy (this is the context in which one might say that RL is 'training' the ANN), but is in general not tied to them (obviously you could argue that right now, they are tied to ANNs in the sense that to get them to work in real world settings, you need a deep network for the function approximations, but this might not always be true).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-30T15:01:39.953Z
Answers.Text: I fully understand the second part, the confusion remain in the first part, I'm trying to clarify : while you say it is possible to use an RL algorithm to learn-optimise the weights of an ANN, you talk about DQN, which I thought were precisely an example of ANN being used inside a RL solution, and that DQN are trained using the classic optimisers we know (like gradient descent etc...). I am rather expecting that if RL can be use to train ANN, it's by incorporating novel optimisers (same way genetic algorithm can be use to optimise ANN weights), but that might not be the case.
Answers.UserDisplayName: Esteban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-01T07:35:29.660Z
Answers.Text: okay, yes, I can see based on the answer provided that this is the angle you were trying to understand. indeed, RL has been used in neural architecture search though it is quite limited currently.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-30T14:41:47.230Z
Answers.LastActivityDate: 2024-06-30T14:41:47.230Z
Answers.UserDisplayName: David
Body: <p>I am learning about deep learning. Currently just at a <strong>superficial</strong> level, I think I am misunderstanding how reinforcement learning and artificial neural networks are used together.</p>
  <p>For what I first understood, artificial neural networks can be used as <strong>part</strong> of a broader RL solution, enhancing the so-called Q-learning algorithms that are normally utilised there. Yet after prompting ChatGPT, it seems that RL can also be used as a framework to train ANNs, but then it gets confusing (maybe it is hallucinating ?).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-30T01:04:05.753Z
Comments.Text: RL has nothing to do with ANN, though if you want to make it scalable, you use it with ANN (chatgpt is a ginormous compressor of a ngram model, thus you can replace with a huge table ideally)
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-30T01:46:42.990Z
Comments.Text: But what about policy gradient methods like PPO that was introduced by OpenAI to fine tune their models ? (Just found that)
Comments.UserDisplayName: Esteban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-30T09:56:24.067Z
Comments.Text: you can use PPO with tabular RL as well, so not sure what you mean
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-30T14:35:50.803Z
Comments.Text: Your first answer suggests that we can use ANN to scale RL (for what I understand, when environment have large state spaces). So that's for "ANN used in RL".
  
  However, I also found after further research that we can apply the RL paradigm (find the best policy that maximise the cumulative reward if I am correct) to **train** ANN, whether by treating ANN as a policy (policy gradient method) or as an actor (actor-critic methods). 
  
  It suggests that there is now two different relationships between ANN and RL to consider, but is it correct ?
Comments.UserDisplayName: Esteban
Tags: <neural-networks><reinforcement-learning>
Title: Do we use RL to train ANNs, or use ANNs as par of a RL solution?
UserDisplayName: Esteban
ThreadText: # Do we use RL to train ANNs, or use ANNs as par of a RL solution?
  
  (Asked by: username_0 on 2024-06-29)
  
  I am learning about deep learning. Currently just at a **superficial** level, I think I am misunderstanding how reinforcement learning and artificial neural networks are used together.
  
  For what I first understood, artificial neural networks can be used as **part** of a broader RL solution, enhancing the so-called Q-learning algorithms that are normally utilised there. Yet after prompting ChatGPT, it seems that RL can also be used as a framework to train ANNs, but then it gets confusing (maybe it is hallucinating ?).
  
  **Tags:** neural-networks, reinforcement-learning
  > **Comments:**
  > **username_1**: RL has nothing to do with ANN, though if you want to make it scalable, you use it with ANN (chatgpt is a ginormous compressor of a ngram model, thus you can replace with a huge table ideally)  
  > **username_0**: But what about policy gradient methods like PPO that was introduced by OpenAI to fine tune their models ? (Just found that)  
  > **username_1**: you can use PPO with tabular RL as well, so not sure what you mean  
  > **username_0**: Your first answer suggests that we can use ANN to scale RL (for what I understand, when environment have large state spaces). So that's for "ANN used in RL".  However, I also found after further research that we can apply the RL paradigm (find the best policy that maximise the cumulative reward if I am correct) to **train** ANN, whether by treating ANN as a policy (policy gradient method) or as an actor (actor-critic methods).   It suggests that there is now two different relationships between ANN and RL to consider, but is it correct ?  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2024-07-01. Score: 1 | Accepted answer
  
  RL is a learning algorithm, just like evolutionary strategies or SGD. ANNs can be used as the value function in RL, which is learned. The classic example is deep RL with AlphaGo. However, there are many RL methods which don’t use NNs, like Q learning as you mentioned.
  
  A clarification by Sutton and Barto which may clear up the area of confusion: RL is both a general field of models/techniques as well as a set of learning algorithms. It depends on the context, but it seems like what you are referring to is RL as a method.
  
  > **Comments:**
  > **username_3**: RL is not **a** learning algorithm. It's a field/area of study. But there are RL algorithms. I think it's important to use the right terminology to avoid further confusion.  
  
  ---
  
  ### Answer by username_0 on 2024-06-30. Score: 2
  
  So now that I inquired further about the question, here are some answers.
  
  Generally speaking, when we use reinforcement learning (RL) and artificial neural networks (ANNs) in the same sentence, it's about using ANNs to approximate RL algorithms. In this case, we train the ANNs as we would normally do (Stochastic gradient descent...).
  
  Here, the ANNs serve as the RL algorithms agents.
  
  However, we can also use RL for :
  
  * Hyperparameter optimization and Neural architecture search (NAS), where ANNs are treated as the environment. We use RL algorithms to find the best architecture (number for layers...) or hyperparameters (activation function...). See Zoph, B., & Le, Q.V. (2017) "Neural Architecture Search with Reinforcement Learning" as an example.
  * Weights optimization (i.e. RL algorithms replacing the classical training algorithms), which is a very rare use case as our traditional optimizers prove to be already pretty efficient. See Bello, I., Zoph, B., Vasudevan, V., & Le, Q.V. (2017) "Neural Optimizer Search with Reinforcement Learning" as an example.
  
  In those cases, the RL algorithms are the agent, and the ANN is the subject being optimised (environment).
  
  
  ---
  
  ### Answer by username_4 on 2024-06-30. Score: 0
  
  I would say that this is just be the semantics of the sentence you read. I would understand the sentence 'RL can be used as a framework to train an ANN' to mean that we are using an RL algorithm to learn the weights of an ANN -- an example would be to learn the parameters of a deep Q-network, which is used to approximate Q-values.
  
  What's important to remember is that, as @Alberto points out in the comments, RL is generally separate to ANNs. RL is a branch of machine learning that *can* use ANNs as function approximators for the Q-function/policy (this is the context in which one might say that RL is 'training' the ANN), but is in general not tied to them (obviously you could argue that right now, they are tied to ANNs in the sense that to get them to work in real world settings, you need a deep network for the function approximations, but this might not always be true).
  
  > **Comments:**
  > **username_0**: I fully understand the second part, the confusion remain in the first part, I'm trying to clarify : while you say it is possible to use an RL algorithm to learn-optimise the weights of an ANN, you talk about DQN, which I thought were precisely an example of ANN being used inside a RL solution, and that DQN are trained using the classic optimisers we know (like gradient descent etc...). I am rather expecting that if RL can be use to train ANN, it's by incorporating novel optimisers (same way genetic algorithm can be use to optimise ANN weights), but that might not be the case.  
  > **username_4**: okay, yes, I can see based on the answer provided that this is the angle you were trying to understand. indeed, RL has been used in neural architecture search though it is quite limited currently.  
  
  ---
  
  