Answers.Body: <p>The normal use case for softmax in the output layer is for a classification problem, where the output is an array of probabilities for each class.</p>
  <p>The normal use case for a linear output is for a regression problem, where the output is an array of floating point numbers that are estimates for some measurement.</p>
  <p>In a DQN, the desired output for the neural network is the action values (or Q values) for each action. The action value is the expected sum of future rewards given the starting condition of state and action choice, plus following the current target policy from that point onwards (this is a little self-referential in DQN in that the target policy is defined by taking the action with the highest action value in all states).</p>
  <p>It is entirely possible for this predicted action value to be greater than <span class="math-container">$1$</span> or less than <span class="math-container">$0$</span>, and there is also no reason to expect a the set of action values for all actions possible in a specific state to sum to <span class="math-container">$1$</span>. This is a regression problem, and you should generally use a linear output for such problems.</p>
  <p>Some Deep RL will learn a policy function, either instead of, or in addition to the value function - for example REINFORCE or A3C. In that case, if the action space is discrete, then the policy function will behave a lot like a classifier (it classifies the preferred action for a given state), and you would use a softmax output for that network. Other action spaces may be described differently though, so you are not guaranteed to see softmax used in output layer of policy function networks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-18T14:56:34.250Z
Answers.Text: Hello Neil, thank you for your answer on this. Will it be possible to elaborate why we would use softmax with an example of REINFORCE or A3C? Because we can also use linear in that case as well, isn't it? Or am I missing something?
Answers.UserDisplayName: Somdip Dey
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-18T14:58:31.053Z
Answers.Text: You would not use linear activation to define a policy for REINFORCE or A3C, at least not for discrete action space. However, some action spaces can be described using linear outputs - the output might be the mean and variance for a Normal distribution for example.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-18T15:37:13.290Z
Answers.Text: @SomdipDey I don't understand sorry. The phrase "the RL has a policy on it" does not make sense to me. In general, you should understand first what the neural network is doing inside the agent before choosing the activation function, because it makes a big difference. Different RL methods, and different action spaces will affect this. If you want to experiment by changing the activation functions, that is fine and might help you learn what they do. Mostly, inside RL agents, you will discover the activation function is constrained by what it is modelling, and changing it will break it.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-18T14:52:06.807Z
Answers.LastActivityDate: 2022-02-18T15:03:24.323Z
Answers.LastEditDate: 2022-02-18T15:03:24.323Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am following the tutorial in this video: <a href="https://youtu.be/cO5g5qLrLSo" rel="nofollow noreferrer">https://youtu.be/cO5g5qLrLSo</a> which implements deep reinforcement learning (DQN) to balance cart pole in OpenAI default environment.</p>
  <p>The DQN model looks like as follows:</p>
  <pre><code>model = Sequential()
  model.add(Flatten(input_shape=(1,states)))
  model.add(Dense(24, activation='relu'))
  model.add(Dense(24, activation='relu'))
  model.add(Dense(actions, activation='linear'))
  </code></pre>
  <p>Full code is available here: <a href="https://github.com/nicknochnack/TensorflowKeras-ReinforcementLearning/blob/master/Deep%20Reinforcement%20Learning.ipynb" rel="nofollow noreferrer">https://github.com/nicknochnack/TensorflowKeras-ReinforcementLearning/blob/master/Deep%20Reinforcement%20Learning.ipynb</a></p>
  <p>The implemented code is also available on Google Collabs for you to run/test here: <a href="https://colab.research.google.com/drive/1oQILItVu6Y8jOCprzwMGzwlztYmVKK-F?usp=sharing" rel="nofollow noreferrer">https://colab.research.google.com/drive/1oQILItVu6Y8jOCprzwMGzwlztYmVKK-F?usp=sharing</a></p>
  <p>I do understand the concept and mathematics behind using &quot;linear&quot; vs &quot;non-linear&quot; (softmax) activation function in the output layer.</p>
  <p>But, what I am struggling to understand is that why in this target application linear activation function is used in the output layer instead of softmax? Can someone specify how to realize that which type of activation function will be the best for what type of target application using DQN?</p>
  <p>P.S. I tried to change the activation function to softmax instead and got completely different result. Hence, I am confused as to why changing the activation function in the output layer could generate completely opposite results.</p>
  
Tags: <reinforcement-learning><dqn><deep-neural-networks><activation-functions>
Title: Using "softmax" (non-linear) vs "linear" activation function in Deep Reinforcement Learning
UserDisplayName: Somdip Dey
ThreadText: # Using "softmax" (non-linear) vs "linear" activation function in Deep Reinforcement Learning
  
  (Asked by: username_0 on 2022-02-18)
  
  I am following the tutorial in this video: https://youtu.be/cO5g5qLrLSo which implements deep reinforcement learning (DQN) to balance cart pole in OpenAI default environment.
  The DQN model looks like as follows:
  model = Sequential()
  model.add(Flatten(input_shape=(1,states)))
  model.add(Dense(24, activation='relu'))
  model.add(Dense(24, activation='relu'))
  model.add(Dense(actions, activation='linear'))
  
  Full code is available here: https://github.com/nicknochnack/TensorflowKeras-ReinforcementLearning/blob/master/Deep%20Reinforcement%20Learning.ipynb
  The implemented code is also available on Google Collabs for you to run/test here: https://colab.research.google.com/drive/1oQILItVu6Y8jOCprzwMGzwlztYmVKK-F?usp=sharing
  I do understand the concept and mathematics behind using "linear" vs "non-linear" (softmax) activation function in the output layer.
  But, what I am struggling to understand is that why in this target application linear activation function is used in the output layer instead of softmax? Can someone specify how to realize that which type of activation function will be the best for what type of target application using DQN?
  P.S. I tried to change the activation function to softmax instead and got completely different result. Hence, I am confused as to why changing the activation function in the output layer could generate completely opposite results.
  
  **Tags:** reinforcement-learning, dqn, deep-neural-networks, activation-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-02-18. Score: 3
  
  The normal use case for softmax in the output layer is for a classification problem, where the output is an array of probabilities for each class.
  
  The normal use case for a linear output is for a regression problem, where the output is an array of floating point numbers that are estimates for some measurement.
  
  In a DQN, the desired output for the neural network is the action values (or Q values) for each action. The action value is the expected sum of future rewards given the starting condition of state and action choice, plus following the current target policy from that point onwards (this is a little self-referential in DQN in that the target policy is defined by taking the action with the highest action value in all states).
  
  It is entirely possible for this predicted action value to be greater than $1$ or less than $0$, and there is also no reason to expect a the set of action values for all actions possible in a specific state to sum to $1$. This is a regression problem, and you should generally use a linear output for such problems.
  
  Some Deep RL will learn a policy function, either instead of, or in addition to the value function - for example REINFORCE or A3C. In that case, if the action space is discrete, then the policy function will behave a lot like a classifier (it classifies the preferred action for a given state), and you would use a softmax output for that network. Other action spaces may be described differently though, so you are not guaranteed to see softmax used in output layer of policy function networks.
  
  > **Comments:**
  > **username_0**: Hello Neil, thank you for your answer on this. Will it be possible to elaborate why we would use softmax with an example of REINFORCE or A3C? Because we can also use linear in that case as well, isn't it? Or am I missing something?  
  > **username_1**: You would not use linear activation to define a policy for REINFORCE or A3C, at least not for discrete action space. However, some action spaces can be described using linear outputs - the output might be the mean and variance for a Normal distribution for example.  
  > **username_1**: @SomdipDey I don't understand sorry. The phrase "the RL has a policy on it" does not make sense to me. In general, you should understand first what the neural network is doing inside the agent before choosing the activation function, because it makes a big difference. Different RL methods, and different action spaces will affect this. If you want to experiment by changing the activation functions, that is fine and might help you learn what they do. Mostly, inside RL agents, you will discover the activation function is constrained by what it is modelling, and changing it will break it.  
  
  ---
  
  