Answers.Body: <p>It is certainly possible for the state features to dominate the action features in the critic.</p>
  <p>There are several strategies you can use:</p>
  <ol>
  <li><p>Replace the action features with a high dimensional learned embedding vector. This way you can scale up it's importance.</p>
  </li>
  <li><p>Introduce the action at a deeper stage of the network. This way, the action is being combined with the state after the state has been compressed by the earlier layers</p>
  </li>
  <li><p>Simply have many action features inputs that are all identical. This is similar to method 1 but easier.</p>
  </li>
  </ol>
  <p>I have used all 3 of these methods. They all worked well in different cases.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-02T23:44:51.900Z
Answers.Text: Hi @chessprogrammer. Thanks for the answer, I have also thought about similar methods. I tried method 2 already, i.e. learning the state embedding by reducing it to a size one representation and combining it with the action. This didn't work, basically my performance was fluctuating around some value and not improving at all. Regarding method 3 I believe you mean just to stack the copies of the action, say 50 times to match the number of the features in the state, right? I am already doing this after failing with method 2, waiting for the results, will post how it did.
Answers.UserDisplayName: Mika
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-02T23:48:54.267Z
Answers.Text: Not sure about the method 1, though. You mean to pass the action with size of one through several layers, then combine the learned representation with the state features, right?
Answers.UserDisplayName: Mika
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-03T00:09:00.663Z
Answers.Text: Hi. First off, you misunderstood method 2. I am not saying to compress the state space to one before combing. That is way too low. I am saying to make it smaller. Not that small. And method 1 I mean passing it through an embedding layer. Look up torch.nn.embedding
Answers.UserDisplayName: chessprogrammer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-05T03:08:44.000Z
Answers.Text: I added a figure of what I did that I thought is similar to method 2; now I am confused. My understanding of torch.nn.embedding is that it's just a trainable look-up table; moreover, my action is continuous, not sure how I'd encode my action space.
Answers.UserDisplayName: Mika
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-02T22:59:16.940Z
Answers.LastActivityDate: 2023-03-02T22:59:16.940Z
Answers.UserDisplayName: chessprogrammer
Body: <p>I have a case where my state consists of relatively large number of features, e.g. 50, whereas my action size is 1. I wonder whether my state features dominate the action in my critic network. I believe in theory eventually it shouldn't matter but considering the sequential nature of RL training I am afraid the state features outweigh the action and its effect will be negligible.</p>
  <p>What I already tried is the following:</p>
  <p><a href="https://i.sstatic.net/JBavI.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JBavI.jpg" alt="enter image description here" /></a></p>
  <p>Here where the state output and action are combined I use tanh activation because my action is in [-1, 1]. This led to almost flat performance from the very beginning with no improvement at all. I understand this might be due to vanishing gradients caused by tanh. I also tried the linear activation instead of the tanh, this time average episode return was fluctuating around some value with no signs of learning.</p>
  <p>What I am currently testing is stacking the action, say 50 times, to match the number of the state features.</p>
  <p>Any other ideas on how to tackle this issue.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-02T22:54:53.557Z
Comments.Text: Hi @Mika! Just to clarify - what is the input to your critic network? I.e., are you using the action as an input at all? It seems that way based on the wording of this question.
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-02T23:36:04.940Z
Comments.Text: Hey @DeepQZero. Yes I am using the action as an input along with the state. But like i said state has a dimension of 50 and action has only one. Let me know if that helped.
Comments.UserDisplayName: Mika
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-03T16:30:38.567Z
Comments.Text: It helped, but I personally would need to understand more details before supplying an answer. It looks like there is an answer on this question already, and I think I'll pass on answering this question for now.
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-05T03:10:57.110Z
Comments.Text: @DeepQZero, I added more information if that helps understand the problem better
Comments.UserDisplayName: Mika
Tags: <deep-rl><actor-critic-methods><soft-actor-critic><td3>
Title: Can action be dominated by state features in actor-critic algorithms?
UserDisplayName: Mika
ThreadText: # Can action be dominated by state features in actor-critic algorithms?
  
  (Asked by: username_0 on 2023-03-02)
  
  I have a case where my state consists of relatively large number of features, e.g. 50, whereas my action size is 1. I wonder whether my state features dominate the action in my critic network. I believe in theory eventually it shouldn't matter but considering the sequential nature of RL training I am afraid the state features outweigh the action and its effect will be negligible.
  
  What I already tried is the following:
  
  [![enter image description here](https://i.sstatic.net/JBavI.jpg)](https://i.sstatic.net/JBavI.jpg)
  
  Here where the state output and action are combined I use tanh activation because my action is in [-1, 1]. This led to almost flat performance from the very beginning with no improvement at all. I understand this might be due to vanishing gradients caused by tanh. I also tried the linear activation instead of the tanh, this time average episode return was fluctuating around some value with no signs of learning.
  
  What I am currently testing is stacking the action, say 50 times, to match the number of the state features.
  
  Any other ideas on how to tackle this issue.
  
  **Tags:** deep-rl, actor-critic-methods, soft-actor-critic, td3
  
  > **Comments:**
  > **username_1**: Hi @username_0! Just to clarify - what is the input to your critic network? I.e., are you using the action as an input at all? It seems that way based on the wording of this question.  
  > **username_0**: Hey @username_1. Yes I am using the action as an input along with the state. But like i said state has a dimension of 50 and action has only one. Let me know if that helped.  
  > **username_1**: It helped, but I personally would need to understand more details before supplying an answer. It looks like there is an answer on this question already, and I think I'll pass on answering this question for now.  
  > **username_0**: @username_1, I added more information if that helps understand the problem better  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-03-02. Score: 1
  
  It is certainly possible for the state features to dominate the action features in the critic.
  
  There are several strategies you can use:
  
  1. Replace the action features with a high dimensional learned embedding vector. This way you can scale up it's importance.
  2. Introduce the action at a deeper stage of the network. This way, the action is being combined with the state after the state has been compressed by the earlier layers
  3. Simply have many action features inputs that are all identical. This is similar to method 1 but easier.
  
  I have used all 3 of these methods. They all worked well in different cases.
  
  > **Comments:**
  > **username_0**: Hi @username_2. Thanks for the answer, I have also thought about similar methods. I tried method 2 already, i.e. learning the state embedding by reducing it to a size one representation and combining it with the action. This didn't work, basically my performance was fluctuating around some value and not improving at all. Regarding method 3 I believe you mean just to stack the copies of the action, say 50 times to match the number of the features in the state, right? I am already doing this after failing with method 2, waiting for the results, will post how it did.  
  > **username_0**: Not sure about the method 1, though. You mean to pass the action with size of one through several layers, then combine the learned representation with the state features, right?  
  > **username_2**: Hi. First off, you misunderstood method 2. I am not saying to compress the state space to one before combing. That is way too low. I am saying to make it smaller. Not that small. And method 1 I mean passing it through an embedding layer. Look up torch.nn.embedding  
  > **username_0**: I added a figure of what I did that I thought is similar to method 2; now I am confused. My understanding of torch.nn.embedding is that it's just a trainable look-up table; moreover, my action is continuous, not sure how I'd encode my action space.  
  
  ---
  
  