Body: <p>Say we want to train an agent <span class="math-container">$A$</span> in an environment <span class="math-container">$E$</span> which provides a continuous loss <span class="math-container">$L$</span>. That is, we want <span class="math-container">$A$</span> to choose its actions <span class="math-container">$a$</span> so that it minimizes the mistake it does, i.e., it minimizes the loss <span class="math-container">$L$</span>.</p>
  <p>In an Actor-Critic scenario, if I understood it correctly, the idea of the Critic is to learn to &quot;evaluate&quot; the current state of the environment by learning from rewards/losses. The Actor will then be adjusted so that its performed actions satisfy the Critic.</p>
  <p>Now, if the optimal loss is always provided already by <span class="math-container">$E$</span>, what would even be an argument to use a Critic? Is the whole idea of an Actor-Critic infrastructure in this case just wrong? Is an Actor-Critic approach only reasonable if the reward/loss sparse so that the Critic can learn a better one?</p>
  <p>ADDED on 22.12.2023:
  As an example for further clarification, <span class="math-container">$A$</span>'s goal could be to adjust settings of various machines in <span class="math-container">$E$</span> so that a measurement, e.g., a pressure inside a tank, stays at a desired value all the time. So the loss would just be the deviation from that value. This is what I meant by &quot;continuous, optimal&quot; loss; a signal that constantly says how well <span class="math-container">$A$</span> is doing. This is very much unlike the case of an inverted pendulum that only provides a loss of, say, <span class="math-container">$-1$</span>, if it fell over, and otherwise constantly <span class="math-container">$0$</span>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-20T16:16:30.037Z
Comments.Text: I'm not sure of the answer here but I would conjecture that A2C still has something to give because although the reward/loss is fixed (I assume this was what was meant by continuous) this is not the Q value of a state.  In a grid environment with a -1 reward for each step, this is not the Q of a state.  The Q should approach -[steps to goal] + [reward for reaching goal].
Comments.UserDisplayName: foreverska
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-20T18:47:51.560Z
Comments.Text: What do you mean exactly by "continuous/optimal loss"? Could you elaborate a bit?
Comments.UserDisplayName: Luca Anzalone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-22T11:06:48.010Z
Comments.Text: @foreverska The reward/loss is not "fixed". Please see my added things.
Comments.UserDisplayName: Mathy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-22T11:06:56.617Z
Comments.Text: @LucaAnzalone Just did that.
Comments.UserDisplayName: Mathy
Tags: <reinforcement-learning><actor-critic-methods><reward-functions><loss>
Title: How and whether to apply Reinforcement Learning in an Environment with a precise and always available Evaluation?
UserDisplayName: Mathy
ThreadText: # How and whether to apply Reinforcement Learning in an Environment with a precise and always available Evaluation?
  
  (Asked by: username_0 on 2023-12-20)
  
  Say we want to train an agent $A$ in an environment $E$ which provides a continuous loss $L$. That is, we want $A$ to choose its actions $a$ so that it minimizes the mistake it does, i.e., it minimizes the loss $L$.
  
  In an Actor-Critic scenario, if I understood it correctly, the idea of the Critic is to learn to "evaluate" the current state of the environment by learning from rewards/losses. The Actor will then be adjusted so that its performed actions satisfy the Critic.
  
  Now, if the optimal loss is always provided already by $E$, what would even be an argument to use a Critic? Is the whole idea of an Actor-Critic infrastructure in this case just wrong? Is an Actor-Critic approach only reasonable if the reward/loss sparse so that the Critic can learn a better one?
  
  ADDED on 22.12.2023:
  As an example for further clarification, $A$'s goal could be to adjust settings of various machines in $E$ so that a measurement, e.g., a pressure inside a tank, stays at a desired value all the time. So the loss would just be the deviation from that value. This is what I meant by "continuous, optimal" loss; a signal that constantly says how well $A$ is doing. This is very much unlike the case of an inverted pendulum that only provides a loss of, say, $-1$, if it fell over, and otherwise constantly $0$.
  
  **Tags:** reinforcement-learning, actor-critic-methods, reward-functions, loss
  > **Comments:**
  > **username_1**: I'm not sure of the answer here but I would conjecture that A2C still has something to give because although the reward/loss is fixed (I assume this was what was meant by continuous) this is not the Q value of a state.  In a grid environment with a -1 reward for each step, this is not the Q of a state.  The Q should approach -[steps to goal] + [reward for reaching goal].  
  > **username_2**: What do you mean exactly by "continuous/optimal loss"? Could you elaborate a bit?  
  > **username_0**: @foreverska The reward/loss is not "fixed". Please see my added things.  
  > **username_0**: @LucaAnzalone Just did that.  
  
  ---
  
  ## 0 answers
  
  