Answers.Body: <p>One representative paper for imitation learning is Ho &amp; Ermon's &quot;<em>Generative Adversarial Imitation Learning</em>&quot; (2016). The goal in imitation learning is to learn a policy that can mimic the expert's behavior.</p>
  <blockquote>
  <p>Consider learning a policy from example expert behavior, without interaction
  with the expert or access to reinforcement signal. One approach is to recover the
  expert’s cost function with inverse reinforcement learning, then extract a policy
  from that cost function with reinforcement learning. This approach is indirect
  and can be slow. We propose a new general framework for directly extracting a
  policy from data, as if it were obtained by reinforcement learning following inverse
  reinforcement learning.</p>
  </blockquote>
  <blockquote>
  <p>Behavioral cloning, while appealingly simple, only tends to succeed with large amounts of data, due to compounding error caused by covariate shift [23, 24]. Inverse reinforcement learning (IRL), on the other hand, learns a cost function that prioritizes entire trajectories over others, so compounding error, a problem for methods that fit single-timestep decisions, is not an issue... Unfortunately, many IRL algorithms are extremely expensive to run, requiring reinforcement learning
  in an inner loop.</p>
  </blockquote>
  <blockquote>
  <p>We desire an algorithm that tells us explicitly how to act by directly learning a policy. To develop such an algorithm, we begin in Section 3, where we characterize the policy given by running reinforcement learning on a cost function learned by maximum causal entropy IRL [31, 32]... The discriminator network can be interpreted as a local cost function providing learning signal to the policy—specifically, taking a policy step that decreases expected cost with respect to the cost function.</p>
  </blockquote>
  <p>Therefore GAIL is mainly imitation learning which learns a policy as its ultimate goal, yet it also has a discriminator network similar to GANs that acts as a local cost function and aims to distinguish expert trajectories against learned policy's trajectories, establishing a more rapid framework for simultaneously learning policy and reward from data due to its adversarial nature and bypassing an additional independent IRL step.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-03T05:59:02.143Z
Answers.LastActivityDate: 2024-03-03T06:09:49.937Z
Answers.LastEditDate: 2024-03-03T06:09:49.937Z
Answers.UserDisplayName: cinch
Answers.Body: <p>I think the reason why some refer to learning a reward policy and others learning an actor policy is that both are possible.</p>
  <p>For example, GAIL: A discriminator (the reward policy) learns to tell apart expert trajectories and the actor's trajectories, and the actor tries to fool the discriminator by learning to imitate expert trajectories.</p>
  <p>In this way, both a reward policy and actor policy are learnt simultaneously.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-03T02:05:56.447Z
Answers.LastActivityDate: 2024-03-03T02:05:56.447Z
Answers.UserDisplayName: Richie Bendall
Answers.Body: <p>In RL, reward networks (also called reward models), say <span class="math-container">$r_\theta$</span>, have the sole role of learning (approximating) the reward function, defined as <span class="math-container">$r(s,a)$</span> or <span class="math-container">$r(s, a,s')$</span>, in which the latter formulation is more general. A reward function, <span class="math-container">$r$</span>, takes the current state <span class="math-container">$s$</span>, selected action <span class="math-container">$a$</span> (by a policy, expert trajectories, or behavioral policy), and optionally the next state <span class="math-container">$s'$</span>, to output the immediate reward, <span class="math-container">$r_t$</span>, of timestep <span class="math-container">$t$</span>. Reward networks should approximate <span class="math-container">$r_t$</span>, and are employed in <em>inverse reinforcement learning</em> (IRL) and sometimes also in model-based RL.</p>
  <p>Moreover, you should not confuse IRL with <em>imitation learning</em> (IL), as IL is more general because it also includes supervised learning methods such as behavioral cloning. In IRL, you want to learn a reward function first that is usually parameterized by a NN (therefore you have a reward network), and then use standard model-free algorithms from the learned reward to learn the policy. This is the general setting, but GAIL for example can learn the policy directly.</p>
  <p>Note also that when you do IRL or IL you don't have access to the environment, and therefore you don't know the true reward function, but you have only access to a finite number of samples (called demonstrations) which can be from either an expert, exploratory, or sub-optimal policy.</p>
  <p>You can refer to this <a href="https://link.springer.com/article/10.1007/s10462-021-10108-x" rel="nofollow noreferrer">survey</a> for a comprehensive introduction about IRL, its foundations, flavors, and applications.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-03T15:37:16.810Z
Answers.LastActivityDate: 2024-03-03T15:37:16.810Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I am reading the following article given over <a href="https://imitation.readthedocs.io/en/latest/main-concepts/reward_networks.html" rel="nofollow noreferrer">here</a> -</p>
  <blockquote>
  <p>The goal of both inverse reinforcement learning (IRL) algorithms (e.g.
  AIRL, GAIL) and preference comparison is to discover a reward
  function. In imitation learning, these discovered rewards are
  parameterized by reward networks.</p>
  </blockquote>
  <p>Does this mean that the output of imitation learning is a reward value? What are the inputs then? The state and action?</p>
  <p>I am currently referring to Levine's lecture <a href="https://rail.eecs.berkeley.edu/deeprlcourse/deeprlcourse/static/slides/lec-2.pdf" rel="nofollow noreferrer">notes</a>. At slide 5, it seems like the output of imitation learning is a policy and not a reward.</p>
  <p>I'd appreciate an answer that refers to papers while explaining these concepts.</p>
  
Tags: <reinforcement-learning><rewards><imitation-learning><inverse-rl>
Title: What are reward networks in reinforcement learning?
UserDisplayName: desert_ranger
ThreadText: # What are reward networks in reinforcement learning?
  
  (Asked by: username_0 on 2024-03-02)
  
  I am reading the following article given over [here](https://imitation.readthedocs.io/en/latest/main-concepts/reward_networks.html) -
  
  > The goal of both inverse reinforcement learning (IRL) algorithms (e.g.
  > AIRL, GAIL) and preference comparison is to discover a reward
  > function. In imitation learning, these discovered rewards are
  > parameterized by reward networks.
  
  Does this mean that the output of imitation learning is a reward value? What are the inputs then? The state and action?
  
  I am currently referring to Levine's lecture [notes](https://rail.eecs.berkeley.edu/deeprlcourse/deeprlcourse/static/slides/lec-2.pdf). At slide 5, it seems like the output of imitation learning is a policy and not a reward.
  
  I'd appreciate an answer that refers to papers while explaining these concepts.
  
  **Tags:** reinforcement-learning, rewards, imitation-learning, inverse-rl
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2024-03-03. Score: 4 | Accepted answer
  
  In RL, reward networks (also called reward models), say $r\_\theta$, have the sole role of learning (approximating) the reward function, defined as $r(s,a)$ or $r(s, a,s')$, in which the latter formulation is more general. A reward function, $r$, takes the current state $s$, selected action $a$ (by a policy, expert trajectories, or behavioral policy), and optionally the next state $s'$, to output the immediate reward, $r\_t$, of timestep $t$. Reward networks should approximate $r\_t$, and are employed in *inverse reinforcement learning* (IRL) and sometimes also in model-based RL.
  
  Moreover, you should not confuse IRL with *imitation learning* (IL), as IL is more general because it also includes supervised learning methods such as behavioral cloning. In IRL, you want to learn a reward function first that is usually parameterized by a NN (therefore you have a reward network), and then use standard model-free algorithms from the learned reward to learn the policy. This is the general setting, but GAIL for example can learn the policy directly.
  
  Note also that when you do IRL or IL you don't have access to the environment, and therefore you don't know the true reward function, but you have only access to a finite number of samples (called demonstrations) which can be from either an expert, exploratory, or sub-optimal policy.
  
  You can refer to this [survey](https://link.springer.com/article/10.1007/s10462-021-10108-x) for a comprehensive introduction about IRL, its foundations, flavors, and applications.
  
  
  ---
  
  ### Answer by username_1 on 2024-03-03. Score: 3
  
  One representative paper for imitation learning is Ho & Ermon's "*Generative Adversarial Imitation Learning*" (2016). The goal in imitation learning is to learn a policy that can mimic the expert's behavior.
  
  > Consider learning a policy from example expert behavior, without interaction
  > with the expert or access to reinforcement signal. One approach is to recover the
  > expert’s cost function with inverse reinforcement learning, then extract a policy
  > from that cost function with reinforcement learning. This approach is indirect
  > and can be slow. We propose a new general framework for directly extracting a
  > policy from data, as if it were obtained by reinforcement learning following inverse
  > reinforcement learning.
  
  > Behavioral cloning, while appealingly simple, only tends to succeed with large amounts of data, due to compounding error caused by covariate shift [23, 24]. Inverse reinforcement learning (IRL), on the other hand, learns a cost function that prioritizes entire trajectories over others, so compounding error, a problem for methods that fit single-timestep decisions, is not an issue... Unfortunately, many IRL algorithms are extremely expensive to run, requiring reinforcement learning
  > in an inner loop.
  
  > We desire an algorithm that tells us explicitly how to act by directly learning a policy. To develop such an algorithm, we begin in Section 3, where we characterize the policy given by running reinforcement learning on a cost function learned by maximum causal entropy IRL [31, 32]... The discriminator network can be interpreted as a local cost function providing learning signal to the policy—specifically, taking a policy step that decreases expected cost with respect to the cost function.
  
  Therefore GAIL is mainly imitation learning which learns a policy as its ultimate goal, yet it also has a discriminator network similar to GANs that acts as a local cost function and aims to distinguish expert trajectories against learned policy's trajectories, establishing a more rapid framework for simultaneously learning policy and reward from data due to its adversarial nature and bypassing an additional independent IRL step.
  
  
  ---
  
  ### Answer by username_2 on 2024-03-03. Score: 2
  
  I think the reason why some refer to learning a reward policy and others learning an actor policy is that both are possible.
  
  For example, GAIL: A discriminator (the reward policy) learns to tell apart expert trajectories and the actor's trajectories, and the actor tries to fool the discriminator by learning to imitate expert trajectories.
  
  In this way, both a reward policy and actor policy are learnt simultaneously.
  
  
  ---
  
  