Body: <p><strong>Consider the following case:</strong></p>
  <p>A reinforcement based web-crawler where:</p>
  <ul>
  <li>State = current page + 1 out-link (reduced to features of some sort)</li>
  <li>Action = Whether to visit that out-link or not (n_actions = 2). This means action = 1 causes the link to be added to the crawlers queue, and action = 0 causes the link to be discarded.</li>
  <li>Reward = Score determined by the page reached by visiting the out-link</li>
  <li>Next state = the next possible link to consider (taken from the scraper prio queue). This is seperate of whether the links in the currently visited page are added to the queue or discarded.</li>
  </ul>
  <p>Can this state-action-reward space be used in a RL implementation, considering we can not calculate the reward (or penalty) for bad pages, since action = 0 never leads to a new page visit where a reward can be calculated. We will thus only ever have feedback and update action = 1.</p>
  <p>Assuming some policy optimization algorithm (such as PPO), can we teach an agent to perform this task, or does it explicitly need feedback on both actions?</p>
  <p>My initial assumption was that to train a binary action space, we either need feedback on both actions that is positive, or we need feedback on either action that is both positive and negative. Evening out the probability distribution over the action space would then take care of &quot;learning&quot; the second action. Is that correct?</p>
  <p><strong>Why is the problem defined like this:</strong></p>
  <p>A different way of defining RL web crawling is by taking the current page as state and considering the list of out-links as the action space, picking a single page to visit. The way I understand policy methods is that a single state is mapped to a single action, meaning that if such a problem definition is used, the agent will pick a single action (link) to take in each page, so only linear paths through the web-content are considered. The problem definition stated above allows the agent to consider all links at all times.</p>
  <p><strong>The goal:</strong></p>
  <p>To find as many relevant pages (scored on, for example, summing hits from a countvectorizer with target terms), while skipping irrelevant links.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-25T16:43:01.767Z
Comments.Text: Your title mismatches the body - there are clearly two actions. Your description of action 0 says "causes the link to be discarded" - what does that look like for the state? You also claim that "action = 0 never leads to a new state", so does the "discarded" link get put back on the state, and action = 0 mean "wait a short while" or "end episode" or is there something else? It is worth explaining the goal of this web crawler, to help someone understand why it might need to choose between visiting or not visiting a link, since this will help decide reward structure.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-26T08:07:07.270Z
Comments.Text: Fair points. I'll rephrase with more info when I have the time.
Comments.UserDisplayName: Kroshtan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-26T09:53:43.787Z
Comments.Text: @NeilSlater updated the title and question
Comments.UserDisplayName: Kroshtan
Tags: <reinforcement-learning><deep-rl>
Title: Is Reinforcement Learning with only feedback on a single action possible?
UserDisplayName: Kroshtan
ThreadText: # Is Reinforcement Learning with only feedback on a single action possible?
  
  (Asked by: username_0 on 2022-05-25)
  
  **Consider the following case:**
  
  A reinforcement based web-crawler where:
  
  * State = current page + 1 out-link (reduced to features of some sort)
  * Action = Whether to visit that out-link or not (n\_actions = 2). This means action = 1 causes the link to be added to the crawlers queue, and action = 0 causes the link to be discarded.
  * Reward = Score determined by the page reached by visiting the out-link
  * Next state = the next possible link to consider (taken from the scraper prio queue). This is seperate of whether the links in the currently visited page are added to the queue or discarded.
  
  Can this state-action-reward space be used in a RL implementation, considering we can not calculate the reward (or penalty) for bad pages, since action = 0 never leads to a new page visit where a reward can be calculated. We will thus only ever have feedback and update action = 1.
  
  Assuming some policy optimization algorithm (such as PPO), can we teach an agent to perform this task, or does it explicitly need feedback on both actions?
  
  My initial assumption was that to train a binary action space, we either need feedback on both actions that is positive, or we need feedback on either action that is both positive and negative. Evening out the probability distribution over the action space would then take care of "learning" the second action. Is that correct?
  
  **Why is the problem defined like this:**
  
  A different way of defining RL web crawling is by taking the current page as state and considering the list of out-links as the action space, picking a single page to visit. The way I understand policy methods is that a single state is mapped to a single action, meaning that if such a problem definition is used, the agent will pick a single action (link) to take in each page, so only linear paths through the web-content are considered. The problem definition stated above allows the agent to consider all links at all times.
  
  **The goal:**
  
  To find as many relevant pages (scored on, for example, summing hits from a countvectorizer with target terms), while skipping irrelevant links.
  
  **Tags:** reinforcement-learning, deep-rl
  
  > **Comments:**
  > **username_1**: Your title mismatches the body - there are clearly two actions. Your description of action 0 says "causes the link to be discarded" - what does that look like for the state? You also claim that "action = 0 never leads to a new state", so does the "discarded" link get put back on the state, and action = 0 mean "wait a short while" or "end episode" or is there something else? It is worth explaining the goal of this web crawler, to help someone understand why it might need to choose between visiting or not visiting a link, since this will help decide reward structure.  
  > **username_0**: Fair points. I'll rephrase with more info when I have the time.  
  > **username_0**: @username_1 updated the title and question  
  
  ---
  
  ## 0 answers
  
  