Answers.Body: <p>The quote from the paper is:</p>
  <blockquote>
  <p>In this work, we built a custom object detector that provides plausible object candidates.</p>
  </blockquote>
  <p>And in their <a href="https://proceedings.neurips.cc/paper/2016/file/f442d33fa06832082290ad8544a8da27-Paper.pdf" rel="nofollow noreferrer">related submisstion to NeurIPS</a>:</p>
  <blockquote>
  <p>In this work, we built a custom pipeline to provide plausible object candidates. Note that the agent is still required to learn which of these candidates are worth pursuing as goals.</p>
  </blockquote>
  <p>I think that this detects and identifies &quot;screen locations of interest&quot; to create parameterised sub-goals - e.g. one goal might be to make objects A and B coincide. It is also not clear whether static objects such as rope and ladders are included, or the detector is more fine-tuned to &quot;active&quot; entities such as the player, keys, doors, enemies etc.</p>
  <p>This also gives a clue:</p>
  <blockquote>
  <p>The internal critic is defined in the space of <code>entity1, relation, entity2</code>, where relation is a function over configurations of the entities. In our experiments, the agent learns to choose entity2. For
  instance, the agent is deemed to have completed a goal (and only then receives a reward) if the agent entity reaches another entity such as the door</p>
  </blockquote>
  <p>It implies that the entity detector does identify the player-controlled entity and sets this as <code>entity1</code> for all the top level goals. All the high level goals are stated in terms of &quot;player coincides with B&quot; or &quot;player avoids C&quot;.</p>
  <blockquote>
  <p>Can you please give me a suggestion on how to Implement this object detector?</p>
  </blockquote>
  <p>The implication is that they created something simple and fast that would specifically detect important objects in the game, so that it would find and identify all the objects that could reasonably be used in sub-goals. Objects in old Atari games are visually distinct and simple, so it is likely to be something quite basic. The <a href="https://github.com/mrkulk/hierarchical-deep-RL" rel="nofollow noreferrer">reference code</a> linked by João Schapke implies perhaps template match because there are suitable images in a &quot;template&quot; directory, but I was unable to find the object detector functions from scanning the code quickly.</p>
  <p>It is also important to note that this custom detector was <em>only</em> used to build a selection of sub-goals and detect whether any had been achieved. It was not used as input to the neural networks (which had to learn their own representations of the same objects in order to make decisions), and did not set internal rewards for achieving those sub-goals. The top level policy selected amongst the sub-goals based on processing the video frames - this selection decided the lower level policy's reward. The algorithm used the detector only to assess whether a sub-goal had actually happened, at which point the reward for achieving it would be granted to the lower level policy. Whilst the upper level policy was granted rewards as scored by the original game.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-12T17:21:00.863Z
Answers.LastActivityDate: 2021-06-13T21:33:55.377Z
Answers.LastEditDate: 2021-06-13T21:33:55.377Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm thinking of implementing &quot;Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation&quot; <a href="https://arxiv.org/abs/1604.06057" rel="nofollow noreferrer">paper</a>. In this paper authors used some custom object detector for entity detection(eg: Key, rope, ladder, etc) but they did not give any information about this custom detector. Can you please give me a suggestion on how to Implement this object detector?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-12T21:00:19.110Z
Comments.Text: The authors of the paper did release the code for the paper on github. You can check it for reference: https://github.com/mrkulk/hierarchical-deep-RL/tree/master/dqn
Comments.UserDisplayName: João Schapke
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-13T21:30:47.413Z
Comments.Text: @JoãoSchapke I had a scan through the code for the custom object detector, but could not find it. I suspect from the inclusion of template images that it is based on template match, which would indeed be very simple as I suggest in my answer. I'd like to extend my answer to confirm this, but sadly not able to.
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><deep-rl><dqn><hierarchical-rl>
Title: How to detect entities in Montezuma's Revenge environment
UserDisplayName: noob_rl
ThreadText: # How to detect entities in Montezuma's Revenge environment
  
  (Asked by: username_0 on 2021-06-12)
  
  I'm thinking of implementing "Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation" [paper](https://arxiv.org/abs/1604.06057). In this paper authors used some custom object detector for entity detection(eg: Key, rope, ladder, etc) but they did not give any information about this custom detector. Can you please give me a suggestion on how to Implement this object detector?
  
  **Tags:** reinforcement-learning, deep-rl, dqn, hierarchical-rl
  > **Comments:**
  > **username_1**: The authors of the paper did release the code for the paper on github. You can check it for reference: https://github.com/mrkulk/hierarchical-deep-RL/tree/master/dqn  
  > **username_2**: @JoãoSchapke I had a scan through the code for the custom object detector, but could not find it. I suspect from the inclusion of template images that it is based on template match, which would indeed be very simple as I suggest in my answer. I'd like to extend my answer to confirm this, but sadly not able to.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-06-12. Score: 0 | Accepted answer
  
  The quote from the paper is:
  
  > In this work, we built a custom object detector that provides plausible object candidates.
  
  And in their [related submisstion to NeurIPS](https://proceedings.neurips.cc/paper/2016/file/f442d33fa06832082290ad8544a8da27-Paper.pdf):
  
  > In this work, we built a custom pipeline to provide plausible object candidates. Note that the agent is still required to learn which of these candidates are worth pursuing as goals.
  
  I think that this detects and identifies "screen locations of interest" to create parameterised sub-goals - e.g. one goal might be to make objects A and B coincide. It is also not clear whether static objects such as rope and ladders are included, or the detector is more fine-tuned to "active" entities such as the player, keys, doors, enemies etc.
  
  This also gives a clue:
  
  > The internal critic is defined in the space of `entity1, relation, entity2`, where relation is a function over configurations of the entities. In our experiments, the agent learns to choose entity2. For
  > instance, the agent is deemed to have completed a goal (and only then receives a reward) if the agent entity reaches another entity such as the door
  
  It implies that the entity detector does identify the player-controlled entity and sets this as `entity1` for all the top level goals. All the high level goals are stated in terms of "player coincides with B" or "player avoids C".
  
  > Can you please give me a suggestion on how to Implement this object detector?
  
  The implication is that they created something simple and fast that would specifically detect important objects in the game, so that it would find and identify all the objects that could reasonably be used in sub-goals. Objects in old Atari games are visually distinct and simple, so it is likely to be something quite basic. The [reference code](https://github.com/mrkulk/hierarchical-deep-RL) linked by João Schapke implies perhaps template match because there are suitable images in a "template" directory, but I was unable to find the object detector functions from scanning the code quickly.
  
  It is also important to note that this custom detector was *only* used to build a selection of sub-goals and detect whether any had been achieved. It was not used as input to the neural networks (which had to learn their own representations of the same objects in order to make decisions), and did not set internal rewards for achieving those sub-goals. The top level policy selected amongst the sub-goals based on processing the video frames - this selection decided the lower level policy's reward. The algorithm used the detector only to assess whether a sub-goal had actually happened, at which point the reward for achieving it would be granted to the lower level policy. Whilst the upper level policy was granted rewards as scored by the original game.
  
  
  ---
  
  