Answers.Body: <p>I do not think there is one standard way to do this, it will depend too much on context. Ultimately you want the agent to output a <em>stop</em> action that is different from a <em>continue</em> action.</p>
  <p>That stop/continue choice could either be part of the existing action encoding, additional data in parallel with the action sequence, or an entirely separate action choice on time steps alternating with the main action choices.</p>
  <p>I do not know anything about pointer networks, so not sure what your action coding options are. Your &quot;dummy&quot; object selection seems like a reasonable choice though, it is part of existing action encoding that the neural network can already output. In that sense it seems a lot like an <code>&lt;END&gt;</code> token that a seq2seq model might output for e.g. translation or summarisation language tasks.</p>
  <p>An alternative might be to add a separate head to the pointer network, that output a <em>stop</em> classifier alongside each object choice. The combined action would be <code>[selection, stop]</code>. If you are feeding previous choice back into the RNN as input (it is not clear to me, but sampling seq2seq networks do this), you have free choice as to whether to use the combined action (with the stop flag as a raw confidence in <span class="math-container">$[0,1]$</span>), or just continue with previous selection as only feedback.</p>
  <p>Finally you could use a different NN for deciding whether to stop or continue and feed it the same sequence.</p>
  <p>Which is better? I cannot say, but your dummy object selection appears to work already to some degree, so I would stick experimenting with that, and the simple all zeroes token. The common 0 total rewards may be an issue with your RL agent exploring, or maybe with a difficult reward metric. E.g. can the agent score less than zero if it overfills the knapsack? If so, not trying anything at all may sometimes look good to the agent, and it will need more training so it can properly predict when this is not the case.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-03T01:02:31.903Z
Answers.Text: Pointer network is very similar with seq2seq but is designed for problems which input does not have any concise/meaningful order e.g. items in knapsack problems, city coordinates in tsp, nodes' coordinates in convex hull. Therefore, the inputs does not have to be encoded by RNN, but only by simple single/multiple layer perceptron. The choice then are based on the attention on the inputs' encoded features.
  
  Yes, I was hoping to replicate **END** symbol in seq2seq. I'm interested in the additional head alternative that you mentioned. I will google it first. Thank you very much!
Answers.UserDisplayName: Sanyou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-02T14:57:15.563Z
Answers.LastActivityDate: 2021-09-02T14:57:15.563Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>You can find information similar to exposed by Neil, but with more theoretical detail, in the book Deep Learning (Goodfellow et al., 2016) in the chapter 10 (Recurrent networks), more specifically in 10.2.3 Recurrent Networks as Directed Graphical Models and other subchapters.</p>
  <p>Additional, related with pointer networks there are people changing the LSTM with Transformer (Learning Heuristics for the TSP by PolicyGradient, 2018)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-06T06:22:39.183Z
Answers.Text: Transformer for TSP, there are some variants of it, I was just started reading the one proposed by Kool, thank you for the book reference!
Answers.UserDisplayName: Sanyou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-06T06:19:58.553Z
Answers.LastActivityDate: 2021-09-06T06:19:58.553Z
Answers.UserDisplayName: StandardIO
Body: <p>I am currently implementing a <a href="https://paperswithcode.com/method/pointer-net" rel="nofollow noreferrer">Pointer Network</a> to solve a simple Knapsack Problem. However, I am a bit puzzled over the correct (or common, or &quot;best&quot;) way to give the agent the option to stop taking the item (terminate episode). Currently, I have done it in 2 ways, adding raw dummy features or adding encoded dummy features (dummy features are all zeros). If the agent selects the dummy item, then the agent will stop taking the item anymore and the episode is terminated.</p>
  <p>I trained both methods for 500K episodes and evaluated their performance on a single predefined test case in each episode, after adding the gradient. I found that concatenating dummy features with the encoded features yielded a higher score earlier, but also scored 0 very often. On the other hand, adding the dummy features to the raw features learned to maximize the score very slowly. Therefore, my questions are:</p>
  <ol>
  <li><p>Is adding the raw dummy features make learning slower because of additional encoding layer learning?</p>
  </li>
  <li><p>What is the most correct (or common or arguably best) way to give the agent the option to terminate the episode (in this case stop taking item)?</p>
  </li>
  </ol>
  
Tags: <reinforcement-learning><ai-design><knapsack-problem>
Title: How to represent "terminate episode" for Knapsack problem with a Pointer Network?
UserDisplayName: Sanyou
ThreadText: # How to represent "terminate episode" for Knapsack problem with a Pointer Network?
  
  (Asked by: username_0 on 2021-09-02)
  
  I am currently implementing a [Pointer Network](https://paperswithcode.com/method/pointer-net) to solve a simple Knapsack Problem. However, I am a bit puzzled over the correct (or common, or "best") way to give the agent the option to stop taking the item (terminate episode). Currently, I have done it in 2 ways, adding raw dummy features or adding encoded dummy features (dummy features are all zeros). If the agent selects the dummy item, then the agent will stop taking the item anymore and the episode is terminated.
  
  I trained both methods for 500K episodes and evaluated their performance on a single predefined test case in each episode, after adding the gradient. I found that concatenating dummy features with the encoded features yielded a higher score earlier, but also scored 0 very often. On the other hand, adding the dummy features to the raw features learned to maximize the score very slowly. Therefore, my questions are:
  
  1. Is adding the raw dummy features make learning slower because of additional encoding layer learning?
  2. What is the most correct (or common or arguably best) way to give the agent the option to terminate the episode (in this case stop taking item)?
  
  **Tags:** reinforcement-learning, ai-design, knapsack-problem
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-09-02. Score: 1 | Accepted answer
  
  I do not think there is one standard way to do this, it will depend too much on context. Ultimately you want the agent to output a *stop* action that is different from a *continue* action.
  
  That stop/continue choice could either be part of the existing action encoding, additional data in parallel with the action sequence, or an entirely separate action choice on time steps alternating with the main action choices.
  
  I do not know anything about pointer networks, so not sure what your action coding options are. Your "dummy" object selection seems like a reasonable choice though, it is part of existing action encoding that the neural network can already output. In that sense it seems a lot like an `<END>` token that a seq2seq model might output for e.g. translation or summarisation language tasks.
  
  An alternative might be to add a separate head to the pointer network, that output a *stop* classifier alongside each object choice. The combined action would be `[selection, stop]`. If you are feeding previous choice back into the RNN as input (it is not clear to me, but sampling seq2seq networks do this), you have free choice as to whether to use the combined action (with the stop flag as a raw confidence in $[0,1]$), or just continue with previous selection as only feedback.
  
  Finally you could use a different NN for deciding whether to stop or continue and feed it the same sequence.
  
  Which is better? I cannot say, but your dummy object selection appears to work already to some degree, so I would stick experimenting with that, and the simple all zeroes token. The common 0 total rewards may be an issue with your RL agent exploring, or maybe with a difficult reward metric. E.g. can the agent score less than zero if it overfills the knapsack? If so, not trying anything at all may sometimes look good to the agent, and it will need more training so it can properly predict when this is not the case.
  
  > **Comments:**
  > **username_0**: Pointer network is very similar with seq2seq but is designed for problems which input does not have any concise/meaningful order e.g. items in knapsack problems, city coordinates in tsp, nodes' coordinates in convex hull. Therefore, the inputs does not have to be encoded by RNN, but only by simple single/multiple layer perceptron. The choice then are based on the attention on the inputs' encoded features.  Yes, I was hoping to replicate **END** symbol in seq2seq. I'm interested in the additional head alternative that you mentioned. I will google it first. Thank you very much!  
  
  ---
  
  ### Answer by username_2 on 2021-09-06. Score: 1
  
  You can find information similar to exposed by username_1, but with more theoretical detail, in the book Deep Learning (Goodfellow et al., 2016) in the chapter 10 (Recurrent networks), more specifically in 10.2.3 Recurrent Networks as Directed Graphical Models and other subchapters.
  
  Additional, related with pointer networks there are people changing the LSTM with Transformer (Learning Heuristics for the TSP by PolicyGradient, 2018)
  
  > **Comments:**
  > **username_0**: Transformer for TSP, there are some variants of it, I was just started reading the one proposed by Kool, thank you for the book reference!  
  
  ---
  
  