Answers.Body: <p>I do not see how you came to choose prices as actions. Normally, actions are something like <code>go left</code>, <code>go right</code>, <code>jump</code>, <code>stay</code> etc.
  Analogously, I would say that in your case the actions are visiting a certain location, whereas locations are what you referred to as <code>states</code>. I'd go for something like that:</p>
  <pre><code>locations = {location1=(c1,cr1), location2=(c1,cr2), ...}
  Actions = {Visit location1, Visit location2, ...}
  States = {--set of all the possible paths the model can generate until the budget is possibly exhausted--}
  </code></pre>
  <p>The <code>reward</code> function could then be a combination of both the <em>acceptance (vs. rejection) of a route/path by the user</em> and the <em>inverse of the cost associated with the suggested path</em> (because you want the model to favor cheap paths in order to keep your own business costs low).
  How you balance these two terms is up to exploration.</p>
  <p>For rapid prototyping, check out <a href="https://stable-baselines.readthedocs.io/en/master/" rel="nofollow noreferrer">stable baselines</a>, which offers a bunch of highly optimized RL algorithms.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-17T22:32:59.283Z
Answers.LastActivityDate: 2020-06-17T22:32:59.283Z
Answers.UserDisplayName: Daniel B.
Answers.Body: <p>I don't know what is your dataset exactly look like. But based on assumption, I would like to refer something --</p>
  <p>You can think your MDP environment this way</p>
  <blockquote>
  <p>action = {stay, go}</p>
  <p>reward = {something like based on visitor's satisfaction maybe <strong>rating</strong>}</p>
  <p>state = {current money in hand, city, other some variable those key feature to make next iteration <strong>action</strong>}</p>
  </blockquote>
  <p>I am working on a project-based stock market trading (sorry, I can not share detail).
  In stock market trading problem, we need to decide action sequentially ( like per hour or maybe day). If your problem (and data as I assume) is a sequential action selection problem.</p>
  <p><em>More detail: the first day you visited NY and enjoyed your moment and cost <span class="math-container">$800. Now you wanted to continue the tour in NY or want to go Washington D.C/ Miami in next day. You need to take action based on several subjects (transportation, time in transport vehicle etc). So, action is either stay or go. what is state possibly? state = {NY, $</span>1200, tired or not maybe etc...}</em></p>
  <p>your <strong>reward function</strong> design may be trickier to design.</p>
  <p>I will suggest studying some RL-based real-life problem which will improve understanding more. You can study <a href="https://towardsdatascience.com/deep-reinforcement-learning-for-automated-stock-trading-f1dad0126a02" rel="nofollow noreferrer">this one</a> and try to relate with your problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-15T17:26:59.773Z
Answers.LastActivityDate: 2020-11-15T17:26:59.773Z
Answers.UserDisplayName: yang2x
Body: <p>I would like to build a model based on reinforcement learning (RL) for the following scenario</p>
  <blockquote>
  <p>Recommend the best route (of cities listed for a given country) that satisfies the required criteria (museum, beaches, food, etc) for a total budget of $2000.</p>
  </blockquote>
  <p>Based on the recommendation, the user will provide its feedback (as a reward), so the recommendations can be fine-tuned (by reinforcement learning) the next time. I modeled the system this way:</p>
  <ul>
  <li><p>States = (c,cr), where <span class="math-container">$c$</span> is the city and <span class="math-container">$cr$</span> is the criteria (history, beach, food, etc)</p>
  </li>
  <li><p>Actions = (p) is the price of visiting the city</p>
  </li>
  <li><p>Reward: acceptance of the cities selected by end user as a route (1 or 0)</p>
  </li>
  </ul>
  <p>The objective is to decide which list of cities together satisfy the
  given budget.</p>
  <p><strong>Is this MDP model right and how can I implement this? May be the only option is using Monte Carlo methods and linear/dynamic programming.. Is there any other way?</strong></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-16T09:03:09.560Z
Comments.Text: If you can easily generate a user feedback for a recommendation given, maybe consider using supervised learning methods? (Not a comment as I have not enough reputation)
Comments.UserDisplayName: oleg.mosalov
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-16T10:16:02.857Z
Comments.Text: You say "delivers the required criteria (museum, beaches, food, etc)". Of course, no one will deliver "museums or beaches", so I suppose that there's a typo in your post. I suggest that you fix that typo! Also, I provisionally added a new title to your post. Change it to make it more descriptive of your question.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-16T12:29:13.903Z
Comments.Text: @oleg.mosalov Thanks. Not sure how supervised learning could provide the list of cities that fulfill the criteria (budget and other parameters)? Could you please elaborate?
Comments.UserDisplayName: Cengiz
Tags: <reinforcement-learning><q-learning><optimization>
Title: How can I model this problem of delivering assets by choosing a route with reinforcement learning?
UserDisplayName: Cengiz
ThreadText: # How can I model this problem of delivering assets by choosing a route with reinforcement learning?
  
  (Asked by: username_0 on 2020-06-16)
  
  I would like to build a model based on reinforcement learning (RL) for the following scenario
  
  > Recommend the best route (of cities listed for a given country) that satisfies the required criteria (museum, beaches, food, etc) for a total budget of $2000.
  
  Based on the recommendation, the user will provide its feedback (as a reward), so the recommendations can be fine-tuned (by reinforcement learning) the next time. I modeled the system this way:
  
  * States = (c,cr), where $c$ is the city and $cr$ is the criteria (history, beach, food, etc)
  * Actions = (p) is the price of visiting the city
  * Reward: acceptance of the cities selected by end user as a route (1 or 0)
  
  The objective is to decide which list of cities together satisfy the
  given budget.
  
  **Is this MDP model right and how can I implement this? May be the only option is using Monte Carlo methods and linear/dynamic programming.. Is there any other way?**
  
  **Tags:** reinforcement-learning, q-learning, optimization
  > **Comments:**
  > **username_1**: If you can easily generate a user feedback for a recommendation given, maybe consider using supervised learning methods? (Not a comment as I have not enough reputation)  
  > **username_2**: You say "delivers the required criteria (museum, beaches, food, etc)". Of course, no one will deliver "museums or beaches", so I suppose that there's a typo in your post. I suggest that you fix that typo! Also, I provisionally added a new title to your post. Change it to make it more descriptive of your question.  
  > **username_0**: @oleg.mosalov Thanks. Not sure how supervised learning could provide the list of cities that fulfill the criteria (budget and other parameters)? Could you please elaborate?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2020-06-17. Score: 0
  
  I do not see how you came to choose prices as actions. Normally, actions are something like go left, go right, jump, stay etc.
  Analogously, I would say that in your case the actions are visiting a certain location, whereas locations are what you referred to as states. I'd go for something like that:
  locations = {location1=(c1,cr1), location2=(c1,cr2), ...}
  Actions = {Visit location1, Visit location2, ...}
  States = {--set of all the possible paths the model can generate until the budget is possibly exhausted--}
  
  The reward function could then be a combination of both the acceptance (vs. rejection) of a route/path by the user and the inverse of the cost associated with the suggested path (because you want the model to favor cheap paths in order to keep your own business costs low).
  How you balance these two terms is up to exploration.
  For rapid prototyping, check out stable baselines, which offers a bunch of highly optimized RL algorithms.
  
  
  ---
  
  ### Answer by username_4 on 2020-11-15. Score: 0
  
  I don't know what is your dataset exactly look like. But based on assumption, I would like to refer something --
  
  You can think your MDP environment this way
  
  > action = {stay, go}
  >
  > reward = {something like based on visitor's satisfaction maybe **rating**}
  >
  > state = {current money in hand, city, other some variable those key feature to make next iteration **action**}
  
  I am working on a project-based stock market trading (sorry, I can not share detail).
  In stock market trading problem, we need to decide action sequentially ( like per hour or maybe day). If your problem (and data as I assume) is a sequential action selection problem.
  
  *More detail: the first day you visited NY and enjoyed your moment and cost $800. Now you wanted to continue the tour in NY or want to go Washington D.C/ Miami in next day. You need to take action based on several subjects (transportation, time in transport vehicle etc). So, action is either stay or go. what is state possibly? state = {NY, $1200, tired or not maybe etc...}*
  
  your **reward function** design may be trickier to design.
  
  I will suggest studying some RL-based real-life problem which will improve understanding more. You can study [this one](https://towardsdatascience.com/deep-reinforcement-learning-for-automated-stock-trading-f1dad0126a02) and try to relate with your problem.
  
  
  ---
  
  