Body: <p>I am trying to model operational decisions in inventory control. The control policy is base stock with a fixed stock level of <span class="math-container">$S$</span>. That is replenishment orders are placed for every demand arrival to take the stock level to <span class="math-container">$S$</span>. The replenishments arrive at constant lead time <span class="math-container">$L$</span>. There is an upper limit <span class="math-container">$D$</span> on the allowed stock out time and it is measured every <span class="math-container">$T$</span> periods, otherwise, a cost is incurred <span class="math-container">$C_p$</span>. This system functions in a similar manner to the M/G/S queue. The stock out time can be thought as the customer waiting time due to all server busy.  So every <span class="math-container">$R$</span> period (<span class="math-container">$R$</span> is less than <span class="math-container">$T$</span>) the inventory level and pipeline of outstanding orders are monitored and a decision about whether to expedite outstanding order (a cost involved <span class="math-container">$C_e$</span>)  or not is taken in order to control the waiting/stock-out time and to minimize the total costs. </p>
  
  <p>I feel it is a time and state-dependent problem and would like to use <span class="math-container">$Q$</span>-learning to solve this MDP problem. The time period <span class="math-container">$T$</span> is typically a quarter i.e. 3 months and I plan to simulate demands as poisson arrivals. My apprehension is whether simulating arrivals would help to evaluate the Q-values because the simulation is for such a short period. Am I not overestimating the Q value in this way? I request some help on how should I proceed with implementation. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-20T10:35:29.793Z
Comments.Text: I have edited the question. I think the question earlier reflected my confused state of mind. I hope there is some clarity now. Basically, I am having trouble in understanding how I should implement q learning for my problem!
Comments.UserDisplayName: ranya
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-20T10:42:07.527Z
Comments.Text: @ranya: Yes it is clearer now, although for those of us not familiar with your stock control problem, it may help to give still more of your MDP model and experiment design. I am not clear how you end up with a short simulated time period for instance - surely you can just run a simulation that includes very many periods of length T, in theory simulating 1000s of years of stock management, in order to learn the value functions?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-20T11:22:21.333Z
Comments.Text: @NeilSlater  Thank you. I understand that I need to simulated for much longer horizon to learn the value function. But the stock-out time is measured only every $R$ (typically a quarter) period and the counter re-set every $R$ interval within the $T$ (typically a year). So I was doubting that simulation in essence would become repetitions of only $R$ period, which is small.
Comments.UserDisplayName: ranya
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-20T11:33:03.023Z
Comments.Text: I'm not 100% sure from your description, but I think that makes each R an episode, and you will want to simulate very many episodes. Whether this is true depends on whether you also reset stock levels every R based on some fixed rule unrelated to your stock control agent's job. If R is just some event that measures and manages ongoing stock, then things continue, then that's different - you will want to allow for it in your model, but it is essentially part of the environment that the agent needs to learn (making your simulations and achieving the goals more complex)
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><ai-design><q-learning>
Title: How can I use Q-learning for inventory decision making?
UserDisplayName: ranya
ThreadText: # How can I use Q-learning for inventory decision making?
  
  (Asked by: username_0 on 2019-07-19)
  
  I am trying to model operational decisions in inventory control. The control policy is base stock with a fixed stock level of $S$. That is replenishment orders are placed for every demand arrival to take the stock level to $S$. The replenishments arrive at constant lead time $L$. There is an upper limit $D$ on the allowed stock out time and it is measured every $T$ periods, otherwise, a cost is incurred $C\_p$. This system functions in a similar manner to the M/G/S queue. The stock out time can be thought as the customer waiting time due to all server busy. So every $R$ period ($R$ is less than $T$) the inventory level and pipeline of outstanding orders are monitored and a decision about whether to expedite outstanding order (a cost involved $C\_e$) or not is taken in order to control the waiting/stock-out time and to minimize the total costs.
  
  I feel it is a time and state-dependent problem and would like to use $Q$-learning to solve this MDP problem. The time period $T$ is typically a quarter i.e. 3 months and I plan to simulate demands as poisson arrivals. My apprehension is whether simulating arrivals would help to evaluate the Q-values because the simulation is for such a short period. Am I not overestimating the Q value in this way? I request some help on how should I proceed with implementation.
  
  **Tags:** reinforcement-learning, ai-design, q-learning
  > **Comments:**
  > **username_0**: I have edited the question. I think the question earlier reflected my confused state of mind. I hope there is some clarity now. Basically, I am having trouble in understanding how I should implement q learning for my problem!  
  > **username_1**: @ranya: Yes it is clearer now, although for those of us not familiar with your stock control problem, it may help to give still more of your MDP model and experiment design. I am not clear how you end up with a short simulated time period for instance - surely you can just run a simulation that includes very many periods of length T, in theory simulating 1000s of years of stock management, in order to learn the value functions?  
  > **username_0**: @NeilSlater  Thank you. I understand that I need to simulated for much longer horizon to learn the value function. But the stock-out time is measured only every $R$ (typically a quarter) period and the counter re-set every $R$ interval within the $T$ (typically a year). So I was doubting that simulation in essence would become repetitions of only $R$ period, which is small.  
  > **username_1**: I'm not 100% sure from your description, but I think that makes each R an episode, and you will want to simulate very many episodes. Whether this is true depends on whether you also reset stock levels every R based on some fixed rule unrelated to your stock control agent's job. If R is just some event that measures and manages ongoing stock, then things continue, then that's different - you will want to allow for it in your model, but it is essentially part of the environment that the agent needs to learn (making your simulations and achieving the goals more complex)  
  
  ---
  
  ## 0 answers
  
  