Answers.Body: <p>Use of reinforcement learning makes most sense over LSTM if control parameters can change during brewing to optimize results, which is not standard brewing policy in low end brewers but more precise control of brewing may produce superior results, especially with regard to temperature control.</p>
  
  <p>Policy does not necessarily need to be predisposed to brewing, if the search is guided by a deep artificial network (DNN), with parameters <span class="math-container">$\theta$</span> randomly initialized and defined as follows. </p>
  
  <p><span class="math-container">$$ (p, v) = f_{\theta}(s) $$</span></p>
  
  <p>The brewing process parameters comprise state <span class="math-container">$s$</span>. The vector of control change probabilities <span class="math-container">$p$</span> have components <span class="math-container">$p_a = \Pr(a|s)$</span> for each action <span class="math-container">$a$</span>. A scalar <span class="math-container">$v$</span> estimates the value of the expected outcome <span class="math-container">$o$</span> from control state <span class="math-container">$s$</span>, where $v \approx \mathbb{E} [o|s].</p>
  
  <p>Rather than characterize the variations in brewing stock as disturbance <span class="math-container">$D$</span>, it is more consistent with most policy learning approaches to consider the measured properties of brewing stock as features <span class="math-container">$\vec{b}$</span> and feed a normalized signal <span class="math-container">$\mathcal{N}(b)$</span> into additional inputs to the deep network. This way, sensitivity to variations in <span class="math-container">$\vec{b}$</span> is learned as a critical factor in brewing policy.</p>
  
  <p>In such a case, a vanilla MCTS (Monte-Carlo tree search algorithm) can be employed. Using a deep network removes the need to extract domain specific knowledge or discover heuristics to effectively guide control change policy.</p>
  
  <p>Control values must be made discrete, based on granularity that fully (but not excessively) covers brewing possibilities. The control state probabilities that guide the search can be learned using coffee drinker feedback collected in any number of ways, depending the relative costs of each form of outcome data acquisition.</p>
  
  <ul>
  <li><p>A focus group</p></li>
  <li><p>A group of independent testers</p></li>
  <li><p>Repurchase statistics at a selected set of coffee shops</p></li>
  <li><p>Feedback controls at a set of offices that provide employees with coffee</p></li>
  </ul>
  
  <p>For the comprehensiveness in control response to variations in brewing stock properties, the distribution of component values in <span class="math-container">$\vec{b}$</span> and their statistical correlation over time may need to be controlled during the training process such that trends and distributions resemble likely real world variations in brewing stock.</p>
  
  <p>Each search during the training process is guided by MCTS control changes, <span class="math-container">$a_t ∼ \pi_t$</span>. The final state at the end of training is <span class="math-container">$s_T$</span>, rated using a fixed scale of taster feedback. Selection of each state <span class="math-container">$s$</span> can be based on a control operation search advantage based on three characteristics.</p>
  
  <ul>
  <li><p>Low past experience (a low count of previous visits in the search)</p></li>
  <li><p>High probability <span class="math-container">$p$</span></p></li>
  <li><p>High average of values <span class="math-container">$\{v, \; ...\}$</span> over the leaf states of brewing trials that selected <span class="math-container">$a$</span> from <span class="math-container">$s$</span> according to the converging neural network parameters <span class="math-container">$f_{\theta}$</span></p></li>
  </ul>
  
  <p>The output of the search is <span class="math-container">$\vec{\pi}$</span> representing a probability distribution over control changes. The deep network parameters <span class="math-container">$\theta$</span> converge via gradient descent using a loss function <span class="math-container">$\ell$</span>.</p>
  
  <p><span class="math-container">$$ \ell = {(o − v)}^2 − {\pi}_T \, \ln{p} + c \, {||\theta||}^2 $$</span></p>
  
  <p>Loss is thus an aggregation of two optimality criteria.</p>
  
  <ul>
  <li><p>Mean square estimated disparity between the predicted outcome <span class="math-container">$v_t$</span>t and the taste outcome <span class="math-container">$o$</span></p></li>
  <li><p>Cross-entropy of policy vector <span class="math-container">$p_t$</span> and search probabilities <span class="math-container">${\pi}_t$</span></p></li>
  <li><p>Hyper-parameter <span class="math-container">$c$</span> weighted L2 regularization</p></li>
  </ul>
  
  <p>The updated parameters are used in subsequent brew and taste trials. All control operations must be included in each trial. Success using this approach has been obtained using either greedy or proportionality with respect to the root visit counts.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-28T23:00:17.277Z
Answers.LastActivityDate: 2018-12-28T23:17:11.170Z
Answers.LastEditDate: 2018-12-28T23:17:11.170Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>I recently encountered an interesting problem and was wondering how RL would solve it.  The objective of the problem is to maximize the coffee quality, given by box X.  The coffee quality objective function is defined by the company.  </p>
  
  <p>To maximize the quality of the coffee, we can perform 2 actions:</p>
  
  <ul>
  <li>change stirring speed of the coffee machine</li>
  <li>change the temperature of the coffee machine</li>
  </ul>
  
  <p>Now to the tricky part.  The coffee bean characteristic coming into the coffee machine is random.  We can measure its characteristics before sending them to the coffee machine, but we <strong>cannot</strong> change them.</p>
  
  <p>I formulated this problem into a control problem, such that <span class="math-container">$X$</span> is a function of my previous states, <span class="math-container">$X(t - k)$</span>, the control input , <span class="math-container">$U$</span>, and the measured disturbance, <span class="math-container">$D$</span>. Given a constant <span class="math-container">$D$</span>, the problem is trival to solve because the disturbance is a constant part of the environment.  However, during times when <span class="math-container">$D$</span> changes rapidly, the policy is no longer optimal.</p>
  
  <p>How do I inject the information of the measured disturbance into my RL agent?</p>
  
  <p><a href="https://i.sstatic.net/WTkQ9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WTkQ9.png" alt="enter image description here"></a></p>
  
Tags: <reinforcement-learning><control-problem>
Title: How does reinforcement learning handle measured disturbances?
UserDisplayName: Rui Nian
ThreadText: # How does reinforcement learning handle measured disturbances?
  
  (Asked by: username_0 on 2018-12-14)
  
  I recently encountered an interesting problem and was wondering how RL would solve it. The objective of the problem is to maximize the coffee quality, given by box X. The coffee quality objective function is defined by the company.
  
  To maximize the quality of the coffee, we can perform 2 actions:
  
  * change stirring speed of the coffee machine
  * change the temperature of the coffee machine
  
  Now to the tricky part. The coffee bean characteristic coming into the coffee machine is random. We can measure its characteristics before sending them to the coffee machine, but we **cannot** change them.
  
  I formulated this problem into a control problem, such that $X$ is a function of my previous states, $X(t - k)$, the control input , $U$, and the measured disturbance, $D$. Given a constant $D$, the problem is trival to solve because the disturbance is a constant part of the environment. However, during times when $D$ changes rapidly, the policy is no longer optimal.
  
  How do I inject the information of the measured disturbance into my RL agent?
  
  [![enter image description here](https://i.sstatic.net/WTkQ9.png)](https://i.sstatic.net/WTkQ9.png)
  
  **Tags:** reinforcement-learning, control-problem
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-12-28. Score: 1
  
  Use of reinforcement learning makes most sense over LSTM if control parameters can change during brewing to optimize results, which is not standard brewing policy in low end brewers but more precise control of brewing may produce superior results, especially with regard to temperature control.
  
  Policy does not necessarily need to be predisposed to brewing, if the search is guided by a deep artificial network (DNN), with parameters $\theta$ randomly initialized and defined as follows.
  
  $$ (p, v) = f\_{\theta}(s) $$
  
  The brewing process parameters comprise state $s$. The vector of control change probabilities $p$ have components $p\_a = \Pr(a|s)$ for each action $a$. A scalar $v$ estimates the value of the expected outcome $o$ from control state $s$, where $v \approx \mathbb{E} [o|s].
  
  Rather than characterize the variations in brewing stock as disturbance $D$, it is more consistent with most policy learning approaches to consider the measured properties of brewing stock as features $\vec{b}$ and feed a normalized signal $\mathcal{N}(b)$ into additional inputs to the deep network. This way, sensitivity to variations in $\vec{b}$ is learned as a critical factor in brewing policy.
  
  In such a case, a vanilla MCTS (Monte-Carlo tree search algorithm) can be employed. Using a deep network removes the need to extract domain specific knowledge or discover heuristics to effectively guide control change policy.
  
  Control values must be made discrete, based on granularity that fully (but not excessively) covers brewing possibilities. The control state probabilities that guide the search can be learned using coffee drinker feedback collected in any number of ways, depending the relative costs of each form of outcome data acquisition.
  
  * A focus group
  * A group of independent testers
  * Repurchase statistics at a selected set of coffee shops
  * Feedback controls at a set of offices that provide employees with coffee
  
  For the comprehensiveness in control response to variations in brewing stock properties, the distribution of component values in $\vec{b}$ and their statistical correlation over time may need to be controlled during the training process such that trends and distributions resemble likely real world variations in brewing stock.
  
  Each search during the training process is guided by MCTS control changes, $a\_t ∼ \pi\_t$. The final state at the end of training is $s\_T$, rated using a fixed scale of taster feedback. Selection of each state $s$ can be based on a control operation search advantage based on three characteristics.
  
  * Low past experience (a low count of previous visits in the search)
  * High probability $p$
  * High average of values $\{v, \; ...\}$ over the leaf states of brewing trials that selected $a$ from $s$ according to the converging neural network parameters $f\_{\theta}$
  
  The output of the search is $\vec{\pi}$ representing a probability distribution over control changes. The deep network parameters $\theta$ converge via gradient descent using a loss function $\ell$.
  
  $$ \ell = {(o − v)}^2 − {\pi}\_T \, \ln{p} + c \, {||\theta||}^2 $$
  
  Loss is thus an aggregation of two optimality criteria.
  
  * Mean square estimated disparity between the predicted outcome $v\_t$t and the taste outcome $o$
  * Cross-entropy of policy vector $p\_t$ and search probabilities ${\pi}\_t$
  * Hyper-parameter $c$ weighted L2 regularization
  
  The updated parameters are used in subsequent brew and taste trials. All control operations must be included in each trial. Success using this approach has been obtained using either greedy or proportionality with respect to the root visit counts.
  
  
  ---
  
  