Answers.Body: <p>One community that has very recently been attacking problems of the type posed by your question is the <strong>Bayesian sequential optimal experimental design</strong> (Bayesian sOED) community. The Bayesian sOED setting generally assumes that there is a current <strong>belief state</strong> <span class="math-container">$x_k \in \mathcal{X}$</span> over an underlying system with some unknown quantity (in your case, unknown parameters) and possible <strong>design choices</strong> <span class="math-container">$d_k \in \mathcal{D}$</span> that may be conditioned on the current belief state. Performing <strong>experiment</strong> <span class="math-container">$k \in \{0, 1, 2, \ldots, N-1\}$</span> is equivalent to choosing a design <span class="math-container">$d_k$</span>, receiving an observation <span class="math-container">$y_k \in \mathcal{Y}$</span> from the underlying system, and earning a <strong>utility</strong> <span class="math-container">$g(x_k, d_k, y_k) \in \mathbb{R}$</span> based on the current belief state <span class="math-container">$x_k$</span>, design choice <span class="math-container">$d_k$</span>, and observation <span class="math-container">$y_k$</span>. The <strong>goal</strong> is to choose a sequence of designs <span class="math-container">$(d_0, d_1, d_2, \ldots, d_{N-1})$</span> that maximizes the expected sum of utility across all experiments. Learning more information about the underlying system (e.g. the unknown parameters) is directly helpful in achieving that goal by producing more informative belief states <span class="math-container">$x_{k+1}$</span> after the conclusion of each experiment.</p>
  <p>The above <strong>experimental design</strong> setting is:</p>
  <ul>
  <li><strong>Bayesian</strong> due to incorporating a belief state over the underlying system,</li>
  <li><strong>sequential</strong> because experiments are performed sequentially to use all information about past experiments as opposed to performing all experiments concurrently (<em>batch</em> design),</li>
  <li><strong>optimal</strong> due to maximizing the expected sum of utility across all experiments as opposed to only performing the maximization over the next experiment (<em>greedy</em> design).</li>
  </ul>
  <p>Since your exact question is simply asking about relevant research communities, I won't make any specific remarks in this answer regarding how to solve your illustrative problem using Bayesian sOED (feel free to make a follow-up question on that topic, if desired). Instead, I will list some references that I personally have found useful for solving problems in this field, with those most directly related to the Bayesian sOED problem listed first:</p>
  <ul>
  <li><a href="https://arxiv.org/abs/2110.15335" rel="nofollow noreferrer">Bayesian Sequential Optimal Experimental Design for Nonlinear Models Using Policy Gradient Reinforcement Learning</a></li>
  <li><a href="https://arxiv.org/abs/2202.00821" rel="nofollow noreferrer">Optimizing Sequential Experimental Design with Deep Reinforcement Learning</a></li>
  <li><a href="https://arxiv.org/abs/1604.08320" rel="nofollow noreferrer">Sequential Bayesian optimal experimental design via approximate dynamic programming</a></li>
  <li><a href="https://conservancy.umn.edu/handle/11299/241402" rel="nofollow noreferrer">Bayesian Sequential Optimal Experimental Design for Inverse Problems using Deep Reinforcement Learning</a></li>
  <li><a href="https://ieeexplore.ieee.org/abstract/document/10069722" rel="nofollow noreferrer">Bayesian Sequential Optimal Experimental Design for Linear Regression with Reinforcement Learning</a></li>
  <li><a href="https://arxiv.org/abs/1107.1445" rel="nofollow noreferrer">Bayesian experimental design for the active nitridation of graphite by atomic nitrogen</a></li>
  <li><a href="https://arxiv.org/abs/1108.4146" rel="nofollow noreferrer">Simulation-based optimal Bayesian experimental design for nonlinear systems</a></li>
  <li><a href="https://www.jstor.org/stable/2246015" rel="nofollow noreferrer">Bayesian Experimental Design: A Review</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-23T12:10:32.947Z
Answers.Text: Marked it as solved. Thanks. Very useful links, some of which i had not seen. I will formulate a subsequent question on how to formulate my problem as an sOED formulation, i make sure to tag you here when i have the question up and running. Thanks alot. Kind regards/paul
Answers.UserDisplayName: paul
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-24T16:43:58.417Z
Answers.Text: i just gave the answer an upvote! thanks a lot. I removed the previous secondary thread i created and instead made another thread which really sums up my problem and addresses my primary concerns when it comes to "solving" this problem. Find that thread in the edit part or:  https://ai.stackexchange.com/questions/41460/methods-for-sequential-decision-optimization-problem-with-nonlinear-bayesian-rew  I would really appreciate you giving it a try, it is incredibly helpful to me to have someone who knows these stuff looking at my problem. Thanks in advance, kind regards.
Answers.UserDisplayName: paul
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-22T19:08:30.517Z
Answers.LastActivityDate: 2023-07-22T19:08:30.517Z
Answers.UserDisplayName: DeepQZero
Body: <p>I am trying to find research papers addressing a problem that, in my opinion, deserves significant attention. However, I am having difficulty locating relevant information.</p>
  <p>To illustrate the problem at hand, consider a multivariate Bayesian regression model represented as <span class="math-container">$y_t = b_1 \cdot x_{1,t}^{c_1} + b_2 \cdot x_{2,t}^{c_2}$</span>. The purpose of this model is to estimate rewards <span class="math-container">$y_t$</span> based on inputs (<span class="math-container">$x_{1,t}$</span>, <span class="math-container">$x_{2,t}$</span>) at each time step denoted by <span class="math-container">$t$</span>. The parameters <span class="math-container">$b_1$</span> and <span class="math-container">$b_2$</span> are assumed to follow a half-normal distribution, while <span class="math-container">$c_1$</span> and <span class="math-container">$c_2$</span> follow a gamma distribution. I aim to plan inputs over a given horizon <span class="math-container">$H$</span> given a specified budget <span class="math-container">$\left(\sum_t x_{1,t} + \sum_{t} x_{2,t} \le \mbox{budget}\right)$</span>, with the main objective of maximizing cumulative rewards. To help with this goal, it is important to also gain knowledge about the model parameters to facilitate improved allocation of inputs for maximizing cumulative rewards in the future.</p>
  <p>Every day we make a decision on how to allocate a portion of the total budget and every day we receive a response (<span class="math-container">$y=\mbox{sales}$</span>). Typically, we have access to 30-300 daily data points to establish the initial posterior, and the planning horizon is typically 30 days.</p>
  <p>This problem exhibits a trade-off between exploration and exploitation, which is commonly encountered in communities such as reinforcement learning and dual control. However, the constraint of only being able to run experiments based on the exact functional form of the model, rather than simulations in the real environment, makes it challenging to apply standard reinforcement learning techniques. While elements of this problem can be found in other communities, such as Bayesian experimental design and active learning, these approaches primarily address the exploration aspect.</p>
  <p>An intuitive strategy could involve incorporating certain aspects of active learning, such as maximum information gain, along with an <span class="math-container">$\epsilon$</span>-greedy approach. In this strategy, the mean of the posterior predictive distribution of the input can be used during exploitation, while maximum information gain can guide exploration by proposing new inputs. I can of course think of more approaches, Thompson sampling being one of them where we sample a single sample of parameters from our posterior and use it as ground truth in the downstream optimization task. Another one is to take multiple samples and use the mean of the posterior predictive distribution. One could be UCB with some tweaks etc. A third one could be to take a Thompson sample and compute the downstream optimization task and repeat it iteratively to get an comprehension of the variance of possible solutions and maybe in the end take the mean or median.</p>
  <hr />
  <p>Despite the belief that this problem falls within the realm of sequential decision optimization and should have received extensive research attention, it is challenging to find specific resources that tackle this type of problem.</p>
  <p><strong>Question</strong>: In which communities can I find more information about my problem type?</p>
  <p><em>Crossposted</em>: <a href="https://or.stackexchange.com/questions/10611/bayesian-regression-model-as-reward-function-in-exploration-vs-exploitation-sett">https://or.stackexchange.com/questions/10611/bayesian-regression-model-as-reward-function-in-exploration-vs-exploitation-sett</a></p>
  <hr />
  <p><strong>EDIT</strong></p>
  <p>After reading the reports provided in the response below and examining various model-based Bayesian reinforcement learning techniques, several concerns arose in my mind, accompanied by potential solution strategies. To address these concerns and demonstrate some techniques applicable to resolving this matter, I initiated a new discussion thread. The problem formulation in that thread bears a striking resemblance to the one above, albeit with additional contextual information that closely aligns with the real problem I am currently engaged in.
  This thread can be read here:
  <a href="https://ai.stackexchange.com/questions/41460/methods-for-sequential-decision-optimization-problem-with-nonlinear-bayesian-rew">Methods for sequential decision optimization problem with nonlinear bayesian reward function</a> In this current question, I am solely interested in hearing about which communities have been trying to attack this problem type.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-28T17:04:02.720Z
Comments.Text: Hi @paul and welcome to AI Stack Exchange! If possible, please outline your specific question in the body of the post. Right now, I'm unsure of what question to answer. I did my PhD in some very similar topic, and I'd like to help out if possible. Thanks, and we look forward to more of your questions in the future!
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-29T07:14:12.807Z
Comments.Text: hi @DeepQZero , just edited: My main question is, where can i find more information about my problem type, under which communities, has this problem-type even been tackled before?
Comments.UserDisplayName: paul
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-29T07:16:54.690Z
Comments.Text: i would love to discuss this problem with someone who actually worked on something similar @DeepQZero because everywhere i look it is extremely difficult to find relevant info on this, bits and pieces can be taken from all the sequential decision communities but it seems so strange that noone had tackled this exact problem type before
Comments.UserDisplayName: paul
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-29T08:24:05.610Z
Comments.Text: also added a couple of more strategies(policies) i been thinking off, it seems so strange however that this problem has not been tackled before extensively in the literature.
Comments.UserDisplayName: paul
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-29T14:14:31.163Z
Comments.Text: Thank you for the update. Another question - is gaining info about the model parameters rewarded somehow, or is it only helpful in maximizing the total return? I'm asking this question since I usually have seen maximizing total rewards as the sole goal in similar problems, whereas gaining info about the model parameters is only a byproduct of trying to maximize the total rewards. If you need clarification, just let me know.
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-29T16:01:23.300Z
Comments.Text: @DeepQZero it is solely helpful in maximizing the total return. No extra reward added for gaining info about them.
Comments.UserDisplayName: paul
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-29T17:59:16.887Z
Comments.Text: i would say that certain bandit-problems, active learning for regression, MPC is the closest i got to finding research that resembles this problem but they all have parts that are very different from this setting
Comments.UserDisplayName: paul
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-30T19:04:21.770Z
Comments.Text: I'll try to give an answer by the end of next week. If I don't give an answer or don't give an update here, please send me another message in this comment chain to get my attention again.
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-30T19:09:55.440Z
Comments.Text: @DeepQZero  Lovely, thank you very much. Are there any specific research papers or Google searches that you recommend right away?
Comments.UserDisplayName: paul
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-30T19:15:54.600Z
Comments.Text: Try this arxiv paper: Bayesian Sequential Optimal Experimental Design for
  Nonlinear Models Using Policy Gradient Reinforcement Learning
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-05T11:29:48.010Z
Comments.Text: @DeepQZero wow, finally i found reports tackling something similar, awesome reports, not just that one, but all the other sOED reports. Will definitely reference them in my final report about my real problem(the one presented here is just a toy one showcasing main considerations). A wide range of questions popped up in my mind as i am digesting the reports and trying to bridge it to my problem. I aim to create maybe separate threads showcasing these questions by friday approx and then link those threads in this one. Hold back until i create those threads, i think you will ace all of them.
Comments.UserDisplayName: paul
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-06T13:56:42.477Z
Comments.Text: Glad to hear it! I'm busy until the weekend (at the very earliest) anyways, so I can't look at any of this until at least then. This sOED problem is very rich, so it doesn't surprise me that it's prompting more questions in your mind.
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-07T18:22:01.020Z
Comments.Text: @DeepQZero no worries, i am just extremely happy someone can actually answer some of my questions, no matter how long it takes :), i created a separate thread/question(read the EDIT part in this question), that sums up my main concerns that popped up in my head when digesting the sOED reports, i would be so happy if you could shed some light on my thoughts in that thread(it is basically a bridge between an algorithm stemming from the problem above and sOED reports): https://ai.stackexchange.com/questions/41198/is-this-deterministic-lookahead-policy-greedy-myopic-and-suboptimal-if-so-can
Comments.UserDisplayName: paul
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-12T15:40:44.113Z
Comments.Text: @DeepQZero hi, hope you are doing well, just wanted to send a little reminder in case you had forgotten, i appreciate your help immensely. kind regards, Paul
Comments.UserDisplayName: paul
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-22T17:27:22.980Z
Comments.Text: I finally got some time to look at this question again. I'll try to give an answer soon, perhaps today.
Comments.UserDisplayName: DeepQZero
Tags: <reinforcement-learning><exploration-exploitation-tradeoff><bayesian-optimization><optimal-policy>
Title: In which community does using a Bayesian regression model as a reward function with exploration vs. exploitation challenges fall under?
UserDisplayName: paul
ThreadText: # In which community does using a Bayesian regression model as a reward function with exploration vs. exploitation challenges fall under?
  
  (Asked by: username_0 on 2023-06-26)
  
  I am trying to find research papers addressing a problem that, in my opinion, deserves significant attention. However, I am having difficulty locating relevant information.
  
  To illustrate the problem at hand, consider a multivariate Bayesian regression model represented as $y\_t = b\_1 \cdot x\_{1,t}^{c\_1} + b\_2 \cdot x\_{2,t}^{c\_2}$. The purpose of this model is to estimate rewards $y\_t$ based on inputs ($x\_{1,t}$, $x\_{2,t}$) at each time step denoted by $t$. The parameters $b\_1$ and $b\_2$ are assumed to follow a half-normal distribution, while $c\_1$ and $c\_2$ follow a gamma distribution. I aim to plan inputs over a given horizon $H$ given a specified budget $\left(\sum\_t x\_{1,t} + \sum\_{t} x\_{2,t} \le \mbox{budget}\right)$, with the main objective of maximizing cumulative rewards. To help with this goal, it is important to also gain knowledge about the model parameters to facilitate improved allocation of inputs for maximizing cumulative rewards in the future.
  
  Every day we make a decision on how to allocate a portion of the total budget and every day we receive a response ($y=\mbox{sales}$). Typically, we have access to 30-300 daily data points to establish the initial posterior, and the planning horizon is typically 30 days.
  
  This problem exhibits a trade-off between exploration and exploitation, which is commonly encountered in communities such as reinforcement learning and dual control. However, the constraint of only being able to run experiments based on the exact functional form of the model, rather than simulations in the real environment, makes it challenging to apply standard reinforcement learning techniques. While elements of this problem can be found in other communities, such as Bayesian experimental design and active learning, these approaches primarily address the exploration aspect.
  
  An intuitive strategy could involve incorporating certain aspects of active learning, such as maximum information gain, along with an $\epsilon$-greedy approach. In this strategy, the mean of the posterior predictive distribution of the input can be used during exploitation, while maximum information gain can guide exploration by proposing new inputs. I can of course think of more approaches, Thompson sampling being one of them where we sample a single sample of parameters from our posterior and use it as ground truth in the downstream optimization task. Another one is to take multiple samples and use the mean of the posterior predictive distribution. One could be UCB with some tweaks etc. A third one could be to take a Thompson sample and compute the downstream optimization task and repeat it iteratively to get an comprehension of the variance of possible solutions and maybe in the end take the mean or median.
  
  ---
  
  Despite the belief that this problem falls within the realm of sequential decision optimization and should have received extensive research attention, it is challenging to find specific resources that tackle this type of problem.
  
  **Question**: In which communities can I find more information about my problem type?
  
  *Crossposted*: <https://or.stackexchange.com/questions/10611/bayesian-regression-model-as-reward-function-in-exploration-vs-exploitation-sett>
  
  ---
  
  **EDIT**
  
  After reading the reports provided in the response below and examining various model-based Bayesian reinforcement learning techniques, several concerns arose in my mind, accompanied by potential solution strategies. To address these concerns and demonstrate some techniques applicable to resolving this matter, I initiated a new discussion thread. The problem formulation in that thread bears a striking resemblance to the one above, albeit with additional contextual information that closely aligns with the real problem I am currently engaged in.
  This thread can be read here:
  [Methods for sequential decision optimization problem with nonlinear bayesian reward function](https://ai.stackexchange.com/questions/41460/methods-for-sequential-decision-optimization-problem-with-nonlinear-bayesian-rew) In this current question, I am solely interested in hearing about which communities have been trying to attack this problem type.
  
  **Tags:** reinforcement-learning, exploration-exploitation-tradeoff, bayesian-optimization, optimal-policy
  > **Comments:**
  > **username_1**: Hi @paul and welcome to AI Stack Exchange! If possible, please outline your specific question in the body of the post. Right now, I'm unsure of what question to answer. I did my PhD in some very similar topic, and I'd like to help out if possible. Thanks, and we look forward to more of your questions in the future!  
  > **username_0**: hi @DeepQZero , just edited: My main question is, where can i find more information about my problem type, under which communities, has this problem-type even been tackled before?  
  > **username_0**: i would love to discuss this problem with someone who actually worked on something similar @DeepQZero because everywhere i look it is extremely difficult to find relevant info on this, bits and pieces can be taken from all the sequential decision communities but it seems so strange that noone had tackled this exact problem type before  
  > **username_0**: also added a couple of more strategies(policies) i been thinking off, it seems so strange however that this problem has not been tackled before extensively in the literature.  
  > **username_1**: Thank you for the update. Another question - is gaining info about the model parameters rewarded somehow, or is it only helpful in maximizing the total return? I'm asking this question since I usually have seen maximizing total rewards as the sole goal in similar problems, whereas gaining info about the model parameters is only a byproduct of trying to maximize the total rewards. If you need clarification, just let me know.  
  > **username_0**: @DeepQZero it is solely helpful in maximizing the total return. No extra reward added for gaining info about them.  
  > **username_0**: i would say that certain bandit-problems, active learning for regression, MPC is the closest i got to finding research that resembles this problem but they all have parts that are very different from this setting  
  > **username_1**: I'll try to give an answer by the end of next week. If I don't give an answer or don't give an update here, please send me another message in this comment chain to get my attention again.  
  > **username_0**: @DeepQZero  Lovely, thank you very much. Are there any specific research papers or Google searches that you recommend right away?  
  > **username_1**: Try this arxiv paper: Bayesian Sequential Optimal Experimental Design for Nonlinear Models Using Policy Gradient Reinforcement Learning  
  > **username_0**: @DeepQZero wow, finally i found reports tackling something similar, awesome reports, not just that one, but all the other sOED reports. Will definitely reference them in my final report about my real problem(the one presented here is just a toy one showcasing main considerations). A wide range of questions popped up in my mind as i am digesting the reports and trying to bridge it to my problem. I aim to create maybe separate threads showcasing these questions by friday approx and then link those threads in this one. Hold back until i create those threads, i think you will ace all of them.  
  > **username_1**: Glad to hear it! I'm busy until the weekend (at the very earliest) anyways, so I can't look at any of this until at least then. This sOED problem is very rich, so it doesn't surprise me that it's prompting more questions in your mind.  
  > **username_0**: @DeepQZero no worries, i am just extremely happy someone can actually answer some of my questions, no matter how long it takes :), i created a separate thread/question(read the EDIT part in this question), that sums up my main concerns that popped up in my head when digesting the sOED reports, i would be so happy if you could shed some light on my thoughts in that thread(it is basically a bridge between an algorithm stemming from the problem above and sOED reports): https://ai.stackexchange.com/questions/41198/is-this-deterministic-lookahead-policy-greedy-myopic-and-suboptimal-if-so-can  
  > **username_0**: @DeepQZero hi, hope you are doing well, just wanted to send a little reminder in case you had forgotten, i appreciate your help immensely. kind regards, Paul  
  > **username_1**: I finally got some time to look at this question again. I'll try to give an answer soon, perhaps today.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-22. Score: 1 | Accepted answer
  
  One community that has very recently been attacking problems of the type posed by your question is the **Bayesian sequential optimal experimental design** (Bayesian sOED) community. The Bayesian sOED setting generally assumes that there is a current **belief state** $x\_k \in \mathcal{X}$ over an underlying system with some unknown quantity (in your case, unknown parameters) and possible **design choices** $d\_k \in \mathcal{D}$ that may be conditioned on the current belief state. Performing **experiment** $k \in \{0, 1, 2, \ldots, N-1\}$ is equivalent to choosing a design $d\_k$, receiving an observation $y\_k \in \mathcal{Y}$ from the underlying system, and earning a **utility** $g(x\_k, d\_k, y\_k) \in \mathbb{R}$ based on the current belief state $x\_k$, design choice $d\_k$, and observation $y\_k$. The **goal** is to choose a sequence of designs $(d\_0, d\_1, d\_2, \ldots, d\_{N-1})$ that maximizes the expected sum of utility across all experiments. Learning more information about the underlying system (e.g. the unknown parameters) is directly helpful in achieving that goal by producing more informative belief states $x\_{k+1}$ after the conclusion of each experiment.
  
  The above **experimental design** setting is:
  
  * **Bayesian** due to incorporating a belief state over the underlying system,
  * **sequential** because experiments are performed sequentially to use all information about past experiments as opposed to performing all experiments concurrently (*batch* design),
  * **optimal** due to maximizing the expected sum of utility across all experiments as opposed to only performing the maximization over the next experiment (*greedy* design).
  
  Since your exact question is simply asking about relevant research communities, I won't make any specific remarks in this answer regarding how to solve your illustrative problem using Bayesian sOED (feel free to make a follow-up question on that topic, if desired). Instead, I will list some references that I personally have found useful for solving problems in this field, with those most directly related to the Bayesian sOED problem listed first:
  
  * [Bayesian Sequential Optimal Experimental Design for Nonlinear Models Using Policy Gradient Reinforcement Learning](https://arxiv.org/abs/2110.15335)
  * [Optimizing Sequential Experimental Design with Deep Reinforcement Learning](https://arxiv.org/abs/2202.00821)
  * [Sequential Bayesian optimal experimental design via approximate dynamic programming](https://arxiv.org/abs/1604.08320)
  * [Bayesian Sequential Optimal Experimental Design for Inverse Problems using Deep Reinforcement Learning](https://conservancy.umn.edu/handle/11299/241402)
  * [Bayesian Sequential Optimal Experimental Design for Linear Regression with Reinforcement Learning](https://ieeexplore.ieee.org/abstract/document/10069722)
  * [Bayesian experimental design for the active nitridation of graphite by atomic nitrogen](https://arxiv.org/abs/1107.1445)
  * [Simulation-based optimal Bayesian experimental design for nonlinear systems](https://arxiv.org/abs/1108.4146)
  * [Bayesian Experimental Design: A Review](https://www.jstor.org/stable/2246015)
  
  > **Comments:**
  > **username_0**: Marked it as solved. Thanks. Very useful links, some of which i had not seen. I will formulate a subsequent question on how to formulate my problem as an sOED formulation, i make sure to tag you here when i have the question up and running. Thanks alot. Kind regards/paul  
  > **username_0**: i just gave the answer an upvote! thanks a lot. I removed the previous secondary thread i created and instead made another thread which really sums up my problem and addresses my primary concerns when it comes to "solving" this problem. Find that thread in the edit part or:  https://ai.stackexchange.com/questions/41460/methods-for-sequential-decision-optimization-problem-with-nonlinear-bayesian-rew  I would really appreciate you giving it a try, it is incredibly helpful to me to have someone who knows these stuff looking at my problem. Thanks in advance, kind regards.  
  
  ---
  
  