Body: <p>I have a time series data with a little unusual cost/reward function (I haven't seen it before)</p>
  <p>The model must predict a <span class="math-container">$Y$</span> value for any <span class="math-container">$X(t)$</span>.</p>
  <p>The reward is computed as follows. The model will receive a reward equal to <span class="math-container">$Y_\text{true} * Y_\text{prediction}$</span>. But if the reward is a positive value, the model won't receive a positive reward in next <span class="math-container">$5$</span> time steps (it will get negative rewards anytime). It means sometimes it is better for the model to predict 0 and wait for a better reward.</p>
  <p>I have two questions:</p>
  <ol>
  <li><p>Is it a supervised learning or reinforcement learning problem?</p>
  </li>
  <li><p>If it is a supervised learning, which optimization method should I use for it?</p>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-16T13:44:42.597Z
Comments.Text: It might be a good idea to describe what $Y$ is in your case (I suppose it's a number given the way you're computing the "reward") and why the reward is computed in the way you're describing, and describe your problem more in a high-level manner, if possible, i.e. what does this time-series data represent? Moreover, please, have a look at my last edit to your post to understand how to format your next post more properly.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-16T16:44:43.960Z
Comments.Text: Is the "prediction" really labelled as such in the original problem? It seems to me that as long as the player/agent in this game is confident in the sign of $X(t)$ it should make the magnitude of its output as high as possible and not really related to actual value of $X(t)$. Is there any statement or element of this problem that rewards $Y_{prediction}$ being close to $Y_{true}$?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-16T16:51:09.217Z
Comments.Text: Some questions which may help decide whether this is RL: Is a "prediction" expected on every timestep? Is the objective to obtain the most reward over some number of timesteps (e.g. a game or episode which ends) or on average? (If not, what exactly is the objective for this game). What form does the value at $X(t)$ take, and is it expected to contain data that can be used to infer the distribution of $Y_{true}$ in isolation (i.e. the value is not dependent on $X(t-1)$ in any way)?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-16T18:54:38.820Z
Comments.Text: @NeilSlater no this is not really a "prediction" problem.I use "Y prediction" label for model output value. Let's assume the problem as a 2d game which the agent is moving forward on a tiled ground. its x position : x = t . it can dig the ground at anytime to find a reward = R(x), the reward can be a positive value (i.e coins in a chest) or negative (some bombs) also it can be zero .if the agent hits a chest it cannot mine another chest in next 5 steps. The model objective is to decide to dig or skip the X(t) based on its last time-series features. we have all R(x) values in train data
Comments.UserDisplayName: MohammadAli Zeraatkar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-16T19:08:48.580Z
Comments.Text: Thanks, that makes it clear. It does seem a lot like an RL problem. Unrelated though, you give the reward equation without setting any bounds on the "prediction". Can I predict a value of 10,000,000 in the hope that the value of $Y_{true}$ for $X(t)$ is +1, and gain 10,000,000 reward? Or are there constraints on what I am allowed to predict?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-16T21:13:49.797Z
Comments.Text: @neilslater yes it have bounds from 0 to 1 for the prediction , It's a little confusing for me to determine it is RL or supervised , because we know the best action to take for the agent.
Comments.UserDisplayName: MohammadAli Zeraatkar
Tags: <reinforcement-learning><supervised-learning><algorithm-request>
Title: Is this a supervised or reinforcement learning problem, and which algorithm should I use to solve it?
UserDisplayName: MohammadAli Zeraatkar
ThreadText: # Is this a supervised or reinforcement learning problem, and which algorithm should I use to solve it?
  
  (Asked by: username_0 on 2021-04-15)
  
  I have a time series data with a little unusual cost/reward function (I haven't seen it before)
  
  The model must predict a $Y$ value for any $X(t)$.
  
  The reward is computed as follows. The model will receive a reward equal to $Y\_\text{true} \* Y\_\text{prediction}$. But if the reward is a positive value, the model won't receive a positive reward in next $5$ time steps (it will get negative rewards anytime). It means sometimes it is better for the model to predict 0 and wait for a better reward.
  
  I have two questions:
  
  1. Is it a supervised learning or reinforcement learning problem?
  2. If it is a supervised learning, which optimization method should I use for it?
  
  **Tags:** reinforcement-learning, supervised-learning, algorithm-request
  
  > **Comments:**
  > **username_1**: It might be a good idea to describe what $Y$ is in your case (I suppose it's a number given the way you're computing the "reward") and why the reward is computed in the way you're describing, and describe your problem more in a high-level manner, if possible, i.e. what does this time-series data represent? Moreover, please, have a look at my last edit to your post to understand how to format your next post more properly.  
  > **username_2**: Is the "prediction" really labelled as such in the original problem? It seems to me that as long as the player/agent in this game is confident in the sign of $X(t)$ it should make the magnitude of its output as high as possible and not really related to actual value of $X(t)$. Is there any statement or element of this problem that rewards $Y_{prediction}$ being close to $Y_{true}$?  
  > **username_2**: Some questions which may help decide whether this is RL: Is a "prediction" expected on every timestep? Is the objective to obtain the most reward over some number of timesteps (e.g. a game or episode which ends) or on average? (If not, what exactly is the objective for this game). What form does the value at $X(t)$ take, and is it expected to contain data that can be used to infer the distribution of $Y_{true}$ in isolation (i.e. the value is not dependent on $X(t-1)$ in any way)?  
  > **username_0**: @username_2 no this is not really a "prediction" problem.I use "Y prediction" label for model output value. Let's assume the problem as a 2d game which the agent is moving forward on a tiled ground. its x position : x = t . it can dig the ground at anytime to find a reward = R(x), the reward can be a positive value (i.e coins in a chest) or negative (some bombs) also it can be zero .if the agent hits a chest it cannot mine another chest in next 5 steps. The model objective is to decide to dig or skip the X(t) based on its last time-series features. we have all R(x) values in train data  
  > **username_2**: Thanks, that makes it clear. It does seem a lot like an RL problem. Unrelated though, you give the reward equation without setting any bounds on the "prediction". Can I predict a value of 10,000,000 in the hope that the value of $Y_{true}$ for $X(t)$ is +1, and gain 10,000,000 reward? Or are there constraints on what I am allowed to predict?  
  > **username_0**: @username_2 yes it have bounds from 0 to 1 for the prediction , It's a little confusing for me to determine it is RL or supervised , because we know the best action to take for the agent.  
  
  ---
  
  ## 0 answers
  
  