Body: <p>I would like to ask for a piece of advice with regard to Q-learning. I am studying RL and would like to do a basic project applied to life science and calculate the reward. I have been trying to get my head around how to define all possible states of the environment. </p>
  
  <p>My states are <span class="math-container">$S = ( \text{health } (4 \text{ levels}), \text{shape } (3 \text{ levels}) \}$</span>. My actions are <span class="math-container">$A=\{a_1, a_2, \dots, a_4 \}$</span>. My possible states are <span class="math-container">$60=4 * 3 * 5$</span>. Could you advise whether these are correct?</p>
  
  <p><span class="math-container">$(s_{w_0, sh_0}, a_1, s'_{w_1, sh_1})$</span> is a tuple of the initial state <span class="math-container">$s_{w_0, sh_0}$</span>, the first action <span class="math-container">$a_1$</span> and the next state <span class="math-container">$s'_{w_1, sh_1}$</span>, where <span class="math-container">$w$</span> is the health level, <span class="math-container">$sh$</span> is the shape of the tumor. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-28T01:40:20.597Z
Comments.Text: Hi and welcome to AI SE! What is "w0,sh0" and "w1,sh1"? Also, when you say that your states are a set composed of "health" and "shape", does this mean that all states have these components and that you have 48 possible states?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-28T10:44:03.920Z
Comments.Text: S(w0,sh0) is the initial state.  The 48 possible states are the 4X3X5 , health levels(w), shape of a malign tumor and  actions of different doses of treatment.  Reading some manuscripts the description of the states are very vague. In my case I have a tuple of variables and the variables also have different levels. The agent is supposed to find the best combination to cure/improve the disease(health and and reduction of the shape of the tumor).
Comments.UserDisplayName: Aze 
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-28T10:44:22.800Z
Comments.Text: This is a fictitious scenario with only one patient to illustrate my understanding of Q-learning and compute the Q matrix manually.  I want to be sure that I got the states correct before moving to use a neural network for complex real data. Any help or comments would be highly appreciated.
Comments.UserDisplayName: Aze 
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-28T21:19:17.657Z
Comments.Text: It looks like you have 12 possible states not 48 (4*3*5 is 60 by the way). What do these 5 different doses of treatment mean? Are those different doses of actions $a_i$. For example does that mean there are 5 different doses of action $a_1$? If yes then that is an action and not part of the state so you would have 12 states (4*3) and 20 actions (4*5).
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-28T22:52:49.707Z
Comments.Text: Thanks Brale, yes these are different doses of a1 and effectively it was four no five. Thanks a lot for your help.
Comments.UserDisplayName: Aze 
Tags: <reinforcement-learning><ai-design><q-learning><applications>
Title: How should I define the state space for this life science problem?
UserDisplayName: Aze 
ThreadText: # How should I define the state space for this life science problem?
  
  (Asked by: username_0 on 2020-02-28)
  
  I would like to ask for a piece of advice with regard to Q-learning. I am studying RL and would like to do a basic project applied to life science and calculate the reward. I have been trying to get my head around how to define all possible states of the environment.
  
  My states are $S = ( \text{health } (4 \text{ levels}), \text{shape } (3 \text{ levels}) \}$. My actions are $A=\{a\_1, a\_2, \dots, a\_4 \}$. My possible states are $60=4 \* 3 \* 5$. Could you advise whether these are correct?
  
  $(s\_{w\_0, sh\_0}, a\_1, s'\_{w\_1, sh\_1})$ is a tuple of the initial state $s\_{w\_0, sh\_0}$, the first action $a\_1$ and the next state $s'\_{w\_1, sh\_1}$, where $w$ is the health level, $sh$ is the shape of the tumor.
  
  **Tags:** reinforcement-learning, ai-design, q-learning, applications
  > **Comments:**
  > **username_1**: Hi and welcome to AI SE! What is "w0,sh0" and "w1,sh1"? Also, when you say that your states are a set composed of "health" and "shape", does this mean that all states have these components and that you have 48 possible states?  
  > **username_0**: S(w0,sh0) is the initial state.  The 48 possible states are the 4X3X5 , health levels(w), shape of a malign tumor and  actions of different doses of treatment.  Reading some manuscripts the description of the states are very vague. In my case I have a tuple of variables and the variables also have different levels. The agent is supposed to find the best combination to cure/improve the disease(health and and reduction of the shape of the tumor).  
  > **username_0**: This is a fictitious scenario with only one patient to illustrate my understanding of Q-learning and compute the Q matrix manually.  I want to be sure that I got the states correct before moving to use a neural network for complex real data. Any help or comments would be highly appreciated.  
  > **username_2**: It looks like you have 12 possible states not 48 (4*3*5 is 60 by the way). What do these 5 different doses of treatment mean? Are those different doses of actions $a_i$. For example does that mean there are 5 different doses of action $a_1$? If yes then that is an action and not part of the state so you would have 12 states (4*3) and 20 actions (4*5).  
  > **username_0**: Thanks Brale, yes these are different doses of a1 and effectively it was four no five. Thanks a lot for your help.  
  
  ---
  
  ## 0 answers
  
  