Body: <p>In deep Q learning, we execute the algorithm for each episode, and for each step within an episode, we take an action and record a reward.</p>
  
  <p>I have a situation where my action is 2-tuple <span class="math-container">$a=(a_1,a_2)$</span>. Say, in episode <span class="math-container">$i$</span>, I have to take the first half of an action <span class="math-container">$a_1$</span>, then for each step of the episode, I have to take the second half of the action <span class="math-container">$a_2$</span>.</p>
  
  <p>More specifically, say we are in episode <span class="math-container">$i$</span> and this episode has <span class="math-container">$T$</span> timesteps. First, I have to take <span class="math-container">$a_1(i)$</span>. (Where <span class="math-container">$i$</span> is used to reference episode <span class="math-container">$i$</span>.) Then, for each <span class="math-container">$t_i\in\{1,2,\ldots,T\}$</span>, I have to take action <span class="math-container">$a_2(t_i)$</span>. Once I choose <span class="math-container">$a_2(t_i)$</span>, I get an observation and a reward for the global action <span class="math-container">$(a_1(i), a_2(t_i))$</span>.</p>
  
  <p>Is it possible to apply deep Q learning? If so, how? Should I apply the <span class="math-container">$\epsilon$</span>-greedy twice?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-05T21:11:35.497Z
Comments.Text: Do you only wait one timestep after taking action $a_1$ before action $a_2$ is executed? Do you have to choose $a_1$ and $a_2$ simultaneously, or are you given an observation of the environment after executing $a_1$ and then allowed to choose $a_2$?
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-05T21:16:18.270Z
Comments.Text: At episode $i$, I choose $a_1(i)$. Now, for each time step $t_i$ in episode $i$, I choose $a_2(t_i)$. Only after choosing action $a_2(t_i)$ I get an observation and receive a reward for my chosen action $(a_1(i), a_2(t_i))$ at $(i, t_i)$.
Comments.UserDisplayName: zdm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-05T21:21:25.617Z
Comments.Text: Just to be sure, after choosing action $a = (a_1, a_2)$, then episode terminates (i.e. you don't choose another action for the remainder of the episode)?
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-05T21:22:40.477Z
Comments.Text: No, the episode has, say, $T$ steps. After choosing action $a_1$, I may choose $a_2$ at time $t$ and action $a_2'$ at time $t'$.
Comments.UserDisplayName: zdm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-05T21:29:56.107Z
Comments.Text: Are $t$ and $t'$ in the same episode? If they are, it seems to contradict what is stated in the question "then for each step of the episode, I have to take the second half of the action $a_2$." It might help to edit your original question with some of this information. It seems like a very interesting question, and I'm eager to try and solve it.
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-05T21:40:38.330Z
Comments.Text: Yes, $t$ and $t'$ are in the same episode. I am editing the question.
Comments.UserDisplayName: zdm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-05T21:58:08.407Z
Comments.Text: Can you explain your MDP further? Once you’ve chosen ($a_1,a_2$) do you then have to choose another two tuple before getting the next state and reward?
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-05T22:04:07.643Z
Comments.Text: Once I choose $(a_1, a_2)$ at episode $i$ and timestep $t_i$, I get a reward and move to the next state at $t_i+1$ and I have to choose another $a_2$, etc.
Comments.UserDisplayName: zdm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-08T11:00:23.077Z
Comments.Text: To me, the fact that you are taking actions at two different time scales suggests a similarity with hierarchical reinforcement learning - HRL([\[1\]](https://ipvs.informatik.uni-stuttgart.de/mlr/wp-content/uploads/2016/06/07-Hierarchical.pdf#page=5),[ \[2\]](http://www.youtube.com/watch?v=e8b0yC6COJ8&t=11m24s)). Have you considered looking at your problem from this perspective? I do not have a deep knowledge about HRL, so I don't know if there would be an issue if you apply existing HRL algorithms to your problem, but it was really the first thing that came to mind when reading your question.
Comments.UserDisplayName: user5093249
Tags: <reinforcement-learning><q-learning><deep-rl><dqn>
Title: How to take actions at each episode and within each step of the episode in deep Q learning?
UserDisplayName: zdm
ThreadText: # How to take actions at each episode and within each step of the episode in deep Q learning?
  
  (Asked by: username_0 on 2020-06-05)
  
  In deep Q learning, we execute the algorithm for each episode, and for each step within an episode, we take an action and record a reward.
  
  I have a situation where my action is 2-tuple $a=(a\_1,a\_2)$. Say, in episode $i$, I have to take the first half of an action $a\_1$, then for each step of the episode, I have to take the second half of the action $a\_2$.
  
  More specifically, say we are in episode $i$ and this episode has $T$ timesteps. First, I have to take $a\_1(i)$. (Where $i$ is used to reference episode $i$.) Then, for each $t\_i\in\{1,2,\ldots,T\}$, I have to take action $a\_2(t\_i)$. Once I choose $a\_2(t\_i)$, I get an observation and a reward for the global action $(a\_1(i), a\_2(t\_i))$.
  
  Is it possible to apply deep Q learning? If so, how? Should I apply the $\epsilon$-greedy twice?
  
  **Tags:** reinforcement-learning, q-learning, deep-rl, dqn
  > **Comments:**
  > **username_1**: Do you only wait one timestep after taking action $a_1$ before action $a_2$ is executed? Do you have to choose $a_1$ and $a_2$ simultaneously, or are you given an observation of the environment after executing $a_1$ and then allowed to choose $a_2$?  
  > **username_0**: At episode $i$, I choose $a_1(i)$. Now, for each time step $t_i$ in episode $i$, I choose $a_2(t_i)$. Only after choosing action $a_2(t_i)$ I get an observation and receive a reward for my chosen action $(a_1(i), a_2(t_i))$ at $(i, t_i)$.  
  > **username_1**: Just to be sure, after choosing action $a = (a_1, a_2)$, then episode terminates (i.e. you don't choose another action for the remainder of the episode)?  
  > **username_0**: No, the episode has, say, $T$ steps. After choosing action $a_1$, I may choose $a_2$ at time $t$ and action $a_2'$ at time $t'$.  
  > **username_1**: Are $t$ and $t'$ in the same episode? If they are, it seems to contradict what is stated in the question "then for each step of the episode, I have to take the second half of the action $a_2$." It might help to edit your original question with some of this information. It seems like a very interesting question, and I'm eager to try and solve it.  
  > **username_0**: Yes, $t$ and $t'$ are in the same episode. I am editing the question.  
  > **username_2**: Can you explain your MDP further? Once you’ve chosen ($a_1,a_2$) do you then have to choose another two tuple before getting the next state and reward?  
  > **username_0**: Once I choose $(a_1, a_2)$ at episode $i$ and timestep $t_i$, I get a reward and move to the next state at $t_i+1$ and I have to choose another $a_2$, etc.  
  > **username_3**: To me, the fact that you are taking actions at two different time scales suggests a similarity with hierarchical reinforcement learning - HRL([\[1\]](https://ipvs.informatik.uni-stuttgart.de/mlr/wp-content/uploads/2016/06/07-Hierarchical.pdf#page=5),[ \[2\]](http://www.youtube.com/watch?v=e8b0yC6COJ8&t=11m24s)). Have you considered looking at your problem from this perspective? I do not have a deep knowledge about HRL, so I don't know if there would be an issue if you apply existing HRL algorithms to your problem, but it was really the first thing that came to mind when reading your question.  
  
  ---
  
  ## 0 answers
  
  