Answers.Body: <blockquote>
  <p>But, I am not sure if that is necessarily the best way to fix the problem of having repetitive sets of fixed action in the replay process.</p>
  </blockquote>
  <p>The best way to handle this is using a <a href="https://github.com/openai/gym/blob/master/gym/spaces/multi_discrete.py" rel="nofollow noreferrer">Multi Discrete Action Space</a>. You don't need a neural network for every action on its own.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-29T17:22:02.850Z
Answers.Text: Hi @tnfru, thanks for your suggestions. I'm not really familiar with this concept. I am under the impression that for each action taken, other actions in the action space would be impacted in someway. Could I simultaneously sample actions from several different action space for every single episode? And if I were to use Multi Discrete Action Space, would it be necessary to adopt methods like actor-critic instead?
Answers.UserDisplayName: Rkz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-30T10:36:55.433Z
Answers.Text: Yes @Rkz , DQN will work on discrete actions and Multi Discrete allows you to select from a set of values per action dimension and thus your action space is discrete. You don't need to switch methods per se, but it is very unual to use DQN for Multi Discrete so I would recommend the switch yes.
Answers.UserDisplayName: tnfru
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-27T10:27:17.530Z
Answers.LastActivityDate: 2021-08-27T10:27:17.530Z
Answers.UserDisplayName: tnfru
Body: <p>I'm trying to build a DQN Agent to take a set of 10 best actions simultaneously (integer values from 1 to 100) as outputs per episode. The input is a float. <strong>The goal is to find the optimal combination of (10) actions per episode.</strong></p>
  <p>Currently, the set up is having a single NN output 10 actions w/ the highest q-valules for each episode. But in the Memory Replay process, each individual <strong>Set</strong> (of 10 fixed actions obtained from the exploration phase) is being treated as a single action. Because the target network also takes the output of the list of 10-action from the main NN. Hence I can see the agent repeatedly trying certain <strong>Set</strong> (with a fixed 10 actions) in the replay/retrain part, whereas our goal is to <strong>find the optimal combination of 10 actions per episode</strong>, Not the optimal Set of fixed combinations. So in essence, I would like the agent to pick out and mix up the actions from the <strong>Sets</strong> with higher Q-values (known from the exploration phase) to form new optimal &quot;Sets&quot; in the Replay process.</p>
  <p>I was thinking maybe instead of using a single NN with 10 outputs I could do 10 NNs with single outputs for each episode so that each action is treated separately. And I suppose I will have 10 q-networks and target networks as well, then I could combine the results by the end of each episodes. But, I am not sure if that is necessarily the best way to fix the problem of having repetitive sets of fixed action in the replay process.</p>
  <p>Alternatively, I think the problem could be treated as a multi-armed bandit problem, except each arm here has &quot;sub-arms&quot; too so to speak, but that could require some changes to the custom environment I am working with and I don't want to touch that unless necessary.</p>
  <p>Maybe there is a clever manipulation within the retrain process given my current setup that I am not seeing. Here is a snippet of the code for some more clarity.</p>
  <pre><code>class DQNAgent():
  
      def __init__(self,optimizer):
          # Initialize atributes
          self._state_size = 1
          self._action_size = 76
          self._optimizer = optimizer
          
          self.experience_replay = deque(maxlen=2000)
          
          # # Initialize discount and exploration rate
          # self.gamma = 0.6
          # self.epsilon = 0.5
          self.gamma = 0.95
          self.epsilon = 1.0
          self.epsilon_min = 0.01
          self.epsilon_decay = 0.95
          self.learning_rate = 0.01
          
          # Build networks
          self.q_network = self._build_compile_model()
          self.target_network = self._build_compile_model()
          
      
      def store(self, state, action, reward, next_state, terminated):
          self.experience_replay.append((state, action, reward, next_state, terminated))
          
      def _build_compile_model(self):
          
          model = Sequential()
          model.add(InputLayer(input_shape=(self._state_size,)))
          model.add(Dense(100, activation='relu'))
          model.add(Dense(100, activation='relu'))
          model.add(Dense(self._action_size, activation='linear'))
          model.compile(loss='mse', optimizer=self._optimizer)
          return model
      
      def alighn_target_model(self):
          self.target_network.set_weights(self.q_network.get_weights())
  
      def retrain(self, batch_size):
          if len(self.expirience_replay) &lt; batch_size:
              return
          minibatch = random.sample(self.expirience_replay, batch_size)
      
          for state, action, reward, next_state, terminated in minibatch:
              
              target = self.q_network.predict(np.reshape(np.array(state), (-1,1)))
              print('target size :', np.shape(target))
              
              
              if terminated:
                  target[0][action] = reward
              else:
                  t = self.target_network.predict(np.reshape(np.array(next_state), (-1,1)))
                  target[0][action] = reward + self.gamma * np.amax(t)
      
              self.q_network.fit(np.reshape(np.array(state), (-1,1)), target, epochs=1, verbose=0)
          
      
      def act(self,state):
          self.epsilon *= self.epsilon_decay
          self.epsilon = max(self.epsilon_min, self.epsilon)
          action_space = [ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
         18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
         35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51,
         52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68,
         69, 70, 71, 72, 73, 74, 75, 76] #all 76 available nodes
          if np.random.rand() &lt;= self.epsilon:
              return np.array(random.sample(action_space,10))-1 #-1 to match control's index
  
          q_values = self.q_network.predict(np.reshape(np.array(state), (-1,1)))
          print(&quot;q_vals shape&quot;,np.shape(q_values))
          print('q_vals type',type(q_values))
    
          top_actions_idx = q_values[0].argsort()[-10:][::-1]
  </code></pre>
  
Tags: <deep-rl><dqn>
Title: Implementing Multiple NNs in one DQN model?
UserDisplayName: Rkz
ThreadText: # Implementing Multiple NNs in one DQN model?
  
  (Asked by: username_0 on 2021-08-24)
  
  I'm trying to build a DQN Agent to take a set of 10 best actions simultaneously (integer values from 1 to 100) as outputs per episode. The input is a float. The goal is to find the optimal combination of (10) actions per episode.
  Currently, the set up is having a single NN output 10 actions w/ the highest q-valules for each episode. But in the Memory Replay process, each individual Set (of 10 fixed actions obtained from the exploration phase) is being treated as a single action. Because the target network also takes the output of the list of 10-action from the main NN. Hence I can see the agent repeatedly trying certain Set (with a fixed 10 actions) in the replay/retrain part, whereas our goal is to find the optimal combination of 10 actions per episode, Not the optimal Set of fixed combinations. So in essence, I would like the agent to pick out and mix up the actions from the Sets with higher Q-values (known from the exploration phase) to form new optimal "Sets" in the Replay process.
  I was thinking maybe instead of using a single NN with 10 outputs I could do 10 NNs with single outputs for each episode so that each action is treated separately. And I suppose I will have 10 q-networks and target networks as well, then I could combine the results by the end of each episodes. But, I am not sure if that is necessarily the best way to fix the problem of having repetitive sets of fixed action in the replay process.
  Alternatively, I think the problem could be treated as a multi-armed bandit problem, except each arm here has "sub-arms" too so to speak, but that could require some changes to the custom environment I am working with and I don't want to touch that unless necessary.
  Maybe there is a clever manipulation within the retrain process given my current setup that I am not seeing. Here is a snippet of the code for some more clarity.
  class DQNAgent():
  
      def __init__(self,optimizer):
          # Initialize atributes
          self._state_size = 1
          self._action_size = 76
          self._optimizer = optimizer
          
          self.experience_replay = deque(maxlen=2000)
          
          # # Initialize discount and exploration rate
          # self.gamma = 0.6
          # self.epsilon = 0.5
          self.gamma = 0.95
          self.epsilon = 1.0
          self.epsilon_min = 0.01
          self.epsilon_decay = 0.95
          self.learning_rate = 0.01
          
          # Build networks
          self.q_network = self._build_compile_model()
          self.target_network = self._build_compile_model()
          
      
      def store(self, state, action, reward, next_state, terminated):
          self.experience_replay.append((state, action, reward, next_state, terminated))
          
      def _build_compile_model(self):
          
          model = Sequential()
          model.add(InputLayer(input_shape=(self._state_size,)))
          model.add(Dense(100, activation='relu'))
          model.add(Dense(100, activation='relu'))
          model.add(Dense(self._action_size, activation='linear'))
          model.compile(loss='mse', optimizer=self._optimizer)
          return model
      
      def alighn_target_model(self):
          self.target_network.set_weights(self.q_network.get_weights())
  
      def retrain(self, batch_size):
          if len(self.expirience_replay) < batch_size:
              return
          minibatch = random.sample(self.expirience_replay, batch_size)
      
          for state, action, reward, next_state, terminated in minibatch:
              
              target = self.q_network.predict(np.reshape(np.array(state), (-1,1)))
              print('target size :', np.shape(target))
              
              
              if terminated:
                  target[0][action] = reward
              else:
                  t = self.target_network.predict(np.reshape(np.array(next_state), (-1,1)))
                  target[0][action] = reward + self.gamma * np.amax(t)
      
              self.q_network.fit(np.reshape(np.array(state), (-1,1)), target, epochs=1, verbose=0)
          
      
      def act(self,state):
          self.epsilon *= self.epsilon_decay
          self.epsilon = max(self.epsilon_min, self.epsilon)
          action_space = [ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
         18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
         35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51,
         52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68,
         69, 70, 71, 72, 73, 74, 75, 76] #all 76 available nodes
          if np.random.rand() <= self.epsilon:
              return np.array(random.sample(action_space,10))-1 #-1 to match control's index
  
          q_values = self.q_network.predict(np.reshape(np.array(state), (-1,1)))
          print("q_vals shape",np.shape(q_values))
          print('q_vals type',type(q_values))
    
          top_actions_idx = q_values[0].argsort()[-10:][::-1]
  
  **Tags:** deep-rl, dqn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-27. Score: 0 | Accepted answer
  
  > But, I am not sure if that is necessarily the best way to fix the problem of having repetitive sets of fixed action in the replay process.
  
  The best way to handle this is using a [Multi Discrete Action Space](https://github.com/openai/gym/blob/master/gym/spaces/multi_discrete.py). You don't need a neural network for every action on its own.
  
  > **Comments:**
  > **username_0**: Hi @tnfru, thanks for your suggestions. I'm not really familiar with this concept. I am under the impression that for each action taken, other actions in the action space would be impacted in someway. Could I simultaneously sample actions from several different action space for every single episode? And if I were to use Multi Discrete Action Space, would it be necessary to adopt methods like actor-critic instead?  
  > **username_1**: Yes @Rkz , DQN will work on discrete actions and Multi Discrete allows you to select from a set of values per action dimension and thus your action space is discrete. You don't need to switch methods per se, but it is very unual to use DQN for Multi Discrete so I would recommend the switch yes.  
  
  ---
  
  