Answers.Body: <pre><code>def action(self, state):
      if np.random.rand() &gt; self.epsilon:
          return np.random.randint(0,4)
      return np.argmax(self.model.predict(state))
  </code></pre>
  <p>You do not need to tell the network it can only take one of the actions.  It is by these lines that a single action is decided upon.  Predict returns the Q-values for all of the actions.  Argmax gets the action with the highest Q-Value (the highest predict future reward).</p>
  <p>As an aside, the randint under epsilon needs to be of the same action_size.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-08T15:51:29.403Z
Answers.LastActivityDate: 2024-05-08T15:51:29.403Z
Answers.UserDisplayName: foreverska
Answers.Body: <blockquote>
  <p>I am not able to understand how can I tell the neural network that it can take only one of the following values</p>
  </blockquote>
  <p>You don't have to. The neural network in Deep Q Learning (the DQN) is not configured to output what the actions <em>are</em>. There are a few different possibilities, but in this case the network has been setup to predict the action value of all possible actions as an array of outputs. This is probably the most common setup for a DQN. The DQN doesn't name the actions or associate them with any value from the environment (e.g. in your case the bandwitch allocation fraction). It only knows the actions by their index position in its output.</p>
  <p>You are free to associate each action's effect on the bandwidth allocation, 0.0, 0.1, 0.2 etc to arbitary action choices. Provided you do it consistently, it doesn't matter which action index from 0 to 10 gets associated with which item. However, you may as well make a simple intuitive mapping, by dividing the action index by 10.</p>
  <p>When interpreting the DQN's output, it might predict for a certain input state e.g. <code>[-0.5, 2.1, 3.7, 1.6, 0.2, 0.5, 0.1, -0.2, -0.5, -0.9, -1.7]</code> - those are the predicted returns from choosing actions. The <em>greedy</em> action from that result would be action index 2, and if you followed the mapping convention I suggested that would be to allocate <span class="math-container">$0.2$</span> of the bandwidth.</p>
  <p>There are other Reinforcement Learning methods that do work directly with numerical actions by predicting either a single number or distribution of numbers that can be used as the action. Q learning does not do that.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-08T15:28:11.243Z
Answers.LastActivityDate: 2024-05-08T15:28:11.243Z
Answers.UserDisplayName: Neil Slater
Body: <p>PROBLEM AT HAND: I have a resource (Bandwidth) of B Hz. I have to distribute the bandwidth B to users as per their requirements. For instance, voice calls would require some amount of bandwidth while video calls would require some amount of bandwidth. Virtual reality applications and Industrial IoT would require more bandwidth than the above two cases. The fraction of bandwidth out of total bandwidth B that I can assign are as follows: [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]. That is to say, anyone among the following fractions can be assigned.</p>
  <p>I am trying to design a Reinforcement Learning Agent that would assign bandwidth to different applications based on the amount of bandwidth that a particular application requires.</p>
  <p>I am trying to use the following template for my application.</p>
  <p><a href="https://github.com/HamedCoding/DQLFrozenLake/blob/main/main_5x5.py" rel="nofollow noreferrer">Deep Q Learning</a></p>
  <p>MY QUESTION:</p>
  <p>I am not able to understand how can I tell the neural network that it can take only one of the following values:</p>
  <p>[0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]</p>
  <pre><code>class Agent:
      def __init__(self, state_size, action_size):
          self.memory = deque(maxlen=2500)
          self.learning_rate=0.001
          self.epsilon=1
          self.max_eps=1
          self.min_eps=0.01
          self.eps_decay = 0.001/3
          self.gamma=0.9
          self.state_size= state_size
          self.action_size= action_size
          self.epsilon_lst=[]
          self.model = self.buildmodel()
  
      def buildmodel(self):
          model=Sequential()
          model.add(Dense(32, input_dim=self.state_size, activation='relu'))
          model.add(Dense(32, activation='relu'))
          model.add(Dense(self.action_size, activation='linear'))
          model.compile(loss='mse', optimizer=Adam(lr=self.learning_rate))
          return model
  
      def add_memory(self, new_state, reward, done, state, action):
          self.memory.append((new_state, reward, done, state, action))
  
      def action(self, state):
          if np.random.rand() &gt; self.epsilon:
              return np.random.randint(0,4)
          return np.argmax(self.model.predict(state))
  
      def pred(self, state):
          return np.argmax(self.model.predict(state))
  
      def replay(self,batch_size):
          minibatch=random.sample(self.memory, batch_size)
          for new_state, reward, done, state, action in minibatch:
              target= reward
              if not done:
                  target=reward + self.gamma* np.amax(self.model.predict(new_state))
              target_f= self.model.predict(state)
              target_f[0][action]= target
              self.model.fit(state, target_f, epochs=1, verbose=0)
  
          if self.epsilon &gt; self.min_eps:
              self.epsilon=(self.max_eps - self.min_eps) * np.exp(-self.eps_decay*episode) + self.min_eps
  
          self.epsilon_lst.append(self.epsilon)
  
      def load(self, name):
          self.model.load_weights(name)
  
      def save(self, name):
          self.model.save_weights(name)
  </code></pre>
  
Tags: <neural-networks><reinforcement-learning><deep-rl><pytorch>
Title: Neural network for specific numbers from a range (Q learning)
UserDisplayName: ANWESA ROY
ThreadText: # Neural network for specific numbers from a range (Q learning)
  
  (Asked by: username_0 on 2024-05-06)
  
  PROBLEM AT HAND: I have a resource (Bandwidth) of B Hz. I have to distribute the bandwidth B to users as per their requirements. For instance, voice calls would require some amount of bandwidth while video calls would require some amount of bandwidth. Virtual reality applications and Industrial IoT would require more bandwidth than the above two cases. The fraction of bandwidth out of total bandwidth B that I can assign are as follows: [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]. That is to say, anyone among the following fractions can be assigned.
  I am trying to design a Reinforcement Learning Agent that would assign bandwidth to different applications based on the amount of bandwidth that a particular application requires.
  I am trying to use the following template for my application.
  Deep Q Learning
  MY QUESTION:
  I am not able to understand how can I tell the neural network that it can take only one of the following values:
  [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
  class Agent:
      def __init__(self, state_size, action_size):
          self.memory = deque(maxlen=2500)
          self.learning_rate=0.001
          self.epsilon=1
          self.max_eps=1
          self.min_eps=0.01
          self.eps_decay = 0.001/3
          self.gamma=0.9
          self.state_size= state_size
          self.action_size= action_size
          self.epsilon_lst=[]
          self.model = self.buildmodel()
  
      def buildmodel(self):
          model=Sequential()
          model.add(Dense(32, input_dim=self.state_size, activation='relu'))
          model.add(Dense(32, activation='relu'))
          model.add(Dense(self.action_size, activation='linear'))
          model.compile(loss='mse', optimizer=Adam(lr=self.learning_rate))
          return model
  
      def add_memory(self, new_state, reward, done, state, action):
          self.memory.append((new_state, reward, done, state, action))
  
      def action(self, state):
          if np.random.rand() > self.epsilon:
              return np.random.randint(0,4)
          return np.argmax(self.model.predict(state))
  
      def pred(self, state):
          return np.argmax(self.model.predict(state))
  
      def replay(self,batch_size):
          minibatch=random.sample(self.memory, batch_size)
          for new_state, reward, done, state, action in minibatch:
              target= reward
              if not done:
                  target=reward + self.gamma* np.amax(self.model.predict(new_state))
              target_f= self.model.predict(state)
              target_f[0][action]= target
              self.model.fit(state, target_f, epochs=1, verbose=0)
  
          if self.epsilon > self.min_eps:
              self.epsilon=(self.max_eps - self.min_eps) * np.exp(-self.eps_decay*episode) + self.min_eps
  
          self.epsilon_lst.append(self.epsilon)
  
      def load(self, name):
          self.model.load_weights(name)
  
      def save(self, name):
          self.model.save_weights(name)
  
  **Tags:** neural-networks, reinforcement-learning, deep-rl, pytorch
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-05-08. Score: 2 | Accepted answer
  
  > I am not able to understand how can I tell the neural network that it can take only one of the following values
  
  You don't have to. The neural network in Deep Q Learning (the DQN) is not configured to output what the actions *are*. There are a few different possibilities, but in this case the network has been setup to predict the action value of all possible actions as an array of outputs. This is probably the most common setup for a DQN. The DQN doesn't name the actions or associate them with any value from the environment (e.g. in your case the bandwitch allocation fraction). It only knows the actions by their index position in its output.
  
  You are free to associate each action's effect on the bandwidth allocation, 0.0, 0.1, 0.2 etc to arbitary action choices. Provided you do it consistently, it doesn't matter which action index from 0 to 10 gets associated with which item. However, you may as well make a simple intuitive mapping, by dividing the action index by 10.
  
  When interpreting the DQN's output, it might predict for a certain input state e.g. `[-0.5, 2.1, 3.7, 1.6, 0.2, 0.5, 0.1, -0.2, -0.5, -0.9, -1.7]` - those are the predicted returns from choosing actions. The *greedy* action from that result would be action index 2, and if you followed the mapping convention I suggested that would be to allocate $0.2$ of the bandwidth.
  
  There are other Reinforcement Learning methods that do work directly with numerical actions by predicting either a single number or distribution of numbers that can be used as the action. Q learning does not do that.
  
  
  ---
  
  ### Answer by username_1 on 2024-05-08. Score: 2
  
  def action(self, state):
      if np.random.rand() > self.epsilon:
          return np.random.randint(0,4)
      return np.argmax(self.model.predict(state))
  
  You do not need to tell the network it can only take one of the actions.  It is by these lines that a single action is decided upon.  Predict returns the Q-values for all of the actions.  Argmax gets the action with the highest Q-Value (the highest predict future reward).
  As an aside, the randint under epsilon needs to be of the same action_size.
  
  
  ---
  
  