Answers.Body: <p>Scale your neural network inputs.</p>
  <p>The raw observations are in range <span class="math-container">$[0,89]$</span>, and neural networks will cope badly with that used as inputs.</p>
  <p>The ideal case for NN for each input feature is a gaussian distribution with mean 0, standard deviation 1. You don't need that to be perfect, though. A simple scale - divide each element by <span class="math-container">$30$</span> and subtract <span class="math-container">$1.5$</span> - will be fine here.</p>
  <p>You can keep the environment as-is, and scale after observations are received. Up to you whether you put ready-scaled observations in the experience replay table or not. In your case it may be very slightly more efficient to do so in terms of CPU effort, but probably not something you would notice.</p>
  <p>There are other ways you might deal with these kinds of numbers in a neural network's input, but pre-scaling to a standard range is usually the simplest and by far the most common solution.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-09T04:03:19.307Z
Answers.Text: Yeh the number 1 mistake a lot of beginners make (myself included!) was not normalizing their data. It makes an enormous difference
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-09T09:18:57.233Z
Answers.Text: Thank you for your answer. I applied the changes you suggested and while it improved the randomness of the predictions, it seems like the model now is only preferring one action over the others. So the maximum of the outputs will always be the fourth one, or always the first one... I tried to change the architecture again and that's how I got those different preferences but for one fixed architecture it's the same preference again and again. I wonder if I should make another post or just stick with the comment, but I'd like to ask you if you have any clue about what's going on. Thank you again
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-09T09:44:07.637Z
Answers.Text: @Daviiid There is probably some other bug in your code. I just quickly scanned it and noticed the lack of scaling, which as Recessive points out is a common beginner's issue. Sorry I do not have time to look deeper at your code and find other implementation issues or bugs - it is very likely that you have something of that nature, where someone needs to work with you in more detail to fix the issue. If you are working alone, try to break the components apart (instead of one long script), and unit test them
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-09T09:46:50.657Z
Answers.Text: @NeilSlater Yes I fixed the scaling issue as you suggest. No need to look at the code in details thank you for guiding me, I'll look at the code in every detail. Should I remove the question since it's not really RL?
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-09T09:52:41.513Z
Answers.Text: @Daviiid: IMO, your code contains one AI conceptual problem (not scaling) which is answered here, so meets the site goals. It gets tricky when there is more going on, but a lot of people do come here with first attempts at RL problems and scripts not quite working. There is some debate between AI SE admins about when a question like this crosses a line from being useful to being too noisy to help next person. But I think it is fine to keep this question.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-08T16:43:13.777Z
Answers.LastActivityDate: 2021-07-09T09:04:15.937Z
Answers.LastEditDate: 2021-07-09T09:04:15.937Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'd like to ask you why, no matter my neural network function approximator for parametrized Q-learning implementation for a Contextual Bandits environment, I'm getting bad results. I don't know if it's a problem with my formulation of the problem and how I'm trying to solve it, or is it the neural architecture. I tried different fully-connected neural networks with different number of layers and different number of neurons (sticking to low numbers since my environment is not complex) but I always get bad results, and it seems the results are random.</p>
  <p>if my implementation of the Q-learning algorithm for the Contextual Bandits problem is right. I made an environment that randomly generates three integers between 0 and 89 and given an action (integer between 0 and 4) it returns a reward following a certain logic (if all three integers are between 0 and 29 and the action is 0 then the reward is 0 otherwise it's -1).</p>
  <p>My environment is:</p>
  <pre><code>class Environment():
  
    def __init__(self):
        
        self._observation = np.zeros((3,))
    
    def interact(self, action):
        self._observation = np.zeros((3,))
        c1, c2, c3 = np.random.randint(0, 90, 3)
        self._observation[0]=c1
        self._observation[1]=c2
        self._observation[2]=c3
        reward = -1.0
        condition = False
        if (c1&lt;30) and (c2&lt;30) and (c3&lt;30) and action==0:
            condition = True
        elif (30&lt;=c1&lt;60) and (30&lt;=c2&lt;60) and (30&lt;=c3&lt;60) and action==1:
            condition = True
        elif (60&lt;=c1&lt;90) and (60&lt;=c2&lt;90) and (60&lt;=c3&lt;90) and action==2:
            condition = True
        else:
            if action==4:
                condition = True
        if condition:
          reward = 0.0
              
        return {&quot;Observation&quot;: self._observation,
                    &quot;Reward&quot;: reward}
  </code></pre>
  <p>The interaction method doesn't return <em>state</em> or <em>time step</em>, not like what TF-Agents environments' step method does. I just thought it's not necessary for the current problem; I don't rely on <em>time steps</em> since each state doesn't influence the next state. I thought that <em>observation</em> is what should be returned, the <em>state</em> being a more general data that could contain information the agent can't observe. I don't return the <em>action</em> too because we can get it outside the environment.</p>
  <p>My function approximator of the Q-values are neural networks, always a fully connected architecture. For instance:</p>
  <pre><code>model = keras.models.Sequential([
          keras.layers.Dense(16, activation=&quot;relu&quot;, input_shape=[n_inputs]),
          keras.layers.Dense(16, activation=&quot;relu&quot;),
          keras.layers.Dense(n_outputs)])
  </code></pre>
  <p>I took the next blocks of code from <a href="https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/" rel="nofollow noreferrer">Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition</a> and adapted them to my situation:</p>
  <pre><code>env = Environment()
  
  n_inputs = 3 #Observations are made of three integers
  n_outputs = 4 #Four actions
  
  def epsilon_greedy_policy(observation, epsilon=0):
    if np.random.rand() &lt; epsilon:
      return np.random.randint(4)
    else:
      Q_values = model.predict(observation[np.newaxis])
      return np.argmax(Q_values[0])
  
  replay_buffer = deque(maxlen=2000)
  
  def sample_experiences(batch_size):
    indices = np.random.randint(len(replay_buffer), size=batch_size)
    batch = [replay_buffer[index] for index in indices]
    observations, rewards, actions = [np.array([experience[field_index] for experience in batch]) for field_index in range(3)]
    return observations, rewards, actions
  
  def play_one_step(env, observation, epsilon):
    action = epsilon_greedy_policy(observation, epsilon)
    observation, reward = env.interact(action).values()
    replay_buffer.append((observation, reward, action))
    return observation, reward
  
  batch_size = 16
  optimizer = keras.optimizers.Adam(learning_rate=1e-3)
  loss_fn = keras.losses.mean_squared_error
  
  def training_step(batch_size):
    experiences = sample_experiences(batch_size)
    observations, rewards, actions = experiences
    target_Q_values = rewards
    mask = tf.one_hot(actions, n_outputs)
    with tf.GradientTape() as tape:
      all_Q_values = model(observations)
      Q_values = tf.reduce_sum(all_Q_values * mask, axis=1, keepdims=True)
      loss = tf.reduce_mean(loss_fn(target_Q_values, Q_values))
    grads = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(grads, model.trainable_variables))
  
  epsilon = 0.01
  obs = np.random.randint(0,90,3)
  for episode in tqdm(range(1000)):
    if episode&lt;250:
      obs, reward = play_one_step(env, obs, epsilon)
    else:
      obs, reward = play_one_step(env, obs, epsilon)
      training_step(batch_size)
  </code></pre>
  <p>I'm not sure at how to evaluate the performance of the agent, but I tried this as a first approach just to see if the predicted Q-values will enable a greedy-policy to choose the best action:</p>
  <pre><code>check0 = np.random.randint(0,30,3)
  
  for i in range(30):
    arr = np.random.randint(0,30,3)
    check0 = np.vstack((check0, arr))
  
  predictions = model.predict(check0)
  
  c = 0
  for i in range(predictions.shape[0]):
    if np.argmax(predictions[i])==0:
      c+=1
  
  (c/predictions.shape[0])*100
  </code></pre>
  <p>Every time I ran the code above it gave me a totally different value. Sometimes it's 0%, sometimes it's 45%, sometimes it's 19%...</p>
  <p>The issue is that no matter my model architecture, at the end, I get random results. I wonder if it's something wrong in the overall approach to solve the problem. I want to solve a Contextual Bandit where the agent observe a continuous context, take actions and try to link together the rewards obtained with the actions and the context in order to &quot;understand&quot; the logic behind it.</p>
  <p>I hope you can help me figure out why do I get these random results.</p>
  <p>Thank you.</p>
  
Tags: <reinforcement-learning><q-learning><dqn><multi-armed-bandits><contextual-bandits>
Title: Why do I get bad results no matter my neural network function approximator for parametrized Q-learning implementation for Contextual Bandits?
UserDisplayName: Daviiid
ThreadText: # Why do I get bad results no matter my neural network function approximator for parametrized Q-learning implementation for Contextual Bandits?
  
  (Asked by: username_0 on 2021-07-08)
  
  I'd like to ask you why, no matter my neural network function approximator for parametrized Q-learning implementation for a Contextual Bandits environment, I'm getting bad results. I don't know if it's a problem with my formulation of the problem and how I'm trying to solve it, or is it the neural architecture. I tried different fully-connected neural networks with different number of layers and different number of neurons (sticking to low numbers since my environment is not complex) but I always get bad results, and it seems the results are random.
  
  if my implementation of the Q-learning algorithm for the Contextual Bandits problem is right. I made an environment that randomly generates three integers between 0 and 89 and given an action (integer between 0 and 4) it returns a reward following a certain logic (if all three integers are between 0 and 29 and the action is 0 then the reward is 0 otherwise it's -1).
  
  My environment is:
  
  ```
  class Environment():
  
    def __init__(self):
        
        self._observation = np.zeros((3,))
    
    def interact(self, action):
        self._observation = np.zeros((3,))
        c1, c2, c3 = np.random.randint(0, 90, 3)
        self._observation[0]=c1
        self._observation[1]=c2
        self._observation[2]=c3
        reward = -1.0
        condition = False
        if (c1<30) and (c2<30) and (c3<30) and action==0:
            condition = True
        elif (30<=c1<60) and (30<=c2<60) and (30<=c3<60) and action==1:
            condition = True
        elif (60<=c1<90) and (60<=c2<90) and (60<=c3<90) and action==2:
            condition = True
        else:
            if action==4:
                condition = True
        if condition:
          reward = 0.0
              
        return {"Observation": self._observation,
                    "Reward": reward}
  
  ```
  
  The interaction method doesn't return *state* or *time step*, not like what TF-Agents environments' step method does. I just thought it's not necessary for the current problem; I don't rely on *time steps* since each state doesn't influence the next state. I thought that *observation* is what should be returned, the *state* being a more general data that could contain information the agent can't observe. I don't return the *action* too because we can get it outside the environment.
  
  My function approximator of the Q-values are neural networks, always a fully connected architecture. For instance:
  
  ```
  model = keras.models.Sequential([
          keras.layers.Dense(16, activation="relu", input_shape=[n_inputs]),
          keras.layers.Dense(16, activation="relu"),
          keras.layers.Dense(n_outputs)])
  
  ```
  
  I took the next blocks of code from [Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/) and adapted them to my situation:
  
  ```
  env = Environment()
  
  n_inputs = 3 #Observations are made of three integers
  n_outputs = 4 #Four actions
  
  def epsilon_greedy_policy(observation, epsilon=0):
    if np.random.rand() < epsilon:
      return np.random.randint(4)
    else:
      Q_values = model.predict(observation[np.newaxis])
      return np.argmax(Q_values[0])
  
  replay_buffer = deque(maxlen=2000)
  
  def sample_experiences(batch_size):
    indices = np.random.randint(len(replay_buffer), size=batch_size)
    batch = [replay_buffer[index] for index in indices]
    observations, rewards, actions = [np.array([experience[field_index] for experience in batch]) for field_index in range(3)]
    return observations, rewards, actions
  
  def play_one_step(env, observation, epsilon):
    action = epsilon_greedy_policy(observation, epsilon)
    observation, reward = env.interact(action).values()
    replay_buffer.append((observation, reward, action))
    return observation, reward
  
  batch_size = 16
  optimizer = keras.optimizers.Adam(learning_rate=1e-3)
  loss_fn = keras.losses.mean_squared_error
  
  def training_step(batch_size):
    experiences = sample_experiences(batch_size)
    observations, rewards, actions = experiences
    target_Q_values = rewards
    mask = tf.one_hot(actions, n_outputs)
    with tf.GradientTape() as tape:
      all_Q_values = model(observations)
      Q_values = tf.reduce_sum(all_Q_values * mask, axis=1, keepdims=True)
      loss = tf.reduce_mean(loss_fn(target_Q_values, Q_values))
    grads = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(grads, model.trainable_variables))
  
  epsilon = 0.01
  obs = np.random.randint(0,90,3)
  for episode in tqdm(range(1000)):
    if episode<250:
      obs, reward = play_one_step(env, obs, epsilon)
    else:
      obs, reward = play_one_step(env, obs, epsilon)
      training_step(batch_size)
  
  ```
  
  I'm not sure at how to evaluate the performance of the agent, but I tried this as a first approach just to see if the predicted Q-values will enable a greedy-policy to choose the best action:
  
  ```
  check0 = np.random.randint(0,30,3)
  
  for i in range(30):
    arr = np.random.randint(0,30,3)
    check0 = np.vstack((check0, arr))
  
  predictions = model.predict(check0)
  
  c = 0
  for i in range(predictions.shape[0]):
    if np.argmax(predictions[i])==0:
      c+=1
  
  (c/predictions.shape[0])*100
  
  ```
  
  Every time I ran the code above it gave me a totally different value. Sometimes it's 0%, sometimes it's 45%, sometimes it's 19%...
  
  The issue is that no matter my model architecture, at the end, I get random results. I wonder if it's something wrong in the overall approach to solve the problem. I want to solve a Contextual Bandit where the agent observe a continuous context, take actions and try to link together the rewards obtained with the actions and the context in order to "understand" the logic behind it.
  
  I hope you can help me figure out why do I get these random results.
  
  Thank you.
  
  **Tags:** reinforcement-learning, q-learning, dqn, multi-armed-bandits, contextual-bandits
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-08. Score: 3 | Accepted answer
  
  Scale your neural network inputs.
  
  The raw observations are in range $[0,89]$, and neural networks will cope badly with that used as inputs.
  
  The ideal case for NN for each input feature is a gaussian distribution with mean 0, standard deviation 1. You don't need that to be perfect, though. A simple scale - divide each element by $30$ and subtract $1.5$ - will be fine here.
  
  You can keep the environment as-is, and scale after observations are received. Up to you whether you put ready-scaled observations in the experience replay table or not. In your case it may be very slightly more efficient to do so in terms of CPU effort, but probably not something you would notice.
  
  There are other ways you might deal with these kinds of numbers in a neural network's input, but pre-scaling to a standard range is usually the simplest and by far the most common solution.
  
  > **Comments:**
  > **username_2**: Yeh the number 1 mistake a lot of beginners make (myself included!) was not normalizing their data. It makes an enormous difference  
  > **username_0**: Thank you for your answer. I applied the changes you suggested and while it improved the randomness of the predictions, it seems like the model now is only preferring one action over the others. So the maximum of the outputs will always be the fourth one, or always the first one... I tried to change the architecture again and that's how I got those different preferences but for one fixed architecture it's the same preference again and again. I wonder if I should make another post or just stick with the comment, but I'd like to ask you if you have any clue about what's going on. Thank you again  
  > **username_1**: @username_0 There is probably some other bug in your code. I just quickly scanned it and noticed the lack of scaling, which as username_2 points out is a common beginner's issue. Sorry I do not have time to look deeper at your code and find other implementation issues or bugs - it is very likely that you have something of that nature, where someone needs to work with you in more detail to fix the issue. If you are working alone, try to break the components apart (instead of one long script), and unit test them  
  > **username_0**: @username_1 Yes I fixed the scaling issue as you suggest. No need to look at the code in details thank you for guiding me, I'll look at the code in every detail. Should I remove the question since it's not really RL?  
  > **username_1**: @username_0: IMO, your code contains one AI conceptual problem (not scaling) which is answered here, so meets the site goals. It gets tricky when there is more going on, but a lot of people do come here with first attempts at RL problems and scripts not quite working. There is some debate between AI SE admins about when a question like this crosses a line from being useful to being too noisy to help next person. But I think it is fine to keep this question.  
  
  ---
  
  