Body: <p>I have created an RL model that uses QBased policy with a neural network for estimating Q values.
  My action space is of 27 actions, where each action is a 3 tuple where each value can be 1, 2 or 3. After training, the model always chooses the same action regardless of the state. For example (1, 2, 3) for all states. But I know this is wrong and not an optimal policy. But I cannot figure out why this is happening. The policy I am using is given below (code). Code is in Julia language and uses <a href="https://juliareinforcementlearning.org/" rel="nofollow noreferrer">ReinforcementLearning.jl library</a>.</p>
  <pre><code># Now we use a QBasedPolicy and neural network to estimate values
  # Create a flux based DNN for q - value estimation
  STATE_SIZE = length(env.channels) # 3
  ACTION_SIZE = length(action_set)    # 27     
  model = Chain(
          Dense(STATE_SIZE, 48, relu),
          Dense(48, 48, relu),
          Dense(48, 48, relu),
          Dense(48, 48, relu),
          Dense(48, ACTION_SIZE)
      ) |&gt; cpu
  
  # optimizer 
  η = 1f-2 # Learning rate  
  η_decay = 1f-3
  opt = Flux.Optimiser(ADAM(η), InvDecay(η_decay))
  
  # Create policies for each agent
  single_agent_policy = Agent(
          policy = QBasedPolicy(;
                  learner = BasicDQNLearner(;
                      approximator = NeuralNetworkApproximator(;
                          model = model,
                          optimizer = opt
                      ),
                      min_replay_history = 500
                  ),
                  explorer = EpsilonGreedyExplorer(
                      kind = :linear,
                      ϵ_stable = 0,
                      ϵ_init = 0.5,
                      warmup_steps = 300,
                      decay_steps = 700,
                      is_training = true,
                      is_break_tie = false,
                      step = 1
                  )
              ),
              trajectory = CircularArraySARTTrajectory(;
                          capacity = 500,
                          state=Array{Float64, 1} =&gt; (STATE_SIZE)
                      )
              )
  </code></pre>
  <p>During training, the model explores and exploits various actions in different states, but, during the testing/exploitation phase, it always outputs the same action for every state.</p>
  <p>I searched for similar questions on the web, but none of the questions were well answered.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-12T10:39:17.167Z
Comments.Text: I am running in to the same problem now. Were you able to solve it?
Comments.UserDisplayName: Simion Becker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-14T12:34:36.440Z
Comments.Text: Unfortunately no :(
Comments.UserDisplayName: KnownUnknown
Tags: <reinforcement-learning><deep-rl><dqn><policies>
Title: DQN learns to always choose the same action for all states
UserDisplayName: KnownUnknown
ThreadText: # DQN learns to always choose the same action for all states
  
  (Asked by: username_0 on 2021-05-27)
  
  I have created an RL model that uses QBased policy with a neural network for estimating Q values.
  My action space is of 27 actions, where each action is a 3 tuple where each value can be 1, 2 or 3. After training, the model always chooses the same action regardless of the state. For example (1, 2, 3) for all states. But I know this is wrong and not an optimal policy. But I cannot figure out why this is happening. The policy I am using is given below (code). Code is in Julia language and uses [ReinforcementLearning.jl library](https://juliareinforcementlearning.org/).
  
  ```
  # Now we use a QBasedPolicy and neural network to estimate values
  # Create a flux based DNN for q - value estimation
  STATE_SIZE = length(env.channels) # 3
  ACTION_SIZE = length(action_set)    # 27     
  model = Chain(
          Dense(STATE_SIZE, 48, relu),
          Dense(48, 48, relu),
          Dense(48, 48, relu),
          Dense(48, 48, relu),
          Dense(48, ACTION_SIZE)
      ) |> cpu
  
  # optimizer 
  η = 1f-2 # Learning rate  
  η_decay = 1f-3
  opt = Flux.Optimiser(ADAM(η), InvDecay(η_decay))
  
  # Create policies for each agent
  single_agent_policy = Agent(
          policy = QBasedPolicy(;
                  learner = BasicDQNLearner(;
                      approximator = NeuralNetworkApproximator(;
                          model = model,
                          optimizer = opt
                      ),
                      min_replay_history = 500
                  ),
                  explorer = EpsilonGreedyExplorer(
                      kind = :linear,
                      ϵ_stable = 0,
                      ϵ_init = 0.5,
                      warmup_steps = 300,
                      decay_steps = 700,
                      is_training = true,
                      is_break_tie = false,
                      step = 1
                  )
              ),
              trajectory = CircularArraySARTTrajectory(;
                          capacity = 500,
                          state=Array{Float64, 1} => (STATE_SIZE)
                      )
              )
  
  ```
  
  During training, the model explores and exploits various actions in different states, but, during the testing/exploitation phase, it always outputs the same action for every state.
  
  I searched for similar questions on the web, but none of the questions were well answered.
  
  **Tags:** reinforcement-learning, deep-rl, dqn, policies
  
  > **Comments:**
  > **username_1**: I am running in to the same problem now. Were you able to solve it?  
  > **username_0**: Unfortunately no :(  
  
  ---
  
  ## 0 answers
  
  