<p>I have created an RL model that uses QBased policy with a neural network for estimating Q values.
My action space is of 27 actions, where each action is a 3 tuple where each value can be 1, 2 or 3. After training, the model always chooses the same action regardless of the state. For example (1, 2, 3) for all states. But I know this is wrong and not an optimal policy. But I cannot figure out why this is happening. The policy I am using is given below (code). Code is in Julia language and uses <a href="https://juliareinforcementlearning.org/" rel="nofollow noreferrer">ReinforcementLearning.jl library</a>.</p>
<pre><code># Now we use a QBasedPolicy and neural network to estimate values
# Create a flux based DNN for q - value estimation
STATE_SIZE = length(env.channels) # 3
ACTION_SIZE = length(action_set)    # 27     
model = Chain(
        Dense(STATE_SIZE, 48, relu),
        Dense(48, 48, relu),
        Dense(48, 48, relu),
        Dense(48, 48, relu),
        Dense(48, ACTION_SIZE)
    ) |&gt; cpu

# optimizer 
η = 1f-2 # Learning rate  
η_decay = 1f-3
opt = Flux.Optimiser(ADAM(η), InvDecay(η_decay))

# Create policies for each agent
single_agent_policy = Agent(
        policy = QBasedPolicy(;
                learner = BasicDQNLearner(;
                    approximator = NeuralNetworkApproximator(;
                        model = model,
                        optimizer = opt
                    ),
                    min_replay_history = 500
                ),
                explorer = EpsilonGreedyExplorer(
                    kind = :linear,
                    ϵ_stable = 0,
                    ϵ_init = 0.5,
                    warmup_steps = 300,
                    decay_steps = 700,
                    is_training = true,
                    is_break_tie = false,
                    step = 1
                )
            ),
            trajectory = CircularArraySARTTrajectory(;
                        capacity = 500,
                        state=Array{Float64, 1} =&gt; (STATE_SIZE)
                    )
            )
</code></pre>
<p>During training, the model explores and exploits various actions in different states, but, during the testing/exploitation phase, it always outputs the same action for every state.</p>
<p>I searched for similar questions on the web, but none of the questions were well answered.</p>

CC BY-SA 4.0
2022-01-12T10:39:17.167Z
I am running in to the same problem now. Were you able to solve it?
Simion Becker
CC BY-SA 4.0
2022-01-14T12:34:36.440Z
Unfortunately no :(
KnownUnknown
<reinforcement-learning><deep-rl><dqn><policies>
DQN learns to always choose the same action for all states
KnownUnknown
# DQN learns to always choose the same action for all states

(Asked by: username_0 on 2021-05-27)

I have created an RL model that uses QBased policy with a neural network for estimating Q values.
My action space is of 27 actions, where each action is a 3 tuple where each value can be 1, 2 or 3. After training, the model always chooses the same action regardless of the state. For example (1, 2, 3) for all states. But I know this is wrong and not an optimal policy. But I cannot figure out why this is happening. The policy I am using is given below (code). Code is in Julia language and uses ReinforcementLearning.jl library.
# Now we use a QBasedPolicy and neural network to estimate values
# Create a flux based DNN for q - value estimation
STATE_SIZE = length(env.channels) # 3
ACTION_SIZE = length(action_set)    # 27     
model = Chain(
        Dense(STATE_SIZE, 48, relu),
        Dense(48, 48, relu),
        Dense(48, 48, relu),
        Dense(48, 48, relu),
        Dense(48, ACTION_SIZE)
    ) |> cpu

# optimizer 
η = 1f-2 # Learning rate  
η_decay = 1f-3
opt = Flux.Optimiser(ADAM(η), InvDecay(η_decay))

# Create policies for each agent
single_agent_policy = Agent(
        policy = QBasedPolicy(;
                learner = BasicDQNLearner(;
                    approximator = NeuralNetworkApproximator(;
                        model = model,
                        optimizer = opt
                    ),
                    min_replay_history = 500
                ),
                explorer = EpsilonGreedyExplorer(
                    kind = :linear,
                    ϵ_stable = 0,
                    ϵ_init = 0.5,
                    warmup_steps = 300,
                    decay_steps = 700,
                    is_training = true,
                    is_break_tie = false,
                    step = 1
                )
            ),
            trajectory = CircularArraySARTTrajectory(;
                        capacity = 500,
                        state=Array{Float64, 1} => (STATE_SIZE)
                    )
            )

During training, the model explores and exploits various actions in different states, but, during the testing/exploitation phase, it always outputs the same action for every state.
I searched for similar questions on the web, but none of the questions were well answered.

**Tags:** reinforcement-learning, deep-rl, dqn, policies

> **Comments:**
> **username_1**: I am running in to the same problem now. Were you able to solve it?  
> **username_0**: Unfortunately no :(  

---

## 0 answers

