Problem in hypeparameter tuning Deep Reinforcement learning
I am trying to train a deep reinforcement learning algorithm in my custom environment which is suitable to work on large discrete action space. This paper seems to provide such solution. Their model is similar of DDPG where there are two networks, actor and critic. The actor network gives the action output in a continuous form. By approximate nearest neighborhood method it selects discrete actions close to the actor output. The critic network evaluates each candidate discrete actions and give a feedback to actor network according to Q value. While training random noise is added to the output of continuous actor network. obviously the magnitude of the noise reduces as the number of episodes goes on. I am using This code and they are using Ornstein–Uhlenbeck process for noise generation.
My problem is that I'm getting extremely inconsistent solution. The number of actions in discrete action space is $2^{25}$ and those discrete values are in the range 0 to $2^{25} -1$. For this reason I have customized the given model; the $tanh$ activation function which gives output of the actor network in the range (-1,1); is multiplied by $2^{24}$. Also I have made the noise range $(-2^{24}, 2^{24})$. 458
Finally, I have made the discrete action in the range $(-2 ^{24}, 2^{24} - 1)$, by setting them as upper and lower range of the action space. But science I need it in the range 0 to $2^{25} -1$; I'm adding $2^{24}$ at the output prescribed by the model ${2^{24} + (-2 ^{24}, 2^{24} - 1) = (0, 2^{25}-1)}$. But I'm getting different training pattern on the same hyper-parameter setting. That $tanh$ output at the actor is sometime moving towards -1 or 1 in rapid speed. Where I'm sure that it is not the optimal action. I am not allowed to show my code here so giving a snapshot of hyper-parameter tuning.
class Arguments(object):
def __init__(self):
self.mode = 'train'
self.env = experiment
self.hidden1 = 100
self.hidden2 = 50
self.rate = 0.00001
self.prate = 0.0001
self.warmup = 30
self.discount = 0.99
self.bsize = 64
self.rmsize = 6000000
self.window_length = 1
self.tau = 0.001
self.ou_theta = 0.15
self.ou_sigma = 0.2
self.ou_mu = 0.0
self.validate_episodes = 1
self.max_episode_length = 30
self.validate_steps = 30
self.output = 'output'
self.debug='debug'
self.init_w = 0.003
self.train_iter= 30*110
self.epsilon=30*100
self.seed=-1
self.max_actions=2**no_switches
self.resume= 'output/Volt_var-run6'
self.k_ratio = 1e-3
Here 100,50 are the size of two hidden layers in both actor and critic network. 0.00001, 0.0001 are the learning rate at actor and critic network respectively. 30 is the initial warmup steps. 0.99 is the discount rate, 64 is the batch size, 6000000 is the reply memory size. target network updation rate 0.001. For Ornstein–Uhlenbeck process theta is the speed of mean reversion is set to 0.15, mean is 0 and standered deviation is 0.2. Also each episode has length 30, which is replayed for 110 time. The noise of exploration is allowed for 100 time here.
Please help me to solve this problem! Please let me know if I need to add more details. 695