Answers.Body: <p>Since you mentioned that you are a beginner in RL, I highly suggest that you engineer the environment to be compatible with off-the-shelf RL libraries to compare with your implemented DQN algorithm. These libraries usually contain episodic algorithms, so the next step in designing this environment is determining the terminal condition(s) for the episode.</p>
  <p>One approach is to only allow a fixed number of time steps <span class="math-container">$T$</span> before the episode terminates. Given a maximum step size <span class="math-container">$M$</span>, it is helpful (albeit unnecessary), to bound the starting position at the beginning of the episode such that the agent is able to reach the maximum objective location before the end of any episode; the location would need to be within <span class="math-container">$M \cdot T$</span> of all starting states. From another perspective, if the greatest distance from a starting point to the location is <span class="math-container">$D$</span>, then <span class="math-container">$D &lt; T \cdot M$</span> would need to hold. With this constraint, the performance of the agent during training can be easily estimated by seeing if the agent ends near the location at episode completion. This method assumes that a general vicinity of the location is known, which seems to be a reasonable assumption.</p>
  <p>We will continue this approach with the function you provided:
  <span class="math-container">$$J(x, y) = -(x^2 - 10\cos(2\pi x) + y^2 - 10\sin(2\pi y) + 20).$$</span>
  Through inspection, we can see that the maximum objective location is at <span class="math-container">$(0, 0)$</span>, the origin. A reasonable number of time steps per episode for baseline DRL algorithms is <span class="math-container">$T=100$</span>. With a maximum step size of <span class="math-container">$M=0.1$</span>, we have <span class="math-container">$D &lt; 100 \cdot 0.1 = 10$</span>. Therefore, if the initial states are sampled from a square grid centered at the origin, then the maximum side length of the grid could be <span class="math-container">$10/\sqrt{2} \approx 7.07$</span>. For simplicity, we make the side length <span class="math-container">$5$</span> to make the problem slightly easier and speed up training. Below is a sample environment coded in Python that adheres to the current Gymnasium specifications.</p>
  <hr />
  <pre><code>import gymnasium as gym
  import numpy as np
  from stable_baselines3 import PPO
  
  
  class BlackBoxEnv(gym.Env):
      def __init__(self):
          self.time_step = None
          self.x_pos, self.y_pos = None, None
          self.observation_space = gym.spaces.Box(
              np.array([-10, -10, 0]), np.array([10, 10, 1]), (3,))
          self.action_space = gym.spaces.Box(
              np.array([-0.1, -0.1]), np.array([0.1, 0.1]), (2,))
  
      def reset(self, seed=None, options=None):
          self.time_step = 0
          self.x_pos, self.y_pos = np.random.uniform(-5, 5, 2)
          return np.array([self.x_pos, self.y_pos, self.time_step/100]), {}
  
      def step(self, action):
          self.time_step += 1
          old_score = self.black_box(self.x_pos, self.y_pos)
          action_magnitude = np.sqrt(action[0] ** 2 + action[1] ** 2)
          self.x_pos += action[0]/action_magnitude * 0.1
          self.y_pos += action[1]/action_magnitude * 0.1
          new_score = self.black_box(self.x_pos, self.y_pos)
          rew = new_score - old_score
          obs = np.array([self.x_pos, self.y_pos, self.time_step/100])
          term = True if self.time_step == 100 else False
          trunc = False
          return obs, rew, term, trunc, {}
  
      @staticmethod
      def black_box(x, y):
          return -((x ** 2) - 10*np.cos(2*np.pi*x) + (y ** 2) - 10*np.cos(2*np.pi*y) + 20)
  
  env = BlackBoxEnv()
  model = PPO(&quot;MlpPolicy&quot;, env, verbose=1)
  model.learn(total_timesteps=250_000)
  </code></pre>
  <hr />
  <p>This environment allows for continuous actions in the range <span class="math-container">$[-0.1, 0.1]$</span> in each dimension, and it scales the resulting action to have magnitude <span class="math-container">$0.1=M$</span>. The observation vector is of length 3 and includes the exact <span class="math-container">$(x,y)$</span> position of the agent. The normalized current time step in the range <span class="math-container">$[0, 1]$</span> is included in the observation to model the environment as a Markov decision process; not including the time step would introduce uncertainty in the current time step which may hamper training. Considering we are only allowing a fixed number of time steps per episode, the reward at each step can be modeled as the difference in the objective between states. In this manner, <strong>the Q-value of a state will estimate the difference in the objective at the final state and current state under the current policy</strong>.</p>
  <p>By sampling random points from the starting region and computing the difference in objective from the starting points and the objective at the origin, the average reward of the optimal policy can be computed as roughly <span class="math-container">$36.7$</span> across all possible start states. <strong>The Q-value being different for various start states is not a concern here since the Q-value is conditioned on a state (and action) and will be trained to simply maximize whatever remaining reward exists until reaching the origin</strong>. I did not see anything wrong with your description of the DQN algorithm. I will provide an example using the PPO algorithm, which is run for <span class="math-container">$0.25\mbox{M}$</span> time steps.  The average training reward and the average distance of the agent's final position to the origin is shown below.</p>
  <p><a href="https://i.sstatic.net/M6W8eaJp.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/M6W8eaJp.png" alt="Training statistics of PPO on black box objective environment." /></a></p>
  <p>The mean reward hovers near the optimal mean reward near the end of training. Similarly, the magnitude of the final position of the agent is near zero, suggesting that the agent has found the maximum of the objective at the origin. There is little exploration performed near the end of training, so running the trained algorithm greedily for a few test episodes should yield the same result.</p>
  <p>Realize that the approach presented in this answer might not be perfectly suitable to your intended problem. To adapt this approach to your intended problem, consider the following:</p>
  <ul>
  <li>DQN is a discrete algorithm. You will need to provide a discrete action space such as the cardinal directions.</li>
  <li>PPO has generally been shown to outperform DQN on many tasks. You may need to add extensions to vanilla DQN such as Double DQN to yield good results.</li>
  <li>If a terminal state is known to the environment (e.g. a vicinity near the maximum objective location), then this terminal state condition can be used in addition to the fixed number of time steps for faster training.</li>
  </ul>
  <p>Good luck, and please provide an update when you have the chance!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-20T17:08:11.197Z
Answers.Text: Hi again! Thanks again for your detailed response earlier! I implemented the PPO version you suggested for my original problem which was adding a new wind turbine to a previously built wind park, and it almost worked! The model converged on two near-optimal solutions, which was great progress.
  
  Now, I’m looking to scale this approach to an objective function with multiple inputs, as I’m optimising the entire layout of the wind park. I’ve posted a new question explaining the problem in more detail and would really appreciate it if you could share your insights there as well.
Answers.UserDisplayName: Shahriar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-20T23:34:58.013Z
Answers.Text: Awesome! I will take a look at it sometime this week, but I might not have a quick answer. Also, if you would upvote this post, it would help me out.
Answers.UserDisplayName: DeepQZero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-22T14:52:23.797Z
Answers.Text: I will as soon as I get the minimum amount of reputation to do that :)
Answers.UserDisplayName: Shahriar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-23T02:32:44.117Z
Answers.Text: I upvoted your other question, so you should be able to upvote now. I'll take a look at the question later this week, but no promises on an answer from me.
Answers.UserDisplayName: DeepQZero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-06T23:42:50.537Z
Answers.LastActivityDate: 2024-09-06T23:42:50.537Z
Answers.UserDisplayName: DeepQZero
Body: <p>I'm a beginner in the field of reinforcement learning, and I'm currently working on a problem that has me a bit stuck. I'm trying to optimize a black-box objective function using reinforcement learning, but I'm not sure if I'm on the right track. I would really appreciate any guidance or insights from the experienced folks in this field.</p>
  <p>So, here's the deal: I have an objective function that takes two inputs (x, y) and outputs a scalar value z. The goal is to find the global maxima of this function. The tricky part is that I don't have access to the inner workings of the function. All I can do is start from a random position (x, y) and take small steps to explore the function's landscape.</p>
  <p>The objective function could look something like this:</p>
  <pre><code>def objective_function(x, y):
      return -(x**2 - 10 * np.cos(2 * np.pi * x) + y**2 - 10 * np.cos(2 * np.pi * y) + 20)
  </code></pre>
  <p>The idea is to use Q-value estimation with a neural network to learn the optimal policy for maximizing the objective function.
  Here's a rough outline of what I've been trying:</p>
  <p>Set up a DQN model using PyTorch that takes the current state (x, y) as input and outputs Q-values for each action.
  Create a replay buffer to store the transitions (state, action, reward, next state) during training.
  Initialize the DQN model and the replay buffer.
  Start the training loop:</p>
  <ul>
  <li>Pick a random starting state.</li>
  <li>For each step, choose an action based on the current state using epsilon-greedy exploration.</li>
  <li>Take the action and observe the next state and reward.</li>
  <li>Store the transition in the replay buffer.</li>
  <li>If the replay buffer has enough samples, sample a batch of transitions and perform gradient descent to update the DQN model.</li>
  </ul>
  <p>After training, find the optimal solution by starting from a random state and greedily selecting actions based on the learned Q-values.</p>
  <p>Now, here's where I'm a bit unsure and could use some help:</p>
  <ul>
  <li>Since there's no clear ending point or terminal state in this problem, I'm not sure how to adjust the Q-value estimation. Should I estimate the Q-value as the cumulative reward over a fixed number of steps instead of until reaching a terminal state? I might be way off here, so please correct me if I'm misunderstanding something! Also, without a terminal state, the Q-value will be different each time due to the random starting point and the limited number of steps. Is this expected, or am I approaching this problem in the wrong way?</li>
  </ul>
  <p>I would be incredibly grateful for any guidance, suggestions, or alternative approaches from the amazing people on this site. If you have any experience with similar optimization problems or have ideas on how to improve the DQN approach for this scenario, please share your wisdom!</p>
  <p>Thank you so much in advance for your help. I'm really excited to learn from all of you and hopefully make some progress on this challenging problem!</p>
  
Tags: <reinforcement-learning><python><dqn>
Title: How Do I Optimise a Black-Box Objective Function with DQN Using Reinforcement Learning?
UserDisplayName: Shahriar
ThreadText: # How Do I Optimise a Black-Box Objective Function with DQN Using Reinforcement Learning?
  
  (Asked by: username_0 on 2024-09-06)
  
  I'm a beginner in the field of reinforcement learning, and I'm currently working on a problem that has me a bit stuck. I'm trying to optimize a black-box objective function using reinforcement learning, but I'm not sure if I'm on the right track. I would really appreciate any guidance or insights from the experienced folks in this field.
  So, here's the deal: I have an objective function that takes two inputs (x, y) and outputs a scalar value z. The goal is to find the global maxima of this function. The tricky part is that I don't have access to the inner workings of the function. All I can do is start from a random position (x, y) and take small steps to explore the function's landscape.
  The objective function could look something like this:
  def objective_function(x, y):
      return -(x**2 - 10 * np.cos(2 * np.pi * x) + y**2 - 10 * np.cos(2 * np.pi * y) + 20)
  
  The idea is to use Q-value estimation with a neural network to learn the optimal policy for maximizing the objective function.
  Here's a rough outline of what I've been trying:
  Set up a DQN model using PyTorch that takes the current state (x, y) as input and outputs Q-values for each action.
  Create a replay buffer to store the transitions (state, action, reward, next state) during training.
  Initialize the DQN model and the replay buffer.
  Start the training loop:
  
  Pick a random starting state.
  For each step, choose an action based on the current state using epsilon-greedy exploration.
  Take the action and observe the next state and reward.
  Store the transition in the replay buffer.
  If the replay buffer has enough samples, sample a batch of transitions and perform gradient descent to update the DQN model.
  
  After training, find the optimal solution by starting from a random state and greedily selecting actions based on the learned Q-values.
  Now, here's where I'm a bit unsure and could use some help:
  
  Since there's no clear ending point or terminal state in this problem, I'm not sure how to adjust the Q-value estimation. Should I estimate the Q-value as the cumulative reward over a fixed number of steps instead of until reaching a terminal state? I might be way off here, so please correct me if I'm misunderstanding something! Also, without a terminal state, the Q-value will be different each time due to the random starting point and the limited number of steps. Is this expected, or am I approaching this problem in the wrong way?
  
  I would be incredibly grateful for any guidance, suggestions, or alternative approaches from the amazing people on this site. If you have any experience with similar optimization problems or have ideas on how to improve the DQN approach for this scenario, please share your wisdom!
  Thank you so much in advance for your help. I'm really excited to learn from all of you and hopefully make some progress on this challenging problem!
  
  **Tags:** reinforcement-learning, python, dqn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-06. Score: 2 | Accepted answer
  
  Since you mentioned that you are a beginner in RL, I highly suggest that you engineer the environment to be compatible with off-the-shelf RL libraries to compare with your implemented DQN algorithm. These libraries usually contain episodic algorithms, so the next step in designing this environment is determining the terminal condition(s) for the episode.
  One approach is to only allow a fixed number of time steps $T$ before the episode terminates. Given a maximum step size $M$, it is helpful (albeit unnecessary), to bound the starting position at the beginning of the episode such that the agent is able to reach the maximum objective location before the end of any episode; the location would need to be within $M \cdot T$ of all starting states. From another perspective, if the greatest distance from a starting point to the location is $D$, then $D < T \cdot M$ would need to hold. With this constraint, the performance of the agent during training can be easily estimated by seeing if the agent ends near the location at episode completion. This method assumes that a general vicinity of the location is known, which seems to be a reasonable assumption.
  We will continue this approach with the function you provided:
  $$J(x, y) = -(x^2 - 10\cos(2\pi x) + y^2 - 10\sin(2\pi y) + 20).$$
  Through inspection, we can see that the maximum objective location is at $(0, 0)$, the origin. A reasonable number of time steps per episode for baseline DRL algorithms is $T=100$. With a maximum step size of $M=0.1$, we have $D < 100 \cdot 0.1 = 10$. Therefore, if the initial states are sampled from a square grid centered at the origin, then the maximum side length of the grid could be $10/\sqrt{2} \approx 7.07$. For simplicity, we make the side length $5$ to make the problem slightly easier and speed up training. Below is a sample environment coded in Python that adheres to the current Gymnasium specifications.
  
  import gymnasium as gym
  import numpy as np
  from stable_baselines3 import PPO
  
  
  class BlackBoxEnv(gym.Env):
      def __init__(self):
          self.time_step = None
          self.x_pos, self.y_pos = None, None
          self.observation_space = gym.spaces.Box(
              np.array([-10, -10, 0]), np.array([10, 10, 1]), (3,))
          self.action_space = gym.spaces.Box(
              np.array([-0.1, -0.1]), np.array([0.1, 0.1]), (2,))
  
      def reset(self, seed=None, options=None):
          self.time_step = 0
          self.x_pos, self.y_pos = np.random.uniform(-5, 5, 2)
          return np.array([self.x_pos, self.y_pos, self.time_step/100]), {}
  
      def step(self, action):
          self.time_step += 1
          old_score = self.black_box(self.x_pos, self.y_pos)
          action_magnitude = np.sqrt(action[0] ** 2 + action[1] ** 2)
          self.x_pos += action[0]/action_magnitude * 0.1
          self.y_pos += action[1]/action_magnitude * 0.1
          new_score = self.black_box(self.x_pos, self.y_pos)
          rew = new_score - old_score
          obs = np.array([self.x_pos, self.y_pos, self.time_step/100])
          term = True if self.time_step == 100 else False
          trunc = False
          return obs, rew, term, trunc, {}
  
      @staticmethod
      def black_box(x, y):
          return -((x ** 2) - 10*np.cos(2*np.pi*x) + (y ** 2) - 10*np.cos(2*np.pi*y) + 20)
  
  env = BlackBoxEnv()
  model = PPO("MlpPolicy", env, verbose=1)
  model.learn(total_timesteps=250_000)
  
  
  This environment allows for continuous actions in the range $[-0.1, 0.1]$ in each dimension, and it scales the resulting action to have magnitude $0.1=M$. The observation vector is of length 3 and includes the exact $(x,y)$ position of the agent. The normalized current time step in the range $[0, 1]$ is included in the observation to model the environment as a Markov decision process; not including the time step would introduce uncertainty in the current time step which may hamper training. Considering we are only allowing a fixed number of time steps per episode, the reward at each step can be modeled as the difference in the objective between states. In this manner, the Q-value of a state will estimate the difference in the objective at the final state and current state under the current policy.
  By sampling random points from the starting region and computing the difference in objective from the starting points and the objective at the origin, the average reward of the optimal policy can be computed as roughly $36.7$ across all possible start states. The Q-value being different for various start states is not a concern here since the Q-value is conditioned on a state (and action) and will be trained to simply maximize whatever remaining reward exists until reaching the origin. I did not see anything wrong with your description of the DQN algorithm. I will provide an example using the PPO algorithm, which is run for $0.25\mbox{M}$ time steps.  The average training reward and the average distance of the agent's final position to the origin is shown below.
  
  The mean reward hovers near the optimal mean reward near the end of training. Similarly, the magnitude of the final position of the agent is near zero, suggesting that the agent has found the maximum of the objective at the origin. There is little exploration performed near the end of training, so running the trained algorithm greedily for a few test episodes should yield the same result.
  Realize that the approach presented in this answer might not be perfectly suitable to your intended problem. To adapt this approach to your intended problem, consider the following:
  
  DQN is a discrete algorithm. You will need to provide a discrete action space such as the cardinal directions.
  PPO has generally been shown to outperform DQN on many tasks. You may need to add extensions to vanilla DQN such as Double DQN to yield good results.
  If a terminal state is known to the environment (e.g. a vicinity near the maximum objective location), then this terminal state condition can be used in addition to the fixed number of time steps for faster training.
  
  Good luck, and please provide an update when you have the chance!
  
  > **Comments:**
  > **username_0**: Hi again! Thanks again for your detailed response earlier! I implemented the PPO version you suggested for my original problem which was adding a new wind turbine to a previously built wind park, and it almost worked! The model converged on two near-optimal solutions, which was great progress.  Now, I’m looking to scale this approach to an objective function with multiple inputs, as I’m optimising the entire layout of the wind park. I’ve posted a new question explaining the problem in more detail and would really appreciate it if you could share your insights there as well.  
  > **username_1**: Awesome! I will take a look at it sometime this week, but I might not have a quick answer. Also, if you would upvote this post, it would help me out.  
  > **username_0**: I will as soon as I get the minimum amount of reputation to do that :)  
  > **username_1**: I upvoted your other question, so you should be able to upvote now. I'll take a look at the question later this week, but no promises on an answer from me.  
  
  ---
  
  