Answers.Body: <p>Your <code>self.Cost</code> variable aggregates rewards over time instead of giving the cost for the current time step as a reward. This causes the gradients to grow larger and everything diverges.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-14T08:41:07.027Z
Answers.Text: Isn't a reward connected to the daily cost more greedy than one that consider the cumulative cost? For example: if a day has really low price of electricity, I would like to charge as much as possible so that I can "save" some energy to shield myself from higher prices in the future. Having a reward coupled only to the daily cost seems to go against this logic.
Answers.UserDisplayName: MiPre
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-17T10:14:34.803Z
Answers.Text: The problem is that it is integrated in the algorithm itself to consider and optimize for the cumulative cost. If you look in the [RolloutBuffer](https://github.com/DLR-RM/stable-baselines3/blob/abffa161982b92e7f4b0186611cb73653bc2553b/stable_baselines3/common/buffers.py#L316) class which is used by the PPO implementation you can see that it computes the return which is the discounted sum of rewards. This is then used as an optimization target.
Answers.UserDisplayName: Dennis Malmgren
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-12T20:54:28.833Z
Answers.LastActivityDate: 2022-11-12T20:54:28.833Z
Answers.UserDisplayName: Dennis Malmgren
Body: <p>I have a problem with my reinforcement learning model.
  I am trying to simulate an electric battery storage. To keep it as simple as possible, the efficiency of charge, storage and discharge are 100%. No depletion either.
  The battery charges when the electricity prices are low and discharge ONLY to the user at fixed hours during the day, every day.
  Therefore, the only cost for the user is power of charge * electricity price at the hour.</p>
  <p><strong>1. Define a dataframe where to store electricity prices for 500 days</strong></p>
  <pre><code>df=pd.DataFrame(np.random.randint(0,500,size=(24, 100)))
  </code></pre>
  <p><strong>2. Define the main parameters</strong></p>
  <pre><code>Lookback_window_size=7
  Current_day=Lookback_window_size
  P_charge=2 #MW
  P_discharge=3 #MW
  </code></pre>
  <p><strong>3. Define the class Battery(Env)</strong></p>
  <pre><code>class Battery(Env):
      
      metadata = {'render.modes': ['human']}
  
      def __init__(self, df):
             
          #Import the dataframe 
  
          self.df = df
  
          # The action space is an array of shape (24,). Since we are simulating day-ahead market, the action space returns
          # the overall daily charge / no charge scenario
          # action = 1 means that we charge our battery, action = 0 means that we don't charge
          
          self.action_space= spaces.MultiBinary(24)
          
          # The observation space is a matrix with 24 rows and N columns. The first N-1 columns represent the electricity 
          # prices for the current day + all the days before included in the lookback window size. The last column is all 
          # zeros except the first two rows: they store SOC (state of charge) at the end of the day and overall cost
          # (how much we paid for electricity).
          # Example: if lookback window is 7, then we have a matrix of 24 rows and 9 columns. The first 8 columns are just 
          # used to store the electricity prices, the 9th column store SOC and costs. 
          
          self.observation_space = spaces.Box(
              low=0, high=np.inf, shape=(24, Lookback_window_size + 2), dtype=np.float16)
  
      def _next_observation(self):
          
          # Add the prices of the last days to the monitor matrix
          
          prices=self.df[[i for i in range(self.Current_day - Lookback_window_size,self.Current_day + 1)]][0:].to_numpy()
                  
          extra_values=np.zeros(24)
          
          # Add extra values to monitor such as SOC, cost
          
          extra_values[[0,1]] = self.SOC, self.Cost  
          
          obs=np.c_[prices,extra_values]
          
          return obs        
      
      def _take_action(self, action):
          
          # Being the action space an array, the for loop will check the action at every hour (action[i]) and update the
          # cost and the state of charge
          
          i=0
                          
          for x in action:
              
              #When action = 1 then we charge our battery, if action = 0 then we don't charge
  
              if x == 1:
                  
                  # The cost increase based on the price of the electricity at that hour
  
                  self.Cost+=self.df[self.Current_day][i]*P_charge
  
                  # If we charge, then the state of charge (SOC) increases as well
  
                  self.SOC+=P_charge
                  
              # Everyday we discharge the battery always at the same hours
  
              if (i in range(8,14)):
  
                  self.SOC-=P_discharge
              
              # if the battery is depleted, then we directly buy electricity from the grid
  
                  if self.SOC&lt;0:
  
                      self.Cost+=self.df[self.Current_day][i+1]*(-self.SOC)
  
                      self.SOC=0
                          
              i+=1
      
      def step(self, action):
                 
          self._take_action(action)
          
          self.Current_day += 1
  
          # Maximizing the reward means minimize the costs
          
          reward =  - self.Cost
          
          # Stop at the end of the dataframe 
          
          done = self.Current_day &gt;= len(self.df.columns)-1
  
          obs = self._next_observation()
     
          return obs, reward, done, {}    
      
      
      def render(self, mode='human', close=False):
  
          print(f'Day: {self.Current_day}')
          
          print(f'SOC: {self.SOC}')
          
          print(f'Cost: {self.Cost}')      
          
          print(f'Actions: {action}')
                    
      def reset(self):
          
          self.Current_day = Lookback_window_size
          
          # Give an initial SOC value
          
          self.SOC = 50
          
          # Cost at day 0 is null
                 
          self.Cost = 0
          
          return self._next_observation()
  </code></pre>
  <p><strong>4. Train</strong></p>
  <pre><code>env = Battery(df)
  model = PPO(&quot;MlpPolicy&quot;, env, verbose=1)
  </code></pre>
  <p>Once I run these lines, I get the following error:</p>
  <p><strong>5. VALUE ERROR</strong></p>
  <pre><code>ValueError: Expected parameter logits (Tensor of shape (64, 24)) of distribution Bernoulli(logits: torch.Size([64, 24])) to satisfy the constraint Real(), but found invalid values:
  tensor([[nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          ...,
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=&lt;AddmmBackward0&gt;)
  </code></pre>
  <p>In more details:</p>
  <pre><code>| time/              |      |
  |    fps             | 784  |
  |    iterations      | 1    |
  |    time_elapsed    | 2    |
  |    total_timesteps | 2048 |
  -----------------------------
  ---------------------------------------------------------------------------
  ValueError                                Traceback (most recent call last)
  Input In [31], in &lt;cell line: 1&gt;()
  ----&gt; 1 model.learn(total_timesteps=100000)
  
  File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\ppo\ppo.py:317, in PPO.learn(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps, progress_bar)
      303 def learn(
      304     self: PPOSelf,
      305     total_timesteps: int,
     (...)
      314     progress_bar: bool = False,
      315 ) -&gt; PPOSelf:
  --&gt; 317     return super().learn(
      318         total_timesteps=total_timesteps,
      319         callback=callback,
      320         log_interval=log_interval,
      321         eval_env=eval_env,
      322         eval_freq=eval_freq,
      323         n_eval_episodes=n_eval_episodes,
      324         tb_log_name=tb_log_name,
      325         eval_log_path=eval_log_path,
      326         reset_num_timesteps=reset_num_timesteps,
      327         progress_bar=progress_bar,
      328     )
  
  File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\on_policy_algorithm.py:283, in OnPolicyAlgorithm.learn(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps, progress_bar)
      280         self.logger.record(&quot;time/total_timesteps&quot;, self.num_timesteps, exclude=&quot;tensorboard&quot;)
      281         self.logger.dump(step=self.num_timesteps)
  --&gt; 283     self.train()
      285 callback.on_training_end()
      287 return self
  
  File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\ppo\ppo.py:214, in PPO.train(self)
      211 if self.use_sde:
      212     self.policy.reset_noise(self.batch_size)
  --&gt; 214 values, log_prob, entropy = self.policy.evaluate_actions(rollout_data.observations, actions)
      215 values = values.flatten()
      216 # Normalize advantage
  
  File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\policies.py:644, in ActorCriticPolicy.evaluate_actions(self, obs, actions)
      642 features = self.extract_features(obs)
      643 latent_pi, latent_vf = self.mlp_extractor(features)
  --&gt; 644 distribution = self._get_action_dist_from_latent(latent_pi)
      645 log_prob = distribution.log_prob(actions)
      646 values = self.value_net(latent_vf)
  
  File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\policies.py:615, in ActorCriticPolicy._get_action_dist_from_latent(self, latent_pi)
      612     return self.action_dist.proba_distribution(action_logits=mean_actions)
      613 elif isinstance(self.action_dist, BernoulliDistribution):
      614     # Here mean_actions are the logits (before rounding to get the binary actions)
  --&gt; 615     return self.action_dist.proba_distribution(action_logits=mean_actions)
      616 elif isinstance(self.action_dist, StateDependentNoiseDistribution):
      617     return self.action_dist.proba_distribution(mean_actions, self.log_std, latent_pi)
  
  File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\distributions.py:380, in BernoulliDistribution.proba_distribution(self, action_logits)
      379 def proba_distribution(self, action_logits: th.Tensor) -&gt; &quot;BernoulliDistribution&quot;:
  --&gt; 380     self.distribution = Bernoulli(logits=action_logits)
      381     return self
  
  File ~\anaconda3\envs\tf-gpu\lib\site-packages\torch\distributions\bernoulli.py:51, in Bernoulli.__init__(self, probs, logits, validate_args)
       49 else:
       50     batch_shape = self._param.size()
  ---&gt; 51 super(Bernoulli, self).__init__(batch_shape, validate_args=validate_args)
  
  File ~\anaconda3\envs\tf-gpu\lib\site-packages\torch\distributions\distribution.py:56, in Distribution.__init__(self, batch_shape, event_shape, validate_args)
       54         valid = constraint.check(value)
       55         if not valid.all():
  ---&gt; 56             raise ValueError(
  
       57                 f&quot;Expected parameter {param} &quot;
       58                 f&quot;({type(value).__name__} of shape {tuple(value.shape)}) &quot;
       59                 f&quot;of distribution {repr(self)} &quot;
       60                 f&quot;to satisfy the constraint {repr(constraint)}, &quot;
       61                 f&quot;but found invalid values:\n{value}&quot;
       62             )
       63 super(Distribution, self).__init__()
  
  ValueError: Expected parameter logits (Tensor of shape (64, 24)) of distribution Bernoulli(logits: torch.Size([64, 24])) to satisfy the constraint Real(), but found invalid values:
  tensor([[nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          ...,
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=&lt;AddmmBackward0&gt;)
  </code></pre>
  <p>What I am doing wrong? Am I defining some variables in a wrong way?</p>
  
Tags: <reinforcement-learning><machine-learning><python><open-ai><gym>
Title: OpeanAI Gym. Train problem: invalid values
UserDisplayName: MiPre
ThreadText: # OpeanAI Gym. Train problem: invalid values
  
  (Asked by: username_0 on 2022-11-04)
  
  I have a problem with my reinforcement learning model.
  I am trying to simulate an electric battery storage. To keep it as simple as possible, the efficiency of charge, storage and discharge are 100%. No depletion either.
  The battery charges when the electricity prices are low and discharge ONLY to the user at fixed hours during the day, every day.
  Therefore, the only cost for the user is power of charge * electricity price at the hour.
  1. Define a dataframe where to store electricity prices for 500 days
  df=pd.DataFrame(np.random.randint(0,500,size=(24, 100)))
  
  2. Define the main parameters
  Lookback_window_size=7
  Current_day=Lookback_window_size
  P_charge=2 #MW
  P_discharge=3 #MW
  
  3. Define the class Battery(Env)
  class Battery(Env):
      
      metadata = {'render.modes': ['human']}
  
      def __init__(self, df):
             
          #Import the dataframe 
  
          self.df = df
  
          # The action space is an array of shape (24,). Since we are simulating day-ahead market, the action space returns
          # the overall daily charge / no charge scenario
          # action = 1 means that we charge our battery, action = 0 means that we don't charge
          
          self.action_space= spaces.MultiBinary(24)
          
          # The observation space is a matrix with 24 rows and N columns. The first N-1 columns represent the electricity 
          # prices for the current day + all the days before included in the lookback window size. The last column is all 
          # zeros except the first two rows: they store SOC (state of charge) at the end of the day and overall cost
          # (how much we paid for electricity).
          # Example: if lookback window is 7, then we have a matrix of 24 rows and 9 columns. The first 8 columns are just 
          # used to store the electricity prices, the 9th column store SOC and costs. 
          
          self.observation_space = spaces.Box(
              low=0, high=np.inf, shape=(24, Lookback_window_size + 2), dtype=np.float16)
  
      def _next_observation(self):
          
          # Add the prices of the last days to the monitor matrix
          
          prices=self.df[[i for i in range(self.Current_day - Lookback_window_size,self.Current_day + 1)]][0:].to_numpy()
                  
          extra_values=np.zeros(24)
          
          # Add extra values to monitor such as SOC, cost
          
          extra_values[[0,1]] = self.SOC, self.Cost  
          
          obs=np.c_[prices,extra_values]
          
          return obs        
      
      def _take_action(self, action):
          
          # Being the action space an array, the for loop will check the action at every hour (action[i]) and update the
          # cost and the state of charge
          
          i=0
                          
          for x in action:
              
              #When action = 1 then we charge our battery, if action = 0 then we don't charge
  
              if x == 1:
                  
                  # The cost increase based on the price of the electricity at that hour
  
                  self.Cost+=self.df[self.Current_day][i]*P_charge
  
                  # If we charge, then the state of charge (SOC) increases as well
  
                  self.SOC+=P_charge
                  
              # Everyday we discharge the battery always at the same hours
  
              if (i in range(8,14)):
  
                  self.SOC-=P_discharge
              
              # if the battery is depleted, then we directly buy electricity from the grid
  
                  if self.SOC<0:
  
                      self.Cost+=self.df[self.Current_day][i+1]*(-self.SOC)
  
                      self.SOC=0
                          
              i+=1
      
      def step(self, action):
                 
          self._take_action(action)
          
          self.Current_day += 1
  
          # Maximizing the reward means minimize the costs
          
          reward =  - self.Cost
          
          # Stop at the end of the dataframe 
          
          done = self.Current_day >= len(self.df.columns)-1
  
          obs = self._next_observation()
     
          return obs, reward, done, {}    
      
      
      def render(self, mode='human', close=False):
  
          print(f'Day: {self.Current_day}')
          
          print(f'SOC: {self.SOC}')
          
          print(f'Cost: {self.Cost}')      
          
          print(f'Actions: {action}')
                    
      def reset(self):
          
          self.Current_day = Lookback_window_size
          
          # Give an initial SOC value
          
          self.SOC = 50
          
          # Cost at day 0 is null
                 
          self.Cost = 0
          
          return self._next_observation()
  
  4. Train
  env = Battery(df)
  model = PPO("MlpPolicy", env, verbose=1)
  
  Once I run these lines, I get the following error:
  5. VALUE ERROR
  ValueError: Expected parameter logits (Tensor of shape (64, 24)) of distribution Bernoulli(logits: torch.Size([64, 24])) to satisfy the constraint Real(), but found invalid values:
  tensor([[nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          ...,
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=)
  
  In more details:
  | time/              |      |
  |    fps             | 784  |
  |    iterations      | 1    |
  |    time_elapsed    | 2    |
  |    total_timesteps | 2048 |
  -----------------------------
  ---------------------------------------------------------------------------
  ValueError                                Traceback (most recent call last)
  Input In [31], in ()
  ----> 1 model.learn(total_timesteps=100000)
  
  File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\ppo\ppo.py:317, in PPO.learn(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps, progress_bar)
      303 def learn(
      304     self: PPOSelf,
      305     total_timesteps: int,
     (...)
      314     progress_bar: bool = False,
      315 ) -> PPOSelf:
  --> 317     return super().learn(
      318         total_timesteps=total_timesteps,
      319         callback=callback,
      320         log_interval=log_interval,
      321         eval_env=eval_env,
      322         eval_freq=eval_freq,
      323         n_eval_episodes=n_eval_episodes,
      324         tb_log_name=tb_log_name,
      325         eval_log_path=eval_log_path,
      326         reset_num_timesteps=reset_num_timesteps,
      327         progress_bar=progress_bar,
      328     )
  
  File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\on_policy_algorithm.py:283, in OnPolicyAlgorithm.learn(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps, progress_bar)
      280         self.logger.record("time/total_timesteps", self.num_timesteps, exclude="tensorboard")
      281         self.logger.dump(step=self.num_timesteps)
  --> 283     self.train()
      285 callback.on_training_end()
      287 return self
  
  File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\ppo\ppo.py:214, in PPO.train(self)
      211 if self.use_sde:
      212     self.policy.reset_noise(self.batch_size)
  --> 214 values, log_prob, entropy = self.policy.evaluate_actions(rollout_data.observations, actions)
      215 values = values.flatten()
      216 # Normalize advantage
  
  File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\policies.py:644, in ActorCriticPolicy.evaluate_actions(self, obs, actions)
      642 features = self.extract_features(obs)
      643 latent_pi, latent_vf = self.mlp_extractor(features)
  --> 644 distribution = self._get_action_dist_from_latent(latent_pi)
      645 log_prob = distribution.log_prob(actions)
      646 values = self.value_net(latent_vf)
  
  File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\policies.py:615, in ActorCriticPolicy._get_action_dist_from_latent(self, latent_pi)
      612     return self.action_dist.proba_distribution(action_logits=mean_actions)
      613 elif isinstance(self.action_dist, BernoulliDistribution):
      614     # Here mean_actions are the logits (before rounding to get the binary actions)
  --> 615     return self.action_dist.proba_distribution(action_logits=mean_actions)
      616 elif isinstance(self.action_dist, StateDependentNoiseDistribution):
      617     return self.action_dist.proba_distribution(mean_actions, self.log_std, latent_pi)
  
  File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\distributions.py:380, in BernoulliDistribution.proba_distribution(self, action_logits)
      379 def proba_distribution(self, action_logits: th.Tensor) -> "BernoulliDistribution":
  --> 380     self.distribution = Bernoulli(logits=action_logits)
      381     return self
  
  File ~\anaconda3\envs\tf-gpu\lib\site-packages\torch\distributions\bernoulli.py:51, in Bernoulli.__init__(self, probs, logits, validate_args)
       49 else:
       50     batch_shape = self._param.size()
  ---> 51 super(Bernoulli, self).__init__(batch_shape, validate_args=validate_args)
  
  File ~\anaconda3\envs\tf-gpu\lib\site-packages\torch\distributions\distribution.py:56, in Distribution.__init__(self, batch_shape, event_shape, validate_args)
       54         valid = constraint.check(value)
       55         if not valid.all():
  ---> 56             raise ValueError(
  
       57                 f"Expected parameter {param} "
       58                 f"({type(value).__name__} of shape {tuple(value.shape)}) "
       59                 f"of distribution {repr(self)} "
       60                 f"to satisfy the constraint {repr(constraint)}, "
       61                 f"but found invalid values:\n{value}"
       62             )
       63 super(Distribution, self).__init__()
  
  ValueError: Expected parameter logits (Tensor of shape (64, 24)) of distribution Bernoulli(logits: torch.Size([64, 24])) to satisfy the constraint Real(), but found invalid values:
  tensor([[nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          ...,
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=)
  
  What I am doing wrong? Am I defining some variables in a wrong way?
  
  **Tags:** reinforcement-learning, machine-learning, python, open-ai, gym
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-11-12. Score: 1 | Accepted answer
  
  Your `self.Cost` variable aggregates rewards over time instead of giving the cost for the current time step as a reward. This causes the gradients to grow larger and everything diverges.
  
  > **Comments:**
  > **username_0**: Isn't a reward connected to the daily cost more greedy than one that consider the cumulative cost? For example: if a day has really low price of electricity, I would like to charge as much as possible so that I can "save" some energy to shield myself from higher prices in the future. Having a reward coupled only to the daily cost seems to go against this logic.  
  > **username_1**: The problem is that it is integrated in the algorithm itself to consider and optimize for the cumulative cost. If you look in the [RolloutBuffer](https://github.com/DLR-RM/stable-baselines3/blob/abffa161982b92e7f4b0186611cb73653bc2553b/stable_baselines3/common/buffers.py#L316) class which is used by the PPO implementation you can see that it computes the return which is the discounted sum of rewards. This is then used as an optimization target.  
  
  ---
  
  