<p>Your <code>self.Cost</code> variable aggregates rewards over time instead of giving the cost for the current time step as a reward. This causes the gradients to grow larger and everything diverges.</p>

CC BY-SA 4.0
2022-11-14T08:41:07.027Z
Isn't a reward connected to the daily cost more greedy than one that consider the cumulative cost? For example: if a day has really low price of electricity, I would like to charge as much as possible so that I can "save" some energy to shield myself from higher prices in the future. Having a reward coupled only to the daily cost seems to go against this logic.
MiPre
CC BY-SA 4.0
2022-11-17T10:14:34.803Z
The problem is that it is integrated in the algorithm itself to consider and optimize for the cumulative cost. If you look in the [RolloutBuffer](https://github.com/DLR-RM/stable-baselines3/blob/abffa161982b92e7f4b0186611cb73653bc2553b/stable_baselines3/common/buffers.py#L316) class which is used by the PPO implementation you can see that it computes the return which is the discounted sum of rewards. This is then used as an optimization target.
Dennis Malmgren
CC BY-SA 4.0
2022-11-12T20:54:28.833Z
2022-11-12T20:54:28.833Z
Dennis Malmgren
<p>I have a problem with my reinforcement learning model.
I am trying to simulate an electric battery storage. To keep it as simple as possible, the efficiency of charge, storage and discharge are 100%. No depletion either.
The battery charges when the electricity prices are low and discharge ONLY to the user at fixed hours during the day, every day.
Therefore, the only cost for the user is power of charge * electricity price at the hour.</p>
<p><strong>1. Define a dataframe where to store electricity prices for 500 days</strong></p>
<pre><code>df=pd.DataFrame(np.random.randint(0,500,size=(24, 100)))
</code></pre>
<p><strong>2. Define the main parameters</strong></p>
<pre><code>Lookback_window_size=7
Current_day=Lookback_window_size
P_charge=2 #MW
P_discharge=3 #MW
</code></pre>
<p><strong>3. Define the class Battery(Env)</strong></p>
<pre><code>class Battery(Env):
    
    metadata = {'render.modes': ['human']}

    def __init__(self, df):
           
        #Import the dataframe 

        self.df = df

        # The action space is an array of shape (24,). Since we are simulating day-ahead market, the action space returns
        # the overall daily charge / no charge scenario
        # action = 1 means that we charge our battery, action = 0 means that we don't charge
        
        self.action_space= spaces.MultiBinary(24)
        
        # The observation space is a matrix with 24 rows and N columns. The first N-1 columns represent the electricity 
        # prices for the current day + all the days before included in the lookback window size. The last column is all 
        # zeros except the first two rows: they store SOC (state of charge) at the end of the day and overall cost
        # (how much we paid for electricity).
        # Example: if lookback window is 7, then we have a matrix of 24 rows and 9 columns. The first 8 columns are just 
        # used to store the electricity prices, the 9th column store SOC and costs. 
        
        self.observation_space = spaces.Box(
            low=0, high=np.inf, shape=(24, Lookback_window_size + 2), dtype=np.float16)

    def _next_observation(self):
        
        # Add the prices of the last days to the monitor matrix
        
        prices=self.df[[i for i in range(self.Current_day - Lookback_window_size,self.Current_day + 1)]][0:].to_numpy()
                
        extra_values=np.zeros(24)
        
        # Add extra values to monitor such as SOC, cost
        
        extra_values[[0,1]] = self.SOC, self.Cost  
        
        obs=np.c_[prices,extra_values]
        
        return obs        
    
    def _take_action(self, action):
        
        # Being the action space an array, the for loop will check the action at every hour (action[i]) and update the
        # cost and the state of charge
        
        i=0
                        
        for x in action:
            
            #When action = 1 then we charge our battery, if action = 0 then we don't charge

            if x == 1:
                
                # The cost increase based on the price of the electricity at that hour

                self.Cost+=self.df[self.Current_day][i]*P_charge

                # If we charge, then the state of charge (SOC) increases as well

                self.SOC+=P_charge
                
            # Everyday we discharge the battery always at the same hours

            if (i in range(8,14)):

                self.SOC-=P_discharge
            
            # if the battery is depleted, then we directly buy electricity from the grid

                if self.SOC&lt;0:

                    self.Cost+=self.df[self.Current_day][i+1]*(-self.SOC)

                    self.SOC=0
                        
            i+=1
    
    def step(self, action):
               
        self._take_action(action)
        
        self.Current_day += 1

        # Maximizing the reward means minimize the costs
        
        reward =  - self.Cost
        
        # Stop at the end of the dataframe 
        
        done = self.Current_day &gt;= len(self.df.columns)-1

        obs = self._next_observation()
   
        return obs, reward, done, {}    
    
    
    def render(self, mode='human', close=False):

        print(f'Day: {self.Current_day}')
        
        print(f'SOC: {self.SOC}')
        
        print(f'Cost: {self.Cost}')      
        
        print(f'Actions: {action}')
                  
    def reset(self):
        
        self.Current_day = Lookback_window_size
        
        # Give an initial SOC value
        
        self.SOC = 50
        
        # Cost at day 0 is null
               
        self.Cost = 0
        
        return self._next_observation()
</code></pre>
<p><strong>4. Train</strong></p>
<pre><code>env = Battery(df)
model = PPO(&quot;MlpPolicy&quot;, env, verbose=1)
</code></pre>
<p>Once I run these lines, I get the following error:</p>
<p><strong>5. VALUE ERROR</strong></p>
<pre><code>ValueError: Expected parameter logits (Tensor of shape (64, 24)) of distribution Bernoulli(logits: torch.Size([64, 24])) to satisfy the constraint Real(), but found invalid values:
tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=&lt;AddmmBackward0&gt;)
</code></pre>
<p>In more details:</p>
<pre><code>| time/              |      |
|    fps             | 784  |
|    iterations      | 1    |
|    time_elapsed    | 2    |
|    total_timesteps | 2048 |
-----------------------------
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [31], in &lt;cell line: 1&gt;()
----&gt; 1 model.learn(total_timesteps=100000)

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\ppo\ppo.py:317, in PPO.learn(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps, progress_bar)
    303 def learn(
    304     self: PPOSelf,
    305     total_timesteps: int,
   (...)
    314     progress_bar: bool = False,
    315 ) -&gt; PPOSelf:
--&gt; 317     return super().learn(
    318         total_timesteps=total_timesteps,
    319         callback=callback,
    320         log_interval=log_interval,
    321         eval_env=eval_env,
    322         eval_freq=eval_freq,
    323         n_eval_episodes=n_eval_episodes,
    324         tb_log_name=tb_log_name,
    325         eval_log_path=eval_log_path,
    326         reset_num_timesteps=reset_num_timesteps,
    327         progress_bar=progress_bar,
    328     )

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\on_policy_algorithm.py:283, in OnPolicyAlgorithm.learn(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps, progress_bar)
    280         self.logger.record(&quot;time/total_timesteps&quot;, self.num_timesteps, exclude=&quot;tensorboard&quot;)
    281         self.logger.dump(step=self.num_timesteps)
--&gt; 283     self.train()
    285 callback.on_training_end()
    287 return self

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\ppo\ppo.py:214, in PPO.train(self)
    211 if self.use_sde:
    212     self.policy.reset_noise(self.batch_size)
--&gt; 214 values, log_prob, entropy = self.policy.evaluate_actions(rollout_data.observations, actions)
    215 values = values.flatten()
    216 # Normalize advantage

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\policies.py:644, in ActorCriticPolicy.evaluate_actions(self, obs, actions)
    642 features = self.extract_features(obs)
    643 latent_pi, latent_vf = self.mlp_extractor(features)
--&gt; 644 distribution = self._get_action_dist_from_latent(latent_pi)
    645 log_prob = distribution.log_prob(actions)
    646 values = self.value_net(latent_vf)

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\policies.py:615, in ActorCriticPolicy._get_action_dist_from_latent(self, latent_pi)
    612     return self.action_dist.proba_distribution(action_logits=mean_actions)
    613 elif isinstance(self.action_dist, BernoulliDistribution):
    614     # Here mean_actions are the logits (before rounding to get the binary actions)
--&gt; 615     return self.action_dist.proba_distribution(action_logits=mean_actions)
    616 elif isinstance(self.action_dist, StateDependentNoiseDistribution):
    617     return self.action_dist.proba_distribution(mean_actions, self.log_std, latent_pi)

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\distributions.py:380, in BernoulliDistribution.proba_distribution(self, action_logits)
    379 def proba_distribution(self, action_logits: th.Tensor) -&gt; &quot;BernoulliDistribution&quot;:
--&gt; 380     self.distribution = Bernoulli(logits=action_logits)
    381     return self

File ~\anaconda3\envs\tf-gpu\lib\site-packages\torch\distributions\bernoulli.py:51, in Bernoulli.__init__(self, probs, logits, validate_args)
     49 else:
     50     batch_shape = self._param.size()
---&gt; 51 super(Bernoulli, self).__init__(batch_shape, validate_args=validate_args)

File ~\anaconda3\envs\tf-gpu\lib\site-packages\torch\distributions\distribution.py:56, in Distribution.__init__(self, batch_shape, event_shape, validate_args)
     54         valid = constraint.check(value)
     55         if not valid.all():
---&gt; 56             raise ValueError(

     57                 f&quot;Expected parameter {param} &quot;
     58                 f&quot;({type(value).__name__} of shape {tuple(value.shape)}) &quot;
     59                 f&quot;of distribution {repr(self)} &quot;
     60                 f&quot;to satisfy the constraint {repr(constraint)}, &quot;
     61                 f&quot;but found invalid values:\n{value}&quot;
     62             )
     63 super(Distribution, self).__init__()

ValueError: Expected parameter logits (Tensor of shape (64, 24)) of distribution Bernoulli(logits: torch.Size([64, 24])) to satisfy the constraint Real(), but found invalid values:
tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=&lt;AddmmBackward0&gt;)
</code></pre>
<p>What I am doing wrong? Am I defining some variables in a wrong way?</p>

<reinforcement-learning><machine-learning><python><open-ai><gym>
OpeanAI Gym. Train problem: invalid values
MiPre
# OpeanAI Gym. Train problem: invalid values

(Asked by: username_0 on 2022-11-04)

I have a problem with my reinforcement learning model.
I am trying to simulate an electric battery storage. To keep it as simple as possible, the efficiency of charge, storage and discharge are 100%. No depletion either.
The battery charges when the electricity prices are low and discharge ONLY to the user at fixed hours during the day, every day.
Therefore, the only cost for the user is power of charge * electricity price at the hour.
1. Define a dataframe where to store electricity prices for 500 days
df=pd.DataFrame(np.random.randint(0,500,size=(24, 100)))

2. Define the main parameters
Lookback_window_size=7
Current_day=Lookback_window_size
P_charge=2 #MW
P_discharge=3 #MW

3. Define the class Battery(Env)
class Battery(Env):
    
    metadata = {'render.modes': ['human']}

    def __init__(self, df):
           
        #Import the dataframe 

        self.df = df

        # The action space is an array of shape (24,). Since we are simulating day-ahead market, the action space returns
        # the overall daily charge / no charge scenario
        # action = 1 means that we charge our battery, action = 0 means that we don't charge
        
        self.action_space= spaces.MultiBinary(24)
        
        # The observation space is a matrix with 24 rows and N columns. The first N-1 columns represent the electricity 
        # prices for the current day + all the days before included in the lookback window size. The last column is all 
        # zeros except the first two rows: they store SOC (state of charge) at the end of the day and overall cost
        # (how much we paid for electricity).
        # Example: if lookback window is 7, then we have a matrix of 24 rows and 9 columns. The first 8 columns are just 
        # used to store the electricity prices, the 9th column store SOC and costs. 
        
        self.observation_space = spaces.Box(
            low=0, high=np.inf, shape=(24, Lookback_window_size + 2), dtype=np.float16)

    def _next_observation(self):
        
        # Add the prices of the last days to the monitor matrix
        
        prices=self.df[[i for i in range(self.Current_day - Lookback_window_size,self.Current_day + 1)]][0:].to_numpy()
                
        extra_values=np.zeros(24)
        
        # Add extra values to monitor such as SOC, cost
        
        extra_values[[0,1]] = self.SOC, self.Cost  
        
        obs=np.c_[prices,extra_values]
        
        return obs        
    
    def _take_action(self, action):
        
        # Being the action space an array, the for loop will check the action at every hour (action[i]) and update the
        # cost and the state of charge
        
        i=0
                        
        for x in action:
            
            #When action = 1 then we charge our battery, if action = 0 then we don't charge

            if x == 1:
                
                # The cost increase based on the price of the electricity at that hour

                self.Cost+=self.df[self.Current_day][i]*P_charge

                # If we charge, then the state of charge (SOC) increases as well

                self.SOC+=P_charge
                
            # Everyday we discharge the battery always at the same hours

            if (i in range(8,14)):

                self.SOC-=P_discharge
            
            # if the battery is depleted, then we directly buy electricity from the grid

                if self.SOC<0:

                    self.Cost+=self.df[self.Current_day][i+1]*(-self.SOC)

                    self.SOC=0
                        
            i+=1
    
    def step(self, action):
               
        self._take_action(action)
        
        self.Current_day += 1

        # Maximizing the reward means minimize the costs
        
        reward =  - self.Cost
        
        # Stop at the end of the dataframe 
        
        done = self.Current_day >= len(self.df.columns)-1

        obs = self._next_observation()
   
        return obs, reward, done, {}    
    
    
    def render(self, mode='human', close=False):

        print(f'Day: {self.Current_day}')
        
        print(f'SOC: {self.SOC}')
        
        print(f'Cost: {self.Cost}')      
        
        print(f'Actions: {action}')
                  
    def reset(self):
        
        self.Current_day = Lookback_window_size
        
        # Give an initial SOC value
        
        self.SOC = 50
        
        # Cost at day 0 is null
               
        self.Cost = 0
        
        return self._next_observation()

4. Train
env = Battery(df)
model = PPO("MlpPolicy", env, verbose=1)

Once I run these lines, I get the following error:
5. VALUE ERROR
ValueError: Expected parameter logits (Tensor of shape (64, 24)) of distribution Bernoulli(logits: torch.Size([64, 24])) to satisfy the constraint Real(), but found invalid values:
tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=)

In more details:
| time/              |      |
|    fps             | 784  |
|    iterations      | 1    |
|    time_elapsed    | 2    |
|    total_timesteps | 2048 |
-----------------------------
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [31], in ()
----> 1 model.learn(total_timesteps=100000)

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\ppo\ppo.py:317, in PPO.learn(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps, progress_bar)
    303 def learn(
    304     self: PPOSelf,
    305     total_timesteps: int,
   (...)
    314     progress_bar: bool = False,
    315 ) -> PPOSelf:
--> 317     return super().learn(
    318         total_timesteps=total_timesteps,
    319         callback=callback,
    320         log_interval=log_interval,
    321         eval_env=eval_env,
    322         eval_freq=eval_freq,
    323         n_eval_episodes=n_eval_episodes,
    324         tb_log_name=tb_log_name,
    325         eval_log_path=eval_log_path,
    326         reset_num_timesteps=reset_num_timesteps,
    327         progress_bar=progress_bar,
    328     )

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\on_policy_algorithm.py:283, in OnPolicyAlgorithm.learn(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps, progress_bar)
    280         self.logger.record("time/total_timesteps", self.num_timesteps, exclude="tensorboard")
    281         self.logger.dump(step=self.num_timesteps)
--> 283     self.train()
    285 callback.on_training_end()
    287 return self

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\ppo\ppo.py:214, in PPO.train(self)
    211 if self.use_sde:
    212     self.policy.reset_noise(self.batch_size)
--> 214 values, log_prob, entropy = self.policy.evaluate_actions(rollout_data.observations, actions)
    215 values = values.flatten()
    216 # Normalize advantage

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\policies.py:644, in ActorCriticPolicy.evaluate_actions(self, obs, actions)
    642 features = self.extract_features(obs)
    643 latent_pi, latent_vf = self.mlp_extractor(features)
--> 644 distribution = self._get_action_dist_from_latent(latent_pi)
    645 log_prob = distribution.log_prob(actions)
    646 values = self.value_net(latent_vf)

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\policies.py:615, in ActorCriticPolicy._get_action_dist_from_latent(self, latent_pi)
    612     return self.action_dist.proba_distribution(action_logits=mean_actions)
    613 elif isinstance(self.action_dist, BernoulliDistribution):
    614     # Here mean_actions are the logits (before rounding to get the binary actions)
--> 615     return self.action_dist.proba_distribution(action_logits=mean_actions)
    616 elif isinstance(self.action_dist, StateDependentNoiseDistribution):
    617     return self.action_dist.proba_distribution(mean_actions, self.log_std, latent_pi)

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\distributions.py:380, in BernoulliDistribution.proba_distribution(self, action_logits)
    379 def proba_distribution(self, action_logits: th.Tensor) -> "BernoulliDistribution":
--> 380     self.distribution = Bernoulli(logits=action_logits)
    381     return self

File ~\anaconda3\envs\tf-gpu\lib\site-packages\torch\distributions\bernoulli.py:51, in Bernoulli.__init__(self, probs, logits, validate_args)
     49 else:
     50     batch_shape = self._param.size()
---> 51 super(Bernoulli, self).__init__(batch_shape, validate_args=validate_args)

File ~\anaconda3\envs\tf-gpu\lib\site-packages\torch\distributions\distribution.py:56, in Distribution.__init__(self, batch_shape, event_shape, validate_args)
     54         valid = constraint.check(value)
     55         if not valid.all():
---> 56             raise ValueError(

     57                 f"Expected parameter {param} "
     58                 f"({type(value).__name__} of shape {tuple(value.shape)}) "
     59                 f"of distribution {repr(self)} "
     60                 f"to satisfy the constraint {repr(constraint)}, "
     61                 f"but found invalid values:\n{value}"
     62             )
     63 super(Distribution, self).__init__()

ValueError: Expected parameter logits (Tensor of shape (64, 24)) of distribution Bernoulli(logits: torch.Size([64, 24])) to satisfy the constraint Real(), but found invalid values:
tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=)

What I am doing wrong? Am I defining some variables in a wrong way?

**Tags:** reinforcement-learning, machine-learning, python, open-ai, gym


---

## 1 answers

### Answer by username_1 on 2022-11-12. Score: 1 | Accepted answer

Your `self.Cost` variable aggregates rewards over time instead of giving the cost for the current time step as a reward. This causes the gradients to grow larger and everything diverges.

> **Comments:**
> **username_0**: Isn't a reward connected to the daily cost more greedy than one that consider the cumulative cost? For example: if a day has really low price of electricity, I would like to charge as much as possible so that I can "save" some energy to shield myself from higher prices in the future. Having a reward coupled only to the daily cost seems to go against this logic.  
> **username_1**: The problem is that it is integrated in the algorithm itself to consider and optimize for the cumulative cost. If you look in the [RolloutBuffer](https://github.com/DLR-RM/stable-baselines3/blob/abffa161982b92e7f4b0186611cb73653bc2553b/stable_baselines3/common/buffers.py#L316) class which is used by the PPO implementation you can see that it computes the return which is the discounted sum of rewards. This is then used as an optimization target.  

---

