@ptrblck, It was my mistake. I cut episode by t_max=1000. After some time, as the agent is getting more intelligent, they all don’t get done after 1000 steps. see the code comment on ‘if np.any(dones): // no one has dones after some time.’. So it will not enter learning step after time, and self.parallel_trajectory keeps accumulating.
def step(self, i_episode, states, actions, action_probs, rewards, next_states, dones):
        self.parallel_trajectory.add(
            parallel_states=states,
            parallel_actions=actions,
            parallel_action_probs=action_probs,
            parallel_rewards=rewards,
            parallel_next_states=next_states,
            parallel_dones=dones)

        if np.any(dones): **// no one has dones after some time.**
            states, actions, action_probs, rewards, next_states, dones = self.parallel_trajectory.numpy()
            returns = self.parallel_trajectory.discounted_returns(self.discount)
            states_tensor, actions_tensor, action_probs_tensor, returns_tensor, next_states_tensor = self.to_tensor(
                states=states,
                actions=actions,
                action_probs=action_probs,
                returns=returns,
                next_states=next_states)
            self.learn(
                states=states_tensor,
                actions=actions_tensor,
                action_probs=action_probs_tensor,
                returns=returns_tensor,
                next_states=next_states_tensor)
            del self.parallel_trajectory
self.parallel_trajectory = ParallelTrajectory(n=self.num_parallels)