Reward Value‑Based Goal Selection for Agents’ Cooperative Route Learning Without Communication in Reward and Goal Dynamism

This paper proposes a goal selection method to operate agents get maximum reward values per time by noncommunicative learning. In particular, that method aims to enable agents to cooperate along to dynamism of reward values and goal locations. Adaptation against to these dynamisms can enable agents to learn cooperative actions along to changing transportation tasks and changing incomes/rewards because of transporting tasks for heavy/valuable and light/valueless items in a storehouse. Concretely, this paper extends the previous noncommunicative cooperative action learning method (Profit minimizing reinforcement learning with oblivion of memory: PMRL-OM) and sets the two unified conditions combined of the number of time steps and the rewards. One of the unified conditions is calculated the approximated number of time steps if the expected reward values are the same each other for all purposes, and the other is the minimum number of time steps divided by the reward value. The proposed method makes all agents learn to achieve the purposes in the order in which they have the minimum number of the condition values. After that, each agent learns cooperative policy by PMRL-OM as the previous method. This paper analyzes the unified conditions and derives that the condition calculating the approximated time steps can be combined both evaluations with almost same weight unlike the value the other condition, that is, the condition can help the agents to select the appropriate purposes among them with the small difference in terms of the two evaluations. This paper tests empirically the performances of PMRL-OM with the two conditions by comparing with the PMRL-OM in three cases of grid world problems whose goal locations and reward values are changed dynamically. The results of this derive that the unified conditions perform better than PMRL-OM without some conditions in grid world problems. In particular, it is clear that the condition calculating the approximated time step can direct the appropriate goals for the agents.

Introduction
Multi-agent system (MAS) simulates real-world problems by several number of agents which have the own roles in the simulated environment, e.g., robots in a storehouse, traffic lights in an intersection and more [2,4]. To control the agents cooperatively, each agent learns the own action selection policy by reinforcement learning (RL). Generally, for cooperation, the most important factor is communication among agents. As for the related works, modeling other agent based on the communicated information becomes important [1]. These techniques make an agent model other agent, predict its action and purpose, and act by the cooperative policy based on the action and the purpose. For example, in storehouse robot problem, the simulated robots (agents) communicate to receive the information of the others' operation and plan the own routes to carry luggage to some points without collisions each other. Although these techniques achieve the agents' complex cooperation, much information is required. This suggests that the cooperation cannot be achieved in some situations, that is, non-or less-communication like field robotics or many agents. In addition, noncommunicative cooperation by reinforcement learning is challenging and important topic. For example, Raileanu et al. proposed cooperative action learning method by modeling the opponent agent (self-other modeling: SOM). Zemzem and Tagina's method (Cooperative multiagent learning approach based on the most recently modified transitions: CMRL-MRMT) makes agents learn cooperative action by copying the learning result of the advantageous agent [5,12]. These methods utilize the information like the agents' states, actions, learning results, and more to achieve the cooperation. However, both methods cannot cooperate without communication.
On the other hand, some researchers aim to propose cooperative learning methods without communication. However, these methods are not complete, that is, they have to be under some conditions. For example, though Arai and Katia mentioned the noncommunicative method, the agent can observe other agent's state [6], and Shiraishi et al. extended that method, but the condition is remain required [7]. Uwano et al. add some direction to learning agents to cooperate each other without communication (Profit minimizing reinforcement learning: PMRL) [10], and they extended PMRL to perform in the dynamic environment (Profit minimizing reinforcement learning with oblivion of memory: PMRL-OM). However, PMRL-OM has two conditions for cooperation: the number of the goals equals to that of the agents and reward values in all goals are the same each other. These are very weak points because the agents' goals might be changed (e.g., changing transportation tasks in a storehouse) and importance ( i.e., reward) of reaching a goal cannot be the same among all purposes (e.g., incomes/ rewards of transporting tasks for heavy/valuable and light/ valueless items are not the same each other.) Therefore, it is important to propose the cooperative-learning method without communication for the dynamism of the number of the goals and the reward values.
To tackle this issue, this paper proposes the two conditions unifying the two local information to select the appropriate purposes and proposes the noncommunicative MARL method based on the unified conditions to adapt the dynamic change of the rewards and the purposes. Concretely, one of the unified conditions is calculated the approximated number of time steps if the expected reward values are the same each other for all purposes, and the other is the minimum number of time steps divided by the reward value. The proposed method makes all agents learn to achieve the purposes in the order in which they have the minimum number of the condition values. After that, each agent learns cooperative policy by PMRL-OM as the previous method. Concluding the above, our contributions are shown as follows: • Appropriate operation for agents with two purposes (e.g., storehouse robots advance both quantity and time to carry luggage) is established • We designed new evaluation for agent's behavior to carry more valuable luggage with smaller time.
• The parameter can control to weight the evaluation whether the value or the time.
The remainder of this paper is organized in the following manner. "Problem and Approach" section explains a maze problem and the goal of this problem in this work. "Baseline Methods" section explains Q-learning and PMRL-OM, and the unified condition is explained and analyzed mathematically in "Proposed Mechanism" section. "Experiment" section describes the experiment and analyzes the obtained results, and finally, our conclusions are summarized in "Conclusion" section.

Problem
We employ route planning tasks in mazes whose number of goals and reward values change dynamically. Equation (1) shows the joint action A from actions of all agents a 1 − a n (n is the number of the agents.) The environment is translated by Eq. (2) from an arbitrary state S and the joint action A. This equation indicates that the joint action can always translate the current state to the next state (When the agents reach the goal, they cannot observe other states by any actions, that is, the goal states are absorbing states.) At the end, Eq. (3) shows a reward function which indicates a value of reward and the condition to get that, where r is the reward value, s i and s −i are the current states of the agents i and the other agents, and S goal is a set of all goal states. This equation shows the agent acquires the reward when its observing state s i is the goal state and the other agents have not observed that state. Figure 1 shows an example of grid world problems. The squares named "Agent A", "Agent B", "Goal X", and "Goal Y" indicate the start states for the agents A and B and the goal states, that is, the states of the agents A and B s A , s B are the start states at the beginning of learning, and the goals X and Y are the goal states in the goal state set S goal . The agents observe the own states s A and s B , select the actions a A and a B , and learn the own action by the acquired reward by the reward function (after that, the agents observe the next state translated by the state transition function, and repeat this cycle.) In addition, this paper focuses on the dynamics for the environmental change as follows: • Reward value change • Goal number change These dynamics influence the reward function and the state transition function for the goal state [Eq. (3) and S goal ]. Concretely, the value r is changed to other value, and the new states are inserted into the goal state set S goal . Note that the environmental change happened once in a trial, and the proposed method aims to learn cooperative policy along to the environmental change.

Motivation and Approach
Our approach sets the discounted rewards according to Manhattan distance from the goals. From this approach, our motivation is that the reward, evaluation for learning, does not become the valuable of the result of the agent's behavior, but we mount information of distance into the reward. For example, Fig. 2 shows our approach and motivation for storehouse robots. In this figure, two robots carry some luggage to two shelves and the situation changes from left to right (i.e., a new shelf appears, the shelves move to other places and their capacities change.) The previous methods focus on the capacity as the reward value, but do not on the distance: Learning result can lead agents to reach a nearest goal. If the number of shelves is the same as that of agents, there is no problem. But actually the numbers are not the same each other like Fig. 2. In this situation, two robots have to select two shelves among three, the previous methods select in order from only the capacity (reward). However, if only five pieces of luggage exist, do you think the robots need to store some of those to the farthest shelf in right side of Fig. 2? Our motivation is to solve this problem, and the proposed method estimates the new reward combined the reward and the distance (written "unified condition" in this paper). For example in Fig. 2, after the environmental change, if the shelf lost the luggage then the reward value increases along to the lost number, if the shelf moves far then the reward value decreases along to the distance, and if the shelf moves near then the reward value increases.

Q-Learning
Reinforcement learning (RL) [8] is a try-and-error method which aims at maximizing an acquired reward per a unit time. As its general framework, an RL agent interacts with an environment: it observes a state from the environment, selects an action, receives a reward from the environment as the result of that action and then learns from the reward. Note that this cycle from the observation to the next observation is called "step" in this paper. Among many RL methods, Q-learning [11] is a very popular RL method for a singleagent task. A Q-learning agent estimates state-action values (called Q-value) for the possible state-action pairs in the environment, i.e., the agent estimates an discounted expected reward that it will receive when its action a is executed in its state s. The agent learns to acquire a policy (s, a) to decide which action should be executed to maximize a received reward. Technically, the policy can be composed of probabilities in selecting any action a in any state s and is calculated by Q(s, a), a ∈ A where Q(s, a) is the Q-value when state and action become s and a, and A is a set of actions that includes the possible actions. To maximize a received reward, Q(s, a) is updated as follows. where s ′ is the next state when a is executed in s, a ′ is the next action executed in the state s ′ , r(s) is the reward received from the environment, max Q(s � , a � ) is the largest Q-value when executing the action a � ∈ A in the state s ′ . In addition to these variables, is the learning rate, while is the discount factor. Precisely, is the real number from 0 to 1 which indicates the learning speed, while is the real number from 0 to 1 which indicates how much the future rewards should be considered as important. Note that the current state s and the current action a is for the general agent, that is, the state and the action for the agent A are s(A) and a(A) as shown in the previous section.

Profit Minimizing Reinforcement Learning with Oblivion of Memory (PMRL-OM)
To learn multi-agent cooperation without communication in the dynamic change environment, Uwano et al. proposed Profit minimizing reinforcement learning with oblivion of memory (PMRL-OM) [9]. PMRL-OM is based on Q-learning and forces the cooperation by controlling the reward for dynamic change environments. By controlling the rewards, PMRL-OM controls all agents to reach the goals with the largest time steps among the goals they can receive the rewards. In the grid world problem like Fig. 1, all agents reach the farthest goals which they can receive the rewards. This is basic idea, that is, the advanced agent reaches the difficult goal to yield the easy goal to the other agents and the multi-agent system behaves cooperatively. The methods employed the internal reward and the goal value to realize the basic idea, and the next section explains those mechanisms.

Internal Reward
The internal reward is the reward inside of the agent, and the agent learns based on the internal reward. PMRL-OM estimates the internal reward for the agent to reach the appropriate goal with the largest goal value. In this section, we assume the appropriate goal is the goal g and explain how to set the internal reward. The internal reward is calculated using Eq. (5) once every iteration. ir g is the internal reward of the goal g, is the discount rate of Q-learning, g ′ is the certain goal among the goal set G which includes all goals, and r g ′ is the external reward in the goal g ′ . t g and t g ′ are the minimum number of the steps until the agent has reached the goals g and g ′ . Note that the internal reward is calculated for only the appropriate goal g and those of the other goals are set by the external rewards. Equation (5) calculates discounted expected rewards as Q-values of actions to go to all goals in the agent's start and adds a constant value to the largest discount expected reward in the internal reward ir g to reach the goal g.

Goal Value
The goal value is the value for the agents to reach the farthest goals, and this value is converged to the minimum number of steps. The agent learns to reach the goal with the maximum goal value using the internal reward. Equation (6) is the goal value update functions. The goal value is updated once every iteration. t g is the minimum number of the steps, and is a constant value. Note that is a positive integer value over than 0, and indicates how much the current minimum number of steps are emphasized in the goal value. if is small, the only present minimum numbers of steps are emphasized, and the more is, the more past minimum numbers of steps are emphasized in the goal value, either. If the agent has received the reward, the goal value is updated by the top function; otherwise, it is updated by the bottom function. From these equations, all agents aim to reach the farthest goals among the goals can be reached by them.

Algorithm
Algorithm 1 is the algorithm of PMRL-OM. In this algorithm, t is the array, including the tuple with the goal number g and the minimum numbers of steps until the agent reaches the goal in each iteration. At first, the goal number for the agent to reach the appropriate goal is initialized by a random value (line 1). The agent observes its own state, selects the action, receives a reward, estimates the internal reward from the reward and updates the Q-value from the internal reward (lines 3-13). As for the internal reward estimation, if the agent is in the goal g ′ , the internal reward is set to reach that; otherwise, the internal reward is set by the reward. The agent repeats this cycle for several times and stops this cycle if it has reached the goal or several steps are spent (lines 3 and 16). Thereafter, if the agent on the arbitrary goal g and the number of the current step is smaller than before, it stores this step into t (lines 17 and 18); otherwise, it stores the maximum number of steps into t (lines 19-21). After this process, if the length of t is over the threshold Cycle, the agent erases the first element in t (lines 22-24). At the end, the agent updates the goal value bid g (lines 25-29) and selects the new goal g ′ with the largest goal value (line 30).
Since this process indicates the flow of PMRL-OM in one iteration, the agent goes to the next iteration and repeats the process. Note that t[g] indicates the set of all steps for the goal g in the array t.

Standardized Step and Simple Condition
Under the environment with the reward change, the agents cannot determine the appropriate goal by the goal value because the goal value is calculated for the spent step to be the smallest. This improvement unifies the two conditions, the minimum number of steps and the reward, using Eq. (7). This value should be smaller for the better evaluation. In this equation, t g is the minimum number of the steps for the goal g, and t ′ g is the standardized steps as the unified condition. r g and r stan are the reward of the goal g and the reward as the standard value, respectively. is the discount rate of Q-learning. Equation (7) approximates the minimum number of steps when the discounted expected reward is r stan . Note that t ′ g can be the positive and negative real value. This equation can be condition to help the agent to select the appropriate goals for cooperation. Concretely, the agents learn cooperative policy for the several goals from the smallest standardized step to the n-th smallest that among all goals, and the agents can obtain maximum gain per time.
In addition, this paper proposes another unified condition more simply shown in Eq. 8. This value should be smaller for the better evaluation, too. The condition value is calculated that the minimum number of steps is divided by the reward value, that is, the minimum number of steps should be smaller and the reward value should be larger.

Mathematical Analysis
To show the difference of the two conditions, the standardized step and the simple condition, we compare the standardized step with the simple condition. Figure 3 shows the behaviors for both conditions when the reward and the minimum number of steps are from 0 to 50. The x-and the y-axes indicate the reward and the number of steps, and the z-axis indicates the condition value. Both conditions show that when the smaller reward and the larger step of the situation are worse and vice versa. However, the gradients are different to each other, the surface of the standardized step is gentle, while that of the simple condition is steep. Equations (9) and (10) show the total derivative values of both conditions. In Eq. (9), since r stan is disappear in the derivative, r stan can be set some values greater than 0. In addition, the smaller reward can emphasize the worse condition, while the larger reward does not emphasize the better condition. As for the minimum number of steps, the gradient is linear and the smaller number is better. This suggests that when the rewards are very large the standardized step evaluates only the minimum number of steps. On the other hand, in Eq. (10), the smaller reward very emphasizes the worse condition and the gradient of the minimum number of steps becomes smaller along to the smaller reward in this condition. In addition, the smaller step makes the gradient of the reward be gentle. This suggests that when the rewards are small, the simple condition evaluates the smaller minimum number of steps is better. From the above analysis, the standardized step is good condition to evaluate without some bias for the minimum number of steps and the reward as the unified condition.
In addition, the gradient can be changed by the parameter for the standardized step, unlike the simple condition. Figure 4 shows the gradient 1 r g ln whose r g is 10. The gradient decreases the influence of the reward: since the reward value is easily larger than the number of the steps, this calculation decreases the difference of the range of the reward and the step. Furthermore, Fig. 5 shows the gradient 1 r g ln where the reward r g is 0 to 50 and the parameter is 0.01 to 1. If the parameter is less than 0.5, the standardized step is influenced by only the step. On the other hand, if that is greater than or equal to 0.5, the reward and the step influences the standardized step with the minus and plus direction, respectively: it is clear that the larger reward and the smaller step are better. The reward value and the number of the step are changed by the environment, e.g., the large step in the large field. However, from Fig. 5, the gradient is 0 for almost all reward values, that is, if the agent has to select one among two goals with large rewards, considers only the numbers of the steps. The standardized step makes agents' learning priorities mainly in order from the largest reward, and for the (10) dC = − t g r 2 g dr g + 1 r g dt g number of the step if the reward value is small. In this paper, is 0.9. Since the gradient 1 r g ln is approximately − 1 by this , the gradients of the standardized step, reward and step, are balanced like dt � g = −dr g + dt g . Therefore, = 0.9 is suitable for integrating the reward and the step.

Algorithm
Algorithm 2 shows the algorithm of PMRL-OM with the standardized step. Note that, for the simple condition, the equation should be changed to that of the simple condition in the process of calculating the unified condition value of the algorithm. In this algorithm, t is the array, including the tuple with the goal number g and the minimum numbers of steps until the agent reaches the goal in each iteration. At first, the goal number for the agent to reach the appropriate goal is initialized by a random value and the array of the standardized steps st is initialized by MaxStep (lines 1 and 2). The agent observes its own state, selects the action, receives a reward, estimates the internal reward from the reward and updates the Q-value from the internal reward (lines 5-15). As for the internal reward estimation, if the agent is in the goal g ′ , the internal reward is set to reach that; otherwise, the internal reward is set by the reward. The agent repeats this cycle for several times and stops this cycle if it has reached the goal or several steps are spent (lines 4 and 17). Thereafter, if the agent on the arbitrary goal g and the number of the current step are smaller than before, it stores this step into t (lines 18 and 19); otherwise, it stores the maximum number of steps into t (lines 20-22). After this process, if the length of t is over the threshold Cycle, the agent erases the first element in t (lines 23-25), and the agent updates the goal value bid g (lines 26-30). After that, the agent calculates the standardized steps (lines 31-33). At the end, the agent selects the new goal g ′ with the largest goal value among the goals from the smallest st to the AgentNumber-th smallest st (line 34). Since this process indicates the flow of PMRL-OM in one iteration, the agent goes to the next iteration and repeats the process. Note that t[g] indicates the set of all steps for the goal g in the array t .

Experimental Design and Setting
To investigate the effectiveness of the unified conditions in PMRL-OM, we verified that the PMRL-OM is applicable to dynamic changes in the grid world problems with the dynamic change of the rewards and the goals. We employ the grid worlds with the three cases as follows. The number of the goals and the goal positions are changed after a certain iteration in Case (1). The rewards in the goals are changed after a certain iteration in Case (2). Both changes exist in Case (3). We test the performance of the improved PMRL-OM by comparing the improved PMRL-OM with the standardized step and the simple condition with the normal PMRL-OM in each case. Concretely, Fig. 6 shows the grid world problems for both cases. The top and the bottom indicate the grid worlds for each case, and the left and the right sides indicate the grid worlds before and after the environmental change. The iteration 25000 is the borderline between two grid worlds in this experiment. In the grid world, the squares named A and B are the starts for the two agents (called the agents A and B), and the squares with some numbers are the goals and the numbers are the reward values. Since the reward values are not changed in Case (1) The experiments were determined by the number of trials with 30 random seeds. Table 1 summarizes the parameters. The learning iterations and the steps are limited to 50,000 and 100 (lines 1 and 2). The initialized Q-value is 0, and the learning rate and the discounted rate of Q-learning is 0.1 and 0.9, respectively (lines 3-5). The constant value in is 10 (line 6). For only parameters of the PMRL-OM, the memory range Cycle is 5000, the constant is 500, and the discount rate is 0.9 (lines 7-9). At the end, the standard reward r stan for the standardized step is 10 (line 10).   of the results with 30 seeds. The higher evaluation value is better in these results. As for Fig. 8, PMRL-OM performs worse than that with the two conditions and PMRL-OM with the conditions performs well. Since the error bars for the PMRL-OM with the conditions do not exist at the last half of the iterations, the agents can learn to reach the appropriate goals stably. On the other hand, in Fig. 9, PMRL-OM with the standardized step performs the best than only PMRL-OM and that with the simple condition. However, the results of all methods cannot converge to some values. Note that when we execute Brunner-Munzel test between the 30 results of PMRL-OM and that with the standardized steps, the p value is less than 2.2e−16 in Case (1) and 2.749e−10 in Case (2), that is, the results of PMRL-OM with the standardized step are better than those of PMRL-OM statistically. Table 2 shows the optimal evaluation values and the median of the evaluation values of PMRL-OM and that with two conditions in the grid worlds before and after the dynamic change. The row indicates each case, while the columns indicate the kinds of case, the optimal evaluation values, the final evaluation values of PMRL-OM and that with the standardized step and the simple condition in order from the left. In the columns of the evaluation values, the left and the right values are the evaluation values before and after the environmental change, respectively. From this table, in Cases (1) and (2), the evaluation values of PMRL-OM with the standardized step are the same as the optimal values. On the other hand, the evaluation values of PMRL-OM with the simple condition are optimal in Case (1), but the evaluation values in Case (2) are optimal only before the dynamic change. As for PMRL-OM, the evaluation values are not optimal before and after the dynamic change in both cases. Table 3 shows the evaluation values of PMRL-OM and that with the two conditions at the iteration just before the dynamic change in each grid world of Case (3). The row indicates each grid world, while the columns indicate the kinds of methods, PMRL-OM, that with the standardized step, and that with the simple condition. In each column, there are the evaluation values before the dynamic change, after the change and after the second change in order from the left. The bold numbers indicate the largest values. From this result, the evaluation values of the two conditions are  (2) better than those of PMRL-OM; in particular, those of the simple condition are the best.

Discussion
From the results, the conditions, the standardized step and the simple condition, perform better than those of PMRL-OM in the dynamic change environment of the goals and the rewards. We discuss the detailed performance of the conditions.

Effectiveness of the Two Conditions
PMRL-OM cannot adapt the dynamic change of the reward and the goal because it makes all agents aim to reach the farthest goals, that is, the agents A and B reach the goal (6,1) and (8,3) in Cases (1) and (2), whereas the conditions limit the agents to reach some goals and lead them to the appropriate goals. In the grid world of Case (1), since all reward values are same with each other, the two conditions are calculated along to the minimum number of steps. On the other hand, in that of Case (2), Table 4 shows the condition values of the standardized step and the simple condition in the grid world of Case (2). The top table has the condition values of the standardized step, while the bottom table has the condition values of the simple condition. In each table, the left and the right sides indicate the condition values before and after the dynamic change. In each side, the rows and the columns indicate the agent and the goal, respectively. Note that the tuple after the word "Goal" shows the location, e.g., (x, y)  . 16 The averaged result on the grid world 2 in Case (3) Fig. 17 The maximum result on the grid world 2 in Case (3) indicates the xth row and yth column as the coordinate of the goal. The italic cell indicates the goals where the agent should reach by following the conditions. From this table, the standardized step can indicate the appropriate goals, while the simple condition cannot indicate like that, i.e., the agents A and B should reach the goals (6,1) and (8,3) after the dynamic change for the largest evaluation value. However, PMRL-OM can cover the mistake of the simple condition; concretely, the agent B can keep reaching the goal (8,3) to prevent from the agent A's reaching that goal. In this grid world, since the agent A has never reached the goal (8,3), the simple condition of the goal (8,3) becomes large (if the agent cannot the goal, the minimum number of steps for the goal is the maximum number of steps.) From the result of that, the agent A can reach the goals (5,2) and (6,1) with the simple condition. The agents aim to reach the farthest goals, that is, the goals (6,1) and (8,3) for the agent A and B, and the evaluation values of the simple condition can be the best in Case (2). However, the standardized step is more suitable than the simple condition. In addition, the simple condition cannot indicate the difference among the goals like Fig. 3. Figures 10, 11, 12, 13, 14 and 15 show the Q-table of each agent in Case (2). The top, the middle and the bottom figures indicate the Q-tables of the agents of PMRL-OM with the standardized step, that with the simple condition and original PMRL-OM, respectively. The left and the right sides are the Q-tables of the agents A and B, respectively. In each figure, the white, the light-colored and the dark-colored squares indicate the starts, the environment states and the goals, respectively. The black arrows from the squares represent the actions, and the thickness of the arrows indicates how much the Q-value of the action. The number nearby the arrow shows the Q-value. From these figures, the agents A and B of PMRL-OM with the two conditions can estimate the appropriate Q-values to reach the goals (6,1) and (8,3), respectively. However, the agents A and B of PMRL-OM estimate the Q-values to reach the goal (8,3). This is bad because the two agents reach the same goal by following the Q-value. From the figures, the above discussion is true and the two conditions can derive the appropriate Q-value.

Difference Between the Two Conditions
the different results of the standardized step and the simple condition in some grid worlds in Case (3), we analyze the difference of the performances of the conditions. Table 5 shows the standardized steps and the simple condition values of the grid world 5 in Case (3). The top and the bottom tables indicate the condition values before and after the dynamic change. In this grid world, the agents A and B should reach the goals (5,2) and (8,2) before the dynamic change, and they should reach the goals (4,2) and (6,1) after the dynamic change, respectively. From this table, the standardized step can indicate the appropriate goals to the agents, whereas the simple condition cannot indicate that. This difference derives that the standardized step performs better than the simple condition in the grid world 5 in Table 3. From this result, it is clear that the standardized step can unify the two conditions. However, since there are many similar numbers of steps to reach the goals in Case (3), the simple condition sensitive for the reward value performs better than the standardized step.

Limit of the Two Conditions
Analyzing the result of the grid worlds in Tab. 3, we discuss the limit of the conditions.
In the grid world 2, the results of both conditions are under that of the previous method at the end. Figures 16  and 17 show the average and the maximum steps on the grid world 2 in Case (3), respectively. From these results, both conditions perform better than PMRL-OM (without the conditions). Both conditions have better performance than PMRL-OM before the second environmental change, but PMRL-OM has the same performance after that. After the first environmental change, the result of the unified condition becomes lower than the simple condition. For the first environmental change, the agents A and B cannot change their learning and should move only right to reach the goals.
The unified condition stops reusing its learning result and explores each agent's goal in some seeds; concretely, the agents A and B reach the goal with 20 as the reward value each other. From this result, the performance of the unified condition becomes lower after the first environmental change, but, in other words, the unified condition weights explore along to the environmental change. The unified condition has to spend more iterations to finish the explore. Figures 18 and 19 show the Q-tables of the agents A and B with the seed 0 on the grid world 2 in Case (3) after the second environmental change, respectively. The joint arrows in these figures are the route of the agents while moving along to the largest Q-values. The agents of all conditions move along to these routes. These routes are not optimal, that is, all methods do not perform the best. That is because, after the second environmental change, both agents did not learn actions on the right side of the environment and have to learn the optimal actions in the only right side. Therefore, the routes are optimal in the only left side, and the agents have to learn more. In the grid world 3, the results of both conditions are under that of the previous method at the end, like the grid world 2. Figures 20 and 21 show the average and the maximum steps on the grid world 3 in Case (3)  SN Computer Science they have to make agents learn more. Note that the results of the grid world 7 are the same as this case, but the condition calculations have a mistake. Table 6 shows both calculated conditions in the grid world 7. The agents have to cooperate among the goal located at (6,3) and (7,3) before the environmental change, and (4,2) and (6,1) after that. However, both conditions evaluate other goals. As for the results of that, the agents A and B do not reach the optimal goals with both conditions. However, the worst case is that the agents cooperate among the goal located at (4,3) and (5,2) before the environmental change, and (4,2) and (4,3) after that.
Since the result of the grid world 5 is wrote above, the analysis of this world is omitted. In the grid world 6, the results of the simple condition are the best, that of the unified condition are the best before the environmental change, and that of PMRL-OM are the worst. Figures 32 and 33 show the average and the maximum steps on the grid world 6 in Case (3)

Conclusion
This paper proposes a noncommunicative learning method to optimally operate multi-agent system with dynamic change, in particular, purposes and rewards. Concretely, this paper proposes the two unified conditions, the standardized step and the simple condition, combined of the number of steps and the rewards, and the proposed method enables the agents to learn cooperative behavior based on the condition. The standardized step is calculated the minimum number of steps if the expected reward values are the same each other for all goals, and the simple condition is calculated the minimum number of steps is divided by the reward value. The proposed method makes all agents learn to achieve the purposes in the order in which they have the minimum number of the condition values. After that, each agent learns cooperative policy by PMRL as the previous method. This paper analyzes the performance of the two conditions and derives that the standardized steps can be combined both evaluations with almost same weight, whereas the simple condition is sensitive for the reward value, that is, the unified condition can help the agents to select the appropriate purposes among them with the small difference in terms of the two evaluations. In this paper, we test empirically the performances of PMRL-OM with the above two conditions by comparing with the PMRL-OM in three cases of grid world problems whose goal locations and reward values are changed dynamically. The results of this derive that the unified conditions performs well than the PMRL-OM without some condition in the grid world problems. In particular, the standardized step estimates the appropriate purposes accurately than the simple condition.

Strengths
The proposed conditions make agents learn to reach the goal with larger rewards and smaller steps than PMRL-OM. In particular, the standardized step, one of the conditions, evaluates the goals to reach ones with the large reward value per time appropriately. Furthermore, the conditions can avoid making agents to reach the goal to acquire the minimum reward with many spent time. Comparing the two conditions, the standardized step estimates the goal priority effectively than the simple condition. For example, on the grid world 5 in Case (3), the standardized step estimates the optimal goal priority for agents, while the simple condition estimates wrong.

Weaknesses
The proposed conditions have two weak points: required number of iterations and incomplete optimization. As for the former, both conditions require many learning iterations to make PMRL-OM apply to the environmental change as states and rewards. Concretely, from the environmental results, though the agents reach the appropriate goal for minimum step if they explore the goals and learn the own actions appropriately, they actually reach the goals through the detour routes and reach the inappropriate goals with some trials.
As for the latter, both conditions cannot enable agents to get the maximum reward value per time in some situations. That is because each agent calculates the proposed conditions for itself, that is, each agent reaches the near goals from the own start point. This paper focuses on summation of reward values all agents acquired become largest per time.
If the agent's own condition is not the same as the purpose for all agents, the proposed conditions do not perform the best (but better). Concretely, on the grid world 7 in Case (3), both conditions cannot estimate the optimal goal priority for agents to reach the goal with the maximum reward sum per time.

Future Work
There are three directions for the future works: proposing an optimal condition, proposing theoretical methods and applying to the real-world problem. For the first direction, the proposed conditions cannot estimate the goal priority optimally in some cases, e.g., the grid world 7 in Case (3). That reason is the conditions' evaluating locally: each agent learns cooperative actions among near goals which have larger reward value for itself, but not for all agents. To tackle this problem, the conditions have to include the information of the agents' locations. Concretely, we propose one rule to the environment that the agents can observe each other if they are in the same state and extend the conditions based on the information which agents have reached the goal at first to enable the agents to calculate new conditions from the reward value, the distance from the own location to the goals and from other's location to the goals.
For the second direction, the proposed conditions estimate the goal priority appropriately. However, this is shown experimentally, that is, they might estimate the worst in some cases. To tackle this problem, we establish the conditions' capability theoretically. Concretely, in this paper's route planning task, the worst case occurs when the arbitrary agent reaches the arbitrary goal. Hence, we should prove that both conditions make the agent avoid the worst goal. We suppose that they can because all agents avoid to reach the own farthest goals for themselves by both conditions, and the worst goal has to be one of these farthest goals. We have to prove that in the future.
For the third direction, we applied PMRL-OM with the conditions to the route planning tasks. This is supposed to apply it to the transportation problem. Therefore, we apply the PMRL-OM to real-world transportation problems as storehouse robots. In fact, Honig proposed the route planning method based on graph theory for Amazon's storehouse robots [3]. However, this method requires the complete information for the storehouse and the environment is stable. Since this situation cannot be occurred in the real world, our method based on a little information performs better than that method in the real-world storehouse robots operation problem.


The Long-Run E ﬀ ects of Trade Openness on Carbon Emissions in Sub-Saharan African Countries

: Using a panel cointegration model developed based on the data extracted from the World Bank indicators, this study quantiﬁed the relationship between carbon emissions, energy consumption, economic growth, and trade openness in sub-Saharan African countries. It discovered from our analysis that there exists a long-run causality association amongst CO 2 emissions, energy consumption, economic growth, and trade openness. The study noted the existence of the Environmental Kuznets Curve (EKC) in the panel using the square term for trade openness; it was found to have a negative impact, thus trade in the long run will somewhat decrease the environmental pollution in this region. The study results imply that there should be stringent policies and rigorous enforcement in sub-Saharan African to ensure sustainable growth without associative environmental issues. openness, real GDP, and energy consumption. This shows the existence of a long-run relationship among the variables.

Introduction
Carbon emissions are topical issues with ever-increasing awareness and policies drawn around the world to curb its production. This stems from the fact that carbon emission is a significant cause of global warming. Since the inception of the industrial revolution, carbon dioxide (CO 2 ) emissions have risen dramatically. There is still an increasing rate of these emissions through economic developments.
The economies of the sub-Saharan African (SSA) countries have experienced substantial growth over the last years, with growth rates rising from an average of around 4% in the 1980s to around 9% in the 2010s. The economies are depending on the production and export of natural and agricultural resources. A few countries, including Ghana and Nigeria, have rich oil reserves that have opened them up for foreign investments and businesses that contribute grossly to the growth of their economy [1]. Trade amongst countries has risen substantially, increasing free trade with the advent of the General Agreement on Tariffs and Trade (GATT). African nations have recently formed a Continental Free Trade Area (ACFTA) to facilitate trade between member countries, aiming to develop a common continental market for goods and services. It is hoped that enhanced trade integration between African countries could yield large economic gains for the nations.
Economic growth is one of the most popular variables used in this research study area. International trade, estimated at $6 trillion a year, maybe a tool to ensure environmental protection and conservation, or not, even though the findings on these topics are conflicting. For example, the work of [2][3][4] asserts that trade liberalization is good for the environment. Discussions have been conducted on the possible adverse environmental consequences of free trade, with concerns that a fully interconnected world economy (for example, the European Union, NAFTA, and the WTO) would intensify resource scarcity, raise global pollution, and encourage dumping in low-income countries.
When countries are signing up a trade agreement with sub-Saharan economies to enhance cooperation and trade through industrialization, higher CO 2 emissions are bound to happen without proper regulation. Conversely, concerns surrounding the possible adverse environmental consequences of free trade have been extensively debated. Hence, the amount of real GDP of every nation is a crucial element in the interpretation of the environment's legislations. In the light of World Bank 2030 plan for Sustainable Climate Change and the background, this paper explores the imminent and long-run effects of trade openness on carbon emissions across selected Sub-Saharan economies based on socio-economic and environmental policies.
Kamara [5] reports that between 2006 and 2010, Foreign direct investment (FDI) grew from an annual average of US$14.9 billion to US$30.3 billion between 2001 and 2005. Sadly, many countries in sub-Saharan Africa (SSA) have not adequately taken environmental considerations in policy formulation to increase trade, FDI, and, consequently, economic growth. Many researchers, including [6][7][8] have shown immense interest in the relationship between economic growth and energy consumption. There have been calls to limit fossil fuels' use and increase the usage of alternative energy sources such as renewable energy to supplement the energy deficit. Limiting fossil fuels aims to reduce CO 2 emissions, but fossil fuels dominate the energy mix of many SSA countries [9]. There is no doubt that industrialized countries seem to contribute more to CO 2 emissions. This is one of the many reasons why most research of this nature concentrates on developed economies. Lindmark [10] also mentioned a high increase in carbon emissions during a country's developing stages.
Reduction of CO 2 emissions through technological means or economic means such as increasing fuel prices, the introduction of taxes to limit fossil fuel consumption are the two main ways previous studies provided on how to reduce emissions. However, these policies and innovations may have growth consequences even though they may reduce carbon emissions [11]. A possible potential solution is to be able to regulate the possible factors affecting CO 2 emissions. Nevertheless, this can only be done if there is an established relationship among these factors.
Several researchers [12][13][14] have tried to establish similar relationships in other parts of the world. This research differs from previous studies because it includes additional variables; trade openness to investigate whether or not they contribute to CO 2 emissions besides the primarily studied variable energy consumption on selected Sub-Saharan economies. Findings from this study provide a better understanding of policymakers and researchers on how to formulate proper economic and environmental policies with regards to key drivers of CO 2 emissions to effectively address the threat of global warming to our environment.
The rest of the manuscript takes the following structures-Section 2 reviews related literature to the study. Section 3 presents the research methodology, and Section 4 addresses findings, results, and discussions. Section 5 concludes the paper and sets out several policy implications for our research work.

Trade and the Pollution Haven
The discussion of whether inter-country differences in environmental regulations are turning developing nations into "pollution haven" has recently drawn much attention. The pollution haven hypothesis (PHH) by Copeland and Taylor [15] explains trade's influence on the environment. It stipulates that the political and regulatory conditions favor the relocation of companies to benefit from environmental conditions that are less demanding than in their territory, contributing to the degradation of the environment [16].
Therefore, the study identifies the pollution haven hypothesis (PHH) as low environmental protections that have competitive benefits for countries and affect foreign trading trends. The effects of trade openness on the world can be explained by composition, scale, and technical effects [2,17,18]. The theory was initially being developed to examine better and understand how trade openness affects industrial air pollution. This theory has been used by many authors [2,17,18] to examine and analyze the effects of trade openness on environmental quality and also the effectiveness of pollution regulations.
The scale effect posits that economic growth has a negative effect on the environment due to increased trading activities, which increases production. This increase in production and consumption causes an increase in environmental damages. The composition effect suggests that the composition of production changes when the demand for traded goods manufactured by polluting methods rises and vice versa. It measures the change in environmental pollution due to changes in goods produced. Environmental effects can be positive or negative depending on trade specialization and techniques used. The technique effect suggests that technological development leads to changes in the effects of production on the environment. It illustrates the environmental effect of transferred expertise and creative manufacturing practices as profits and trade increase. With free trade, the transition of new and more environmentally sustainable technologies with more energy-efficient processing methods will lead to lower environmental harm. The technique effect could also represent technological advances that lead to more serious environmental damages.

CO 2 Emissions and Trade Openness
Using the cointegration, error correction model, and the granger causality test, ref. [19] investigated the relationship between CO 2 emissions and economic growth in the United Kingdom. A short and long-run causality running between the variables was observed. It also found a unidirectional causality from the real gross domestic product (RGDP) to foreign trade ratio, foreign trade ratio to CO 2 emissions, and foreign trade ratio to final energy consumption. The results further revealed an inverted U-shape relationship between CO 2 and GDP. Hence the EKC hypothesis confirmed in the UK and trade openness can increase CO 2 emissions.
Akin [20] examined the impact of foreign trade, energy consumption, and income on CO 2 emissions using panel data cointegration analysis. The results revealed a significant positive relationship. Findings suggest that the unidirectional causality of CO 2 emissions to trade openness persists in the short term.
In Turkey, Ozturk and Acaravci [21] found a positive and significant relationship between CO 2 and trade openness, while Boutabba [22] indicated that trade openness negatively influences CO 2 levels in India. Additionally, many studies have found a link between CO 2 emissions and trade openness Jalil [23] with contradictory results. A unidirectional causality running from trade openness to CO 2 emissions was also identified by Omri et al. [24]. Using instrumental variables, modus operandi Managi et al. [25] measured trade's aggregate effect on environmental quality [25]. The findings revealed that foreign trade reduces pollution in non-advanced economies, with opposing responses in advanced economies: this change was attributed to the scale and composition of trade impacts.
On the other side, trade openness will minimize CO 2 emissions in the long run. Al Mamun et al. [26] examined the role of real income, trade openness, population growth, and energy consumption on CO 2 emissions using data from 82 developing countries from 1980 to 2012 using various mean group (MG) methods. The findings revealed that the proportional change in trade, the holding of all other explanatory variable constants, decreases CO 2 emissions by 0.3%. Interestingly, the findings for low-income, middle-income, and full survey countries were inconclusive.

CO 2 Emissions, Energy Consumption and Economic Growth
Fossil energy has become the symbol of modern industrial civilization. This has resulted in a tremendous increase in greenhouse gas emissions. The relationship between energy consumption and economic growth is still unclear, but the energy consumption is undoubtedly associated with a vital factor; the environment. Till now, a substantial amount of literature has diverged on the link between economic growth, energy consumption, and environmental pollution.
Adewuyi and Awodui [27] examined the relation amid biomass energy use, economic growth, and carbon emissions in West Africa from 1980 to 2010 by incorporating the role of pollution production and energy demand into an increased endogenous growth model. They also employed a simultaneous equation model estimated with three stages least squares (3SLS). They analyzed individual West African countries and a panel of countries in their work. The results showed an entirely significant relationship with feedback effects between GDP, biomass consumption, and carbon emissions in five West African countries (Nigeria, Burkina Faso, Gambia, Mali, and Togo). Partial significant ties have been identified between the variables in the remaining West African countries.
Long et al. [28] investigated the relationship between energy consumption, carbon emission levels, and economic growth in China from 1952 to 2012. They used unit root and cointegration analysis to evaluate the stationarity. They examined the mutual effects of energy consumption, carbon emissions, and economic growth through the Granger's causality test. They also performed a static and dynamic regression study of the determinants of carbon emissions and economic development. The findings showed that coal has a significant effect on economic growth and carbon emissions. Gross domestic product has a bi-directional relationship with CO 2 emissions, coal, gas, and energy consumption.
The long-term causal relationship between economic development, energy use, and pollutant emissions with labor and capital has been investigated as additional variables in South Africa [29] using the bounds testing approach to cointegration and the Granger causality test in a multivariate framework. Their results showed a unidirectional causality from pollutants' emissions to economic growth, from energy consumption to economic growth, and from energy consumption to environmental pollution, with no causal feedback. These results suggest that South Africa, a developing country, cannot increase economic growth or energy consumption without increasing emissions; hence, it has to reduce growth and energy consumption to reduce carbon emissions.
Conversely, Zhang, and Cheng [30] used the Toda and Yamamoto approach to research the relationship between energy use, economic development, and carbon emissions in China. They observe unidirectional long-term causality from economic development to energy consumption and from energy consumption to carbon dioxide emissions. Their findings indicate that both carbon emissions and energy consumption does not lead to economic growth.

The Environmental Kuznets Curve
In extending educational and academic objectives, the environmental Kuznets curve (EKC) has become an instrument or standard for explaining the link amid assessed natural environmental and economic growth. The critical theoretical development explaining the likelihood of an EKC interaction was discussed by Grossman and Krueger [31], indicating three potential effects of a rise in economic-global trade treaties' growth. Researchers have considered this hypothesis globally, considering both developing and developed economies with different approaches, turning points, and outcomes. Since the commencement of the EKC hypothesis proposal, not all research works fit this hypothesis.
Series of previous research, which includes [32][33][34][35][36], among many others, have examined the association linking CO 2 emissions and income and confirmed the EKC concept throughout the range of nations and regions. To mention few, multiple studies support the EKC hypothesis through the panel data or time-series Skaza and Blais [37] examined 190 developed and developing nations; Al-Mulali and Sheau-Ting [38] also investigated 189 economies from Asia Pacific, Eastern Europe, the Americas, Middle East and North Africa (MENA), Sub-Saharan Africa and Western Europe; ref. [24] also analyzed MENA nations.
Another study [39] and Al-Mulali et al. [40] presented contradictory findings that do not endorse the EKC's legitimacy in Tunisia and Vietnam. Dogan and Turkekul [41] also analyzed the connection amid CO 2 emission, real output, energy usage, trade openness, financial growth, and urbanization in the United States for the timeframe 1960 to 2010 by utilizing the bound test for cointegration. Their analysis concluded that the United States of America does not accept the logic behind the EKC model; hence, EKC was not verified. On the other hand, Amissah and Clottey [36] employed the common correlated effects mean group (CCEMG) approach for 25 emerging economies in Africa spanning from 1990 to 2015 by examining environmental effects on economic growth. Their conclusion verified the EKC framework across all the panels (25 nations, oil-exporting and non-oil exporting) with turning points of $3197.65, $1433.67, and $5909.16 amid the three panels in their study.
In another study by Shahbaz et al. [42] also examined EKC's presence among 19 African nations from 1971-2012, utilizing the ARDL bound test method. The researchers concluded the existence of EKC amid African nations; moreover, concerning Tanzania and Sudan, the association amid CO 2 emission and economic growth depicted U shaped. Shahbaz et al. [43] investigated the impact of financial progress, economic development, trade openness, and coal usage on environmental sustainability through the utilization of data from 1965 to 2008 in South Africa by applying ARDL bound test approach. Their evidential observation depicted the existence of the EKC hypothesis, while trade openness increases the environment's efficiency by reducing the production of CO 2 emissions. Ref. [44] explored the effect regarding CO 2 emissions and economic production in 22 OECD nations for the period 1975-1998. The authors ascertained the EKC presence across the nations by utilizing the pooled mean group (PMG) methodology.
Farhani and Shahbaz [45] examined the causal association amid CO 2 emission, economic development, and renewable and nonrenewable energy usage from 1980-2009 for ten nations within the Middle East and North Africa (MENA). They utilized the FMOLS and the dynamic ordinary least squares (DOLS) to approximate the long-term projections. Their findings provided proof of the EKC phenomenon in the economies concerned. Also, a study by Rehman et al. [46] considering the relevance of the EKC hypothesis in India, Sri Lanka, Bangladesh, and Pakistan. Data are spanning from 1984-2018 by applying the fixed effects model (FEM) considering corruption and trade openness relationships. The outcome depicted a verification of the EKC hypothesis among the economies under study. To conclude, this study examines the long-run effect of trade openness on carbon emissions in selected Sub-Saharan economies while testing the validity of the EKC hypothesis.

Methodology and Data Collection
In this study, the data on CO 2 emissions, GDP, trade openness, and energy consumption complied from the World Development Indicators for selected Sub-Saharan African economies, namely: Benin, Botswana, Cameroon, Cote d'Ivoire, Gabon, Kenya, Mauritius, Mozambique, Namibia, Nigeria, Senegal, South Africa, Sudan, Togo, Tanzania, Zimbabwe, Zambia, and Ghana. The data gathered were yearly, ranging from 1990 to 2014. Some elements used in measuring pollution in other literature includes SO 2 , nitrogen oxide (NO), and CO 2 . In our study, however, CO 2 was used considering its global effects. The variables used in the study are defined as follows: (1) Carbon emissions (CO 2 )-Carbon emissions measured in metric tons per capita.
(2) Energy consumption (EN)-consumption of energy measured in kg of oil equivalent per capita.
(3) Gross Domestic Product (GDP)-real gross domestic product per capita measured in current USD. (4) Trade Openness (TO)-Trade openness is the sum of imports and export measured in current USD.
In specifying the models applied for the analysis, trade openness and energy consumption are considered critical forces of economic growth. Environmental pollution is therefore characterized as the result of energy consumption in association to trade openness and economic growth. This study, therefore, came up with: The chosen variables were transformed into natural logarithms to evaluate the progression level of the parameters. This is achieved by taking their log differences to estimate Equation (2). This also helps to reduce the problem of heteroscedasticity. The transformed selected variables are used to explore the association between the dependent and independent variable to obtain: where lnCO 2it signifies the natural log of CO 2 emissions, lnEN it depicts the natural log of energy usage, while lnGDP it represents the natural log of real GDP, and lnTO it also depicts the natural log of trade openness. The ε it represent the unobserved term. The elasticity of environmental pollution concerning energy consumption, real income, and trade openness is indicated with the coefficients α 1 , α 2 and α 3 respectively. The constant parameter is shown with α 0 .
In the study, we expected α 1 to have a positive outcome. This was because higher energy consumption (fossil fuels) is expected to increase environmental pollution. We also expected to obtain a positive α 2 , since an increase in economic growth will increase environmental pollution if the economies rely on fossil fuels. We expected α 3 to be either positive or negative, as seen in other economic literature such as [20,47], who suggests that the sign of trade openness varies depending on the degree of economic development.
Employing the theory behind the EKC analysis, we lastly examined the reality of an inverted U-shape relationship between trade openness and CO 2 emissions. Shahbaz and Sinha [48] argued that EKC's outcome might differ based on explanatory variables, empirical approaches, period, and economies in question. To obtain Equation (3), the square of trade was incorporated into Equation (2). The equation applied is to verify the validity of the EKC hypothesis in the model. The study, therefore, defined the EKC model as below: where α 0 in the model is the interception parameter, which differs between country i and year t.

Slope Homogeneity Test
As the first preliminary test, the Pesaran and Yamagata's slope homogeneity test was conducted. The results of the slope homogeneity test, shown in Table 1, the p-values are smaller than 5%, proving the fact that the slope coefficients are not homogeneous.

Cross-Sectional Dependence Test
Trade openness primarily involves various economies; thus, it has become vital when considering the effect of cross-sectional reliance on the dataset specified. This study, therefore, commenced by analyzing the cross-sectional dependency test. Cross-sectional dependency among panels, if not checked, can lead to biased estimations. Using the parametric test proposed by Breusch and Pagan [50], Pesaran [51], and Pesaran et al. [52], This research investigated the presence of cross-sectional dependency across the chosen variables. Ref. [50] suggested a Lagrange Multiplier for null cross-section error correlation testing. It is determined by applying the squares and including the correlation coefficients amid cross-section residues after the ordinary least square (OLS) approximation. The null hypothesis H0 for this test is that there is no cross-section dependence and the alternative hypothesis H1 for this test is that there is cross-section dependence.
The Lagrange Multiplier (LM) test is more pertinent and does not necessitate a unique cross-section unit sequence. However, it is only relevant if N is significantly smaller and T is higher: whereρ ij denotes the result of the pairwise residual correlation sample, which can be defined as: The null and alternative hypotheses to be verified are specified as: H0.ρ ij =ρ ji = cor ε it ε jt = for i j, H1.ρ ij =ρ ji = cor ε it ε jt for i j Pesaran [51] proposed an alternative to the Lagrange multiplier (LM) test by Breusch and Pagan [50] due to the latter's limitations when N is large. The alternative test by Pesaran et al. is based on the pairwise correlation coefficient rather than the squares used in the LM test. The test is computed as below: This method is utilized when T > N or N > T, and is asymptotically standard and normally distributed. The null and alternative hypotheses were coherent with the CD LM1 the test utilized by Breusch et al. [50]. The study presents the outcome of the test in Table 2.

Panel Unit Root Test
To conduct panel data analysis, the premiere procedure estimates the degree of integration of each variable, thus testing for the existence of unit root in our variables. In the study, we reduced the problem of inconsistency and invalid test statistics in our estimation by employing two alternative unit root test approaches. The Levin-Lin-Chu (LLC) approach by Levin et al. [53] and the Im-Pesaran-Shin (IPS) test by Im et al. [54]. They were used to examine stationarity's availability in our panel data and presented in (Table 3). Therefore, the null hypothesis of non-stationarity within the sequence is assessed against the alternative hypothesis that the proportion of the variables is entirely fixed. Equation (7) specifies the test based on Levin-Lin-Chu (LLC) approach: where U it is white noise and U it ∼ N(0; σ 2 ). The Im-Pesaran-Shin (IPS) test by Im et al. [54] proposed a new concept that rectifies the OLS methodology's shortfalls by using the autoregressive methods across panels.  The symbol * and *** indicate 1% and 10% level of significance. Source: authors' calculation.
All the series are tested for non-stationarity after their logarithmic transformation. The non-stationarity for each variable is tested under two conditions: using intercepts, and intercept and trend in the models. All variables are stationary at a significant level of 1% as shown in Table 3. After the unit root test, the study proceeded to check for the presence of long-term cointegrating associations among the variables.
In addition, the Pesaran [55] cross-sectionally augmented Dickey-Fuller (CADF) as a second generation unit root test that considers cross-sectional dependence. As shown in Table 4, the null hypothesis can be strongly rejected and therefore all series are integrated of order 1.

Panel Cointegration Test
In this section, we applied the Westerlund panel cointegration test [56] to find out the existence of cointegrating relationship. The test provides four statistics of G T , G α , P T , and P α . The first two statistics can detect cointegration in cross-sectional units, while the last two statistics determine cointegration in the whole panel. Table 5 reports the findings of the Westerlund panel cointegration test. According to the results, we can reject the H0 (no cointegration) and confirm the long-run cointegrating relationship among variables. Besides, to check the cointegration with the structural break, we conducted the Westerlund and Edgerton's test [57], including two LM statistics as follows: where,φ i andω i denotes the least square estimate of ϕ i . Table 6 presents the results of this test, expressing that the H0 of no cointegration can be rejected.

Cointegration Estimation
Many studies proposed that the presence of long-run effects between the dependent and independent variables should be estimated using the two main methods of ordinary least square (OLS) based estimators. That is the fully modified OLS (FMOLS) and dynamics OLS (DOLS). The key difference between the two methods lies in how autocorrelation is resolved in regression [58] Whereas with FMOLS, the Newey-West can be used for correction, DOLS embraces to add more lagged and lead variables. Pedroni [59] proposed the approach to estimating the coefficients used to measure the long-run effects.
This article employed three error estimators, the above mentioned and the Pool Mean Group (PMG), to examine the validity of openness of trade on carbon emissions in Sub-Saharan economies: in which: Considering Ω as asymptotic covariance matrix for the long-term variance and a dynamic covariance, and L is a lower triangular matrix with partition calculation, then, DOLS estimator is utilized to take the below form: (11) in which q is defined as the numbers of lags specified for the models. Subsequently, the DOLS estimation method is linked to Kao and Chiang [60] for the use of finite sample properties. The study presents the results of cointegration estimates in Tables 7 and 8.

Panel FMOLS Estimate for EKC Results
To examine the long-run impact of trade on the environment, we include the square of trade openness. We relate trade openness to the EKC framework since many studies uphold the theory that trade openness correlates to growth. Thus, an increase in trade has the effect of increasing economic growth. Tables 9 and 10 display FMOLS and DOLS estimation.  The symbols * and ** indicate the 1% and 5% level of significance, respectively. Source: authors' calculation.

Robustness Test
The mean group estimator, according to Pesaran and Smith [61], offers coherent assessments of the average values. This requires the parameters to be independently variable within groups and may not recognize the capacity for homogeneity among the groups. The follow-up step is the usual pooled method. Considering on merit of PMG over the DOLS and FMOLS methods shows that it permits the long-term coefficients are bound to be the same. In contrast, the short-term dynamic specification varies from nation to nation. The PMG is which is considered as the intermediary estimation method since it includes averaging and pooling. This is estimated as below: The cross-section divisions (nations) are depicted by i = 1, 2, . . . . . . , N, t = 1, 2, . . . . . . , T stands for the timelines, X it (k,1) is a vector response variable for the nation i, µ depicts the static effect, β ij the coefficient of the dependent lagged variable, while δ ij are k × 1 coefficient vectors. The study presents the outcome of the test in Table 11.

Panel Causality Test
Once cointegration and long-run relationship have been established, the study proceeds to estimate the error correction model (ECM). The augmented Granger type causality test model with a one-period lagged error correction term (ECTt-1). Engle and Granger [62] indicate that the Granger causality test, which is conducted in the first-differenced variables using a VAR, will be ambiguous in the presence of cointegration. The VECM allows us to capture both the long-run and short-run causality. The short-run causal effects can be obtained by the F-test of the lagged independent variables, but the t-statistics on the coefficient of the lagged error correction term shows the significance of the long-run causal effect. A general ECM is derived as follows: The purpose of the error correction term (ECT) is to identify the speed and adjustment of the cointegration vector θ and shows how the variable returns to long-run equilibrium in a short-run period. It has a statistically significant value with the negative coefficient. α 2ik , α 3ik , α 4ik indicate the short-term elasticity for the break values of energy consumption, economic growth, and trade openness. The direction of both causalities within our model was analyzed using the multiple regression causal relationships proposed by Granger with an optimal lag length n (Aikaike Info Criterion, AIC = 2), results are shown in Table 12:  The symbols * and ** indicates 1%, 5% level of significance, respectively. Source: authors' calculation.

Results and Discussion
In this section, we discuss the empirical outcomes of our analysis. Before testing for stationarity (panel unit root test), we examined the presence of slope homogeneity and also if there existed a cross-sectional dependency in our panel. Using the approaches of Pesaran and Yamagato [49], Pesaran et al. [51] and Breusch et al. [50], we report the slope homogeneity test and cross-sectional dependency test outcomes as utilized both the dependent and independent variables in Tables 1 and 2.
In light of our outcomes, we dismissed the null hypothesis of the cross-sectional dependency test for the underlined variables at a significant level of 1%. Accordingly, a unit root test was performed to check for stationarity. To debunk any doubts, we decided to use two unit root tests to determine a more reliable inference of the data. The LLC test measurements of Levin et al. [53] and the IPS test measurements of Im et al. [54] were utilized. The results indicated that the variables CO 2 emissions, trade openness, economic growth, and energy consumption were stationary at first difference with a 1% significant level. This shows that in the first difference, all variables have a distinctive order of integration.
After confirming the stationarity of our variables, we move on to explore the long-run relationship between the variables utilizing the Pedroni panel cointegration method [63]. The null hypothesis posits that there is no cointegration in all tests, whereas the alternate hypothesis posits that there is cointegration. The panel cointegration test result indicates that, in most circumstances, the alternative hypothesis of the cointegration test could not be rejected. Along these lines, this study concluded that there was plentiful verification for the presence of cointegrating connections amid the variables CO 2 emissions, trade openness, real GDP, and energy consumption. This shows the existence of a long-run relationship among the variables.
The outcome of the panel FMOLS and DOLS tests so shows the long-run effects of the independent variables trade openness, GDP, and energy consumption on the dependent variable CO 2 emissions. The variables showed significance with energy consumption, GDP, and trade openness with the highest level of significance at 1% in both FMOLS and DOLS. The study also finds that trade openness has a positive relationship to environmental quality (CO 2 emissions) in the region (Sub-Saharan Africa). Thus a percentage increase in trade will increase carbon emissions by 0.33% t a 1% level of significance. This outcome is in line with [64][65][66].
As a robustness check, the PMG estimation outcome depicted that the coefficients estimated are respectively 0.62, 0.26, and 0.37 for all the economies (nations) pooled. This is seen in all cases such as FMOL, DOLS, and PMG; all coefficients are statistically significant. Comparison from the above three tests can be seen that trade openness has relatively higher coefficients across all three estimates representing 0.33 (FMOLS), 0.40 (DOLS), 0.37 (PMG), also supports the result from the FMOL and DOLS but with different coefficients. In terms of energy consumption, we discovered that it had a positive impact on CO 2 emissions in the Sub-Saharan region. The panel results indicated a positive significant impact of economic growth on CO 2 emissions in the Sub-Saharan region (1%). This shows the existence of a significant long-run nexus between GDP and carbon emissions. This outcome supports the findings of [36].
The effect of the square of trade openness using the FMOLS indicates the existence of EKC for carbon emissions in the sub-Saharan region. The square of trade openness had a significant negative relationship with carbon emissions, meaning in the long run, as trade increases, carbon emissions will decrease hence creating the inverted U-curve. Consequently, the result implies that in the initial stages of economic development in SSA economies, economic expansion is mainly concerned and environmental factors are ignored, focusing both on growing trade with other countries. Investment through this will increase as income increases, thereby increasing energy demand and the deterioration of the environment as a result of the rise in carbon emissions. The upsurge in level of economic growth based on income therefore brings social and environmental awareness that helps reduce pollution of the environment.
This could indicate that as the sub-region develops, the need and desire for economic growth and development outweighs that of environmental quality; hence, higher carbon emissions at higher income levels. The rest of the results in the study showed that energy consumption has a positive significant (1%) relationship, like the result of the previous model without the square of trade. Like the impact of energy consumption on carbon emissions, GDP also showed a highly positive significant relationship with carbon emissions. On the other hand, trade openness in this model showed a positive and highly significant impact on carbon emissions.
The output of the panel VECM Granger causality results in Table 12 indicates a long-run relationship between CO 2 emissions and the variables energy consumption, economic growth, and trade openness. Similarly, there is a long-run relationship amongst the variables when energy consumption and trade openness are dependent variables. There was no short-run relationship recorded.

Conclusions and Policy Recommendations
This research mainly investigated the long-run relationship between trade and CO 2 emissions, with other critical explanatory variables such as energy consumption and economic growth for 18 selected sub-Saharan African countries. The data for this research spanned from 1990 to 2014 and were derived from the World Development Indicators. A current panel estimation method was used for this study. The variables employed in this study were stationary at the first difference and cross-sectional dependent on the test outcome. Moreover, the cointegration test outcome depicted that in the long-term energy usage, CO 2 emissions, economic growth, and trade openness were integrated. Besides, the panel finding employing the FMOLS, DOLS, and PMG methods depicted that trade openness increases CO 2 emissions in the employed panel. Using the error correction model, the short and long-term causal correlation was established. The panel VECM Granger causality findings demonstrated no short-run panel causality among the variables. Alternatively, the findings depicted that there were long-run causal associations amid the variables. This lagged error correction term in the carbon emissions and energy is statistically significant. Hence the results showed that there are unidirectional causal relationships among these variables and trade openness Granger-caused environmental pollution in these Sub-Saharan countries in the long run.
The paper also attempted to test the EKC hypothesis for our chosen panel. The findings based on the FMOLS estimation suggest the existence of an inverted u shape of trade and carbon emissions in the panel; hence, the existence of EKC. Our results demonstrated the existence of a statistically significant long-term relationship between CO 2 emission and energy consumption, economic growth, and trade openness. There was no record of any short-run relationship implying that trade, as well as other variables, will preferably be more influential on environmental pollution in the long run.
The study's empirical outcome also leads us to conclude that energy consumption and economic growth cause an increase in carbon emissions in sub-Saharan African countries. This suggests that the substantial and steady increase in economic growth, as witnessed over the last few decades, which is projected to continue, would have adverse consequences on the environment. Therefore, efforts should be made in reducing carbon emissions through means such as a reduction in coal usage. Efforts need to be made for practical support for renewable energy, vigorously advocating industrial electrification and renewable building heating systems, and saving carbon effectively. As a region, Africa must follow the cap and trade scheme to guarantee an atmosphere free of pollution. Policymakers in Africa have to set a limit on the amount of total CO 2 emissions that can be generated each year by an organization or a given business. The cap and trade program uses emissions trading; thus, it allows businesses that have used up their quota to buy from other organizations that have not used up their total emissions quota. This will provide economic incentives and motivate businesses and organizations to use green energy and other efficient production methods to limit carbon emissions.
The study noted the existence of the environmental Kuznets curve (EKC) in our panel as the square term for trade openness was found to have a negative impact, thus trade in the long run will somewhat decrease the environmental pollution. The results show that growth in trade will harm the environment initially, but in the long run, the trade will increase environmental quality; thus, there will be a reduction in carbon emissions. This means there should be stringent policies, and rigorous enforcement is undertaken to ensure sustainable growth without the associative environmental issues. Also, a carbon tax policy will need to be implemented. High taxes should be levied on firms that emit high harmful gases. Upon implementing the taxes, companies in Africa would calculate the cost of lowering their emissions against the tax they would pay if they continued to produce at their present rates.
The scope of this study considered the long-run effects of trade openness on CO 2 emissions in Sub-Saharan Africa. The restructuring of trade reforms across the continent, like establishing the African Continental Free Trade Area (ACFTA) would enhance trade cooperation amid various African economies. Therefore, by far, trade is one of the key drivers that will either enhance CO 2 emissions or help emerging economies minimize it while sustaining the growth of the economy concurrently.
It can also be established from the study that pollution levels in the atmosphere are the outcome of trade openness negatively impacting air quality. Removing tariffs and other trade barriers to environmentally friendly goods are likely to increase green innovations at a reduced cost. As pollution reduction initiatives to be implemented in the energy sector are identified, firms and organizations should be rewarded for the integration of renewable energy sources to increase the adoption of advanced energy-saving technologies.
The study's results show that more trade does not necessarily induce emissions, confirming various past studies. The inverse is also true, particularly as nations are progressively embracing environmentally friendly production technologies. These technologies will reduce the heavy dependence on old and conventional forms of energy production. This has many policy implications that require diverse techniques to enhance economic growth and improve environmental quality.
Overall, the study suggests that African economies should develop and enforce efficient laws and policies to prevent or reduce environmental pollution while favorable for sustainable growth. Besides, a new detailed economic development evaluation framework in the selected African countries should be developed to reduce contaminant emissions to the atmosphere. The region should adopt initiatives to encourage low carbon as well as sustainable and high economic development, reinforce national structures to create a compelling and functional climate change system, and also massively to educate the general public on tackling climate change challenges.