\section{Introduction}

Rapid progress in AI has been enabled by remarkable advances in computer systems and hardware, but it is not widely used in cyberspace security protection. However, the intellectualization of cyberspace security protection system is an important problem facing the current cyberspace security protection \cite{DBLPSS10}. In the security management of cyberspace, actually the cyberspace should be regarded as a space composed of physical domain, digital domain, network domain and social domain, and its security protection should also be conducted as a whole \cite{DBLPnips2016}. In this process, it mainly includes the intelligence discovery of hidden attack paths, intelligent deployment of security equipment \cite{DBLPtsmc05}, intelligent monitoring of network traffic, intelligent awareness of security situation and other parts \cite{inbook009}, which comprehensively constitute the cyberspace security protection system. We believe that it is AI itself will provide the means to constitute the cyberspace security protection system, creating a symbiotic relationship between cyberspace security and AI with each fueling advances in the other.

In this work, we present a learning-based approach to discover hidden attack paths in cyberspace. Our objective is to find the shortest hidden attack path, to discover the weakest flaw in the cyberspace, thus it can effectively repair the cyberspace weaknesses and improve the security of the multi-domain cyberspace. Despite of research on this problem, it is still rely on experts to judge and evaluate the security risks existing in current cyberspace, this method is not highly intelligent and consumes a lot of human resources. Besides, the problem complexity arises from multiple domain interaction each other in cyberspace, and only digital domain or network domain can be considered in current research. In general, the cyberspace should include physical domain, cognitive domain, social domain, network domain and digital domain. Physical domain includes space and physical entity. Space includes city, campus, building, room, etc. Physical entity includes network equipment and terminal equipment, such as switch, router, computer, etc. Cognitive domain is different people have different understandings of cyberspace, resulting in different influences on cyberspace. Social domain refers to the relationship between people, for example, an attacker is a friend of the network administrator, so that it is easier to obtain higher network privileges than other attackers. Network domain is what we refer to as the traditional network space. Digital domains represent information entities used to represent digital information, such as user's names, passwords, secret keys, messages, etc. Because of multiple domain interaction each other in cyberspace, this leads to the high complexity of discovering potential attack paths. Therefore, it cannot be solved by exhaustive method to discover the shortest attack path. Even after breaking the problem into sub-problems, the state space is still larger so that the traditional methods can't to solve it.

To our knowledge, in order to realize the target, the following problems need to be solved:

\begin{itemize}

	\item{First, the problem of multiple domain which can interaction each other. That is the problem of multiple domain cyberspace, and how can change the existing cyberspace security risk analysis to focus on multiple domain cyberspace, enforce the pertinence and relevance of business level;}

	\item{Second, the problem of the alternative actions in different states is different. In traditional RL, the number of actions which agent can select is the same in any state. But in multi-domain cyberspace, the agent have different alternative actions in different states. How can solve this problem in order to make the alternative actions in different states is different.}

\end{itemize}

To address this challenge, we propose discover the multi-domain united attack paths as a Reinforcement Learning (RL) problem, where we train an agent (as an attacker) to discover the hidden shortest attack path in the multi-domain cyberspace. In each iteration of training, the attack paths are sequentially found by the RL agent. Training is guided by a fast but approximate reward signal for each of the RL agent discover the attack paths. At the same time, we proposed improved DDPG algorithm, to solve the agent can select different actions in different states.

We believe that the ability of our approach to learn from experience and improve over time unlocks new possibilities for network administrator. The experiment shows that we can achieve superior result on simulated cyberspace environment, as compared to the baselines method. Furthermore, our methods can discover the shorter attack paths comparable to human expert based method and other baseline method in same time. Although we evaluate primarily on a simulated cyberspace, our proposed method is broadly applicable to many real cyberspace.

The main contributions of this paper are as follows:
\begin{itemize}

	\item{We propose a unified representation method of multi-domain semantics and user's cyberspace permission, to simulate multi-domain cyberspace, which can describe the entities and entity relations in cyberspace, so that network entities in different domains can be described and expressed in a unified way, and the existing network multi-domain security state is inferred through the unified description, so as to effectively grasp the relationship between the multi-domain attacks;}
	\item{We first proposed RL on cyberspace operation and maintenance, proposed to discover the hidden attack paths by RL in multi-domain cyberspace, our experimental proved that the shorter the hidden attack path, the cyberspace security is low. It is through our proposed method that we can find the weak in cyberspace, so as to provide a reference for administrators, and improve the operation and maintenance capacity of cyberspace security;}
	\item{To address the alternative actions in different states is different, we proposed improved DDPG algorithm to solve this challenge. By introduced multi-domain action selection module, by this module, we can know an action whether or not can be executed in a state, and we can obtain a execution action ${a_t}'$ in a state. We can realize the reasonable choice of actions in different state through this method.}

\end{itemize}

The structure of this paper is as follows: Section 1 is introduction. In Section 2, we discuss the related works. Section 3 provides our proposed method, including the discover hidden attack paths model and improved DDPG algorithm. Section 4 presents our experiment and analysis of the experimental results. Section 5 is the conclusion.

\section{Related Work}
\subsection{Intelligent Security Protection}

Intelligent security protection mainly studied the user's behaviors and rules of user's network monitoring. In general, there is an attack in the cyberspace, when it appears, the traffic will change, we can take advantage of the attack mode type to detect cyberspace anomalies. Collect the original message of the data in the network and extract it, take the destination address and other information, establish the normal traffic model, and then use discrete wavelet transform technology analyzes and detects the data traffic to judge the cyberspace anomalies \cite{TNET902685}.

At present, the intelligent security protection technology based on cyberspace user's action mainly relies on web data mining, user abnormal action detection and neural network based method to distinguish.

Combining traditional data mining techniques with the Internet for web mining is to extract potentially useful patterns and hidden information from web documents, web structures and service logs. Generally, according to the different objects of web mining, researcher divide web data mining into four types: web content mining, web structures mining, use record mining and web comprehensive mining \cite{Badea2015}.

In the operation process of users will retain a lot of action information, effective use of this information is the basis and key to the realization of abnormal action determination. Multi-layer log collection is implemented to support the decision of user access action, by this method, we can find the abnormal user's behavior. Using multi-level user access log, and integrate web front, user click action and URL access logic, to extract the user's access action characteristics, by a large number of calculating the average user behavior baseline characteristics, use of effective monitoring abnormal access action scoring algorithm, trace the action of the abnormal IP, the above methods can find out whether the user's behavior is abnormal or not \cite{Beutel2015}.

As an important method to deal with nonlinear systems, the neural network method has been successfully applied in the fields of pattern recognition and probability density estimation. Compared with the statistical analysis theory, the abnormal behavior analysis method based on neural network can better express the nonlinear relationship between one variable and another. The changing of abnormal network action requires the ability of behavior analysis system to analyze a large number of network packets. Moreover, many common attacks may be coordinated by multiple attackers on the cyberspace, which requires the intelligent security protection system must have the ability to deal with a large amount of nonlinear data. The method based on neural network has a fast response ability, especially for the processing of noisy data and incomplete data, so it provides a great flexibility for the analysis of intelligent security protection \cite{Kawazu2016Analytical}.

In recent years, the emergence of machine learning has made intelligent security protection become a new trend. There are many new attempts, including SVM \cite{DBLP123} \cite{Gao2017A}, K-nearest neighbors \cite{Xu2017Incremental}, Naive Bayes \cite{DBLPBK19}, random forests \cite{DBLPZH08}, neural network \cite{DBLPMK17}, deep learning and so on. The methods based on deep learning have become mainstream in the field because of their better performance. Gao proposed an model based on deep belief network, which uses a multi-layer unsupervised learning network and a supervisor-based back-propagation network \cite{DBLPQuZSQ17}. Shone used asymmetric depth self-encoders to learn network traffic characteristics in an unsupervised, not only achieved good performance on large data sets, but also reducing training time \cite{DBLPTPS18}. Yin proposed a model using RNN, compared the effectiveness of the non-depth model, and achieved good performance \cite{DBLPZFH17}. Kim proposed a model using LSTM and gradient descent strategy. The experiment result which proved the LSTM can achieve a better performance \cite{Le2017}. Sheraz conducted a comprehensive study on deep learning model, and proved that the deep learning method can not only be used in this field, but also can achieve better performance \cite{DBLPSKBHIH18}.

\subsection{Cyberspace Simulation}
Elderman et al. (2017) focus on a simulation game in cyberspace. The game is an adversarial sequential decision making problem played with two agents, the attacker and defender. The simulation cyberspace is modeled as the graph network composed of nodes where the attacker and defender move. The state of the attacker or defender is the node where they are located. Each of attacker’s actions has an attack value, and the exploit succeeds when it is larger than the defense value of the defender. The disadvantage of this method is the two agents use reinforcement learning against each other and examined their effectiveness against learning opponents. This
research showed that we can modeled an environment to simulation the cyberspace, and the agents trying to attack the environment by reinforcement learning.

However, there are actually few environment like the previous work: in this environment, the attacker and defender will adapt and deal with each other in real-time. But according to Sharma et al. (2011), when the attackers achieved 62 of their attack goals, their attacks are detected. This means that attackers and defenders rarely compete in real time. Therefore, in this paper, we train the attack agents in the cyberspace environments that does not set the defense agents.

Both of the previous works have been validated the effects of reinforcement learning by the simulation. By this way, The second contribution of our work is to validate the effects of reinforcement learning in a simulation cyberspace environment. Our work embodies the previous works in the following points: the cyberspace configuration, the agent state, the agent actions, and the agent reward. We trained agents to learning the attack sequences steps in this simulation environment.

\subsection{Reinforcement Learning}
Reinforcement learning is commonly considered as a general machine learning model, it mainly studies how agent can learn certain strategies by interacting with the environment, to maximize long-term reward. RL is based on the Markov Decision Process(MDP) \cite{DBLPSutton98}. A MDP is a tuple $(\emph{S}, \emph{A}, \emph{T}, \emph{R}, \gamma$), which \emph{S} is the set of states and \emph{A} is the set of actions. $\emph{T}$($s_i$$\vert$$s_j$, \emph{a}): $\emph{T}\times\emph{A}$$\to$${R}$ is the reward after executing action $\emph{a}$ at stage $s_i$, and ${\gamma}$ is the discounting factor. We used $\pi$ to denote a stochastic policy, $\pi$($\emph{s},\emph{a}$): $\emph{T}\times\emph{A}$ $\to$[0,1] is the probability of executing action $\emph{a}$ at state $\emph{s}$ and $\sum_{\emph{a}\in\emph{A}}$ $\pi$($\emph{s},\emph{a}$)=1 for any $\emph{s}$. The goal of RL is to find a policy $\pi$ that maximizes the expected long-term reward. Besides, the state action value function is

\begin{equation}\label{first_equation}
	\begin{array}{l}
		{Q}^\pi(s,a)=E\left[ \sum_{t=0}^{\infty}{\gamma}^t R(s_t,a_t)|s_0=s,a_t\sim\pi(s_t) \right]
	\end{array}
\end{equation}

which $\gamma \in (0,1]$ measure the importance of future reward to current decisions.

For different policies $\pi$, they represent the possibility of different actions selected in the same state, and also correspond to different rewards. A better policy can select better action in the same state, to obtain more reward.

In traditional RL, the action-value function is calculated interactively, and will eventually converge and obtain the optimal strategy, mainly including Dynamic Programming, Monte Carlo Method and Temporal-Difference Learning. After deep learning was proposed, the deep reinforcement learning method formed by combining RL is the mainstream method at present.

In the following, we introduce the mainstream RL algorithm A2C and DDPG.

\textbf{Advantage Actor Critic (A2C)}: A2C is similar to A3C (Asynchronous Advantage Actor-Critic) (Mnih et al, 2016), only there is no asynchronous part. A2C uses the advantage function instead of the raw returns in the critic network, which can be used as a measure of how good or bad the selected action values and the averages of all actions are. Both A2C and A3C are learning algorithms using the advantage. The advantage is expressed in the following equation: $A(s, a) = Q(s, a) $ - $ V(s)$. The advantage is denoted as $A(s, a)$. We denote the state of the agent as $s$ and the action as $a$. $V(s)$ represents the pure value of the state $s$ ; therefore, $A(s, a)$ represents the pure value of action $a$. According to previous studies, the advantage can stabilize the learning. A2C lacks the asynchronous part but A2C performs better than A3C. Thus, in our paper, we use A2C as a baseline, so as to stabilizes and improves the learning progress.

\textbf{Deep Deterministic Policy Gradient (DDPG)}: DDPG \cite{lillicrap2015} is a learning method that integrates deep learning neural network into Deterministic Policy Gradient(DPG) \cite{Silverarticle}. Compared with DPG, the improvement the use of neural network as policy network and \emph{Q}-network, then used deep learning to train the above neural network. DDPG has four networks: actor current network, actor target network, critic current network and critic target network. In addition to the four network, DDPG also uses experience playback, which is used to calculate the target \emph{Q}-value. In DQN, we are copying the parameters of the current \emph{Q}-network directly to the target \emph{Q}-network, that is $\theta^{Q '}=\theta^Q$, but DDPG use the following update:

\begin{equation}\label{equation5}
	\left  \{
	\begin{array}{l}
		{\theta ^{Q'}} \leftarrow \tau {\theta ^Q} + (1 - \tau ){\theta ^{Q'}}\\\\
		{\theta ^{\mu '}} \leftarrow \tau {\theta ^\mu } + (1 - \tau ){\theta ^{\mu '}}
	\end{array}
	\right.
\end{equation}

where $\tau$ is the update coefficient, which is usually set as a small value, such as 0.1 or 0.01. And this is the loss function:

\begin{equation}
	L(w) =\frac{1}{m}\sum\limits_{j=1}^m(y_j-Q(\phi(S_j),A_j,w))^2
\end{equation}

In our experiments, A2C and DDPG were compared to our proposed method so that demonstrate the superiority of our method.

\section{Methods}

In this section, we introduced our problem definition, an overview of how we formulate the problem as a reinforcement learning (RL) problem, followed by a detailed description of the reward function, action, state, reward, policy and policy updates. At last, we introduced our proposed improved DDPG algorithm to solve this problem.
\subsection{Overview}

This section describes the proposed method, that is, the components of the training of the reinforcement learning agents. First, we define the state of the agent $s$. Second, we define the action $a$ selected by the agent in the cyberspace environment (how to select from the action lists). Third, we set rewards $r$ according to the result of the action $a$. Lastly, we introduced our method: how to improved the DDPG algorithm in this cyberspace environment. The agent accumulates the set of $s, a,$ and $ r$ observed from the learning environment as experience and proceed with learning. Figure 1 shows an overview of our method. We simulation an multi-domain cyberspace environment. Our method uses our proposed improved DDPG as the algorithm for reinforcement learning.

\subsection{Problem Definition}
In this work, our propose is discovering hidden attack paths in multi-domain cyberspace, and the objective is to discover the shortest hidden attack paths, thus the administrators can realize the weakest in the cyberspace, so he can take measures to enhance cyberspace security. The problem is defined as given a cyberspace environment, as Figure 1 show, we stored the security information in S2. Under the current cyberspace configuration, the attacker cannot access the S2 and get the security information. Our purpose is training an attacker agent by reinforcement learning to access the S2 and get the security information. If an attacker can access S2 and to get the security information, meaning the attack is successfully. The shortest hidden attack path define: The attacker can access S2 and to get the security information and the attack sequences steps is the minimum.

We treat the problem as a MDP, that is $M = (S,A,P,R,\gamma )$, where is $s \in S$  is the current state of the cyberspace,  $a \in A$ is an attack action that is currently available, $P$ is the probability of transitions between states, $R$ is the reward value after taking an action to reach the next state. $\gamma$ is the discount factor. For the transfer probability, it can be expressed as $p(\hat s|s,a) = p({S_{t + 1}} = \hat s|{S_t} = s,{A_t} = a)$. For the reward function, it can be formally expressed as $R(s,a) = E[{R_{t + 1}}|s,a]$.

In our setting, at the initial state, $s_0$, we have a RL agent as an attacker, and an configured cyberspace. The final state $s_t$ corresponds to an attacker attack successfully or not in limited steps. At each step, the RL agent will take an action to complete an attack step. Thus, $T$ is equal to the total number of attack steps. At each step $t$, the agent begins in state $s_t$, take an action $a_t$, arrives at a new state $s_ {t+1}$, and receives a reward $r_t$ from the cyberspace environment.

We define $s_t$ to be a concatenation of features representing the state at time $t$, including where the agent stand, which computer the agent operating, which services the agent can access and so on. We will describe it in detail in Section 3.4.

$s_{t+1}$ is the next state, which includes an updated representation containing information about the newly attacker location or the attacker obtain the newly service information.

The action space is all valid actions of the state $s_t$, which is the agent will take in state $s_t$. Action $a_t$ is the agent states space of the $s_t$ that was chosen by the agent.  We will describe it in detail in Section 3.5.

Our goal in this work is to discover the shortest attack path, subject to constraints on cyberspace security configuration or security equipment. Our final reward is the a fixed value divided by number of attack steps. We will describe it in detail in Section 3.6.

\subsection{Definition of Multi-domain Cyberspace}
With the deepening of the understanding of the concept of network, especially the proposal of the concept of cyberspace, more and more scholars realize the cyberspace is affected by multi-domain behavior. Cyberspace should be defined as integrated into multiple domains such as physical domain, information domain, network domain and social domain, and takes the interconnected information technology infrastructure network as the platform, transmits signals and information through radio and cable channels, and controls the actions of entities, with special emphasis on the multi-domain attribute of the cyberspace. In this paper, because social domain mainly involves the social relationship between network administrator and attacker, the social domain attribute of multi-domain cyberspace is not discussed in this paper, we define the multi-domain cyberspace have physical domain, information domain and network domain. The physical domain mainly describes the spatial information of the equipment, the network domain mainly describes the interface, path and action related to network transmission, and the information domain mainly describes the digital information in the cyberspace.

Besides, we have some security rules in this cyberspace, including physical domain security protection rules, network domain security protection rules and information domain security protection rules. Physical domain security protection rules mainly describe the methods to prevent illegal access in the physical domain and to prevent illegal personnel from entering a certain space. The network domain security protection rules mainly describe the methods to prevent illegal access in the network domain. In general, access control lists(ACLs), static routing, VLAN partition and other methods can be used to achieve network isolation. In this paper, we focus on ACLs, which are described as allowing data to flow through a port at a source address, a port at a destination address, and a service at a destination address. The information domain security protection rules mainly describe the methods to prevent illegal access in the information domain. The main method is to encrypt the information when it is stored or transmitted. Whether encryption is through symmetric cryptography or public key cryptography, a secret key is needed to decrypt the file.

We take a deep reinforcement learning approach to the discovering the hidden attack paths problem, where an RL agent (policy network) discover the attack paths in multi-domain cyberspace; once the RL agent attack the service successfully, we will give the RL agent an positive reward, and the final reward is the reward divided by number of attack sequences steps. If the RL agent failure to attack the service or exceeds the limited number of attack steps, we will give the RL agent an negative reward. RL problems can be formulated as MDPs, consisting of three key elements: states, actions and reward. We define the following elements respectively.

\subsection{Definition of Agent State}

States: the set of possible states of the multi-domain cyberspace, and the states include the RL agent's location, the device he is operating on, and the device permissions he has. The core of cyberspace states is attacker's permissions which he can obtain by series actions. In this paper, we discuss 9 types of attacker's permissions: Space-Enter, Object-Use, Object-Dominate, Port-Use, Port-Dominate, Service-Reach, Service-Dominate, File-Dominate, and Information-Know, respectively. Space-Enter, Object-Use and Object-Dominate is a physical domain's permission, the Space-Enter is meaning attacker enters a physical space, Object-Use is meaning attackers have permission to use a device, terminal or equipment, Object-Dominate meaning attackers can dominate an equipment, Object-Use and Object-Dominate are the differences between the former only can use the equipment with the current state of the equipment, while the latter can change the configuration of equipment. Port-Use, Port-Dominate, Service-Reach are network domain's permission, Port-Use is meaning the attacker can use the port to access network, Port-Dominate is meaning the attackers can change the state or configuration of a port, Service-Reach is meaning the service request information flow to achieve the service, but cannot use the service. File-Dominate is meaning the attacker can through the secure authentication service, to use the service. File-Dominate, and Information-Know are digital domain's permission. File-Dominate is meaning the attacker can read, delete and modify the file or even more. Information-Know is meaning the attacker obtain the security information, for example, administrator user name, administrator password, administrator key or so on.

In this paper, we set the attacker's cyberspace permission to the RL states, We set a state list, including the attacker's devices permissions in cyberspace. For example, if an attackers in the outer space, we set the attacker's  state as follows: Room A Space-Enter is 0, Room B Space-Enter is 0, Terminal A Object-Use is 0, Terminal A Object-Dominate is 0, Port B Port-Use is 0, Port B Port-Dominat is 0, et al. And then, the attacker have an action: enter room A, the the state set as Room A Space-Enter is 1, and other state value is also 0. When the attacker have an action, the states is also changing because the attacker's action.

\subsection{Definition of Agent Action}
Actions: the set of actions that can be taken by the agent and can change the cyberspace environment's states (e.g., enter a room, operate or control a computer, access a service by a port).

But in this cyberspace environment, action space is very large, and the attacker's actions is limited in a state, for example, if the attacker's state is in the outer space, he only have the enter-room or stay still two actions to choose, he don't have the operate a computer, control a computer or other actions, this is different from standard RL architecture, therefore, we will give some constraints in states when the attacker select actions, and we'll cover this in more detail in Section 3.7.

\subsection{Definition of Agent Reward}
Reward: the reward for agent taking an action in a state.

This section describes the reward setting. In our experiment of Section 4, the goal of the agent is to obtain the security information in S2 and obtain the key to decrypt the information. In that case, the first compromised is that the attacker how can obtain the Object-Use and Object-Dominate of S2, so as to get the password to decrypt the security information, that is the attacker obtain the Information-Know permission of the security information, it is means the attack is successfully. Because the multi-domain security rules, the attacker cannot access the S2 directily, in order to access the S2, the attacker will access the FW2 to modify the ACLs, to allow the attacker can access the server S1 and S2, then the attacker can access S2, if he obtain the security information successfully, he also back to modify the ACLs to restore the original state. Therefore, reward $r$ setting is as follows:
\begin{itemize}
	\item $r$=500000/$t$ if the attacker obtain the Information-Know of security information in S2, the $t$ is the attacker's successfully sequences steps;
	\item $r$=10 if the attacker obtain the File-Dominate of the security information in S2;
	\item $r$=5 if the attacker obtain the Object-Dominate and Service-Dominate of S2;
	\item $r$=5 if the attacker obtain the Object-Dominate and Service-Dominate of S1;
	\item $r$=1 if the attacker obtain the Port-Use of S1;
	\item $r$=1 if the attacker obtain the Port-Use of S2;
	\item $r$=-0.1 if the attacker have an action but obtain no permissions.

\end{itemize}
Reward $r$=-0.1 means punishment for the agents. The agents try to maximize the reward. Therefore, if $r$=-0.1 is set for each action, the agents try to reach the goal as soon as possible. This corresponds to a situation where attacker achieve the goal as soon as possible to minimize attack sequences steps. Besides, the episode ends when the agent attack successfully and the attack sequences exceeds 10000 steps.

\subsection{Improved DDPG Algorithm}

The main function of our model is to discover the hidden attack path by a RL agent under a certain cyberspace configuration.

The model takes DDPG algorithm, we are using an agent to represent the attacker. In the process of discovering the optimal hidden attack path, the agent first selects the action in the current state, which can change the environment and the agent's state. At the same time the agent will obtain certain reward or negative reward. Besides, the change in the agent's state enables it to perform other actions to obtain more rewards. As a result, the agent discover the hidden attack path in this cyberspace configuration by trial and error. The model is shown in Figure \ref{figure2222}.

The process of to find the shortest attack sequences steps to select the corresponding action \textbf{\emph{a}} according to the current state \textbf{\emph{s}} of the cyberspace, it also means find the corresponding policy mapping function $R(s) \to A$, to make the long-term reward of agent maximum. In this process, the policy can be divided into two categories, namely the deterministic policy and the stochastic policy, deterministic policy is for the state, the conviction of corresponding output(action). In general, the deterministic policy algorithm efficiency is high, but the lack of ability to explore and improve. Rather than the deterministic policy, the stochastic policy join corresponding random value, enables the stochastic policy to have certain ability of exploration. For our model, since the action value range is generally not large in practical problems, a deterministic policy is adopted to ensure better performance of the model.

The model is a standard RL model, through the study of the awareness of environment, the agent will take the action and get a reward, the goal of the agent is to maximize rewards, and then to further training of the agent. In this paper, we will to take DDPG algorithm and improved it, its main architecture as shown in Figure \ref{figure11}.

With the standard DDPG algorithm, it also consists of four networks and one experience replay memory. Among them, the experience replay memory is mainly responsible for storing the state transfer process of $ <s,a,r,s'> $, then, by means of small batch sampling, the corresponding transferred samples are extracted to train the corresponding neural network so as to avoid the strong correlation between samples. Among the four networks, there are two policy networks(Actor) and two \emph{Q}-networks(Critic), namely the online policy network, the target policy network, the online \emph{Q}-network and the target \emph{Q}-network. The policy network mainly simulates the attacker's policy through the deep neural network, which takes the current state as the input and the output as the corresponding action. The \emph{Q}-network is mainly used to estimate the expectation of the final reward value obtained if the policy is continuously executed after the current action is executed in a certain state. The input is the current state, the current action and the output is the \emph{Q}-value. If only a single neural network is used to simulate policy or \emph{Q}-value, the learning process is unstable. So in DDPG algorithm respectively, policy network and \emph{Q}-value network create copies of two networks, two networks are known as the online network, two networks are known as the target network, online network is the current training of network, the target network is used to calculate the training goal, and after a short period of time, the model of online networks parameter updates to the target networks, so as to make the training process is stable, easy to convergence.

We have improved the standard DDPG algorithm, which is different from the standard DDPG algorithm in two aspects:

\begin{itemize}
	\item  In improved DDPG algorithm, we introduced the multi-domain action selection module.

	Different from standard DDPG algorithm, the biggest change is that the introduction of multi-domain action selection module. In the standard DDPG, the actions which agent can choose in each state is the same. But in this environment, when the attacker select the attack paths in cyberspace, he have different alternative actions in each state. For example, if the attacker's state is in the outer space, he only have the action 'enter room' or 'keep still', but in standard DDPG algorithm, he have all the actions to select, obviously, when attacker in outer space, he cannot have the action 'control terminal'. In order to make DDPG algorithm can choose different actions in different states, joined the multi-domain action selection module. This module's input is online policy network's output, we called this is theory action $a_t$, then a linear change under this current state, and perform the actual action $a_t'$, the actual execution action $a_t'$ into multi-domain action execution module, get the corresponding reward $r_t'$. In the end, the corresponding actual execution of action $a_t'$ and the corresponding reward $r_t'$ return online policy network. Through this method, the reasonable choice of actions in different states can be realized.

	\item Second, the input of experience playback memory is different.

	In order to ensure that the multi-domain action selection conforms to the constraints of the actions on the state, the input of the experience playback memory is increased, not only by the online policy network to store the sequence $({s_t},{a_t},{r_t},{s_{t + 1}})$, that is, execute the action $a_t$ in the state $s_t$, get the reward value $r_t$, and convert to the next state to $s_{t+1}$. Moreover, since the corresponding relationship between the state and the action needs to be considered when selecting the action, it is avoided that the policy network chooses the action that is not feasible in the state. Therefore, when the policy network selects an inoperable action $a_t$ in the state $s_t$, it is not only necessary for the multi-domain action selection module to use a linear transformation to map it to a feasible action $a_t'$, in addition, relevant action sequences $({s_t},{a_t},{- \infty},{s_{t + 1}})$ need to be taken to indicate that actions $a_t$ are executed in the state $s_t$, and the subsequent state obtained is still $s_t$, and the reward at this time is a huge negative value, so as to ensure that relevant actions are not selected in the process of training the policy network.
\end{itemize}

In terms of network architecture, the two policy networks have the same architecture, whose input is the state of network and output is the action to be selected. Structurally, a RNN hidden node is added between the original DDPG input layer and the hidden layer. The transformed policy network is divided into 5 layers. The first layer is the input layer; The second layer is the RNN hidden layer, which contains 32 GRU nodes. The third layer and the fourth layer are the full connected layer, including 48 full connected nodes. The activation function uses the ReLu function. The fifth layer is the output layer, use the sigmoid function as the activation function, and finally output a multi-dimensional vector representing the multi-domain action that needs to be performed.

In addition, the two \emph{Q}-networks have another architecture, whose input is not only the state of the network, but also includes a multidimensional vector, representing the corresponding multi-domain actions, and the output is a scalar. The network is divided into four layers. The first layer is the input layer; The second layer and the third layer respectively contain 48 fully connected nodes. The activation function uses the ReLu function. The fourth layer is the output layer, which outputs a scalar and uses the linear function as the activation function, representing the \emph{Q}-value of the corresponding state and action.

\section{Experiments}

\subsection{Experiment Environment}

In this experiment, we modeled an cyberspace environment as our experiment data, to verify the effectiveness of our method. In this experiment environment, there are five spaces in total. The outermost space is the whole physical space, representing a region. P1 is the region where terminal located, P2 is the region where VPN equipment located, P3 is the location of the communication team, and P4 is the communication hub. There are 5 kinds of equipment, including computer(T1 and T2, respectively stored in P1 and P3), firewall(FW1, FW2, respectively stored in P3 and P4), sensor (D1, stored in P2), router (R, stored in P4) and switch (SW, stored in P4), server (S1, S2, stored in P4) and its equipment connection relationship as shown in Fig. \ref{figure336}. The security information is stored in S2. If an attacker can access S2 to get the security file and obtain its 'Information-Known' permission, meaning the attack is successfully. Also, our purpose is not only attack successfully, but also the attack sequences is shortest.

\begin{figure}[h]

	\centering
	\includegraphics[height=4.2cm, width=6.8cm]{336.jpg}

	\caption{Experiment Environment Topology}

	\label{figure336}

\end{figure}

In this network, there are 15 network services, as shown in Table \ref{tab111}

\begin{table*}
	\caption{Network Services in the Experiment Environment}
	\label{tab111}
	\begin{tabular}{ccccc}

		\toprule

		Web Service & Web Services's Role & Service Support Equipment & Service
		Dependent Port & Service Password\\
		\midrule

		T2\_manager & Remote Management equipment & T2 & T2\_E0 & None\\
		FW1\_manager & Remote management equipment & FW1 & FW1\_E2 &
		FW1\_password\\
		FW2\_manager & Remote management equipment & FW2 & FW2\_E2 &
		FW2\_password\\
		S1\_web & Web services in server S1 & S1 & S1\_E0 & S1\_web\_password\\
		S2\_web & Web services in server S2 & S2 & S2\_E0 & S2\_web\_password\\
		\bottomrule

	\end{tabular}
\end{table*}

In this environment, because of the firewall FW1 equipment are in need of
remote management, FW1-password remains in FW1, at the same time, due to the T2
maintains FW2 and S1, so T2 store password FW2\_password and S1\_web\_password,
in this environment, to ban other flow of information. But we can know, an
attacker can through the multiple domain joint attack, which can obtain the
security information stored on the server S2, a possible attack path is as
follows:

First, attacker enter space P2 and obtain the management service password
FW1\_password of firewall FW1;

Second, use device T1 or D1 to access the management service of FW1, add access
control list: allow T1 or D1 to access the management service of T2, that is
T2\_manager;

Third, get the password FW2\_password of firewall FW2 stored on T2 and the
password S1\_web\_password of service S1\_web through T2\_manager;

Fourth, use T2\_S1 port, access firewall FW2\_manager, add access control list:
allow T1 or D1 access service S1\_web and S2\_web;

Fifth, use T1 or D1 to access the service S1\_web and get the password
S2\_web\_password of S2\_web, at this point, the attacker's higher permissions
have been obtained.

At last, the attacker can use T1 or D1 to access the service S2\_web to get the
security information by the S2\_web\_password.

In this process, three key firewall security policy changes are involved: on
firewall FW1, T1 or D1 are allowed to access T2's management service
T2\_manager; On firewall FW2, allow T1 or D1 access to service S1\_web; On
firewall FW2, allow T1 or D1 access to the service S2\_web.

\subsection{Experiment Process}

During the experiment, an agent (attacker) is introduced, located in the outermost space, and then, in the cyberspace environment shown in Figure \ref{figure336}.

Each episode's termination condition is the attack sequence steps exceeds 10000. And we train 500 episode. If an attacker attack successfully, the attack sequence steps $n$ will be recorded, and he will go back to the outermost space, start looking for the attack sequences again, until the number of  attack sequences steps exceeds 10000 in an episode.

\begin{figure*}[h]
	\centering
	\subfigure[IDDPG Method]{\includegraphics[width=8cm]{IDDPGResult.png}}
	\subfigure[DDPG  Method]{\includegraphics[width=8cm]{TRAI11N.png}}
	\subfigure[A2C Method]{\includegraphics[width=8cm]{A2C11.png}}
	\subfigure[DQN Method]{\includegraphics[width=8cm]{DQNResult.png}}
	\caption{ Different Methods Reward Results }
	\label{fig4}
\end{figure*}
According to the RL model and improved DDPG algorithm, define the corresponding state, action, reward, etc. The relevant settings are as follows:

On the set of state, we set a length of 106 vector to represent a state of different position on the value of the attacker, respectively from the spaces, the ports, services or information, in setting a state vector, for example, if the attacker is in a certain space, the value representing that space is set to 1, otherwise it is 0. If the attacker can use a port, set the value representing the port to 1, otherwise it is 0; If the attacker is connected to a service, set the value representing the service to 1, otherwise 0; If the attacker obtain security information, set the value representing that information to 1, otherwise it is 0.

Attackers have different action in different states. We have introduced in Section 3.5. For example, when the attacker can dominate the management service FW1\_manage of FW1 in the current state, he can add the corresponding access control list for firewall FW1 in the current state. Otherwise, he cannot add the access control list for FW1.

In terms of the reward setting, different reward are set for the attacker according to the degree of the completion of the attack path. Among them, in an episode, the reward is set in Section 3.6.

\subsection{Baseline}
In order to verify the effectiveness of our proposed method, we propose three comparative RL methods to verify the effectiveness of our proposed method:

DQN method: DQN is a RL standard algorithm, DQN uses a neural
network to predict Q-value and constantly updates the neural
network to learn the shortest attack path. There are two neural networks in DQN. One is a relatively fixed network, called target network, is used to obtain the value of Q-target, the other is evaluate-network, used to get the value of Q-evaluate.

A2C method: A2C is a RL standard algorithm, we have introduce this method in section 2.3. This method similar to A3C, but there is no asynchronous part.

DDPG method: DDPG is also a RL standard algorithm, we have introduce this method in section 2.3.

Improved DDPG method(IDDPG): this method is this paper proposed.

\subsection{Results and Discussions}

\begin{table}[!htbp]
	\caption{Attack Sequences Steps in Different Security Rules}
	\label{extremos456}
	\centering
	\begin{tabular*}{\hsize}{ p{2cm} p{1.2cm} p{1.2cm} p{1.2cm} p{1.2cm}}
		\hline
		security rules  & 3 & 4 & 5 & 6 \\
		\hline
		DQN & 3533  & 4323 & 7231 & 8313 \\
		A2C & 3222 & 4213 & 7332 & 8812 \\
		DDPG & 2434 & 4765 & 7240 & 8551 \\
		IDDPG & 995 & 3322 & 6321 & 7233 \\

		\hline
	\end{tabular*}
\end{table}
Under the above conditions, learn 500 episodes in environments, and we recorded these results:

Result A: We added some security rules, to compare the average successfully attack sequences steps, that is, we added up all the successfully attack sequences steps and divided by the number of successfully attack.

From Tabla \ref{extremos456}, we added security rules, respectively the number of security rules is 3, 4, 5, 6. In general, when the more security rules there are, the attack sequences steps is longer, even so the attacker cannot attack successfully. From the Result A, Tabla \ref{extremos456}, it also confirm the opinion. As we can see, as the number of security rules increases, the attack sequences steps get longer and longer in all methods. Therefore we can confirm, if the cyberspace have no security rules, the attackers more easily to the obtain their target, this also proved that the important of the cyberspace configure. Result A shows the we can use the attack sequence steps to assess the level of cyberspace security. That is, if the attack successfully sequences is shorter, the cyberspace security is low, that is the cyberspace configuration is bad. On the contrary, if the attack sequences steps is longer, that is the cyberspace configuration is better. Therefore, in our experiment, We can use the attack successfully sequences steps to represent the cyberspace security.

Following the experiment, we use the number of security rules is 3, to verify our proposed method is useful and efficient.

Result B: The reward the attacker obtain.

We can see the Result B form Figure 5, first of all, with the increasing the number of training, \emph{R} present a slow upward trend, until finally tend to be convergent, it is in a learning process of RL, shows that the proposed model conformed to the characteristics of reinforcement learning, can discover the hidden attack paths in the cyberspace environment, and discover the shorter hidden attack paths constantly, so as to verify the effectiveness of the proposed model is presented in this paper.

Secondly, Figure 5 represent four methods obtain the reward, we can see our proposed method IDDPG have the highest reward than other methods; This may account for the following reasons: our proposed method has more successfully attacks, and it has shorter attack sequences steps.

We define the Attack Successfully Number is the attacker attack successfully in an experiment, the Minimum Steps is the in an episode the shortest attack successfully sequences steps.

Result C: The Attack Successfully Number;

Result D: The Minimum Steps;

Result C and D is recorded in Tabla 3.

\begin{table}[htbp]
	\caption{Experiment Results}
	\label{extremos45}
	\begin{tabular*}{\hsize}{lcc}
		\hline
		& Attack Successfully Number & Minimum Steps  \\
		\hline
		DQN &  1203 & 1800   \\
		A2C & 2533 & 2400   \\
		DDPG & 3711 & 2100   \\
		IDDPG & 6705 & 750   \\

		\hline
	\end{tabular*}
\end{table}

Thirdly, we can see the Table 3, in Result C and Result D, our proposed method have the minimum attack sequences, it is shown that our proposed method can discover the shortest attack sequences steps under the same conditions, also, the attack successfully number is also maximum. From the experiment results, our proposed method is superiority than existing methods.

From the experiment result, first we confirm the cyberspace configuration can affect the cyberspace's security, then we can use the attack sequences steps as the measure of the cyberspace security; Following, we use the different method in the same cyberspace environment, we compared the reward, the Attack Successfully Number, the Minimum Steps, our proposed method have the higher reward in the experiment, and the most Attack Successfully Number, the minimum Minimum Steps, from this experiment results, we can conclude our proposed method can attack more successfully, to get more reward in an episode. As a result, the attacker will have shortest steps, will discover the shortest attack sequences steps in cyberspace, then will provide reference to the administrator where the weakest in the cyberspace. In a word, our proposed method is superiority than existing methods.

\section{Conclusion}
In order to discover the vulnerability of the cyberspace, we proposed a learning-based approach to discover the hidden attack path. Meanwhile, we has been learn about the cyberspace weakness metrics, that is the attacker attack successfully sequences steps, and finally has carried on the experimental verification. This method can comprehensively consider the mutual influence of the multiple domain configuration in the cyberspace, and can take an intelligent method to analysis the weakness of the cyberspace, which has a strong practical value.

This paper analysis a typical cyberspace environment and applies the reinforcement learning method to discover the hidden attack paths in configured cyberspace, which has achieved better results. However, the cyberspace environment in this paper is limited. In the next step, we hope to apply the reinforcement learning to more cyberspace operation and maintenance management, and achieve better results.

\subsection{State Representation}
Figure \ref{fig:architecture} shows the overall RL architecture. Our state contains information about the candidate node (macro) to be placed: the embedding of that node, embedding of the netlist, and some metadata about the netlist including the total number of wires, macros, and standard cell groups. In the following we discuss how we generate node and graph embeddings for this problem.

\subsection{Policy architecture} In this section, we propose a new graph neural network method that embeds information about the netlist. The role of graph neural networks is to distill information about the type and adjacency structure of a node within a large graph into low-dimensional vector representations which can later be used in various downstream tasks such as node classification \citep{}, device placement \citep{}, and link prediction \citep{}. In a recent work, graph neural networks have been used to predict DRC from a netlist once the placement is done and before doing detailed routing \citep{}.

In the following, we propose a new graph neural method that suits the placement problem. The input to this graph neural network is the netlist's adjacency graph and node types (macros vs. standard cell groups). We explain the steps in Algorithm \ref{ref:algorithm}. The input to the algorithm is information about the node features including the type (macro vs. standard cell cluster), node width, height. We also get node neighborhood information as input. We first create a vectorized representation of the nodes by concatenating the node features. We then repeatedly perform the following updates: first each edge updates its representation by applying a fully connected network on an aggregated representation of node intermediate embeddings. We then for each node, take the mean of the embeddings of its adjacent edges and update the node representation.

\begin{algorithm}[H]
\label{ref:algorithm}
\SetAlgoLined
\KwResult{}
 initialization: netlist's adjacency graph and node types (macros vs. standard cell groups)
 Node are represented with $v_i's$ for $1<=i<=N$\;
 \While{Repeat until convergence}{
  $e_{ij} = fc^1(concat[fc^0(v_i) | fc^0(vj) | w^e_{ij}])$ \;
  $v_i = mean_{j\in N(v_i)}(a_{ij})$ \;
  }
 }
 \caption{Edge based graph neural network embedding.}
\end{algorithm}

\subsection{Action representation} As described earlier, for policy optimization purposes, we convert the canvass into a $m\times n$ grid. Thus, for any given state, the action space (or the output of the policy) is the probability distribution of placement (of the current candidate macro) over the $m\times n$ grid. The action is the argmax of this probability distribution.

\textbf{Density Constraints}: A feasible node (macro or standardd cell group) placement should meet the following criteria: the density of the placed items on each grid cell should not be more than a target density threshold ($max_{density}$. To meet this criteria, during each RL step we calculate the current density mask, a binary $m\times n$ matrix that represents grid cells onto which we can place the center of the current node without violating the density threshold criteria. Before choosing an action from the policy output, we first take the dot product of the mask and the policy output and then take the argmax from the feasible locations. This approach inherently disables the creation of macros to overlap, or dense standard cell areas.

\subsection{Policy Update: Training Parameters $\theta$}
To train the parameters of the policy, we use Proximal Policy Optimization (PPO) \citep{}.