Partial Gated Feedback Recurrent Neural Network for Data Compression Type Classification

Owing to the widespread use of digital devices such as mobile phones and tablet PCs that are capable of easily viewing contents, the number of digital crimes committed using these digital devices has increased. One of the most common digital crimes is to hide the header information of the compressed data, which makes the user’s data unusable. It is difficult to restore original data without the header because header contains the compression type. In this paper, we propose a Partial Gated Feedback Recurrent Neural Network (PGF-RNN) for the identification of lossless compression algorithms. We modify the gated recurrent units to improve the correlation of layers by grouping the fully-connected layers to effectively determine the characteristics of the compressed data. We emphasize on the temporal features, which consider a wide range of data, and spatial features from fully-connected layers to extract the feature vectors of each compression type. To improve the performance of the proposed PGF-RNN, we apply post-processing that considers the frequency of bit sequences on some compression types with similar compressed data. The proposed method is evaluated on 31 well-known lossless compression algorithms of the Association for Computational Linguistics dataset. The average top 1 accuracy of the proposed method is 92.63%.

I. INTRODUCTION
As the use of digital devices has increased, data exchange has become more convenient, and various types of information can now be efficiently stored and transmitted by data compression [1]- [3]. However, digital crimes are also increasing daily owing to an increase in the data exchange. One of the most common digital crimes is to hide the header information of the compressed data, rendering the data unusable until a financial demand is met. This is effective because the header contains information regarding the compression type, and thus the compressed data cannot be used when the header is modified or removed.
In digital crimes, users are often unable to restore their data because the data header is eliminated, and there is no clear solution for restoring the data header. When the header is modified or removed, restoring the original data is nearly The associate editor coordinating the review of this manuscript and approving it for publication was Sabah Mohammed . impossible because it is extremely difficult to determine the compression type. If a method is proposed to reduce the number of candidate compression types for compressed data, the compressed data could potentially be restored even without the header of the compressed data. Therefore, we propose a compression type classification method that uses only the overall characteristics of compressed data.
In previous studies [4]- [8], the proposed methods used feature vectors of the compressed data by extracting various hand-crafted features by conducting tests, such as a frequency test and runs test. While these methods can be applied easily, there is a limitation in expressing the data because it is difficult to increase the dimensionality of the feature vector using only the hand-crafted features. To overcome this problem, some studies have suggested methods that use deep learning to represent the characteristics of the compressed data, rather than extracting hand-crafted features.
Lossless compression algorithms compress data using a prestored index for a specific data sequence or using 6 of different length depending on the frequency of a sequence's appearance. Because of these characteristics of lossless compression, it is important to understand the data entirely to determine the frequently used text sequences. Additionally, generating features that can represent each compression type is difficult because it depends upon various parameters, such as a prestored index and sign of a substituted sequence. Recently, a convolutional neural network (CNN)-based method of distinguishing dictionary-based compression algorithms has been proposed [9]. In general, it is difficult to apply to text data that cannot be controlled by the scale (length), unlike image or video, because the input size of the CNN is fixed. To overcome this limitation, a spatial pyramid pooling (SPP) layer is used for extracting a fixed-sized features vector of each text data. However, using a CNN that partially extracts features from entire bit sequences is not an efficient method. Since lossless compression algorithms check the entire data when indexing for compression, the characteristics of compression types cannot be analyzed properly if the feature extraction is performed partially as CNN. Although the method uses various filters of SPP layer to extract fixed-sized feature vectors, it cannot represent the characteristics of each compression type by sliding a fixed-sized window. The average accuracy of CNN-based method as low as 80% proves this.
To know the characteristics of each lossless compression type, it is necessary to consider the entirety of the text data, rather than only a specific part of the data. Therefore, we apply a recurrent neural network (RNN), which is generally used for voice or text data. Unlike CNN, RNN considers the temporal dimension, enabling it to obtain compression type characteristics considering a wider area of data. However, in the case of a conventional RNN, only temporal features of data can be reflected, rather than including spatial features, because its structure has one-to-one matching for the connection between hidden states in each timestep. In other words, it is possible to find the relationship between the values within a certain interval of data, but it is difficult to determine the overall relationship of the data.
In this paper, we extract features that account for the relationship between bit sequences instead of extracting local features. Therefore, we propose a Partial Gated Feedback Recurrent Neural Network (PGF-RNN) that reflects the characteristics of lossless compression algorithms, such as structural features, indexing methods, and the organization of bitstreams. By extracting the temporal features of bitstreams using RNN, we handle the features of lossless compression algorithms effectively. In addition, the PGF-RNN has a structure in which several hidden states of each layer are grouped and fully-connected to hidden states of the next layer. By enhancing the correlation between layers through the fully-connected hidden states of each group, we generate a feature vector optimized for the classification of lossless compression algorithms by extracting both the temporal and spatial features of compressed text data. Finally, the compressed data are analyzed in bit units after performing PGF-RNN to increase the classification accuracy of the proposed method. For some entropy-based lossless compression types, such as Shannon, Shannon Fano Elias, and Huffman, it is insufficient to analyze using only the form of bit sequences because the encoded data are generated similarly. Therefore, the overall performance of the proposed method is improved by reclassifying misclassified data through a frequency test of input bit sequences. The main contributions of this paper are summarized as follows: 1) An RNN-based architecture, which extracts the temporal features of compressed text data in consideration with the characteristics of lossless compression algorithms, enables the improved performance of compression type classification; 2) The architecture of PGF-RNN, which groups the hidden states of each timestep and fully-connects them to the group of next timestep, helps to generate temporal and spatial features of the compressed data to more effectively determine the characteristics of the compression algorithms; 3) To demonstrate the validity and performance of PGF-RNN, we present a comparison of compression type classification accuracy between the gated recurrent unit (GRU)-based method and PGF-RNN and an analysis of the feature vectors. The rest of the paper is organized as follows. In Section II, we present an overview of lossless compression algorithms and two basic methods for PGF-RNN. In Section III, the proposed PGF-RNN and post-processing are described. In Section IV, the analysis of the proposed architecture and performance comparison between the proposed and existing models are given. Conclusions are presented in Section V.

II. RELATED WORK A. LOSSLESS COMPRESSION ALGORITHMS
There are two types of compression methods, i.e., lossy compression and lossless compression [10]- [13]. Lossy compression has the advantage of a high compression rate, but it has the disadvantage of being unusable for voice or text data because they are sensitive to data loss. In comparison with lossy compression, lossless compression has a low compression rate but has the advantage of no data loss during compression. Usually, the compression methods used to generally transmit data are.7z or.zip, which are examples of lossless compression. There is no difficulty in restoring images or videos to a state that is comparable to the original because the data lost from images or videos are not critical even if lossy compression is used. However, in the case of text data, it is difficult to estimate some lost data through surrounding values. Thus, since compressed text data can be restored differently from the original data, lossless compression is predominantly used in everyday life.
There are two types of lossless compression methods, i.e., entropy-and dictionary-based compression. Entropy-based compression is a method of compressing continuous data values by defining them as a shortcode for reducing the length of data. Entropy specifies the amount of essential information that can be considered redundant and compressed to determine the lowest bit value of data occurring during communicating or writing of the given information. Then, it is proved that the limit of compression is defined as entropy to prevent the loss of original information. In other words, irrespective of how ideal the compression method is, the data size cannot be smaller than its entropy. Entropy-based compression has compression types that are affiliated with Shannon [14], Huffman [15], [16], and Golomb [17]. One of the general compression types in entropy-based compression is Shannon encoding. Shannon encoding follows the mechanism below. 1) Find the probability of each symbol and use them to find the code length and sum of probabilities. 2) Convert the sum of probabilities to binary and assign each symbol following a code length. A symbol S i is the i th sequence of the data, and the probability p i of each symbol can be obtained as the number of bits of the entire data divided by the number of appearances of each symbol. The code length for each symbol can be calculated as follows: where l i represents the code length for each symbol and x denotes a ceiling function that rounds x up to the next integer value. Therefore, the method uses shorter code lengths for symbols that are more frequent over all the data. Table 1 shows an example of Shannon encoding. If there are 6 symbols and their probabilities p i are 0.36, 0.18, 0.18, 0.12, 0.09, 0.07, the code length of each symbol can be obtained using (1). For obtaining the binary values of each symbol as shown in the fifth column of Table 1, p i is listed in ascending order and the cumulative sum of p i is calculated. From the obtained binary values, the binary code is decided by cutting off the decimal point as the code length of each symbol. In contrast, dictionary-based compression is a method of indexing a specific character sequence rather than consecutive or overlapped characters and then converting the entire bitstream based on predefined indices. Typically, algorithms using the dictionary-based compression method are kinds of Lempel-Ziv algorithms. Among the various Lempel-Ziv series, we describe the process of encoding and decoding of using LZ77 as shown in Fig. 1. LZ77 is an internal dictionary-based compression algorithm proposed in 1977 by Lempel and Ziv [1]. LZ77 compresses data by indexing the overlapping parts in a predefined sliding window to the dictionary and then replacing repeated sequences with the relative positions of the existing text sequences. Sliding windows comprise a search buffer that is used as a dictionary and look-ahead buffer that performs encoding using the search buffer. The output of the encoded data consists of three parts, i.e., an offset indicating the relative position of the overlapping text sequences, length of the overlapping string, and codeword. The encoding process of LZ77 is as follows: 1) Set the size of the search buffer and look-ahead buffer.
2) Determine if there is any redundancy between look-ahead buffer and search buffer. 3) If redundancy exists between the look-ahead buffer and search buffer, the position of the duplicated sequence is represented by the offset and length values. Then, the character after the duplicated part is set as the codeword. In this case, the offset is a value indicating the first part of the duplicated string, and the length indicates the length of the duplicated string. 4) If there is no redundancy with the search buffer, the offset and length values are set as 0, and the codeword is the first character of the look-ahead buffer. 5) The offset, length, and codeword determined by steps 3) and 4) are stored as tuples. Then, the sliding window is moved by the length of the encoded data. 6) By repeating steps 2) to 5), the encoding is performed. The encoding processes of the two previous compression algorithms are those of Shannon and LZ77, which are the most representative compression algorithms in entropy and dictionary-based compression, respectively. The entropy-based compression algorithm is a method of making codes according to the frequency of the bits, whereas a dictionary-based compression algorithm is a method of generating codes according to a dictionary which depends on bit alignments. The two lossless compression schemes have different compression schemes, and thus the characteristics of the compressed data are also different. Fig. 2 shows the visualization of the feature vectors of Shannon and LZ77 obtained through GRU for arbitrary compressed text data. It is an image of all values obtained from hidden states in each layer. The x-and y-axis represents the number of hidden states in each layer and the number of layers in the GRU, respectively. This shows the difference between the two different compression algorithms. As shown in Fig. 2, in the case of Shannon, most of the hidden states of each layer are activated as indicated by the red box. In contrast, in the case of LZ77, only a few hidden states of each layer are activated, as indicated by the yellow box, and are gradually deactivated. When checking feature vectors of other compression algorithms except for Shannon and LZ77 as shown in Fig. 2, there are a big difference between the feature vectors of entropy-and dictionary-based compression algorithm. The parameters of the entropy-based compression algorithm are determined by checking through the overall data sequences. In contrast, the dictionary-based compression algorithm only has to know the index stored in the dictionary, which means that it only needs to activate in front of the layers. Therefore, deep learning-based lossless compression type classification is possible because the feature vector for each compression algorithm differs due to the characteristics of the compression algorithm. In total, we used 31 well-known lossless compression algorithms, comprising 16 dictionary-and 15 entropy-based compression algorithms, as shown in Table 2.

B. RECURRENT NEURAL NETWORK
RNN is a kind of neural network that forms a circular structure by connecting hidden states using directional edges. It is mainly used for sequential data, such as speech recognition, speech enhancement, and text classification, because the structure of the network can accept data regardless of the length of the data sequence [18], [19]. For an explanation of RNN, let h t be the hidden state at the t th timestep. Then, h t can be represented as follows: where x t is input at the t th timestep, h t−1 is hidden state at the (t − 1) th timestep, φ is usually a logistic sigmoid function or a hyperbolic tangent function, W is weight matrix suggesting how much information is obtained from h t−1 , and U is the weight matrix suggesting how much information is obtained from x t . RNN learns the data characteristics by determining the given data by a timestep. Although RNN has the advantage of understanding the flow of the data, the vanishing gradient problem may occur, meaning the gradient value gradually decreases during back-propagation as the number of h increases. GRU has been proposed to avoid this problem.

C. GATED RECURRENT UNITS
GRU was proposed by Bengio et al. in 2014, and evolved from RNN [20], [21]. For reducing the power of gate computation, the GRU structure is lighter than that of long short-term memory, which is also used for solving the gradient vanishing problem [22]. The value of the hidden state at the t th timestep of the GRU is expressed as: The update gate z t provides the information for updating the current state by reflecting the ratio of previous and current information. z t is calculated using h t−1 and x t : where W z and U z are the weights which represent the extent of reflection of x t and h t−1 at update gate z t , and σ is a VOLUME 8, 2020 hyperbolic tangent function.h t in (3) is calculated as: where is the Hadamard product [23]. The main difference from (2) is that the GRU has a reset gate r t to determine about how much the past information is reflected in the current state. r t is calculated using h t−1 and x t : where W r and U r are the weights that represent the extent of reflection of x t and h t−1 at reset gate r t , respectively.

III. PARTIAL GATED FEEDBACK RECURRENT NEURAL NETWORK
A. THE STRUCTURE OF PGF-RNN PGF-RNN for lossless compression type classification is based on GRU. PGF-RNN improves the conventional GRU by grouping fully-connected layers between timesteps [24]. Fig. 3 shows the structural difference between the conventional GRU and the proposed PGF-RNN. As shown in Fig. 3(a), conventional GRU takes the value only from h j t−1 and reflects the value in h j t . However, PGF-RNN transmits the information of several hidden states from the (t − 1) th timestep to the t th timestep using the partial fully-connection of hidden states. Gate g i→j reflecting fully-connected layers between timesteps can be expressed as: where the superscript i→j denotes the interaction between the i th layer at the (t − 1) th timestep and j th layer at the t th timestep, superscript i (5) ⇒j (5) denotes the fully-connected layers in a group between two timesteps, and h i (5) t−1 contains five layers, h i t−1 , · · · , h i+4 t−1 . In PGF-RNN, it is used as a successive fully-connected structure of five hidden states at the (t − 1) th timestep and five hidden states at the t th timestep for improving the layer to layer interaction and the time interaction as shown in Fig. 3(b). In a conventional GRU, the information from the (t − 1) th timestep is simply obtained through U h t−1 as in (5). However, in the PGF-RNN,h j t , which reflects g i→j , is calculated as: where H is the number of hidden layers andh j t is the new candidate memory of the content of the j th layer at the t th timestep. The amount of information obtained from the group of hidden states of the previous timestep can be obtained by applying g i→j . g i→j represents the connection between timesteps toŨ , which means the extent of reflection of x t in (5). Therefore, a wide range of data can be understood for the extraction of the characteristics of input data by sharing the information about a plurality of layers using PGF-RNN.
The specific parameters used to implement PGF-RNN for lossless compression type classification are as follows. The number of hidden states in each timestep is 125, and the number of cells (i.e., the number of timesteps) is set to 50 to learn the information at each timestep sufficiently. Compressed text data are converted to decimal after separating data by 8 bits. The reason why the bitstream is divided into decimal codes is that it is difficult to know the characteristics of the compression algorithms because the variation between bits is constant when binary values are used. The compressed text data converted to decimal are normalized and are finally used as the input of the network, as shown in Fig. 4. The dimension of X t is the value resulting from dividing the total length of the decimal value converted from the compressed text data  by the number of cells, which is 50. It means that PGF-RNN accepts text data of arbitrary length. We also used the 1 × 125 vector, which is the concatenated output Y t , as the input of the fully-connected layer. The class label for each data is a 1×31 dimensional one-hot vector. The probability of classifying compression types for the data is obtained and we finally determine the compression type with the highest probability.

B. THE FEATURES USED FOR PGF-RNN
The proposed PGF-RNN improves the performance of compression type classification by extracting feature vectors differently for each compression type, unlike the conventional CNN-and GRU-based methods. The feature vector for the compression type can be analyzed for two aspects: temporal feature and spatial feature. The temporal feature is not considered to be obtained by simply referring to some surrounding values such as CNN, but instead considers the sequence of text data by obtaining the characteristics of a wider region. The spatial feature results from the structural difference between the GRU-based method and PGF-RNN. The GRU-based method has one-to-one connections between the hidden states of each layer, but in the proposed PGF-RNN, it has partially fully-connected. It can be said that PGF-RNN gives more consideration to the correlation between layers. By analyzing both aspects, we show the structural validity of PGF-RNN.

1) TEMPORAL FEATURES
The temporal feature is a feature vector that considers the overall values of the compressed data by extracting the characteristics referring to a wider region. Fig. 5 shows the cross-correlation matrix whose values are the average of cross-correlation between feature vectors of each compression algorithm learned by the CNN-based method, GRU-based method, and PGF-RNN. In the cross-correlation matrix, a lighter color indicates that two feature vectors are similar, whereas a darker color indicates that the correlation of two feature vectors is lower. The diagonal line in the matrix is always bright because it represents a correlation value between the same feature vectors. In other words, it is ideal that the cross-correlation matrix has only a bright diagonal line. The difference in performance according to the usage of temporal features is shown by identifying the correlation between feature vectors of each compression algorithm through the cross-correlation matrix. As shown in Fig. 5(a), there are many brightly colored areas in addition to the diagonal line in the case of determining compression algorithms using the CNN-based method. This means that the CNN-based method is not good for compression type classification because the feature vectors of each compression algorithm are not distinguishable. In Fig. 5(b), the feature vectors of some compression types have high correlations that show brighter colors than the CNN-based method. However, as shown in Fig. 5(c), the correlation of feature vectors of each compression type is significantly lower using PGF-RNN than that using the GRU-based method. This result shows that the features of lossless compression algorithms are effectively extracted by considering both temporal and spatial features by exchanging information between layers through partially gated feedback.

2) SPATIAL FEATURES
As shown in Fig. 3, the structure of the conventional GRU can show the association between bit sequences because hidden states in the same order are connected between layers. However, there is a limitation to the association between neighboring sequences because it uses a concatenation structure. In contrast, PGF-RNN has a structure in which several hidden states belonging to the same layer are assembled in one group and are fully-connected with groups of other layers. As shown in Fig. 6, the final outputs of RNN are obtained by passing through several concatenated cells. We analyze the cross-correlation of the output Y i at the i th cell (i ∈ [1, · · · , N ]) that can be obtained from each cell to take advantage of the structural advantages of PGF-RNN using fully-connected layers. Fig. 7 is a cross-correlation matrix between the conventional GRU and the output of each cell of the PGF-RNN. The x and y axes of each matrix represent the number of each compression type defined in Table 2. Additionally, the cross-correlation matrix is calculated by the mean values of cross-correlation between the {Y 1 , Y 2 , · · · , Y N } obtained from each data of each class. As shown in Figs. 7(a) and 7(b), the two cross-correlation matrices make a distinct difference. Higher the correlation between Y i is indicated with red, whereas lower correlation is indicated with blue. Additionally, the larger the red area around the diagonal line, the greater the number of cells affected by one cell. In the case of PGF-RNN, the red region is wider around the diagonal line than that in the GRU-based method. Fig. 7(c) shows the difference of correlation matrix values of Figs. 7(a) and 7(b). This clearly shows the performance difference between the GRU-based method and PGF-RNN. The more the diagonal line goes away, the higher the difference of correlation values is. In other words, it can be seen that the PGF-RNN extracts the characteristics of the compression type by considering the front and back relationship of the compressed data more widely than the conventional GRU by connecting being fully-connected between each timestep.

C. POST-PROCESSING
Entropy-based compression types generate the code of each symbol S i for encoding according to the frequency of symbols in the given data. When the functions F i and l i are similar, which are the functions to help p i to make a binary form and determine the length of the code, the encoding results are similar. Fig. 8 is an example of encoding with Shannon and Shannon Fano Elias, which are the basic methods of entropy-based lossless compression types. As shown in Fig. 8, the codes of Shannon and Shanno Fano Elias are similar because they have similar functions F i and l i . For this reason, the classification accuracies of some entropy-based compression types are very low. In particular, the top 1 classification accuracies of Shannon and Shannon Fano Elias are more than 40% lower than those of other compression types. Shannon is most often misclassified as Shannon Fano Elias and Shannon Fano Elias is often confused as Shannon. In addition, both compression types are misclassified as Huffman. Huffman creates a code tree by combining the two symbols with the smallest probability and assigns code to each symbol, as shown in Fig. 9. Huffman has a different encoding method from Shannon and Shannon Fano Elias. However, there is a high probability that the symbol code is generated similarly, as shown in Fig. 9(b), because the entropy-based compression type has a characteristic that the symbol with the smallest probability has the longest code.  Therefore, high-performance compression type classification cannot be performed using only deep learning-based PGF-RNN for some misclassified compression types.
In this paper, post-processing is used for reducing the misclassification to reclassify some blind data after primary classification with PGF-RNN. The overall flowchart of compression type classification with post-processing is shown in Fig. 10. Firstly, the compressed text data T is estimated its class C using the PGF-RNN. If the class C is not in N = {Shannon, Shannon Fano Elias, Huffman}, T is C compression type. If C is in N , then the text data T is reclassified by the Frequency test. The frequency test makes the data be reclassified based on the ratio R of bit 0 in the entire bit sequences which are generated as shown in Fig. 4. Reclassified class C (T) is defined as follows: We use the average values obtained from other text databases as the standard for dividing the three compression types. The top 1 classification accuracies of misclassified compression types N have increased by more than 8% on average using post-processing. This result is shown in Table 5.

IV. EXPERIMENT A. ENVIRONMENT
We used the Association for Computational Linguistics dataset with more than 100,000 text files to evaluate the performance of our proposed compression type classification method. Using the open-source code of each lossless compression algorithm, all the text files were compressed individually with 31 lossless compression algorithms. The total number of training data is 1,240,000 text files (40,000 text files per compression type), and the number of testing data is 155,000 text files (5,000 text files per compression type). The batch size is 620 (20 files for each compression type) and the learning rate is 0.001.

B. THE PERFORMANCE OF PGF-RNN
The experiments were conducted in two directions to evaluate the proposed method. One is to show the superiority of feature vectors obtained through PGF-RNN. We analyze the results of the support vector machine (SVM) and k-nearest neighbor (KNN), which are trained using feature vectors from the CNN-based method [9] and PGF-RNN. The other experiment is to measure the top 1 and top 3 accuracies of the compression type classification of PGF-RNN. It shows the performance of the proposed method as a compression type classifier by comparing it with the GRU-based method.

1) THE SUPERIORITY OF PGF-RNN's FEATURES
We proved the superiority of the feature vectors obtained from our proposed method using SVM and KNN algorithms. The feature vectors for training SVM and KNN are obtained from the last layer of each method. By applying the two basic classification algorithms to each feature vector and comparing their accuracy, we checked whether the feature vectors obtained through each machine learning method are  Table 3.
suitable for lossless compression type classification, i.e., how well each feature vector reflects the characteristics of the compression type [25]. Table 3 shows the top 1 accuracy of classification by learning SVM and KNN using the feature vector of the last layer of two methods. Since the classification accuracy of each compression type is similar between the CNN-based and proposed methods, the characteristics of the compression type reflected by the feature vector of the two methods are similar. However, the accuracies of some compression types such as Arithmetic coding (2), LZ77 (12), LZSS (19), SNAPPY (28), and Tunstall (29) of SVM and KNN based on the proposed method are more than 10% higher than those of the CNN-based method. It can be interpreted that the CNN-based method does not reflect the characteristics of various compression types, unlike the proposed method. Additionally, it can be seen that the average classification accuracy of the two methods differs by more than 10%. To determine the characteristics of the compression type, it is important to understand the relationship between bit sequences. The proposed method has a structure that increases the correlation between each timestep. It means that this structure is better for understanding the characteristics of the compression type by learning about the data. Fig. 11 shows the classification results of the CNN-based and our proposed method as confusion matrix heatmaps. A confusion matrix, also known as an error matrix, shows performance for statistical classification. Each row of the matrix represents the predicted class and each column of the matrix represents the actual class. If the matching ratio between the predicted class and the actual class is high, the diagonal line of the confusion matrix looks clearer. In contrast, if the probability that the predicted class is not similar to the actual class, the non-diagonal parts become more visible. As shown in Fig. 11(a), although most of the compression types look to be properly classified, compression types, which have the numbers 1, 2, or 25, are not properly classified by comparing with the confusion matrix heatmap of the proposed method of Fig. 11(b). Additionally, there are more misclassified data using the CNN-based method than using the proposed method including the type numbers 1, 2, and 25. The average accuracy of both methods can also show this analysis of confusion matrix heatmaps.

2) RESULTS OF COMPRESSION TYPE CLASSIFICATION
We proved the performance of the proposed method by comparing the classification accuracy of the proposed method with that of the GRU-based method, which is a base of our method. Furthermore, we proved the post-processing efficiency by showing the average top 1 accuracies of some entropy-based lossless compression types with and without post-processing. Table 4 shows the average accuracy of classification for each compression type using the GRU-based method and the proposed method. The average top 1 accuracy of the proposed method is 92.63%, whereas that of the GRU-based method is 86.36%, which is more than 6% lower than that of the proposed method. The classification accuracies of some compression types such as numbers 9, 14, 16, and 19 (Huffman, LZJB, LZO, and LZSS) have significantly increased. This demonstrates that PGF-RNN represents the characteristics of various compression types by simultaneously considering the temporal and spatial features. The average top 3 accuracy of the proposed method is also higher than the result of the GRU-based method. Even though the top 1 accuracies of some compression types are significantly lower than other compression types, the top 3 accuracies of most lossless compression types are 95% or higher. This shows that PGF-RNN can be effectively used to determine the compression type for blind compressed data.
For some entropy-based compression types that have low average top 1 accuracy, we apply the post-processing after classifying the data through PGF-RNN, as shown in Fig. 10. Among several compression algorithms with low average top 1 accuracy, we performed post-processing on Huffman (9), Shannon (25), and Shannon Fano Elias (27). Table 5 shows the average top 1 accuracy with and without post-processing for three compression types. As shown in Table 5, post-processing has a good performance on all three compression types. The accuracies of three compression types have increased by more than 5%. By improving the top 1 accuracies of the three compression types through postprocessing, the average top 1 accuracy of the 31 compression types tested also increased from 91.78% to 92.63%. In the case of entropy-based compression, it is important to determine the overall bit pattern from compressed data. However, there is a limitation in identifying the characteristics of the algorithm as a pattern of data, especially for a similar type of compression algorithm. Therefore, through the additional processing using bit frequency information, the overall performance of the proposed method is improved by making up for the parts represented by a deep learning-based method.

V. CONCLUSION AND FUTURE WORK
We propose PGF-RNN for lossless compression type classification. We performed classification by modifying the conventional GRU to use a model that considers the timestep due to the characteristics of the lossless compression algorithm that compresses data by scanning the whole bit sequences. PGF-RNN has a structure that utilizes information between layers to enhance the interaction of hidden states by sharing the weights of layers. This structure overcomes the limitations of GRU, which only temporal characteristics of data, by also using the spatial features of compressed data.
For enhancing the accuracies of some misclassified compression types, we apply post-processing which considers the frequency of bit sequences. Our proposed compression type classification method has good performance on lossless compression type classification. We expect that the proposed method will be useful for finding digital forensics evidence.


extfieldsofstudy: Geography Physics

text:
Analysis on Influencing Factors of Urban Road Traffic Noise and Its Control

With the continuous improvement of people’s requirements for a better life, the use of motor vehicles is also gradually increasing, thus road traffic pressure is also gradually increasing. Especially in recent years, urban construction has developed rapidly. Newly built and expanded streets and roads have turned the originally remote and quiet areas into bustling and noisy downtown, thus increasing the impact of traffic noise on the surrounding environment. Obviously, this goes against people’s pursuit of a better life, so the optimization of urban traffic noise is particularly important. This paper analyzes the main factors that affect traffic noise and puts forward corresponding control measures.

Influencing Factors of Traffic Noise
Influencing Factors of Traffic NoiseIn recent years, with the continuous development of economy and the continuous improvement of people's income level, the use of motor vehicles has gradually become popular and generalized, thus bringing about increasingly serious problems such as traffic congestion and traffic noise. Obviously, this goes against people's pursuit of a better life. Therefore, the optimization of urban traffic soundscape is particularly important. Then we will analyze from the aspects of speed, power system, traffic flow and road conditions.
Traffic noise and speed. The speed of a car is fast and slow, and constant changes will cause friction between the ground and tires, the power of the engine, and changes in the acting force with the wind, thus causing changes in the noise value. According to the general physics theory, the noise will increase with the acceleration of vehicle speed, and how the vehicle speed affects the traffic noise. In this study, the vehicle speed and the corresponding noise data are collected and analyzed.
In order to ensure the accuracy of the obtained data and make the test data free of interference from other factors, the test weather was chosen to be conducted under the condition of sunny weather, with the wind force level 1 on that day. The sound level meter is selected to be fixed on the measuring tripod, which is placed between the motor vehicle lane and the non-motor vehicle lane, 1.2m above the ground to reduce the reflection effect as much as possible and to keep it away from other reflectors. Because this test is mainly to analyze the influence of vehicle speed and noise, the place of measurement is chosen on a quiet road, which is equipped with two motor lanes and one non-motor lane in the same direction. In order to prevent the mutual influence of vehicles coming and going, each test is conducted without interference from other vehicles.
In order to avoid the influence of different vehicle types on the test data, Toyota Corolla was selected for this test, and the test lane was selected near the non-motorized lane, which was closer to the sound level meter, so the test result was more accurate. In order to reduce the error of the test results, the test AESEE 2020 IOP Conf. Series: Earth and Environmental Science 512 (2020) 012044 IOP Publishing doi:10.1088/1755-1315/512/1/012044 2 involved three participants, including one driver with 6 years of driving experience. The main task is to control the gear and accelerator to make the vehicle speed at the required speed. One passenger. The main task of the passenger is to read the speed of the car when it passes through the terminal sound level meter on the mobile phone software. The test is about 200 meters away, and the sound level meter is set at the terminal of 200 meters. The test driver adjusts different speeds through the 200 meters, and tests 3 times each at the same speed, because the speed test of the same value may not be completely consistent 3 times. If a speed value of ± 1 appears, the test is considered successful. As can be seen from Table 1, the minimum vehicle speed for this test is 30 km/h, the noise values obtained are 56.8 dB(A) and 56.4 dB(A), respectively, and the maximum vehicle speed for the test is 100 km/h, and the noise values obtained are 82.4 dB(A) and 80.6 dB(A), respectively. At the same speed, the measured noise value is different, because the test will be more or less affected by wind, pedestrians in non-motorized lanes, etc. The data are not different and the error is within a reasonable range. The data obtained from this experiment are valid.
In this paper, SPSS19.0 software is used to analyze the relationship between car speed and traffic noise, and the correlation coefficient is 0.98, which is highly correlated. The experimental results show that the noise value gradually increases with the increase of vehicle speed in the range of 30-100 km/h. Traffic noise and automobile power system. A long-term theoretical study shows that automobile noise is generated by engine noise. As far as the current situation is concerned, more and more people choose electric vehicles. Fuel cell vehicles are also pure electric vehicles, which convert chemical energy in the fuel cell into electric energy as the power of the vehicle when hydrogen and oxygen react. In daily life, whether electric cars are compared with ordinary cars or electric buses are compared with ordinary gasoline buses, the noise heard from people's ears is smaller than that of the latter. then, at the same speed, how much difference is there between the noise generated by the two, which will be verified by the following experiments.
(1) Preparation before testing In the study of the influence of vehicle speed on traffic noise, Toyota Corolla was selected. In this section, the comparison of the influence of vehicle power system on traffic noise, the noise value of ordinary vehicles at different speeds was based on the previous Corolla data. In order to control the influence of irrelevant variables on the experimental results as much as possible, electric vehicles with similar vehicle length, width, height and shape are selected this time. In this experiment, on the same 3 day and at the same place as the last experiment, the sound level meter did not change its position, so the test conditions also meet the requirements, and the data obtained from the experiment is valid.
(2) Testing process There are three participants in this test, including one driver who has been driving for 6 years. The main task is to control the gear and accelerator so that the car can reach the required speed. The main task of the passenger is to read the speed of the car when passing through the terminal sound level meter on the mobile phone software. This test is about 200 meters away. The sound level meter is set at the end of 200 meters. The driver is tested to adjust the speed through the 200 meters. Three times for each same vehicle speed test, the test speeds are 30km/h, 40 km/h, 50 km/h, 60 km/h, 70 km/h, 80 km/h, 90 km/h, 100 km/h respectively. Another tester is responsible for recording the data on the sound level meter. the specific test data are shown in Table 2. It can be seen from Table 2 that under the condition of the same vehicle speed, the noise value measured by electric vehicles is smaller than that measured by Toyota vehicles, and the difference between the noise values generated by the two vehicles gradually increases as the vehicle speed gradually increases. Therefore, the data obtained from the experiment are the same as the results felt by people in normal hearing. Electric cars produce less traffic noise than gasoline cars, and the power system of cars also has some influence on traffic noise.
Traffic noise and traffic flow. Every moving car is generating traffic noise. According to the logical reasoning of normal thinking, the greater the traffic flow, the greater the traffic noise. In order to verify the positive correlation between traffic flow and traffic noise, the following experiments were carried out.
(1) Pre-test preparation The experiment in this section is mainly to study the relationship between traffic flow and traffic noise. In order to test the accuracy of the test results, the same road section is selected for testing. So this section can meet the conditions of large traffic flow and small traffic flow. In some sections, the traffic flow varies greatly with different time, so the auxiliary road of South Outer Ring Road is selected.
The test was conducted on a sunny day with a wind force of 1~2. The sound level meter was still mounted on a tripod 1.2 meters above the ground. The tripod was placed in the green belt in the middle of the road.
(2) Testing process This test requires two testers. One tester is responsible for photographing the traffic flow of the road section, and the other tester is responsible for recording the data of the sound level meter. The two testers must set the test time in advance and turn on the machine at the same time. This test is from 7: 30 a.m. to 10: 30 a.m. and takes about 3 hours. It goes through the process of traffic flow from small to large and then from large to small.
(3) Data analysis It is not difficult to see the data curve of fig. 1 derived from the combination of video and sound level meter: when cars pass through the sound level meter one by one, the curve changes obviously, and the AESEE 2020 IOP Conf. Series: Earth and Environmental Science 512 (2020) 012044 IOP Publishing doi:10.1088/1755-1315/512/1/012044 4 fluctuation in instantaneous time is large; When two vehicles pass through the sound level meter at the same time or almost at the same time, the instantaneous fluctuation of the noise curve decreases, and the highest point of the noise curve is obviously higher than that of one vehicle. As the number of vehicles increases gradually, the fluctuation of the noise curve decreases obviously for a period of time, and the noise curve generally shows an upward trend. However, when 20 vehicles pass through the sound level meter every 10 seconds or so, the noise curve gradually tends to a high level and peaceful state. Therefore, it is believed that the noise value increases significantly with the increase of traffic flow in the initial stage. When the traffic flow reaches a certain amount, the noise value tends to a relatively stable value, and the increasing trend is not obvious, which shows that the two are logarithmically positively correlated. Traffic noise and road conditions. The influence of urban road conditions on urban traffic noise mainly includes the use of pavement materials and slope etc. At present, asphalt, cement concrete and flexible pavement are mainly used as pavement materials for urban traffic roads. The use of pavement materials for traffic roads will affect the rolling sound power level and vibration level of motor vehicles. Among them, the parameters in the rolling sound power level of motor vehicle tires are selected when building roads. The vibration level is mainly affected by pavement materials, including the total weight of motor vehicles and the running speed, etc. In addition, the road surface gradient of urban roads is also an important factor affecting urban traffic noise. Usually, there are transverse gradient and longitudinal gradient. The gradient will directly affect the running speed of vehicles, thus indirectly affecting the running noise of motor vehicles. The theory holds that if the width of the road surface is large and the number of lanes is large, the carrying capacity of the road will be relatively large, thus reducing traffic flow density and traffic noise. Therefore, widening the road surface and increasing the number of lanes are one of the methods to reduce traffic noise. We should also scientifically integrate greening and other methods to reduce noise, so as to achieve a pleasant state of mind.
Traffic noise and green belts. Green belts with a width of 30m were selected for the survey. In order to avoid interference, the test points should be as far away from other buildings as possible, with a distance of more than 2m. Use recording equipment to record a section of traffic sound source when the traffic flow is large, select 10 seconds with small fluctuation of sound pressure level from the recorded sound source, and use audio processing software Adobe Audition to repeatedly copy the 10 seconds of audio to make the 30-minute sound source required for the test. Because the test sound source is artificially synthesized, interference from other sound sources should be avoided as much as possible during the test, so the test time should be from 4: 30 a.m. to 6: 00 a.m. when the traffic volume is small. The test shall be conducted in clear, windless or breezy weather. If there is wind, the wind speed shall be below 3m/s. The equivalent continuous A sound level is used in the test. The measured parameters include Leq, L10, L50, L90 and Lmax. The Leq and Lmax are mainly used as reference values. The time

Analysis of test results
For the data measured in Table 4, in order to make the analysis process simpler and clearer, and to make the comparison result more obvious, Lmax attenuation value and Leq attenuation value are compared to detect and evaluate the sound insulation condition of the green belt. The distance between cedar trees is only 1 meter, the plant height is 1.3 meter, and the lowest branch height of cedar is only 8 cm, which can be well connected with the lawn on the ground. The distance between the purple leaf plum trees is 3 meters, and the crown width is about 1.9 meters, which means that some of the two purple leaf plum trees will cross together, while the lowest branch height of the purple leaf plum tree is 1 meter, the height of the cedar tree is 1.3 meters, and the purple leaf plum tree is well connected with the cedar tree. The spacing between Chinese locust trees is 8 meters and the crown width is about 9 meters, then some of the two Chinese locust trees will also cross, with a height of about 12 meters and a minimum branch height of 4 meters, while the height of purple leaf plum is about 4.5 meters. The Chinese locust tree and the purple leaf plum are well connected in height. The measuring point 7 forms a continuous green sound barrier in both width and height, so the Lmax attenuation value and Leq attenuation value of the measuring point 7 are the largest. The lowest layer of measuring point 4 is also a lawn with a height of about 5 cm, followed by shrub Ligustrum lucidum with a height of 110 cm. There are two kinds of arbors. The height of purple leaf plum is about 4. 5 meters, the lowest branch height is about 1.5 meters, the crown width is 2.4 meters, and the plant spacing is about 2 meters. The height of Sophora japonica is about 10 meters, the lowest branch height is about 3.8 meters, the plant spacing is about 6.8 meters, and the crown width is about 7.2 meters. From these data, it is not difficult to see that the height of Ligustrum lucidum is less than the lowest branch height of Prunus salicina, thus the green sound barrier with a height of 1.1m to 1.5m is missing. Therefore, the Lmax attenuation value and Leq attenuation value at measuring point 4 are slightly smaller than those at measuring point 7.
The planting distribution of measuring point 3 and measuring point 10 are similar, both of which are the combination of lawn and arbor, but the Lmax attenuation value and Leq attenuation value of measuring point 3 are slightly larger than the Lmax attenuation value and Leq attenuation value of measuring point 10. The trees at measuring point 3 are apricot trees with an average height of 4.2 meters, the lowest branch height is about 0.6 meters, and the crown width is about 4 meters. The trees at measuring point 10 are hibiscus syriacus with a height of 3.5 meters, the lowest branch height is 0.8 meters, and the crown width is about 1 meter. The Lmax attenuation value and Leq attenuation value of AESEE 2020 IOP Conf. Series: Earth and Environmental Science 512 (2020) 012044 IOP Publishing doi:10.1088/1755-1315/512/1/012044 8 measurement point 3 are slightly larger because the plant height of measurement point 3 is higher than that of measurement point 10, the lowest branch height is lower, and the crown width is larger.
The planting distribution of measuring point 02 is similar to that of measuring point 09, which is a combination of lawn, shrubs and trees. The vegetation of measuring point 02 is grass with an average height of 5 cm. Shrub Damascus Rose, with an average height of 50 cm; The tree Platanus acerifolia has an average height of 9 meters, the lowest branch height is about 2.5 meters, the plant spacing is 4.5 meters, and the crown width is between 3.6 and 5.2 meters. Weeds are found at measuring point 09; Shrub is Iris tectorum with average plant height of 35 cm and plant spacing of 40 cm. There is a tree Platanus acerifolia in the bush, with a plant height of about 8m, a plant spacing of 2.5m, a minimum branch height of about 5m, and a crown width of 3.8-5.4m. Based on the data of measuring point 02 and measuring point 09, it is found that the plant spacing is the largest difference, and other data are all close. Therefore, the plant spacing is also a factor that affects Lmax attenuation value and Leq attenuation value.
Through the comparative study of each measuring point, it can be found that the sound insulation effect of vegetation combination from low to high is as follows: only lawn without shrubs and trees; Trees with higher lowest branching height are more than shrubs. The combination of arbor and shrub whose lowest branch height is higher than that of shrub; The combination of arbor and shrub with the lowest branch height of arbor lower than that of shrub; Under the same combination condition, the green belt with better growth, dense branches and leaves and higher planting density has the best noise reduction effect.

Suggestions on the Control of Urban Road Traffic Noise
Improve the information level of traffic management. In order to reasonably control the speed of a city's main traffic arteries, it is not necessary to cause large noise due to too fast speed or serious traffic congestion due to too slow speed. Through reasonable adjustment of traffic flow, the traffic efficiency of motor vehicles is improved, and the impact of traffic peak on traffic noise is reduced. Through improvement of traffic efficiency, the impact of traffic noise on the surrounding environment of roads is reduced. Noise violation detection equipment can be installed in hospitals, residential areas and other places to regulate civilized driving of motor vehicle drivers and protect people's living environment by legal means.
Strengthen the construction of urban road infrastructure. Researchers should continue to study ways to reduce the noise of the car's power system and low noise tires to reduce the noise generated by the car itself. Advanced pavement materials are used to reduce the friction noise between the pavement and tires, while widening the width of the road and increasing the number of motor vehicles. We will improve the public transportation and subway systems, develop urban public transportation, encourage citizens to use public transportation modes, increase publicity on environmental protection and public welfare, and attract more citizens to change their modes of travel by lowering fares.
Increase green belts and optimize traffic sound. According to the principles of aesthetics and so on, do a good job in greening the city's main roads. When designing the road green belts, attention should be paid to the combination of deciduous trees and evergreen trees, so as to ensure that no matter what period a single plant is in, when combined, it can ensure a good sound insulation effect for one year.
Attention should be paid to the combination of surface vegetation, shrubs and trees to build composite green belts of different levels. As different vegetation has different attenuation effects on noise of different frequencies, a variety of plants should be planted.
The height of the grass on the ground should be reasonable, and the height of shrubs should be higher than the lowest branches of trees, so that the heights of different plants can be connected in space, thus forming a closed space, allowing the green belt to form a complete barrier, and playing a very good sound insulation and noise reduction function.
In order to make the leaves play a good sound-absorbing effect, it is necessary to select some porous and hairy trees with luxuriant branches and leaves. Broad-leaved trees and coniferous trees have


extfieldsofstudy:
Computer Science
Mathematics

text:
Proximal Distilled Evolutionary Reinforcement Learning

Reinforcement Learning (RL) has recently achieved tremendous success due to the partnership with Deep Neural Networks (DNNs). Genetic Algorithms (GAs), often seen as a competing approach to RL, have run out of favour due to their inability to scale up to the DNNs required to solve the most complex environments. Contrary to this dichotomic view, in the physical world, evolution and learning are complementary processes that continuously interact. The recently proposed Evolutionary Reinforcement Learning (ERL) framework has demonstrated the capacity of the two methods to enhance each other. However, ERL has not fully addressed the scalability problem of GAs. In this paper, we argue that this problem is rooted in an unfortunate combination of a simple genetic encoding for DNNs and the use of traditional biologically-inspired variation operators. When applied to these encodings, the standard operators are destructive and cause catastrophic forgetting of the traits the networks acquired. We propose a novel algorithm called Proximal Distilled Evolutionary Reinforcement Learning (PDERL) that is characterised by a hierarchical integration between evolution and learning. The main innovation of PDERL is the use of learning-based variation operators that compensate for the simplicity of the genetic representation. Unlike the traditional operators, the ones we propose meet their functional requirements. We evaluate PDERL in five robot locomotion environments from the OpenAI gym. Our method outperforms ERL, as well as two state of the art RL algorithms, PPO and TD3, in all the environments.

Introduction
The field of Reinforcement Learning (RL) has recently achieved great success by producing artificial agents that can master the game of Go (Silver et al. 2017), play Atari games (Mnih et al. 2015) or control robots to perform complex tasks such as grasping objects (Andrychowicz et al. 2017) or running (Lillicrap et al. 2015). Most of this success is caused by the partnership with Deep Learning (Goodfellow, Bengio, and Courville 2016), generically called Deep Reinforcement Learning (DRL).
At the same time, Genetic Algorithms (GAs), usually seen as a competing approach to RL, have run out of favour due Copyright c 2019, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
to their inability to scale up for evolving Deep Neural Networks (DNNs). As the recent progress in RL has shown, DNNs are required to solve complex environments. However, GAs are particularly suitable for optimisation problems where the dimensionality of the search space is small because of their substantial computational requirements. Therefore, they are unable to cope with the explosion in the number of parameters that DNNs brought.
Contrary to this dichotomic view of RL and GAs, in the physical world, evolution and learning are complementary processes that continuously interact. Perhaps, the most famous result of this interaction is the Baldwin effect (Simp- son 1953), which explains how the genotype can assimilate learnt behaviours over the course of many generations. A more spectacular by-product of this interplay, which has received more attention in recent years, is the epigenetic inheritance of learnt traits (Dias and Ressler 2013).
Despite these exciting intricacies of learning and evolution, the two have almost always received separate treatment in the field of AI. Even though they have been analysed in computational simulations multiple times (Belew and Mitchell 1996), (Ackley and Littman 1992), (Suzuki and Arita 2004), they have rarely been combined to produce novel algorithms with direct applicability. This is surprising given that nature has always been a great source of inspiration for AI (Floreano and Mattiussi 2008).
For the first time, Khadka and Tumer (2018) have recently demonstrated on robot locomotion tasks the practical benefits of merging the two approaches in their Evolutionary Reinforcement Learning (ERL) framework. ERL uses a reinforcement learning-based agent alongside a genetically evolved population, with a transfer of information between the two. However, ERL has not fully addressed the scalability problem of GAs. While the gradient information from the RL agent can significantly speed up the evolutionary search, the population of ERL is evolved using the same variation operators that have been used by previous methods. Paired with directly encoded DNNs, which is the most common genetic representation in use, these operators are destructive.
This paper brings the following contributions: • Demonstrates the negative side-effects in RL of the traditional genetic operators when applied to directly encoded arXiv:1906.09807v1 [cs.
• Proposes two novel genetic operators that are based on backpropagation. These operators do not cause catastrophic forgetting in combination with simple DNN representations.
• Integrates these operators as part of a novel framework called Proximal Distilled Evolutionary Reinforcement Learning (PDERL) that uses a hierarchy of interactions between evolution and learning.

Background
This section introduces the Evolutionary Reinforcement Learning (ERL) algorithm and the genetic operators it uses.

Evolutionary Reinforcement Learning
The proposed methods build upon the ERL framework proposed by Khadka and Tumer (2018). In this framework, a population of policies is evolved using GAs. The fitness of the policies in the population is based on the cumulative total reward obtained over a given number of evaluation rounds. Alongside the population, an actor-critic agent based on DDPG (Lillicrap et al. 2015) is trained via RL. The RL agent and the population synchronise periodically to establish a bidirectional transfer of information.
The first type of synchronisation in ERL, from the RL agent to the genetic population, is meant to speed up the evolutionary search process by inserting the policy gradient information into the population to guide the genetic search. This information is inserted by cloning the actor of the RL agent into the population every few generations. The synchronisation period, ω, is a hyperparameter that controls the amount of information flowing from the RL agent to the population.
The second type of synchronisation consists of a reverse information flow coming from the population to the RL agent. The actors in the population can be seen as experience collectors from which the RL agent can learn offpolicy. All the transitions coming from rollouts in the population are added to the replay buffer of the DDPG agent. The population experiences can be seen as being generated by evolution-guided parameter space noise (Plappert et al. 2018).

Genetic encoding and variation operators
The policies in the ERL population are represented by neural networks with a direct encoding. In this genetic representation, all the weights of a network are stored in a list of real numbers in an arbitrary order. The traditional variation operators are based on simple stochastic modifications of this list. As the number of parameters increases, it is becoming increasingly unlikely that these random weight changes produce better policies.
In the physical world, mutations and crossovers rarely have catastrophic phenotype effects because the phenotype is protected by the complex biological and chemical processes that translate the DNA. In a direct genetic encoding, this protective layer is lacking because of the trivial representation that is used. The biologically inspired variation operators commonly used in the literature, including ERL, do not have the desired functionality when paired with a direct encoding. Ideally, crossovers should combine the best behaviours of the two parents, while mutations should produce a slight variation in the behaviour of the parent.
However, because DNNs are sensitive to random modifications of the weights (genes), these operators typically cause catastrophic forgetting of the acquired behaviours and do not meet their functional requirements. ERL evolves the population using two variation operators commonly used for representations consisting of a list of real numbers: n-point crossovers and Gaussian mutations (Eiben and Smith 2015). Both of these operators can have devastating effects on the behaviour of a directly encoded neural network.
n-point crossovers produce an offspring policy by randomly exchanging segments of the lists of weights of the two parents, where n endpoints determine the segments. ERL uses a custom version of this operator where the segments are the rows of the dense layer matrices. The resulting child policy contains a mix of rows coming from the matrices of the parents. These two types of weights can destructively interfere with each other, and the behaviour of the child policy can diverge from that of the parents.
Similarly, the damaging effects of adding Gaussian noise to the parameters of a DNN have been discussed at great length by Lehman et al. (2018). To limit them, a common approach, which is also employed by ERL, is to mutate only a fraction of the weights. Nonetheless, even with this trick, these mutations are still destructive. Furthermore, evolving just a small part of the weights can slow down the evolutionary search for better policies.

Method
This section introduces our proposed learning-based genetic operators and describes how they are integrated with ERL.

The genetic memory
A significant problem of the population from ERL is that it does not exploit all the individual experiences of the members of the population directly. The only way the population benefits from them is indirect, through the RL agent, which uses them to learn and improve. The individual experiences of the agents are an essential aspect of the new operators, and, therefore, the agents also need a place to store them.
The first modification we perform to ERL is to equip the members of the population and the RL agent with a small personal replay buffer containing their most recent experiences and, depending on the size of the buffer, experiences of their ancestors. Because these experiences span over multiple generations, we refer to this personal replay buffer of each agent as the genetic memory. When the policies are evaluated, they not only store their experiences in DDPG's replay buffer, but also their genetic memory.
As a usual replay buffer, the buffer evicts the oldest experiences when it becomes full. At the same time, the buffer is entirely inherited during mutations and partially during crossovers. Therefore, the size of this buffer determines how much the memories span in the genetic tree. A smaller buffer would usually contain only the memories of the current agent, whereas a larger buffer would also contain ancestral experiences.

Q-Filtered distillation crossovers
In this section, we proposed a knowledge distillation-based crossover to selectively merge the behaviour of two parent policies into a child policy. The proposed algorithm radically differs from the previous approaches that perform the crossovers in the parameter space.  Figure 1: Q-filtered distillation crossover compared to npoint crossover. The contour plots represent the state visitation distributions of the agents. The Q-filtered distillation crossover selectively merges the behaviours of the two parents, whereas the modes of the state visitation distribution obtained by the traditional crossover are disjoint from the modes of the parent distributions.
For a pair of parent agents from the population, the crossover operation works as follows. A new agent with an initially empty associated genetic memory is created. The genetic memory is filled in equal proportions with the latest transitions coming from the genetic memories of the two parents. The weights of the child DNN are initialised with the weights of one of the parents. The child agent is then trained via a form of Imitation Learning (Attia and Dayan 2018) to selectively imitate the actions the parents would take in the states from the newly created genetic memory.
Equivalently, this process can also be seen as a more general type of policy distillation (Rusu et al. 2016) process since it aims to "distil" the knowledge of the parents into the child policy. Unlike the conventional policy distillation proposed by Rusu et al. (2016), two parent networks are involved, not one. This introduces the problem of divergent behaviours. The two parent policies can take radically different actions in identical or similar states. The main problem is how the child policy should decide whom to imitate in each state.
The key observation of the proposed method is that the critic of the RL agent already knows the values of certain states and actions, and therefore it can be used to select which actions should be followed in a principled and globally consistent manner. Thus, we propose the Q-Filtered behaviour cloning loss from Equation 1, where the sum is taken over a batch of size N C sampled from the genetic memory of the agent. µ x and µ y represent the deterministic parent policies, while µ z is the deterministic policy of the child agent.
The indicator functions use the Q-Network of the RL agent to decide which parent takes the best action in each state. The child policy is trained to imitate those actions by minimising the first two terms. The final term is an L 2 regularisation that ensures that the child policy does not overfit in capturing the behaviour of the parents. Figure 1 contains a diagram comparing this new crossover with the ERL n-point crossover. We refer to ERL with the distillation crossover as Distilled Evolutionary Reinforcement Learning (DERL).

Parent selection mechanism
An interesting question is how parents should be selected for this crossover. A general approach is to define a mating score function m : Π × Π → R that takes as input two policies and provides a score. The pairs with higher scores are more likely to be selected. Two ways of computing this score can be distinguished: greedy and distance-based.
can be greedily determined by the sum of the fitness of the two parents. This type of selection generally increases the stability of the population and makes it unlikely that good individuals are not selected.
Distance based. The score m(µ x , µ y ) = d Π (µ x , µ y ) can be computed using a distance metric in the space of all possible policies. "Different" policies are more likely to be selected for mating. The exact notion of "different" depends on the precise form of the distance metric d Π . While previous approaches generally compute distances between individuals in the parameter space (Eiben and Smith 2015), here we propose a novel distance between behaviours. This distance metric takes the form from Equation 2, where ρ x and ρ y are the state-visitation distributions of the two agents: This distance metric measures the expected difference in the actions taken by the two parent policies over states coming from a mixture of their state visitation distributions. This expectation is in practice stochastically approximated by sampling a large batch from the genetic memories of the two agents. While this strategy does indeed introduce more novel behaviours into the population, it also makes the population more unstable by increasing the probability that fit individuals are not selected. We note that a linear combination of the two methods could also be performed with m(µ to exploit the benefits of both methods. In the beginning, α could be set to zero to encourage more exploratory behaviour, and as the training approaches the end, α could be increased towards one to shift the population towards exploitation. We leave this possible improvement for future work.

Proximal mutations
As showed by Lehman et al. (2018), Gaussian mutations can have catastrophic consequences on the behaviour of an agent. In fact, the stability of the policy update is a problem even for gradient descent approaches, where an inappropriate step size can have unpredictable consequences in the performance landscape. Methods like PPO (Schulman et al. 2017) are remarkably stable by minimising an auxiliary KL divergence term that keeps the behaviour of the new policy close to the old one. This section introduces a stable mutation operator with the same incentive of preventing a significant divergence from the behaviour of the parent.
Ideally, one would like to compute numerically how much the visitation distribution changes with tiny variations in each of the parameters. In the absence of a differentiable model of the environment, this is not possible. A proxy for this measurement is the expected absolute gradient of the actions with respect to the weights as in Equation 3, where |A| is the size of the action space, and ρ µ is the visitation distribution of the policy µ θ . This equation quantifies the sensitivity of the actions to parameter variations.
This expression is effectively approximated by sampling states from the genetic memory of the agent. This measure can be used to scale the magnitude of the mutations as in Equation 4. Thus, parameters with high sensitivity are scaled down, and the effective mutation size is reduced. The scaling has an opposite effect on parameters with low sensitivity.
With the absolute value removed, s can be computed efficiently at the cost of cancelling opposing signs in the sum over the states. The use of a sum instead of an expected value empirically helps reduce the effect of these cancellations. This approximation works well in practice.
This mutation produces child policies that are in the proximity of their parent's behaviour. Therefore, we refer to this operator as a proximal mutation (Figure 2), and the version of ERL using it as Proximal Evolutionary Reinforcement Learning (PERL). While the proximal mutations do not explicitly use learning, they rely on the capacity of the policies to learn, or in other words, to be differentiable. Without this property, these behaviour sensitivities on the parameters cannot be computed.

Integration
To obtain the full benefits of the newly introduced operators, they should be used together. The Q-Filtered distillation crossover increases the stability of the population and drives the agents towards regions of the state-action space with higher Q-values. The proximal mutations improve the exploration of the population and its ability to discover better policies. As will be seen in the evaluation section, the two operators complement each other. We refer to their integration with ERL as Proximal Distilled Evolutionary Reinforcement Learning (PDERL). Ultimately, PDERL contains a hierarchy of interactions between learning and evolution. A high-level interaction is realised through the information exchange between the population and the RL agent. The newly introduced operators add a low-level layer of interaction, at the level of the genetic operators. A diagram of PDERL is given by Figure 3.

Evaluation
This section evaluates the performance of the proposed methods, and also takes a closer look at the behaviour of the proposed operators.

Experimental setup
The architecture of the policy and critic networks is identical to ERL. The full list of hyperparameters in given in the Appendix. Those hyperparameters that are shared with ERL are identical to those used by Khadka and Tumer (2018), with a few exceptions that are also detailed in the Appendix.
The PDERL specific hyperparameters were set as follows. The distillation crossover batch size is N C = 128. The proximal mutation batch size is N M = 256. The genetic memory contains 8000 experiences. The learning rate for the distillation crossover is 1e −3 , and the child policy is trained for 12 epochs. All the training procedures use the Adam optimiser. Greedy parent selection is used unless otherwise indicated. As in ERL, the population is formed of k = 10 policies.
When reporting the results, we use the official implementations for ERL 1 and TD3 2 , and the OpenAI Baselines 3 implementation for PPO.

Performance evaluation
This section evaluates the mean reward obtained by the newly proposed methods as a function of the number of environment frames experienced. The results are reported across five random seeds. Figure 4 shows the mean reward and the standard deviation obtained by all algorithms on five MuJoCo environments.
While PERL and DERL bring improvements across multiple environments, they do not perform well across all of them. PERL is effective in stable environments like HalfCheetah and Hopper, where the total reward has low variance over multiple rollouts. At the same time, DERL is more useful in unstable environments like Walker2d and Ant since it drives the population towards regions with higher Q values. In contrast, PDERL performs consistently well across all environments, demonstrating that the newly introduced operators are complementary. PDERL significantly outperforms ERL and PPO across all environments. Despite being generally less sample efficient than TD3, PDERL catches up with it eventually. PDERL significantly outperforms TD3 on Swimmer, HalfCheetah and Ant, and marginally on Hopper and Walker2d. Table 1 reports the final reward statistics for all the tested models and environments. Side by side videos of ERL and PDERL running on simulated robots can be found at the following link: https://youtu.be/7OGDom1y2YM. The following subsections take a closer look at the newly introduced operators and offer a justification for the improvements achieved by PDERL.

Crossover evaluation
A good indicator for the quality of a crossover operator is the fitness of the offsprings compared to that of the parents. Figure 5 plots this metric for ten randomly chosen pairs of parents in the Ant environment. Each group of bars gives the fitness of the two parents and the policies obtained by the two types of crossovers. All these values are normalised by the fitness of the first parent. The performance of the child obtained via an n-point crossover regularly falls below 40% the fitness of the best parent. At the same time, the fitness of the policies obtained by distillation is generally at least as good as that of the parents.  The state visitation distributions of the parents and children offer a clearer picture of the two operators. Figure 6 shows these distributions for a sample crossover in the Ant environment. The n-point crossover produces a behaviour that diverges from that of the parents. In contrast, the Qfiltered distillation crossover generates a policy whose behaviour contains the best traits of the parent behaviours. The new operator implicitly drives each new generation in the population towards regions with higher Q values. Figure 7 shows the fitness of the children obtained by the two types of mutation for ten randomly selected parents  Figure 8: As before, the blue contours represent the state visitation distribution of the parent, whereas the red ones represent the difference. The child obtained by proximal mutation inherits the behaviour of the parent to a large degree and obtains a 600 fitness boost. The behaviour obtained by Gaussian mutation is entirely different from that of the parent. The KL divergence between the parent and child distributions (0.03 and 0.53) quantitatively confirm this .

Mutation evaluation
As in the previous section, the analysis of the state visitation distribution of the policies reveals the destructive behaviour of the Gaussian mutations. The contours of these distributions for a sample mutation are given in Figure  8. The policy mutated by additive Gaussian noise completely diverge from the behaviour of the parent. This sudden change in behaviour causes catastrophic forgetting, and the new offspring falls in performance to a total reward of −187. In contrast, the proximal mutation generates only a subtle change in the state visitation distribution. The offspring thus obtained inherits to a great extent the behaviour of the parent, and achieves a significantly higher total reward of 5496.
Another benefit of the proximal mutations is that the size of the change in the behaviour can be directly adjusted by tuning the magnitude of the mutations. Figure 9 shows how increasing the mutation magnitude gradually induces a more significant change in the behaviour of the child policy. This is not the case for the Gaussian mutations where a given mutation size can be unpredictably large and non-linearly related to the behaviour changes. The red contours show how the differences from the parent distribution increase as the mutation magnitude becomes higher. This is also confirmed by the KL divergence between the parent and child distributions that strictly increases with the mutation magnitude.

Related work
This paper is part of an emerging direction of research attempting to merge Evolutionary Algorithms and Deep Reinforcement Learning: Khadka and Tumer (2018), Pourchot and Sigaud (2019), Gangwani and Peng (2018), Khadka et al. (2019). The closest to this work remain the papers of Lehman et al. (2018) and Gangwani and Peng (2018). Both papers address the destructive behaviours of classic variation operators. Lehman et al. (2018) focus exclusively on safe mutations, and one of their proposed operators is directly employed in the proximal mutations. However, their paper is lacking a treatment of crossovers and a tighter integration with learning.
At the same time, the methods of Gangwani and Peng (2018) are focused exclusively on safe operators for stochastic policies, while the methods proposed in this work could be applied to stochastic and deterministic policies alike. The closest aspect of their work is that they propose a crossover operator with the same goal of merging the behaviour of two agents. Their solution reduces the problem to the classic single parent distillation using a maximum-likelihood approach to combine the behaviours of the two parents. They also propose a mutation operator based on gradient ascent using policy gradient methods. However, this deprives their method of the benefits of derivative-free optimisation such as the robustness to local optimums.

Conclusion
The ERL algorithm has demonstrated how genetic algorithms can be scaled to DNNs using an integration with learning. In this paper, we propose the PDERL algorithm that achieves a new scalability level through a tighter, hierarchical integration of learning and evolution. While PDERL maintains the bi-directional transfer of information between the population and the RL agent, it also uses learning at the level of the genetic operators. Unlike the traditional operators, the learning-based ones have the desired functionality and do not cause catastrophic forgetting when applied to directly encoded DNNs. Finally, we show that PDERL outperforms ERL, PPO and TD3 in all the tested environments.
The hyperparameters that are shared with ERL have the same values as those reported by Khadka and Tumer (2018), with a few exceptions. For Walker2D, the synchronisation rate ω was decreased from 10 to 1 to allow a higher information flow from the RL agent to the population. In the same environment, the number of evaluations ξ was increased from 3 to 5 because of the high variance of the total reward across episodes. Another modification is that the fraction of elites in the Hopper and Ant environments was reduced from 0.3 to 0.2. Generally, a higher number of elites increases the stability of the population, but the stability gained through the new operators makes higher values of this parameter unnecessary. Table 2 includes the hyperparameters that where kept constant across all environments.

Interactions between learning and evolution
One of the motifs of this work is the interaction between learning and evolution. In PDERL, this can be quantitatively analysed by looking at the number of times the RL agent was selected, discarded or became an elite in the population. Figure 10 shows how these numbers evolve during training and reveals that each environment has its own unique underlying interaction pattern.
Swimmer. Swimmer is an environment where the genetic population drives the progress of the agent almost entirely. The RL agent rarely becomes good enough to be selected or become an elite.  HalfCheetah. HalfCheetah is at the other end of the spectrum from Swimmer. In this environment, the RL agent becomes an elite after over 90% of the synchronisations early in training. As the population reaches the high-reward areas, genetic evolution becomes slightly more important for discovering better policies.
Hopper. On Hopper, the RL agent drives the population in the first stages of training, but the population obtains superior total rewards beyond 1.5 million frames. Therefore, evolution has a greater contribution to the late stages of training.
Walker2d. In this environment, the dynamics between learning and evolution are very stable. The selection rates do not change significantly during the training process. This is surprising, given the instability of Walker2d. Overall, evolution has a much higher contribution than learning.
Ant. Unlike in the other environments, in the Ant environment, the interactions between learning and evolution are more balanced. The curves converge towards a 40% probability of being discarded, and a 60% probability of becoming an elite or being selected. Table 4 shows the final mean selection rates and their standard deviations side by side with the ones reported by ERL. This comparison indicates how the dynamics between evolution and learning changed after adding the new crossovers and mutations. The general pattern that can be seen across all environments is that the probability that the RL agent becomes an elite decrease. That probability mass is mainly moved towards the cases when the RL agent is discarded. This comes as another confirmation that the newly introduced variation operators improve the performance of the population and the RL agent is much less often at the same level of fitness as the population policies.

Parent selection mechanism
The previous experiments used the greedy parent selection mechanism for choosing the policies involved in the crossovers. This section offers a comparative view between this greedy selection and the newly proposed distance-based selection.
The mechanisms are compared in Figure 11 on Hopper and Walker2d. On a relatively unstable environment like Walker2d, the distance-based DERL and PDERL perform significantly worse than their fitness-based equivalents. However, on Hopper, which is more stable than Walker2d, the distance-based PDERL surpasses the fitness-based one, (b) Walker2d-v2 Figure 11: A comparison between the greedy (fitness-based) parent selection mechanism and the distance-based parent selection. The distance-based selection works better in stable environments like Hopper, but worse in environments with high variance across different episodes like Walker2d.
while also having very low variance. The fact that the distance-based selection performs better in the early stages of training for both environments further supports the idea of using a convex combination of the two selections.