Research on Continuous Berth Allocation Problem based on Genetic-Harmony Search Algorithm

In order to solve the berth allocation problem and improve the efficiency of the dock, a mixed integer nonlinear programming model is built with the length of port stay. The genetic-harmony search algorithm is used to solve it for the first time. Finally, the algorithm and Lingo are used to solve the large-scale berth allocation problem at the same time. Under the same data of arrival ships, several groups of experiments are designed to verify the algorithm. The results show that the genetic-harmony search algorithm can be used to solve the berth allocation problem. On the basis of the consistency between the genetic-harmony search algorithm and Lingo, the genetic-harmony search algorithm is faster and more efficient.

Introduction
The berth system, one of the most important operations of container terminal, includes berthing and unberthing of ships and cargo handling. In addition, a reasonable berth allocation can make full use of water-front resources and improve the operational efficiency of berth system. In current research, berth allocation can be divided into static berth allocation and dynamic berth allocation according to the different arrival ways of ships. The former assumes that all ships have arrived at the port when the plan is executed. Its research significance is to be applied in busy ports with a large number of ships waiting to berth. However, the study is only applicable to some ports and still needs to be improved. [1,2]. Relatively speaking, the latter is more in line with the actual situation. According to berth space, the berth allocation problem can be divided into discrete berth and continuous berth. The former model is relatively simple and easy to analyze, but the berth resources are not sufficiently used [3,4]. In addition, with the increasing development of competitiveness, the discrete berth is not suitable for the actual situation of the port, therefore more and more scholars focus on the problem of continuous berth allocation. Because of the above analysis, this paper will study the dynamic continuous berth allocation.
The berth planning problem has been proved to be NP-Complete, which full name is the uncertainty problem of polynomial complexity [5].The characteristics of berth planning problem show that it is difficult to find the global optimal solution in polynomial time with accurate algorithm, so experts usually use heuristic algorithm to solve it. Imai et al. regarded the solution of discrete berth allocation as a guide and changed the limit of berth length. They proposed a heuristic algorithm which was able to obtain the approximate optimal solution by solving the continuous berth allocation model [6]. Giallombardo [7,8]. Ting et al. proposed a mixed integer programming model of the dynamic discrete berth planning problem, and particle swarm optimization algorithm was used [9]. Li used three types of heuristic algorithm to solve the combined scheduling of continuous berth and quay crane [10]. For the berth allocation problem, tabu search algorithm, genetic algorithm, particle swarm optimization algorithm and other kinds of heuristic algorithm are commonly used. In this paper, genetic-harmony search algorithm will be used to solve the dynamic continuous berth allocation and it will be proved to be an effective algorithm in solving the berth planning problem.

Problem statement of continuous berth allocation
The continuous berth allocation can be transformed into geometric form of the two-dimensional packing problem in which the length of the Quay Line is taken as the X-axis and the time as the Y-axis to constitute a two-dimensional space [6].The ship is represented in the form of a rectangle, the length of which is the length of the ship, and the width of which is the time of cargo handling, as shown in Fig  Take ship 1 as an example, t 1 is the berthing time of ship 1, t 2 is the departure time of ship 1, (p 1 , p 2 ) is the berth location of ship 1 on the quay. The continuous berth allocation can be described as how to put vessels (replace with rectangle) efficiently on this this two-dimensional space-time graph which aim is to determine the berthing time and berth location of ships.
To analyze the problem more simply and formulate the mathematical model more accurately, the following assumptions are made: (1) Each vessel must be serviced only once.
(2) Each vessel can only be served after arriving at the port.
(3) The quay depth meets the requirement for vessels to berth, which means vessels can berth at any location of the quay.
(4) Each vessel has a preferred berth location, and the actual time of cargo handling depends on the preferred berth location.

Model building
In formulating a mathematical model, we define the notations as follows: i, jV, ship A i , arrival time of vessel i L i , the length of vessel i Q, the length of quay (including inter-ship clearance) H, the planning duration M i , the preferred berth location of vessel i. Generally, the vessel would like to berth near a specific yard, which is the preferred berth location of the vessel [11].
CM i , the handling time of vessel i at the best quay location Tan i , a ratio of increasing handling time to the distance for vessel i The mathematical model is formulated as follows: (8) Function (1) minimizes the sum of service times for all vessels. Constraints (2)-(6) are the non-overlapping restriction both in time and space. Constraints (7)-(8) ensure that vessel i must be berthed within the length of the quay. Constraint (9) ensures that vessel i is berthed after its arrival. Constraint (10) defines the decision variables. Equation (11) indicates the handling time of vessel i that it consists of two pieces: the minimum handling time at the best quay location, and the increasing handling time.
Equation (12) indicates the time vessel i departs berth equals the time vessel i starts berthing plus the time vessel i operations berthing and unberthing plus the time of vessel i's cargo handling.  Harmony search algorithm (HS) imitates the music improvisation process where musicians improvise their instruments' pitch by searching for a perfect state of harmony [12]. HS analogizes the solution vector of the optimization problem to the harmony of the instrument's pitch, and finding the best state of harmony is to find the optimal value of the objective function. In order to prevent HS from falling into the local optimum and make the output more accurately, this paper uses genetic algorithm (GA) to optimize HS firstly. GA, a parallel random search optimization method, was proposed by Professor Holland. Its principle is to simulate the genetic mechanism of nature and the theory of biological evolution. The main operations of GA are selection, crossover and mutation.

Harmony search algorithm and genetic algorithm
The parameters of HS are: harmony memory size (HMS), harmony memory size (HMZ), harmony memory considering rate (HMCR), pitch adjusting rate (PAR), band width (BW), Tmax which means the stopping criteria. The procedure of HS is shown in Fig.2.

The procedure of genetic-harmony search algorithm
In this paper, genetic-harmony search algorithm is used to solve the berth allocation problem. First of all, GA is used to select and cross the initial population so as to get a new suboptimal population with a certain quality. Then, HS is used to iteratively search for the optimal solution of the new suboptimal population. The procedure of genetic-harmony search algorithm is: Step 1: Initialize parameters: HMS, HMCR, PAR, BW, the range of value, [XMIN, XMAX], the probability of crossover operator pc in GA and so forth.
Step 2: Initialize harmony memory HM: Randomly select solution vectors in the range of the objective function to fill in the matrix HM, and calculate the objective function value of each solution vector. The generation mode of initial population is shown in formula (13). (13) Step 3: Select the initialization harmony memory: this paper uses roulette wheel selection to select the individuals, and a New Harmony memory called HMSelect is generated.
Step 4: make the crossover on HMSelect: All individuals of the population are paired in pairs, and whether they need to cross is judged according to the probability of crossover operator pc. The crossover mode is shown in formula (14) and formula (15), and new population HMNew is generated. (14) (15) c1, c2 represent the newly generated solution vector, and x1, x2 represent the original solution vector in the population.
Step5: HS is used to iteratively search for the optimal solution of HMNew so as to find the optimal solution of the problem.

Specific example
In this paper, the data is from the reference [13], the length of quay is 1200m, the planning period is 24h, the time that each vessel operations berthing and unberthing is 0.5h respectively, tan i is 0.005, and other relevant data is shown in Table 1. We have implemented our heuristic using MATLAB, and the parameters are the size of HM=200, HMCR=0.98, PAR=0.3, BW=0.5, pc=0.9, and Tmax=10000. The solution is shown in Table 2  The results show that the sum of service times for all vessels is 63. 85h.The distance deviation of V1, V2, V3, V4, V5, V6, V7 and V8 is 29m, 22m, 38m, 47m, 478m, 172m, 62m and 82m respectively. In addition, V4 needs to be berthed and the waiting time is 0.2h.

Test example
In order to verify the effectiveness of genetic-harmony search algorithm, we have designed 5 test sets to conduct experiments. The number of vessels in each set is 10,15,20,25, and 30 respectively. The length of vessel (m) is randomly selected from [100, 400], the arrival time (h) is selected randomly from [0,24], the minimum handling time (h) is selected randomly from [4,8], the length of quay is 1000m, 1400m, 1800m, 2200m and 2600m respectively. Other data are the same as the data of example 3.1. According to the method of verification from reference 14, we set the maximum time of Lingo's calculating to 10800s.That means when calculating time of Lingo is over 10800s, the operation is terminated, and the feasible solution is obtained. The parameters are that the size of HM is 200, Tmax which ensures the running time of genetic-harmony search algorithm is more than 300s. At the same time, the running time of genetic-harmony search algorithm is set to 300s.When the program runs for 300s, it stops running and outputs the current optimal solution. The results and running time of Lingo and genetic-harmony search algorithm are shown in Table 3, while the error percentage is shown in Fig  3.  Figure 3. Error percentage It can be seen from Table 3, the optimal solution can be obtained within 10800s by Lingo in first example. However, the running time of latter four examples has exceeded 10800s and the feasible solutions are obtained. From the solutions, from the result, the solution of genetic-harmony search algorithm is basically the same as that of Lingo. It can be seen from Fig.3, the maximum error percentage is 2.38%, the minimum is 0.11%, and the average error percentage is 1.08%, which proves that the genetic-harmony search algorithm can be used to solve the berth allocation problem. Meanwhile, in terms of computing time, the running time of Lingo is much longer than that of the genetic-harmony search algorithm with the increase of the number of vessels. Therefore, we can draw a conclusion that when the number of vessels is too large, the running time of genetic-harmony search algorithm is faster and more efficient than that of Lingo on the basis of basically consistent results.

Conclusion
This paper establishes a mixed integer nonlinear programming model with the minimum total service time of vessels at the port as the objective function. To facilitate the solution procedure, genetic-harmony search algorithm that harmony search algorithm is optimized by genetic algorithm is used to solve the mathematical model and the berth allocation plan is obtained. Furthermore, the effectiveness of geneticharmony search algorithm is verified by test examples of a large number of vessels. This paper compares the results and running time of the algorithm with those of Lingo. The result shows that genetic-harmony search algorithm is faster than lingo when there are many vessels arriving at the port. However, the preferred berth location of vessels is set as 300m, 600m, and 900m in this paper. But the preferred berth location of the ship is closely related to the location of the yard in reality. In the future research, the berth and yard can be jointly scheduled to calculate the preferred berth location of vessels so as to make it more conducive to guiding the actual work of the port in the future.


Use of Modern Information Technologies for Countering Corruption in the Executive Authorities

The article describes the examples of experience of using information systems in anti-corruption activities of executive agencies in the Republic of Tatarstan, Yamal-Nenets Autonomous area, Volgograd and Tula region of Russian Federation. The points raised in the article are: tactics and methodology of countering corruption within the executive agencies of Russian Federation territorial entities. Within the framework of further development of anti-corruption system in the executive agencies of Russian Federation territorial entities, it was suggested to create a new and unique information analysis system. It is expected to test this system with the public authorities of Volgograd region. It was considered, that an active interregional cooperation is needed in the field of anti-corruption activities as well as the most progressive practices introduction in anti-corruption activity throughout all territorial entities of the country.

I. INTRODUCTION
The efficient activity and existence of the State, its functioning as a system, as an integral "organism", depends directly on the orderly operation of all State structures, established specifically to exercise powers of government authorities. Corruption at the level of the above-mentioned structures, disturbs their normal performance, that would result in malfunction of the entire State "organism" and impedes the achievement of the State objectives.
Every year, the International non-governmental organization Transparency International rates countries based on corruption degree, using the corruption perception index. Thus, the index indicator equal to zero shows total Manuscript received May 20, 2019; revised January 11, 2020. The reported study was funded by RFBR according to the research project № 18-29-16119 "Formation of the anti-corruption environment in the state and municipal authorities through the introduction of the information and analytical system "Methods and tactics of countering corruption for state and municipal employees"".
Olga A. Astafurova and Anna S. Borisova are with the Information Systems and Mathematical Modelling Department, Volgograd Institute of Management -the Branch of Russian Presidential Academy of National Economy and Public Administration RANEPA, Volgograd, Russia (e-mail: astoa@vlgr.ranepa.ru, Borisova-AS@mail.ru).
Eyda V. Golomanchuk  absence of corruption in a country; the indicator 100 displays the highest level of corruption prevalence in a country. As for 25 August 2018, Russian Federation took the 135 place in this rating with corruption perception index equal to 29, sharing this position with such countries as Honduras, Laos, Dominican Republic, Kyrgyzstan, Paraguay [1]. The index of corruption degree in Russia has not changed since 2015 [2] and it is a matter of concern. Such a situation means that the anti-corruption measures put into place are ineffective and at the same time, it shows the necessity of their improvement and expansion.

II. PROBLEM STATEMENT
Accordingly to the regulations of the art. 1 of the Federal Law dated 25 December 2008 № 273-ФЗ «Concerning anti-corruption activity» [3] anti-corruption activity concerns all levels of state authority, local authorities, civil society institutions, organizations and individuals within their competences to prevent corruption. It also means detection and further elimination of corruption causes (prevention of corruption), detection, prevention, suppression, detection and investigation of corruption-related offences (fight against corruption), minimization and (or) dealing with impact of corruption offences.
Anti-corruption activity, as well as any other goal-oriented activity, includes such integral parts as tactics and methodology, which ultimately determine its efficiency. If we turn to dictionaries, we may find out, that tactics is an integral part of the martial art, which describes theory and practice of combat, and also has several transferred meanings. For example, according to S.I. Ozhegov's glossary tactics is a combination of means and techniques needed to achieve the intended goal [4]. The legal dictionary gives the following definition: tactics is a short-term behavior, a course of action, intended for a relatively short period, based on the current situation [5]. With regard to anti-corruption activity, tactics should be understood as a complex of specific measures used in this activity.
The complex of measures, reflecting anti-corruption activity tactics, should be made sustainable through regional programs and departmental anti-corruption plans, developed under the National anti-corruption plan. The anti-corruption program in Volgograd region for 2018 -2020 years was approved by the Volgograd region Governor's decree № 622 [6] dated 11 September 2018. In particular, this program will focus on creation and maintenance of conflict commissions, on advanced professional training organization, on legal education of public and municipal officials. Is should also enable anti-corruption expertise of normative legal acts and projects, develop anti-corruption mechanisms in personnel policies within executive and local-government bodies etc.
In various dictionaries, methodology is defined as a set of methods for doing something in practice [4], and as specification of a method, its transformation into an instruction and a clear description of how it works [7]. Within the context of the present topic, methodology should be understood by the modalities of anti-corruption activities and measures realization.
Since corruption within the public authorities is considered as the most dangerous one, it makes sense to analyze practically applied anti-corruption measures specifically within the above-mentioned bodies.
One of the basic directions of anti-corruption activities within the executive agencies of Russian Federation territorial entities includes work with data on incomes and expenses, work with data about property and property related obligations (hereinafter referred to as data on income), reported by civil officers. This kind of work involves consultations, information and awareness-raising sessions for public officials regarding the rules of filling the income certificates, the procedure of such data reporting, analysis of data, provided by the officials, with a view to identifying inconsistencies and to establishing grounds for conducting monitoring activities etc [8].
The analysis of data on income implies comparing it with the same data, that had been reported by the public officers in the previous year. In order to reduce the amount of ineffective and non-resultative control measures, a practice has been introduced in the matter of requesting information about the officials' financial status from the competent authorities and organizations of Volgograd region. It helps to make prior assessment of accuracy and completeness of data on income reported by them.
In many constituent entities of the Russian Federation, including the Volgograd Region, the system for the monthly issuance of a newsletter containing information on amendments to the anti-corruption legislation has been well established; judicial practice analytics on holding responsible for corruption offenses; investigation of typical mistakes made when state and municipal employees provide information on income, expenses, property and property obligations.
When conducting an analysis of income information, personnel services are endowed with the following powers:  send inquiries,  conduct interviews with employees with their consent,  receive from the employees with their consent the necessary explanations,  study the information obtained during the analysis. However, as the study of practice shows, not all bodies use the granted powers when conducting an analysis, not all send requests. Unfortunately, employees often categorically refuse to give explanations on controversial issues, which often entails bringing to a more stringent form of legal liability.
Note that article 59.1 of Federal Law No. 79-ФЗ for non-compliance by civil servants with restrictions and prohibitions, requirements to prevent or resolve a conflict of interest and non-fulfillment of obligations established in order to combat corruption by Federal Law No. 79-ФЗ, Federal Law No. 273-ФЗ and other federal laws, the following penalties are imposed: 1) remark; 2) reprimand; 3) a warning of incomplete job compliance.
Article 59.2 of Federal Law No. 79-FZ stipulates that a civil servant is subject to dismissal in connection with the loss of confidence in the event of: 1) failure to take measures to prevent and (or) resolve a conflict of interest; 2) failure to provide information on their income, expenses, on property and liabilities of a property nature, as well as on income, expenses, on property and property liabilities of their spouse (spouse) and minor children, or the submission of knowingly false or incomplete information; 3) participation on a paid basis in the activities of the governing body of a commercial organization; 4) entrepreneurial activity; 5) joining the composition of governing bodies, board of trustees or supervisory boards, other bodies of foreign non-profit non-governmental organizations and their structural units operating in the territory of the Russian Federation, unless otherwise provided by an international treaty of the Russian Federation or legislation of the Russian Federation; 6) violation by the employee, his spouse and minor children of the ban on opening and having accounts (deposits), storing cash and valuables in foreign banks located outside the territory of the Russian Federation, owning and (or) using foreign financial instruments. The use of all the powers granted during the analysis would allow the personnel services, on the one hand, to more carefully analyze the information provided and respond to detected violations in a timely manner, and on the other hand, minimize the number of so-called «zero» checks. It would also lead to the prosecution of truly guilty employees.
Analytical work with data on income is a quite labor-intensive process. In order to facilitate and accelerate this process, the authorities of several regions develop their own methodologies or apply methodologies, created by other organizations for the purposes of data processing using computer technologies and information systems.

A. Inquiry System EGRIP (Unified State Register of Private Entrepreneurs) and EGRUL (Unified State Register of Legal Entities)
For the purposes of rising the efficiency of anti-corruption measures, Volgograd region executive agencies use the software complex «Inquiry system EGRIP (Unified State Register of Private Entrepreneurs) and EGRUL (Unified State Register of Legal entities) ». For example, by means of electronic access to the above-mentioned resource, it is possible to check whether public officers and citizens, seeking for employment in the civil service, are engaged in business activity as private entrepreneurs, or participate in the administration of any commercial and non-commercial organization, as well as to reveal facts of conflict of interests.

B. SPARK-R
The anti-corruption program in Volgograd region for 2018 -2020, provides access to the information product designed by the international information group "Interfax"the SPARK-R system, for the officials of the Corruption and other offences prevention unit of the Civil service and Personnel directorate of Volgograd region Governor's administration. It has been done in the frame of efficiency-raising activities of Volgograd region authorities in the sphere of countering corruption. This system has been designed to check information accuracy and contractual partners reliability by organizations; it is supposed to use SPARK-R in the Volgograd region Governor's administration for detecting signs of affiliation, in other words the relationship between public officials and persons, holding public positions in Volgograd region and different companies and other physical persons. For this purpose, the system reflects not only the current state of the relationships, but their existence in the past, that will make possible to establish facts of possible violations with greater precision.

C. Software Application for Verification of Data on Income, Reported by Public Officials
The government of Tula region has developed an autonomous software application for verification of data on income, reported by public officials. The use of this software application supposes public officials' personal data entry in the database as well as information placement about their property status. To start the process of income data analysis and verification it is necessary to scan a QR-code, which is placed on the lower-left of the paper copy of the certificate, submitted by the official. This form should be filled in a specially designed program "Certificates-BK". In the process of scanning, the software application automatically reads data about the person, contained in the certificate and compares it with personal data stored in the electronic database. In the event of discrepancy of such data, the software application delivers the result of verification «Verification failed» and specifies data, which does not match while compared. This situation arises, for example, in case of personal identification document replacement, when its number is changed, or a women changes her surname after marriage.
After public official's data verification, the software application proceeds to the next stage in which it compares personal data on income for the current and prior reporting periods, also shows discrepancies in the data, and what is more, it reflects their justification.

D. Unified Information System for Human Resources
The Unified information system for human resources was developed in the Republic of Tatarstan, which is aimed to provide quick access to the information about staff composition of any executive body, to undertake staff review, to standardize human resource procedures and to transfer and update data online. The number of functionalities of the system include: maintenance of personnel management system, as well as staff members' personal data and personal files processing, keeping of the executive body staff registry, maintenance of personnel reserve data, reports generation, including statistic ones, and also anti-corruption functions.
The objective of anti-corruption operation of the Unified information system for human resources is to fill in income certificates in electronic form; to register public officials' property status and to keep records of its history; to enter data about any other remunerated activity, carried by the officials; to enter data and results of any in-service reviews held against public officials; to enter data about meetings of the commissions on compliance with the requirements of the officials' conduct and about management of conflicts of interests; to enter data about reasons for such meetings and the following decisions.
The Integrity of this information systems helps to monitor personal data of any public official and to update automatically personal data for filling out income certificates, as well as to take into consideration any other remunerated activity, conducted by the officials in the process of verification of their data on income.

E. Unified Information System of Volgograd Region
At present, the Civil service and Personnel directorate of Volgograd region Governor's administration is developing a unified information system, which will include both personnel management functions and a wide range of procedures in the field of anti-corruption activities. It is supposed that except options, analogical to similar systems, this information system will make possible to forward requests to the appropriate authorities by means of electronic workflow, as well as to analyze automatically data on income, reported by public officials, and to receive information, in response to requests, from designated authorities and organizations.

F. Testing System INDIGO
From 2016, the Department for specific activities of the Yamal-Nenets Autonomous area, has been successfully implementing the «Plan of legal education in the field of anti-corruption activities», which includes holding of thematic sessions with the participation of public officials, distribution of the relative legal documentation, development of guidance materials on the sessions issues, advisory work, final assessment organization on a regular basis (on a quarterly basis) aimed to assess the officials level of knowledge on the subject matter, further performance analysis and implementation of "work on errors". By means of a systematic approach, this practice helps to enhance effectiveness of the advisory work, to raise the public officials' competence level in the field of anti-corruption activity and to integrate separate methods of corruption offences prevention into a unique practice. The «Testing system INDIGO» is applied in order to automate the process of public officials testing in the Department for specific activities of the Yamal-Nenets Autonomous area. This electronic system helps to organize tests, contests and academic contests, to conduct surveys, to assess the officials' qualification level.

G. Unified System of Anti-corruption Work Monitoring
Among the number of innovative methodologies of anti-corruption activities, a special software for the automated information system called "Unified system of anti-corruption work monitoring" is being used in the executive agencies of Russian Federation territorial entities. It helps to automate the processes of reporting and statistic information production and storage in the field of anti-corruption activity.

H. The Proposed Information Analysis System «Methodology and Tactics of Anti-corruption Activity»
The descriptive analysis of information technologies in the area of anti-corruption activity affirms the need for continued work on support and development of information resources in this area.
The Volgograd region information system in the sphere of anti-corruption activity is not efficient on a number of grounds. Firstly, it is not systematic and focuses exclusively on systematizing the process of providing details about incomes, expenditures, property and property obligations by the public officials. Secondly, it is not a universal system and it is not used at the level of the municipal service. At the level of federal service (both civil and other kind of service) there are some personnel related software applications, but they are not used within all authorities and organizations. Thirdly, none of the existing platforms provides access to tutorial materials and further control of the level of learning.
The most pressing challenge therefore is to create a new information product, which would reflect an integrated approach to the questions of personnel training and retraining and would help to form an anti-corruption environment of state and municipal bodies.
As part of research carried by the authors of the article, an information analysis system is being created, designed to contain data in the following areas: regulatory support of anti-corruption activity at the level of state and municipal service, practicalities of countering corruption, guidelines and methodical recommendations for anti-corruption practices, video lectures, video tutorials and workshops. In the future, it is planned to test the designed system in the public authorities of Volgograd region. The software program to be developed, includes a database, created on the basis of a wide scope of theoretical and practical sources, a training and testing unit, which comprises both training of public officials and metrical methods of knowledge and competence assessment, as well as a unit of additional materials, which will contain regulatory and consultative data [9]. The programs are coded in C# language, within the development environment Visual Studio. The information analysis system will give the possibility to carry out monitoring, analysis, planning and forecasting of key indicators of anti-corruption processes with further visualization of results and report generation. The proposed development is unique both in structure of the information analysis system and in data contained, and it will be possible to replicate it.
The information analysis systems operation starts with user authorization window. It is necessary to differentiate user roles for trainees and supervisors (HR representatives).
The designed interface of a user, undergoing training in the information analysis system, includes the following tabs: 1) information tab (contains information about the training complex), 2) rules and regulations (contains all the necessary legal acts to review), 3) tutorial unit (infographics, video-lectures and video-tutorials), 4) test block (test of proficiency in learned material, psychological tests), 5) control block (consists of the subsystem "my results", which shows the results of tests passed and information messages from the reviewing officers; «user account management», which helps to change the user's password). Except the user's interface, the system contains the reviewing officers' interface. This interface includes the following tabs: 1) Commandsmake possible to add or delete any user of the system, to assign a login, a password, to determine his/her position and internal affiliation, to mention the lengths of service, and public service period. All data about new users are stored in the database. The database is established in the Database management system Microsoft Access. 2) Data viewingthe analytical component, which helps to track results of all users testing. 3) Mailing listmakes possible to add users into the mailing list, appoint day and time for the next testing. Such message will appear in the tab "Organizer tab page" under the user mode of the program. The whole correspondence will be stored in the database. The anti-corruption measures proposed by the authors are based on many years of experience and can help ensure the effectiveness of the various departments of state and municipal authorities on the implementation of anti-corruption measures. The information and analytical system being developed, in addition to the functions already listed, is designed to facilitate the work of personnel services in maintaining documentation regarding the prevention of corruption offenses. In particular, it is planned to automate the work of:  maintaining a register of training on the prevention and combating corruption,  keeping a journal of checking the level of knowledge of employees in the field of anti-corruption in the form of testing,  generating a report on the implementation of training activities,  generation of a report on checking the level of knowledge of the anti-corruption legislation and public service,  formation of a report on the identification of the psychological prerequisites of corruption among public servants,  generating a report of corruption risks in personnel work and making managerial decisions. Anti-corruption procedures for the employment of relatives. In addition, it is planned to add methodological recommendations on conducting trainings aimed at anti-corruption legal education and studying anti-corruption mechanisms that form a negative attitude of employees towards any facts of corruption.

IV. CONCLUSION
The system of fight against corruption includes many different elements, including measures and means of prevention of corruption and struggle with it.
The article presents the results of the analysis of the use of information technologies in the anti-corruption activities of the Executive authorities of a number of subjects of the Russian Federation. It is concluded that none of the currently operating platforms provides access to training materials together with the ability to control their assimilation. The work of the existing information systems is mainly aimed at systematization of the process of providing information about property and property obligations, as well as the income and expenses of civil servants.
The information and analytical system proposed by the authors will allow to develop a preventive direction, provide an objective assessment of the anti-corruption environment and forecast its development, as well as facilitate the work of human resources services to document the learning processes, monitor the quality of training.
Other territorial units of Russian Federation possess an interesting in the area of countering corruption. In our opinion, an active interregional cooperation is needed to create an efficient anti-corruption system. The most progressive practices should be implemented over all the territory of constituent entities of the country. "Russian Presidential Academy of National Economy and Public Service" In 2004 she graduated with honors from the law faculty of FGOU VPO "Volgograd Academy of civil service", in 2016 she accomplished with honors Master's degree program majoring in "Social psychology" in VB of RANEPA. In 2006, she defended her candidate thesis for the degree of Candidate of legal sciences majoring in constitutional right; municipal right -12.00.02.
She has been working in Volgograd Branch of RANEPA from 2010 in the capacity of Senior lecturer of the department of constitutional and administrative right. Before doing that, from 1999 she was in public service at Federal Service of State Registration, Land Register and Mapping (Rosreestr) to Volgograd region. She pursues cooperation with the legislative, executive and judicial authorities of Volgograd region. She is member of Commissions for control over compliance with requirements of official conduct, formed in Volgograd regional Duma, in Volgograd region Governor's administration, in the Federal Penitentiary service of Volgograd region. She delivers lectures and conducts tutorial workshops in governmental authorities of different levels, holds public training events for state and municipal officials in the field of anti-corruption management. Research interests include: constitutional social and economic rights of a person and a citizen; issues of administrative responsibility; administrative Dr. Borisova is the Prize laureate of RF President for support of talented young people, the Prize laureate of Volgograd region in science and technology.
process; legal regulation of public service; legal aspects of anti-corruption management.
Tatyana Y. Yagotinsteva was born in Russia on 25 January 1994.
In 2016, she accomplished with honors the Bachelor's degree program of VB of FGBOU VPO RANEPA in the field of study "Jurisprudence", in 2018 she accomplished with honors the Master's degree program of Volgograd Institute of Management majoring in "Law enforcement in criminal proceedings". In 2019, she accomplished with honors the Master's degree program of Volgograd Institute of Management -the branch of RANEPA in "Civil, family and international private law".
She has been working at the Volgograd region Governor's administration in the capacity of a Chief specialist of the Anti-corruption department of the Directorate for public service and personnel. Research interests include: legal regulation of public service, anti-corruption management, criminal law, protection of human and civil rights.


extfieldsofstudy:
Computer Science
Mathematics

text:
Mimicking Full-Duplex Secure Communications for Buffer-Aided Multi-Relay Systems

This paper considers secure communication in buffer-aided cooperative wireless networks in the presence of one eavesdropper, which can intercept the data transmission from both the source and relay nodes. A new max-ratio relaying protocol is proposed, in which different relays are chosen for reception and transmission according to the ratio of the legitimate channels to the eavesdropper channels, so that the relay selected for reception and the relay selected for transmission can receive and transmit at the same time. It is worth noting that the relay employs a randomize-and-forward (RF) strategy such that the eavesdropper can only decode the signals received in the two hops independently. Theoretical analysis of the secrecy throughput of the proposed scheme is provided and the approximate closed-form expressions are derived, which are verified by simulations. Through numerical results, it is shown that the proposed scheme achieves a significant improvement in secrecy throughput compared with existing relay selection policies.

I. INTRODUCTION
Wireless communication technologies play an important role in military and civil applications, and their rapid developments have been promoting the evolution into the fifth generation (5G) communication [1], [2]. However, the broadcast nature of wireless medium makes the communication over wireless networks susceptible to the interception attacks from unauthorized users (eavesdroppers), and thus guaranteeing the security of wireless communication is becoming an increasingly urgent demand [3].
Traditionally, security issues are addressed by applying cryptographic methods, which utilize secret keys and encryption/decryption algorithms to provide secure data streams, in the upper layers of the network protocol stack [4]. However, their applications may be limited by the inherent difficulty of secret key management and the increasingly powerful computation capability of the eavesdroppers. In the seminal work [5], Wyner introduced the wiretap channel model and established the possibility of creating perfectly secure communication links without relying on secret keys. A rate at which information can be transmitted securely from the source to its intended destination is termed as achievable secrecy rate, and secrecy capacity is the maximal achievable secrecy rate. The secrecy capacity of the scalar Gaussian wiretap channel was analyzed in [6]. Recently, physical layer (PHY) security techniques have been considered as a promising solution to guarantee everlasting secure communication for wireless networks by exploiting the inherent randomness of wireless channels and noise, and have received a lot of attention [7]- [18].
Of particular interest is the secure communication over relay channels, which is one of the fundamental building blocks of communication systems. In [8], the relayeavesdropper channel was studied and different node cooperation strategies were analyzed. It has been shown that cooperative communication not only significantly improves the transmission capacity for wireless networks, but also provides an effective way to improve the secrecy capacity. On top of that, several relay strategies have been designed in literature [9]- [11]. Moreover, recent works have shown that use of buffers at the relays make it possible to store packets and transmit them in more favorable wireless conditions, which greatly improves the security performance of wireless communications [12]- [18]. For instance, in [12], we have designed the link selection and power control policies for secure communications over a buffer-aided two hop communication link.
Note that a relay usually operates in either full-duplex (FD) or half-duplex (HD) mode. In FD relaying, the relays transmit and receive at the same time and frequency, at the cost of hardware complexity [19], [20]. We consider a bufferaided relay system with multiple HD relays in this paper. Inspired by the space full-duplex max-max relay selection (SFD-MMRS) scheme in [21], which mimics FD relaying with HD relays via link selection, we propose a new maxratio relay selection scheme for secure transmission in HD randomize-and-forward (RF) buffer-aided relay networks in this paper. In the proposed scheme, we select different relays for reception and transmission according to the ratio of the legitimate channels to the eavesdropper channels. We analyze the secrecy throughput of the proposed scheme in independent and identically distributed (i.i.d.) Rayleigh fading channels and derive the approximate closed-form expressions. Numerical results in accordance with theoretical analysis show the superiority of the proposed scheme over the existing relay selection schemes.
The reminder of this paper is organized as follows. The system model and two existing max-ratio relay selection scheme for secure buffer-aided cooperative wireless networks are briefly introduced in Section II. In Section III, the relay selection policy is proposed, comprehensive analysis of the secrecy throughput is presented and the approximate

A. System Model
We consider a two-hop wireless communication system consisting of one source node S, one destination node D, a set of K relays R 1 , ..., R K adopting the RF decoding strategy, and one eavesdropper E which can intercept signals from both the source and relay nodes, as shown in Fig. 1. Under the RF relaying strategy, the source and relay use different codebooks to transmit the secret message, so the eavesdropper cannot combine the data transmitted by source and relay [11], [14]. We assume that each node is equipped with a single antenna and operates in the HD mode. We assume that there is a buffer of infinite length at each relay such that each relay can store the information received from the source and transmit it in later time.
We assume that there is no direct link between the source and destination due to high attenuation, and the communications can be established only via relays. The channel coefficients for S − R k , R k − D, S − E and R k − E links at time t are denoted as h sr k (t), h r k d (t), h se (t) and h r k e (t), respectively. The channel is assumed to be stationary and ergodic. We consider the block fading, in which the channel coefficients remain constant during one time slot and vary independently from one to the other. In addition to fading, all wireless links are impaired by additive white Gaussian noise (AWGN) with variance N 0 . Without loss of generality, we assume that the noise variances at the receiving nodes are equal to one, i.e., N 0 = 1. The source and relays are assumed to transmit with power P S and P R respectively.
Throughout this paper, we consider the case of i.i.d. Rayleigh fading for S − R i , R i − D, and R i − E links, which is a typical assumption to facilitate analysis [21]. We assume that the mean of S − R k , S − E, R k − D and R k − E channel gains are E[|h sr k | 2 ] = γ sr , E[|h se | 2 ] = γ se , E[|h r k d | 2 ] = γ rd and E[|h r k e | 2 ] = γ re , respectively, where E[·] denotes the expectation.

B. Existing Relaying Schemes
In this part, we review two existing relay selection protocols for secure buffer-aided cooperative wireless networks.
1) Max-Min-Ratio Relay Selection: With the RF relaying strategy applied at the relays, the instantaneous secrecy rate for the buffer-aided multi-relay systems is obtained as [8] C k (t) = where [·] + = max{·, 0}. The secrecy throughput is given by If the exact knowledge of the eavesdropping channels are available, the best relay node can be selected with the maximum C k (t). This scheme is termed as the max-minratio relay selection in this paper. For convenience, the time index t is ignored in the rest of the paper unless necessary.

2) Max-Link-Ratio Relay Selection:
This protocol chooses the best link with the highest gain ratio among all available source-to-relay and relay-to-destination links [17]. If the exact knowledge of all channels, including the eavesdropping channels h se and h r k e , are available, the max-link-ratio selects the best relay R k as

III. FULL CSI AT THE TRANSMITTERS
In this section, we assume the instantaneous channel state information (CSI) of legitimate channels (i.e., |h sr k | 2 and |h r k d | 2 ) are always known. Regarding the knowledge of eavesdropper CSI, we consider the perfect CSI case where the instantaneous eavesdropper CSI (i.e., |h se | 2 and |h r k e | 2 ) are available in this paper.
Note however that, when the eavesdropper is passive and its behavior can not be monitored, the assumption of the exact CSI of the eavesdroppers link might be unrealistic. We also consider the partial CSI case where only the average gains of the eavesdropping channels are available. Since the derivations are similar, they are omitted in this paper due to the limit of space.

A. Link Selection Policy
Inspired by the SFD-MMRS scheme, if the channel gains of both the legitimate receiver and the eavesdropper are known at the transmitter, the best and the second best relay for reception R r1 and R r2 are selected respectively based on r1 = arg max k∈{1,...,K} and the best and the second best relay for transmission R t1 and R t2 are selected respectively according to Denote Let Then, in the proposed imitating full-duplex max-max-ratio relay selection (IFD-MRRS) policy, the relays selected for reception Rr 1 and transmission Rt 1 are chosen as which means if the best S − R and the best R − D channels do not share the same relay, i.e., r 1 = t 1 , we select the relay with the best S −R channel for reception and the relay with the best R − D channel for transmission. However, if the same relay has the best S − R and R − D channel, i.e., r 1 = t 1 , we choose the best bottleneck link, i.e., if Q > 0, and select the relay with the best S − R channel for reception and the relay with the second best R − D channel for transmission. Conversely, if Q < 0, we select the relay with the second best S − R channel for reception and the relay with the best R − D channel for transmission. Note that the use of buffers make virtual full duplex operations at the relays possible when two different relays are selected for the reception and transmission (see, e.g., [17] and [21]).

B. Secrecy Throughput Analysis
Considering that we employ the RF relaying strategy [10], the eavesdropper cannot combine the data transmitted by source and relay at each time slot. Therefore, when either the source wishes to transmit confidential information to the relay or the relay sends private message to the destination, we can view as a single hop transmission in the presence of the interference from the other hop. Assuming that the eavesdropper employs decoding with successful interference cancellation and the decoding order at the eavesdropper is not known at the source or the relays, the maximum eavesdropping data rate is assumed to be upperbounded by log 2 (1 + P S |h se | 2 ) for the link S − E, and log 2 (1 + P R |h r k e | 2 ) for the link R k − E. If the relay R k is selected for the data transmission, the instantaneous secrecy rate of the first and second hop are lowerbounded by respectively. In the following, we adopt the lowerbound for the analysis, which represents the worst case and specifies the minimum throughput achievable. Then the secrecy throughput for the buffer-aided multirelay system is given by [22] where E[C SR ] and E[C RD ] denote the average secrecy throughput of the S − R and R − D links, respectively. The average secrecy rate of the first and second hop, C k , k ∈ {SR, RD}, can be expressed as [21, (9)] where p s denote the probability that r 1 = t 1 , C k,1 is the average throughput of the best channel of the first and second hop, k ∈ {SR, RD}, C k,21 and C k,22 denote the average throughput of the best channel and the second best channel of the first and second hop when Q > 0 and Q < 0, respectively.
Note that our selection policy involves eight independent channel coefficients, and hence finding the closed-form expressions is very tricky, if not intractable. So we derive the approximate closed-form expressions for the average secrecy throughput of the proposed scheme.
Proposition 1: The average secrecy rate of the first hop E[C SR ] can be approximately expressed as where p s denotes the probability that r 1 = r 2 , p 12 denotes the probability that Q > 0, p 21 denotes the probability that Q < 0, E[C SR,1 ] and E[C SR,2 ] denote the average secrecy capacity of the best and the second best S − R channel, respectively. Similarly, the average secrecy rate of the second hop E[C RD ] can be approximately expressed as where E[C RD,1 ] and E[C RD,2 ] denote the average secrecy capacity of the best and the second best R − D channel, respectively. Proof: Based on (7), we can divide the time index t into three cases correspondingly. If r 1 = t 1 , we select the relay with the best S − R channel for reception and the relay with the best R − D channel for transmission. We denote such indices as t ∈ Ω 1 . If r 1 = t 1 , we need to determine whether Q is positive or negative, if Q > 0, we select the relay with the best S − R channel for reception and the relay with the second best R − D channel for transmission. We denote such time indices as t ∈ Ω 2 . Inversely, if Q < 0, we select the relay with the second best S − R channel for reception and the relay with the best R − D channel for transmission, and denote such time indices as t ∈ Ω 3 . Let N 1 denote the number of times in N transmissions that r 1 = t 1 , and hence N 12 denote the number of time instances in N 1 transmissions that Q > 0, and N 21 denote the number of time instances in N 1 transmissions that Q > 0, i.e., p s = N1 N , p 12 = N12 N1 , and p 21 = N21 N1 . Consider the first hop. The average secrecy rate is given by The average secrecy rate of the second hop can be proved by the same logic.
Proposition 2: Assume that full CSI of the eavesdropper's channels are known. Given P S and P R , the detailed expressions for the terms in (11) and (12) can be expressed as follows: where fZ r 2 (z) = Remark 1: Note that E[C SR ] can be obtained by substituting the above equations into (11).
Similarly, we can obtained the expressions for E[C RD ].
Proof: The probability that the best S − R and the best R − D channels share the same relay is p s = 1 K [21], it follows directly from the fact that the channels for both S−R and R − D links are i.i.d. And it is obvious that we have p 21 = 1 − p 12 . So, to compute the secrecy throughput of IFD-MRRS, we need to find Computation of E[C SR,1 ]: In this case, we denote where x (e −t /t)dt, x > 0 is the exponential integral function. See Appendix A for the derivation of (18).
Computation of E[C RD,1 ]: In this case, we denote z t1 = max k∈{1,2,...,K} we first compute the CDF of z k , then we can calculate the CDF of z t1 . E[C RD,1 ] is given by See Appendix B for the derivation of (19).

Computation of E[C SR,2 ] and E[C RD,2 ]:
To compute E[C SR,2 ] and E[C RD,2 ], we need to compute the CDF of z r2 and z t2 , so we consider the following theorem of order statistics: Theorem 1 ( [23]): Let Z 1 , ..., Z n be n independent variates, each with cdf F (z). Let Z 1 , ..., Z n denote the increasing order of Z 1 , ..., Z n , i.e., Z (1) ≤ Z (2) ≤ Z (n) . Let F (r) (z), (r = 1, ..., n) denote the cdf of the r th order statistic Z (r) . Then the cdf of Z (r) is given by Based on (20), we obtain Since And E[C RD,2 ] can be obtained as Computation of p 12 : Given that z r1 > z r2 , z t1 > z t2 , we have Since we have obtained the CDF of z r2 and z t2 , we can express the probability of Q > 0 , i.e., p 12 , as an integral form and calculate its value numerically.
Note that E[C SR ] can be obtained by substituting (18), (23) and p 12 into (11), E[C RD ] can be obtained by substituting (19), (24) and p 12 into (12). Finally, the approximate closed-form expression of the secrecy throughput of the proposed scheme is obtained by substituting (11) and (12) into (9).
Given the total power constraint SNR of the network, we can allocate the total power to the source and relays to achieve the best performance.
Regarding IFD-MRRS scheme, we need to allocate transmit energy to source and K relays. The sources works for all time slots, therefore, we should have (P S + KP R ) ≤ SNR. For max-link-ratio scheme, we should allocate transmit power to the source and K relays for each time slot to enable each link to be capable of being selected for reception or transmission, so we should have (P S + KP R ) ≤ SNR as well. With max-min-ratio scheme, similarly, we should allocate transmit energy to the source and K relays, albeit the data transmission occupies two time slots, so we should have 1 2 (P S + KP R ) ≤ SNR. Consider the derived expressions of secrecy throughput, once given the total power SNR, it is obvious that when P S is small, the throughput is limited by first hop. On the other hand, when P R is small, the second hop will be the bottleneck of the system. Therefore, there is always an optimal power allocation that maximizes the secrecy throughput.
Definition 1: The maximum secrecy throughput of IFD-

MRRS is given by
Similarly, we can define the maximum secrecy throughput for max-min-ratio scheme and max-link-ratio scheme.

IV. NUMERICAL RESULTS
In this section, simulation results are given to verify the secrecy throughput for the proposed IFD-MRRS scheme. We assume that γ sr = γ rd = γ se = γ re = 2, unless specified otherwise. Fig. 2 plots the secrecy throughput versus P S /P R for each scheme, where the relay number is set as K = 3. We assume SNR = 10 dB. We can find that the secrecy throughput always has a peak value as P S /P R varies, and the proposed scheme achieves the largest throughput. It is interesting that, each scheme does not achieve the maximum secrecy throughput when PS PR = 1, i.e., P S = P R . This is because, the distribution of the channels for the two hops is not symmetric, which can also be seen from the selection strategy (5). We also note that the analytical results obtained based on the derivation in Section III are very close to the simulation results, which verifies the approximate closedform expressions.
In Fig. 3, we compare the maximum secrecy throughput of the proposed scheme with that of two existing max-ratio schemes as SNR varies, where the relay number is set as K = 3. We can find that the proposed scheme achieves the best performance with power allocation. Also, we can find that the approximate expression holds for a wide range of SNR values. Fig. 4 plots the maximum secrecy throughput of each scheme versus the number of relays for SNR = 10 dB. We can find that the proposed scheme achieves the best performance in all cases. It is clearly shown that, the increase of the relay number can significantly improve the secrecy throughput performance.

V. CONCLUSION
In this paper, we have proposed an IFD-MRRS protocol for secure communications over buffer-aided cooperative relay networks. Notice that we considers the RF strategy such that the eavesdropper can only independently decode the signals received in the two hops. With the help of buffers at the relays, different relays for reception and transmission have been selected with the largest or the second largest ratio among S − R and R − D links. Approximate closedform expressions for the secrecy throughput were derived. Numerical results in consistence with the analytical expressions show that the distribution of the channels of the two hop is not symmetric, and the proposed IFD-MRRS scheme achieves significantly higher secrecy throughput compared with two existing relay selection protocols for secure bufferaided cooperative wireless networks.

APPENDIX
A. Proof of expression (18) We first compute the CDF of x, the CDF of x can be obtained as Since x is independent of y, we have f XY (x, y) = f X (x)f Y (y). Then the CDF of z r1 can be calculated as where E 1 (x) = ∞ x (e −t /t)dt, x > 0 is the exponential integral function.

B. Proof of expression (19)
The CDF of z k can be obtained as Then the CDF of z t1 can be calculated as (32)


Stacked Residual Recurrent Neural Networks With Cross-Layer Attention for Text Classification

Text classification is a fundamental task in natural language processing and is essential for many tasks like sentiment analysis and question classification etc. As we all know, different NLP tasks require different linguistic features. Tasks such as text classification requires more semantic features than other tasks such as dependency parsing requiring more syntactic features. Most existing methods focus on improving performance by mixing and calibrating features, without distinguishing the types of features and corresponding effects. In this paper, we propose a stacked residual recurrent neural networks with cross-layer attention model to filter more semantic features for text classification, which named SRCLA. Firstly, we build a stacked network structure to filter different types of linguistic features, and then propose a novel cross-layer attention mechanism that exploits higher-level features to supervise the lower-level features to refine the filtering process. Based on this, more semantic features can be selected for text classification. We conduct experiments on eight text classification tasks, including sentiment analysis, question classification and subjectivity classification and compare with a broad range of baselines. Experimental results show that the proposed approaches achieve the state-of-the-art results on 5 out of 8 tasks.

I. INTRODUCTION
Text classification is a fundamental task in natural language processing (NLP). It is essential for a number of other tasks, such as sentiment analysis [1], question classification [2], and subjectivity classification [3]. Naturally, it has attracted considerable attention from many researchers, and various types of models have been proposed. A conventional method, e.g. a the bag-of-words (BoW) model, would treat a text as a set of unordered words [4]. In recent years, the deep learning models have been widely used for the task to more extensively incorporate feature information, for example, recursive neural networks [1], [5], convolutional neural networks The associate editor coordinating the review of this manuscript and approving it for publication was Biju Issac .
However, most existing methods merely improve the performance via combining different features without distinguishing them by types. It would be very useful to investigate the corresponding effects with the different types. As we all know, different NLP tasks require different linguistic features. The tasks like text classification require more semantic features than other tasks such as dependency parsing relying more syntactic features, as shown in Figure 1.
Inspired by the previous works [10]- [12], which have shown that the different layers of stacked model encode different types of information. Specifically, the higher level layers capture more semantic features and the lower level  layers capture more syntactic features, as shown in Figure 2.
More specifically, if we assume that each layer of features contains three kinds of features: morphology, syntactic and semantics, the proportion of the three types of features is 1. From the low level to the high level, the proportion of three types of features in each layer will change roughly as shown in the left part of Figure 2. In addition, due to the selectivity of attention mechanism, we propose a novel crosslayer attention mechanism that exploits higher-level features to supervise the lower-level features to filter more semantic features for text classification.
In this paper, we propose a stacked residual model with cross-layer attention method (SRCLA) capable of filtering and selecting features for a specific task. Therefore, we first utilize stacked structure to roughly filter linguistic features, and then propose a cross-layer attention method to refine the filtering process, and finally introduce morphological features, e.g. character-level representation, at the lower level of the model to validate the filtering process. The cross-layer attention method uses information reflux to supervise lowlevel linguistic features with high-level semantic features, which can also be referred as the high-to-low attention in the paper. Furthermore, the cross-layer attention method is also able to mitigate the problem of long-distance dependence.
The contributions of this paper can be summarized as the following: • We propose a stacked residual cross-layer attention model (SRCLA) to filter linguistic features. The model first utilizes the stacked structure to filter the linguistic features. Then, the proposed cross-layer attention is used to refine the filtering process. To the best of our knowledge, we are the first to propose the cross-layer attention method.
• We validate SRCLA on eight classification tasks, including sentiment analysis, question classification and subjectivity classification. The experimental results indicate that our approaches achieve the state-of-the-art results on 5 out of 8 tasks compared with a broad range of baselines.
• We have done detailed experiments to verify the validity and rationality of our model, including the effect of cross-layer attention, the effect of model layers, the convenience and effectiveness of integrating external knowledge, and the visualization of attention weights.
The remainder of the paper is organized as the following.
In section 2, the related work about text classification is reviewed. Section 3 presents the proposed model structures for text classification in detail. Section 4 describes the details about the setup of the experiments. Section 5 presents the experimental results and the analyses. The conclusion and the future work are in section 6.

II. RELATED WORK
Deep learning based neural network models have achieved great improvement on text classification tasks. These models generally consist of a projection layer that maps words of text to vectors, and then combines the vectors with different neural networks to make a fixed-length representation. According to the relevance of our work, we divide it in the following categories.

A. RECURRENT NEURAL NETWORKS
RNN has obtained much attention because of the superior ability to preserve the sequence information over time. Some variants(e.g., LSTM [13] and GRU [14]) have also been proposed because RNN can't handle the gradient vanishing and long-distance dependencies problem. In order to utilize both the past and the future information, BiLSTM [15] are proposed to extend the unidirectional LSTM by introducing a second hidden layer. Reference [8] generalized LSTM to Tree-LSTM where each LSTM unit can gain information from its children units. Reference [9] introduced BiLSTM with attention mechanism to automatically select features that have decisive effect on classification.

B. STACKED NEURAL NETWORKS
Due to the strong representation ability of stacked neural networks, stacked neural networks have been widely used in many tasks. Network depth is of central importance for neural networks as a powerful machine paradigm [16]. Theoretical evidence indicates that deeper networks can be exponentially more efficient at representing certain function classes [17], [18]. Although the strong representation ability of deep model, deep networks suffer from what are commonly referred to as the vanishing and exploding gradient problems [19], [20]. In order to overcome the vanishing and exploding gradient problems of deep model, Highway networks [21] and Residual networks [22] are proposed. In recent years, deep networks have been widely used in many NLP tasks, such as text classification [23], language modeling [24], etc.
Besides, the Transformer model [25] of using attention only also takes six blocks. Reference [10]- [12] have shown that different layers of deep BiRNNs encode different types of information, specifically that the higher-level layers capture context-dependent aspects of word meaning while the lowerlevel capture aspects of syntax, as shown in Figure 2. Inspired by this, we propose our model to better use information of different layers.

C. ATTENTION-BASED NEURAL NETWORKS
Attention-based Neural Networks have attracted growing interest due to their ability to explicitly capture the importance of context words. Reference [9] proposed a BiLSTM-Atten model for text classification, which applies attention mechanism in BiLSTM hidden outputs to select the important features for the task. Reference [26] proposed a hierarchical attention model for document classification, which consists of two-level attention mechanisms (wordlevel and sentence-level). [27] proposed a directional selfattention for RNN/CNN-free natural language understanding. Besides, Attention-based networks and its variants have also been widely used in other tasks, such as machine translation [25], [28], etc.

D. OTHER NEURAL NETWORKS
In addition to the models described above, many other neural networks have also been proposed for text classification. [1], [5] introduced recursive neural tensor network to build representations of phrases and sentences by combining neighbour constituents based on the parsing tree. Reference [6], [7] utilized Convolution neural networks for text classification. Reference [29] attempted to leverage external linguistic knowledge for sentiment classification. Some works based on supervised topic model [30], semantically rich hybrid model [31], noisy label aggregation [32] and other methods are also proposed for text classification. Figure 3 is the structure of the SRCLA model, which contains four processes: 1 , 2 , 3 , 4 . In this section, We will elaborate the details of each process in the model corresponding to 1 , 2 , 3 , 4 in Figure 3.

A. WORD EMBEDDING
As shown in Figure 3, 1 represents the process of word embedding. In a NLP task, a sentence is often treated as a sequence of discrete tokens, i.e. words or characters. We denote a sentence as V = [v 1 , v 2 , . . . , v n ], in which v i could be a one-hot vector which dimension length equals to the number of distinct tokens N . A pre-trained token embedding, e.g. Glove [33], is applied to V and all discrete tokens are transformed to a sequence of low-dimensional dense vector This preprocess can be written as the equation, where W (e) ∈ R d×N denotes word embedding weight matrix, X ∈ R d×n denotes the low-dimensional dense vector representations of a sentence.

B. STACKED RESIDUAL BiLSTM
The specific stacked residual BiLSTM structure in SRCLA model is shown in Figure 4, which corresponding to the process 2 in Figure 3. LSTM was firstly proposed by [13] VOLUME 8, 2020 to overcome gradient vanishing problem of RNN. The main idea is to introduce an adaptive gating mechanism, which decides the degree to keep the previous state and memorize extracted features of the current data input. The stacked BiLSTM is illustrated in 2 in Figure 3. Given a sentence X = [x 1 , x 2 , . . . , x n ], where n is the length of the input sentence. x i ∈ R d is the d-dimensional word vector corresponding to the i-th word in the sentence X . LSTM processes it word by word, at time-step t, the memory c t and the hidden state h t are updated with the following equations: Here, h t ∈ R m , in which m denotes the hidden dimension of LSTM. x t is the input at the current time-step. i, f and o are the input gate activation, the forget gate activation and the output gate activation respectively.ĉ is the current cell state. σ denotes the logistic sigmoid function and denotes elementwise multiplication. For the sequence modeling tasks, it is beneficial to have access to the past context as well as the future context. In this paper, BiLSTM is utilized to capture past and future information. So, the output of the i-th word is shown in the following equation: Here, ⊕ denotes the concatenation of hidden states of the forward LSTM and the backward LSTM. So, h i ∈ R 2m . In order to prevent the gradient vanishing in the stacked model, we introduce a residual connection between any two adjacent layers. The specific structure of stacked residual BiLSTM is shown in Figure 4. As shown in Figure 4, the concatenation of the hidden state h l t of the l th BiLSTM unit and the hidden state h l−1 t of the (l − 1) th BiLSTM unit is used as the input to the (l + 1) th BiLSTM unit at the time-step t, which is shown in the following equation: Here, ⊕ denotes the concatenation operation.

C. CROSS-LAYER ATTENTION
As shown in Figure 3, 3 denotes the process of the Cross-Layer Attention. Let the number of BiLSTM be L. The input sentence X = [x 1 , x 2 , . . . , x n ], where X ∈ R n×d . n denotes the length of the sentence and d denotes the dimension of word embedding. The hidden output of the K th BiLSTM . . , h L n of L th BiLSTM as query, as shown in the following equation: where q ∈ R 2d L , d L denotes the hidden dimension of the L th LSTM. We copy q n times to get query matrix Q, where Q ∈ R n×2d L . Then, we compute the attention weight by the additive method [28], as shown in the following equations: Here, we take tanh function as the activation function. p(z = i|H K , Q) denotes the importance of the i th hidden output of the K th BiLSTM. According to the normalized attention weight p(z|H K , Q), we get the weighted average representation of the hidden output of the K th BiLSTM, as shown in the following equation: where h K s ∈ R 2d K , d K denotes the hidden dimension of the k th LSTM. We concatenate h L s and h K s to get the final sentence representation h s , as shown in the following equation: where h s ∈ R n×2(d K +d L ) . The value of K depends on the value of L. K is a subset of all sets made up of elements from {1, 2, . . . , L − 1}. Considering the semantic features contained in each layer, the amount of dataset, the running time and the parameters, we specify the values of L and K , as shown in Table 2. Later, we will conduct a detailed experiment to analyze why we set the values of L and K in this way.

D. MLP CLASSIFIER
For text classification, we take a two-layer perceptron and softmax as the classifier to predict the labelŷ from a discrete set of classes Y . The classifier takes the final sentence representation h s as input and takes relu as the activation function, as shown in the following equations: We take the cross-entropy loss with L2 regularization as the training objective, as shown in the following equation: where t ∈ R m is the one-hot represented ground truth. p(y) ∈ R m is the estimated probability for each class by softmax. m is the number of target classes. λ is an L2 regularization hyper-parameter. θ is the model parameters.

IV. EXPERIMENTAL SETUP A. DATASETS
As we all know, text classification is a fundamental task for a number of more advanced NLP tasks, such as sentiment analysis, problem classification, subjective recognition, etc. In order to test the validity and robustness of our model, eight datasets are carefully selected from many as being representative. The statistics of the datasets are in Table 1.
• SST-1: Stanford Sentiment Treebank. It is an extension of MR with train/dev/test splits provided and finegrained labels (very positive, positive, neutral, negative, very negative), re-labeled by [1].
• SST-2: It is the same as SST-1 with neutral reviews removed and binary labels. For both experiments, phrases and sentences are used to train the model, but only the results on the sentences are marked while testing [1].
• TREC: TREC question dataset. The task involves classifying a question into one of six question types (whether the question is about person, location, numeric information, etc.) [2].
• MR: Movie reviews with one sentence per review. Classification involves detecting positive/negative reviews [34]. • Subj: Subjectivity dataset. The task is to classify a sentence as being subjective or objective [3].
• CR: Customer reviews of various products (cameras, MP3s etc.). The task is to predict positive/negative reviews [35].

B. HYPER-PARAMETER SETTINGS AND TRAINING
For the datasets without a development set, we randomly select 10% of the training data as the development set. In the experiments, all word vectors are initialized by Glove 840B pre-trained vectors [33]. The out-of-vocabulary word embedding is initialized by randomly sampling from uniform distribution in [-0.25, 0.25]. The word embedding is fine-tuned during training to improve the classification performance. We minimize objective functions by Adagrad [41] with batch size 25 and initial learning rate 0.003. In order to prevent overfitting, we adopt early stop with the tolerance of 5. For regularization, we use L2 penalty with coefficient 10 −5 over the parameters, besides, we also employ Dropout operation [42] with dropout rate of 0.5 for the word embedding, character embedding, the BiLSTM layer and the penultimate layer. Considering the running time and parameters, we specify the values of L and K , the values are shown in Table 2. The set of LSTM hidden dimension is [512, 1024, 2048, 4096]. All weight matrices are initialized by Glorot Initialization [43]. All models are implemented with Pytorch and run on single Nvidia GTX 2080Ti graphic card.

V. RESULTS AND ANALYSIS A. OVERALL PERFORMANCE
This work implements two models, Stacked-BiLSTM and SRCLA. Table 3 presents the performance of the two models along with the state-of-the-art models on seven classification tasks. Our models achieve the excellent performances on seven classification tasks, especially SRCLA achieves the state-of-the-art results on SST-1, SST-2, MR and CR dataset respectively. According to the results in Table 3, we have the following observations: Firstly, the proposed model SRCLA is effective. Compared with all baselines, our models have considerably improved the performance on all datasets. For example, compared with BiLSTM-Att, SRCLA achieves 3.0% improvement over SST-1 datatset, 2.0% improvement over SST-2 dataset, 2.5% improvement over MR dataset and 1.5% improvement over  [1]. DRNN: Deep recursive neural networks for compositility in language [5]. DCNN: A convolutional neural network for modeling sentences [7]. CNN-nonstatic/MC: Convolutional neural networks for sentence classification [6]. TBCNN: Disriminative neural sentence modeling by tree-based convolution [38]. BiLSTM/Tree-LSTM: Improved semantic representations from tree-structured long short-term memory networks [8]. BiLSTM-Att/2DCNN: Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling [9]. DiSAN: DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding [27]. AdaSent: Self-adaptive hierarchical sentence model [39]. NSCL: Context-Sensitive Lexicon Features for Neural Sentiment Analysis [40]. LR-BiLSTM: Linguistically Regularized LSTM for Sentiment Classification [29]. The best result for each dataset is in bold. The result marked with # are retrived from [29]. * indicates that our model is significantly better than most baseline models such as LSTM, BiLSTM, Tree-LSTM, LSTM-Att, DiSAN, etc with p < 0.05 based on one-tailed unpaired t-test.
Subj dataset. Compared with BiLSTM, the Stacked-BiLSTM improves the result by a larger margin, which reflects that stacked structure can filter linguistic features. Compared with Stacked-BiLSTM, SRCLA achieves about 1.5% improvement over all datasets, which proves that the cross-layer attention method can refine the filtering process and select more semantic features. In summary, our model is able to filter linguistic features and select more semantic features for text classification.
Secondly, SRCLA can mitigate the long-distance dependence problem. Compared with DiSAN model that adopts the directional self-attention method making the distance between any two words one, our models outperform it on all datasets. This demonstrates the capability on the longdistance dependence problem. To further validate the observation, we apply SRCLA model to the document-level dataset IMDB. The results are in Table 4. As shown in Table 4, compared with a broad range of baselines, SRCLA model achieves the state-of-the-art results by a larger margin, which further prove the observation.
Thirdly, SRCLA can further improve the results by integrating external knowledge conveniently. The previous works, such as RNTN/TBCNN/Tree-LSTM/LR-BiLSTM, have used external knowledge. Our current models outperform them without introducing external knowledge. We can further improve the result by integrating morphological knowledge (e.g. character-level embedding) at the lower layer, introducing syntactic knowledge (e.g. parsing) in middle layer and introducing semantic knowledge (e.g. sentiment lexicon) in higher layer. In the following subsection, we will do further experiments to verify the observation.
Lastly, compared with context feature, n-gram feature is more important in short text classification. Compared the   TABLE 4. Classification results about IMDB dataset on several standard benchmarks. LSTM+LA: Neural sentiment classification with user and product attention [44]. LSTM+CBA+LA: A cognition based attention model for sentiment analysis [45]. LSTM+dynamic skip: Long Short-Term Memory with Dynamic Skip Connections [46]. The result marked with # are retrived from [46]. * indicates that our model is significantly better than LSTM, LSTM+LA, LSTM+CBA+LA, LSTM-Att, LSTM+dynanmic skip with p < 0.05 based on one-tailed unpaired t-test. BiLSTM-2DCNN model that capture more n-gram feature by CNN model, we don't get the best result on TREC dataset. Compared the AdaSent model that captures more n-gram feature by complex mathematical methods, we don't get the best result on MPQA dataset. We think that the sentence length of TREC and MPQA dataset is relatively short, so n-gram feature may play more important role than context feature in short text classification.

B. EFFECTIVENESS OF CROSS-LAYER ATTENTION
In order to analyse the effectiveness of cross-layer attention, we conduct the ablation experiments on MR, Subj, CR and MPQA dataset. The experimental results are shown in Table 5. We have the following observations: Firstly, the cross-layer attention is effective. Compared with SRCLA, SRCLA-CLA decreases 1.4% on average over all dataset, which demonstrates the observation on the whole. Compared with SRCLA, SRCLA-CLA+ decreases 1.1% on  TABLE 5. Effectiveness of Cross-layer attention. SRCLA-CLA: Without the cross-layer attention, and is actually a stacked residual model (the representation of L th layer as the final presentation). SRCLA-CLA+: Based on SRCLA-CLA, concatenate directly the representation of K th layer and the representation of L th layer as the final representation. The best result is in bold. * indicates that our model is significantly better than Stacked-BiLSTM, SRCLA-CLA with p < 0.05 based on one-tailed unpaired t-test. average over all datasets by concatenating directly the representation of previous layer and the representation of last layer, which further proves the filtering effect of cross-layer attention. In summary, our model with the attention method is able to select relevant and important features for tasks.
Secondly, more semantic features are beneficial for text classification. Compared with SRCLA-CLA that only uses the representation of the highest layer, SRCLA-CLA+ improves the result on all datasets by concatenating the representation of both the highest layer and other high layers. As shown in Figure 2, high layers contain more semantic features. From this perspective, it is shown that text classification requires more semantic features.
Lastly, residual structure is effective in stacked model. Compared with the Stacked-BilSTM, SRCLA-CLA further improves the result on all datasets by introducing the residual structure into the Stacked-BiLSTM, which proves the observation.

C. EFFECTIVENESS OF L AND K SELECTION
In order to explore the influence of L and K on the model, we enumerate all combinations of L and K in four layers. The experiments are conducted on MR, Subj, CR and MPQA datasets. The experimental results under different L and K are shown in Table 6. According to Table 6, we have the following observations: Firstly, Higher layers contain more semantic features than lower layers. As shown in Table 6, in all datasets, when L = 3, the result of K = {2} is better than that of K = {1}. When L = 4, the result of K = {3} is better than that of Secondly, different datasets need different L and K combinations to achieve the best results. According to the results in Table 6, MR and CR datasets achieve the best results when L = 3, K = {2}, Subj dataset achieves the best result when L = 2, K = {1}, and MPQA dataset achieves the best result when L = 3, K = {1, 2}. It is supposed to be related with the size of datasets, the number of model parameters and the linguistic features required for tasks.
Lastly, how to determine the optimal combination of L and K. As shown in Table 6, different values of L have great influence on the results, and when the value of L is determined, the value of K has little influence on the results. So, we suggest that we first use k = {L − 1} to to get the optimal value of L, then use K = {L − 1}, K = {L − 1, L − 2}. . . to obtain the optimal value of K , and we suggest that the number of elements in K should meet the following formula: Here, |K | represents the number of elements in K . indicates rounding up.

D. CONVENIENCE AND EFFECTIVENESS OF INTRODUCING EXTERNAL KNOWLEDGE
In order to verify the convenience and effectiveness of introducing external knowledge into our model, we introduce the morphological knowledge (e.g. character-level embedding) at the lower layer of SRCLA model, which we call Char-CNN-SRCLA. Specifically, we apply CNN model to character embedding, take the output of CNN as character representation, and use the concatenation of character-level representation and corresponding word embedding as the input of SRCLA model. Here, the character embedding is initialized by randomly sampling from uniform distribution in [−0.25, 0.25], the dimension of character embedding is 50. The character embedding is fine-tuned during training to improve the classification performance. We set CNN filters of width [3,4,5]   Firstly, introducing morphological knowledge into SRClA is effective. Compared with SRCLA, Char-CNN-SRCLA increases 0.3% on average over all datasets, which demonstrate the observation.
Secondly, introducing external knowledge into SRCLA is intuitive and convenient. As shown in Figure 2, different layers contain different types of information, so we can introduce corresponding types of external knowledge into corresponding layers.

E. VISUALIZATION OF ATTENTION
In order to get a better understanding of our model and validate that this model is able to filter and select more semantic features, we visualize the obtained attention weight p(z|H K , Q) in Equation (9). Figure 5 shows the attention visualizations for some sentences sampling from MR dataset. The color depth indicates the importance degree of attention weight for the sentence, the darker the more important.
MR belongs to sentiment classification dataset, so we will analyze the results from the perspective of sentiment classification. As we all know, analyzing the emotion of a text is to recognize the emotion expression in the text. According to the habit of daily expression, the emotional expression in the text can be roughly divided into two categories: explicit emotional expression and implicit emotional expression. The explicit emotional expression can be divided into one expression, multiple expressions with the same polarity, multiple expressions with opposite polarity, as shown in the Figure 5. Next, we will analyze the results of four samples one by one.
For the first sample, it belongs to explicit emotional expression and contains only one expression, which is easy to identify for most models. As shown in the Figure 5, our model can select positive emotion words or phrases such as ''enjoyable'' and ''feel good'', which proves the validity of our model.
For the second sample, it belongs to explicit emotional expression and contains multiple expressions, but the polarity is the same, and it is easy to distinguish for some models. As shown in the Figure 5, our model can choose emotional phrases such as ''never dull'' and ''looks good'', which shows that our model can be considered from a semantic perspective.
For the third example, it belongs to explicit emotional expression and contains multiple expressions, but the polarity is opposite, which is difficult to identify for such samples. As shown in the Figure 5, our model can accurately identify the negative affective phrase ''extremely silly peace'', which will not be affected by the positive affective phrases such as ''fast packed'', ''glitzy'', etc., which proves the robustness of our model.
For the fourth sample, it belongs to implicit emotion expression, which is the most difficult to identify. Because there is no clear emotional expression, it is necessary to distinguish the deep semantics of the text, and the requirements for the model are high. As shown in the Figure 5, our model can select a relatively negative emotional phrase ''get ready to take off'', which further verifies that our model can filter and select more semantic features.

VI. CONCLUSION
In this paper, two models for text classification are presented, which are SRCLA and Char-CNN-SRCLA. They are different from the previous methods that improve performance by mixed linguistic features. The novel models are able to filter linguistic features and select more semantic features for text classification. SRCLA first utilizes a stacked structure to filter linguistic features, and then uses a novel attention method. The method is a cross-layer attention capable of refining the filtering process. The Char-CNN-SRCLA further introduces morphological features at the lower layer of SRCLA to validate the filtering process. The experiments are conducted on eight text classification tasks, and the results show that both SRCLA and Char-CNN-SRCLA achieve excellent performance compared with a broad range of baselines. Especially, the SRCLA and Char-CNN-SRCLA achieve state-of-the-art results on 5 out of 8 tasks. To better understand the effectiveness of the proposed models, a series of elimination experiments are carried out and analysed in the paper. It is demonstrated that filtering linguistic features enhances performance in text classification and possibly in other NLP tasks.
In the future, we would continue the study in three directions, Firstly, we will experiment the different directions of attention. The current direction of SRCLA is from high to low, which can select more semantic features. Could a lowto-high attention assist in selecting features for the more syntactic-relied tasks like part-of-speech tagging? Secondly, we would integrate different external knowledge at the different layer (e.g., Char-CNN-SRCLA). Last but not least, we will apply our model to other NLP tasks, such as natural language inference, aspect-level sentiment analysis, etc.
YANGYANG LAN received the bachelor's degree in software engineering from Northwestern Polytechnic University, in 2017. He is currently pursuing degree with the Department of Computer Science and Technology, Xi'an Jiaotong University. His research interests include representation learning, sentiment analysis, and natural language processing.
YAZHOU HAO received the bachelor's degree in computer science and technology from Chang'an University, in 2013. He is currently pursuing the Ph.D. degree with the Department of Computer Science and Technology, Xi'an Jiaotong University. His research interests include representation learning, sentiment analysis, and natural language processing.
KUI XIA received the bachelor's degree in computer science and technology from Xidian University, in 2018. He is currently pursuing the degree with the Department of Computer Science and Technology, Xi'an Jiaotong University. His research interests include representation learning, sentiment analysis, and natural language processing. BUYUE QIAN received the Ph.D. degree from the University of California, Davis, in 2013. He was a Former Researcher at the IBM Watson Research Center. He is currently an Associate Professor with the Department of Computer Science and Technology, Xi'an Jiaotong University. In the past few years, he has participated in or led many large research projects, including NSF, onr, NIH, Yahoo, and IBM. He have been published more than 20 articles (CCF class A or B) in several top conferences and journals of data mining and artificial intelligence. In terms of invention patents, there are 23 U.S. patents, involving algorithms, finance, e-commerce, medical, and other fields, of which more than half are the first patent inventors. He has a wide range of interests in data mining and machine learning, specializing in active learning, machine learning-based sorting, spectral clustering, semi supervised learning, and big data analysis-based on hash algorithm. In terms of academic review, he is currently a member of the procedure Committee of several international top academic conferences,