Revisiting three anonymous two-factor authentication schemes for roaming service in global mobility net- works

Designing a secure and efficient anonymous authentication protocol for roaming services in global mobile networks is a hot topic in the field of information security protocols. Based on the widely accepted attacker model, this paper analyzes the security of three representative anonymous authentication protocols in global mobile networks. It is pointed out that: (1) Xu et al.’s protocol cannot resist the claimed offline password guessing attack and mobile user impersonation attack, and do not achieve mobile user untraceability and forward security; (2) Gupta et al.’s protocol cannot resist offline password guessing attacks, and temporary information disclosure attacks; (3) Madhusudhan et al.’s protocol cannot resistmobile user impersonation attack, foreign agent impersonation attack, replay attack, offline password guessing attack and session key disclosure attack, and cannot realize the anonymity and untraceability and forward security of users. It is emphasized that the fundamental reason for the failure of these protocols lies in the violation of the four basic principles of protocol design: Public key principle, Forward security principle, User anonymity principle and Anti offline guessing attack principle. The specific mistakes of these schemes are clarified, and the corresponding correction methods are proposed.

INTRODUCTION
With the rapid growth of Internet application demand, the Global Mobility Network (GLOMONET) gradually shows a wide range of application prospects in various fields closely related to people' s lives. This kind of network makes it easy for people to enjoy the convenience of mobile network. In the GLOMONET, when a travel mobile user with wireless device wants to get network service, she can pass the authentication of global mobile network with the help of home agent (HA) and be allowed to use the roaming service of foreign agent (FA) anywhere. Due to the openness and mobility of mobile networks and the limited resources of mobile devices, communication is vulnerable to various attacks, such as offline guessing attacks and failure to provide forward security. According to O'Dea [1] , the forecast number of mobile users worldwide in 2024 will be 7.41 billion, 6.6% more than the 6.95 billion users in 2020. In other words, everyone in the world has at least one mobile device on average. The huge personal data of users is in urgent need of privacy protection. In the Internet of things, access control and authentication technology has been effectively studied [2][3][4][5][6][7] . Nevertheless, how to ensure the authenticity of communication entities, prevent the abuse of services and illegal access to resources, without reducing system availability, remains a serious challenge to the GLOMONET.

Related work
In 1997, Suzuki and Nakada [8] proposed an authentication technique for GLOMONET. The proposed authentication technique which only consists of two phases: registration phase and authentication phase, is suitable for the distributed security management of GLOMONET. Since then, a large number of authentication and key agreement protocols have been proposed for GLOMONET. In 2005, Lee et al. [9] proposed an authentication scheme without password. In the proposed scheme, the home network cannot obtain the authentication key between the roaming user and the visited network. In 2006, Lee et al. [10] proposed an enhanced scheme for eliminating the security weaknesses of Zhu and Ma' s scheme [11] . However, in 2009, Chang et al. [12] pointed out Lee et al.' s scheme suffers from the impersonation attack. Afterwards, Chang et al. [12] proposed an authentication scheme for roaming service that only used one-way hash functions and exclusive-OR operations in order to obtain security goals.
In 2010, Wu et al. [13] proposed a novel lightweight authentication scheme used one-way hash functions and symmetric cryptographic operations in GLOMONET for roaming service to provide user anonymity. In 2011, Zhou and Xu [14] also proposed a provable secure two-factor authentication protocol with anonymity for roaming service based on Diffie-Hellman assumption. In 2013, to overcome two kinds of impersonation attacks, He et al. [15] proposed anonymous two-factor authentication protocol for Consumer Roaming Service. However, He et al.'s scheme [15] is vulnerable to time synchronization attack.
In 2013, Jiang et al. showed that He et al.' s scheme [16] cannot achieve two-factor security, and it suffers from multiple known attacks. In order to improve security, Jiang et al. [17] proposed a scheme which based on quadratic residue assumption for GLOMONET. But it can be observed that Jiang et al.' s scheme [17] suffers denial of service attack. Moreover, Wen et al. [18] showed that Jiang et al.' s scheme [17] is vulnerable to replay attack and the stolen-verifier attack.
In 2017, Lee et al. [19] showed that Mun et al.' s scheme [20] is insecure against impersonation attack and manin-the-middle attack, and it cannot achieve anonymity. Subsequently, Lee et al. [19] only used one-way hash function and exclusive-OR operation to propose an improved scheme for GLOMONET.
In 2018, Xu et al. [21] showed that Gope-Hwang' s scheme [22] cannot resist replay attack and synchronous attack. Afterwards, they proposed an authentication and key agreement protocol for GLOMONET used only hash functions and symmetric cryptosystem. While Gupta et al. [23] showed that Wu et al.' s scheme [24] cannot provide untraceability of the mobile user.What' s more, it' s inefficiency for the verification of the wrong password. Because there are many attacks in the existing protocols, in order to eliminate these problems, Madhusudhan and Shashidhara [25] proposed a secure authentication and key agreement scheme for mobile roaming users in 2019.
Combining with a large number of related literatures, we can observe that such authentication protocols in GLOMONET can be divided into three categories based on the different basic cryptography techniques used: (1) based on hash function and exclusive-OR operation; (2) based on hash function, exclusive-OR operation and symmetric cryptography; (3) based on public key cryptography. The authentication protocols of (1) and (2) always have some security problems, such as offline password attack and perfect forward secrecy. However, when the public key cryptography is not used properly, the authentication protocols of (3) are also vulnerable to various attacks.

Contribution
We provide a better understanding of user anonymous and untraceability, offline password guessing attack and perfect forward secrecy, etc, and we believe it would facilitate the design of secure and usability authentication and key agreement schemes for GLOMONET. Specifically, a summary of our contributions are as follows: a) We analyze Xu et al. [21] 's, Gupta et al. [23] ' s and Madhusudhan et al. [25] ' s protocols, and find that none of the three anonymous authentication protocols in GLOMONET environment can achieve the user anonymity and untraceability, and they are vulnerable to offline password guessing attacks, and there are forward secrecy issues and mobile user impersonation attack, etc. b) We highlight four basic design principles of anonymous two-factor authentication protocol in GLOMONET: (1) Public key technology principle. Under the assumption of non tamper resistant smart card, using public key technology is a necessary condition to resist offline password guessing attack; (2) Perfect forward secrecy principle. Public key technology is a necessary condition for preserving perfect forward secrecy; (3) Mobile users anonymity and untraceability principle. Using public key technology is a necessary condition for realizing user anonymity and untraceability; (4) Anti offline password guessing principle. At present, using "Fuzzy-Verifiers" and "Honeywords" technology is a good choice for realizing anti offline password guessing attack [26] .

Roadmap of this paper
The remainder of this paper is as follows: Section 2 describes the system model and attacker model. Section 3 reviews the efficient anonymous authentication scheme proposed by Xu et al. And the security of the scheme is analyzed in Section 4. Section 5 describes the two-factor authentication scheme based on quadratic residue hypothesis proposed by Gupta et al. And Section 6 points out the security problems of the scheme. Section 7 and Section 8 respectively review and analyze the scheme of Madhusudhan et al. Section 9 highlights four basic design principles of two-factor authentication scheme in GLOMONET. Finally, Section 10 summarizes the conclusion.

SYSTEM MODEL AND ATTACKER MODEL
This section introduces the system model of authentication and key agreement in GLOMONET and attacker model. The notations used in this paper are presented in Table 1.

System model
In a two-factor authentication and key exchange protocol for roaming service in GLOMONET, there exist there participants namely the mobile user (MU), the FA and the HA. First of all, MU needs to register themselves with HA before she wants to get mobile network roaming service. In the registration phase, MU sends the registration request to HA, and sends the identity or password information after privacy processing to HA on the secure channel. Then, HA stores some key parameters processed by cryptography in a new smart card and sends the smart card to the corresponding MU. Then, in order to obtain the access rights of FA, MU needs the assistance of HA. The specific process is as follows: (1) MU sends roaming service login request to FA; (2) FA sends authentication request to HA; (3) HA sends response to FA after authenticating FA; (4) FA sends response to user after authenticating HA; (5) After MU authenticates FA, the session key is calculated. Therefore, mobile users can use the session key to enjoy roaming service safely.

1)
All parameters stored in the smart card of the mobile users can be extracted using side channel attack by the adversary A.  [37,38] the space of identities and passwords is very limited in real life, |D | ≤ |D | ≤ 10 6 . 4) Any adversary A can register as a legitimate mobile user if anyone can do this. 5) A may can obtain previous session keys by improper erasure(e.g. using digital forensic techniques). 6) A can obtain the private key of the mobile user, the home agent and the foreign agent when carrying out the perfect forward secrecy attack.

XU ET AL.'S SCHEME
In 2018, Xu et al. [21] pointed out that Gopa and Hwang' s scheme [22] is vulnerable to replay attack and has the problem of computational burden. Afterwards, Xu et al. [21] designed an improved authentication scheme for roaming service in GLOMONET. However, here we show that Xu et al.' s scheme [21] still has several serious defects, including lack of mobile user untraceability and perfect forward secrecy, offline password guessing attack, and mobile user impersonation attack.

S1.
A new mobile user MU sends her real identity to the home agent HA through the secure channel. S2. On receiving the , HA generates two random numbers ℎ and 0 and then calculates ℎ = ℎ( Finally, MU replaces , ℎ with * , * ℎ , respectively. And the smart card SC contains these param-

Authentication and key agreement phase
In this part, with the help of the home agent HA, the mobile user MU and the foreign agent FA will authenticate each other and establish a common session key. S1. MU generates a random number and inputs her identity and password into the smart card SC. Then, SC computes ℎ = *  || ℎ ) and replaces with .

Password update phase
The mobile user MU can change her password by itself. In order to change the password, MU needs to use her old password and enters the new password * . After that, she calculates ℎ = * ℎ , * * } in the smart card, respectively.

Lack of mobile user untraceability
We suppose that A gets the message { , , ℎ , 1 , 1 }. Since = ( || 0 ) is a fixed value, A can track the login request behavior of legitimate mobile user . Therefore, Xu et al.' s scheme cannot provide mobile user untraceability.

Offline password guessing attack: Case I (via special parameter in smart card)
Suppose that the adversary A extracts these parameters { * , ℎ()} and gets the message { , , ℎ , 1 , 1 }.
The adversary A can guess the user password offline. The specific process is as follows: 1) A first selects * from the password dictionary space D and selects * from the identity dictionary space D .

4)
A checks whether * is equal to .
If equal, A finds the correct password and identity of MU. Otherwise, A repeat steps 1)-4) until she finds the correct password and identity.
The time complexity of the above attack is: O(|D | * |D | * ℎ ), where |D | and |D | denote the number of passwords in D and the number of identity in D , ℎ is the running time of hash computation. Usually |D | ≤ |D | ≤ 10 6 [32,37] , therefore, the above attack is very efficient. In fact, why the above attack is successful is that, A can obtain the parameter * in smart card and in public channel, and directly figures out the exact parameter ℎ( || ) directly. Finally, A just needs to traverse the space of passwords and identities.
The adversary A can guess the user password offline. The specific process is as follows: 1) A first selects * from the password dictionary space D and selects * from the identity dictionary space D .

2)
If equal, A finds the correct password and identity of MU. Otherwise, A can repeat steps 1)-4) until the equation holds.
The adversary A can guess the user password offline. The specific process is as follows: 1) A first selects * from the password dictionary space D and selects * from the identity dictionary space D .

4)
A checks if * 1 is equal to 1 . If equal, A finds the correct password and identity of MU. Otherwise, A can repeat steps 1)-4) until the equation holds. The time complexity of the above attack is: O (|D | * |D | * 2 ℎ ), therefore, the above attack is also very efficient.

No perfect forward secrecy
In Xu et al.' s scheme [21] , A can obtain the established session key between the mobile user MU and the foreign agent FA if A gets the private key of HA.

1) A eavesdrops the message {
, , ′ } in public channel and extracts the parameter { * , * ℎ , ℎ()} in smart card. 2) A decrypts using the long term private key of HA, that is , (

Mobile user impersonation attack
According to Section 2), the adversary A can figure out the shared key ℎ between MU and HA. Thus, if A obtains the identity of the mobile user MU by using of offline guessing attack, she becomes capable to impersonate MU. To do so, A captures the login request message { , , ℎ , 1 } in public channel and extracts the parameter { * , * ℎ , ℎ()} in smart card. Afterwards, A performs the following steps.

2)
A chooses a random number * ,and then calculates * = ℎ( Since FA only checks the validity of * 1 , the forged message * 1 is easy to pass the authentication of FA. On the other hand, since the forged message * 1 } is indistinguishable from the real 1 , MU also can pass the authentication of HA. Therefore, Xu et al.' s scheme [21] cannot resist mobile user impersonation attack.

GUPTA ET AL.'S SCHEME
Gupta et al.' s scheme [23] uses public key encryption based on quadratic residue assumption. Quadratic residue assumption is described as follows: Assume , are large primes, = , and and is given. It' s hard to get from the equation = 2 mod . However, if the factors of i.e. and are known, Chinese remainder theorem can solve this problem. More detailed description can be found in Jiang et al.' s scheme [17] .
Moreover, in order to solve the problems of efficient typo detection, DoS attack and password guessing attack, the proposed scheme uses the "Fuzzy-Verifier" technique [26] . And Gupta et al.' s scheme [23] has four phases. However, registration phase and mutual authentication phase are needed in this paper. The detailed descriptions of the two phases are as follows. . After that, HA stores { , , 0 , , ℎ(·), } in the smart card. is a public key of the home server which used for the encryption by the mobile users. Then HA sends it to MU. On receiving the smart card, MU enters into the smart card.

Mutual authentication phase
In this phase, the mobile user MU can get access of the foreign server by performing authentication and key agreement. Assuming is prime, is an Elliptic curve over the field ( ) where has large embedding degree, and is a base point in Elliptic curve. S1. MU inserts the smart card into a card reader and inputs and . , are the identities of the foreign agent and home agent respectively, 1 is the timestamp at which the message 1 is sent, is a random number generated by the mobile user. Afterwards, MU sends the 1 to the foreign agent FA. S3. On receving 1 , FA chooses a random number and calculates 2 = ( 1 || 2 || ). 2 is the timestamp when the message 2 is generated. Subsequently, FA uses ECDSM and private key on the message 2 to generate digital signature . Then FA publishes its public key to all the servers periodically, which is corresponding to the private key and certified by the certificate authority CA. In the end, FA sends the home agent HA the message 2 and signature . S4. On receiving 2 and , HA verifies the timestamp 2 . If the timestamp 2 is not valid, HA end this session. Otherwise, HA checks the signature using the public key of FA. If the verification is not successful, HA sends the failure notice to FA. If so, HA using the private key , decrypts 1 where = × .Then HA obtains

S2. The smart card figures out
where is digital signature generated using ECDSM, 3 is the timestamp when 3 and 4 are sent, Sign is ECDSM signature generation algorithm, is the private key of HA for ECDSM signature generation. HA publishes the public key to all the servers which is corresponding to and certified by the certified authority CA. HA sends ( 3 , 4 , 5 , ) to FA. Otherwise, MU terminates this session.

CRYPTANALYSIS OF GUPTA ET AL.'S SCHEME
Here, we show that Gupta et al.' s scheme [23] still has two serious flaws, namely, offline password guessing attack and session-specific temporary information attack.

Offline password guessing attack: Case I
Suppose that the adversary A extracts these parameters { , } from the smart card, gets the message { 1 , 3 } and the public key of HA. The adversary A can guess the user password offline. The specific process is as follows: 1) A first selects * and three identities { * , , * , * } from the password dictionary space D and three identity dictionary space D , respectively. Moreover, A chooses a time-stamp * 1 from the appropriate time interval Δ . 2) A calculates * = ℎ( * || ).
The time complexity of the above attack is: O(|D | * 3|D | * Δ * ℎ ), therefore, the above attack is efficient.

Offline password guessing attack: Case II
Suppose that the adversary A extracts these parameters { , }, gets the message { 4 , 5 }. The adversary A can guess the user password offline. The specific process is as follows: 1) A first selects * from the password dictionary space D and selects three identities { * from three identity dictionary space D .

5) A verifies whether *
5 is equal to 5 . If they are equal, A gets the correct password and identity of MU. Otherwise, A can repeat steps 1)-5) until the equation holds.
The time complexity of the above attack is: O(|D | * |D | * 2 ℎ ), therefore, the above attack is very efficient.

Sessionspecific temporary information attack
In Gupta et al.'s scheme [23] , if all temporary information , are compromised, then A can compute = . Therefore,in the case of temporary information disclosure, Gupta et al.' s scheme [23] is vulnerable to sessionspecific temporary information attack.

MADHUSUDHAN ET AL.'S SCHEME
In 2019, Madhusudhan et al. [25] only used hash function and symmetric password to construct an authentication scheme [25] in GLOMONET, and they claimed this scheme to be able to resist various attacks and provide user anonymity. But here, we show that Madhusudhan et al.' s scheme [25] cannot provide user anonymity and perfect forward secrecy, and it is vulnerable to at least five types of attacks. The specific cryptanalysis process is as follows:

Initialization phase
Suppose that HA computes = , where , are two prime numbers. And ′ and ′ are public primes, HA selects G (multiplication group) and an element ∈ with order ′ . Then HA choose a symmetric key = (< ′ ) and computes the public key = mod ′ . Similarly, FA chooses a private key = (< ′ ) then computes the public key = mod ′ .

S1. A new MU randomly chooses
, and and a random number . Afterwards, MU submits

Login and authentication phase
In this phase, MU and FA agree on a session key and perform mutual authentication through HA to access the required services. The login and authentication phases' procedures are depicted in Fig. 3. S1. MU inputs * , and calculates * = ℎ( * || * || ). After, MU checks whether * = or not. If not, MU end the session. Otherwise, the legality of MU is ensured. Then MU generates a nonce and Finally, the mobile user MU sends FA the message 1 = { , , }. S2. Upon receiving 1 , FA generates a random number . Afterwards, FA encrypts the message 1 with . Subsequently, FA sends the encrypted information with FA' s identity to HA. S3. Upon receiving 2 , HA checks and searches the secret key corresponding to . Then HA decrypts the received information and authenticates on it. If the authentication is sucessful, a session key is generated by HA for communication between FA and MU. If not, HA refuses the login request 2 . Otherwise, HA calculates MU performs further routine verification. If both pass the verification, the authentication and key agreement process are completed successfully.

No provision of moboile user anonymity and untraceability
Since the adversary A can get the parameters { . Therefore, Madhusudhan et al.' s scheme [25] cannot provide mobile user anonymity and untraceability.

Offline password guessing attack
Suppose that the adversary A extracts these parameters { , , , , , ℎ()} from the smart card. The adversary A can guess the user' s password in the offline way, and the specific process is as follows: 1) A first selects * and three identities * from the password dictionary space D and three identity dictionary space D , respectively. 2) A calculates * = ℎ( * || * || ).

3) A verifies whether * is equal to .
If it is equal, A finds out the correct password and identity of MU. Otherwise, A can repeat steps 1)-3) until the equation holds.
The time complexity of the above attack is: O(|D | * |D | * ℎ ), therefore, the above attack is very efficient. On the other hand, according to Section , A has been able to get . Hence, the time complexity of the above attack can be reduce to O (|D | * ℎ ).

Replay attack
The attacker resends the 4 to the mobile user, and the mobile user is unable to check the freshness of 4 . A method is: the user constructs the session key and the new message 5 to FA, FA checks the validity of 5 and figures out by using of its secret key.

Mobile user impersonation attack
According to Section , A has been able to get and . A must forge a real login request message so as to impersonate the legitimate mobile user. In fact, A can take the following steps: 1) A chooses a random number * and computes * = ⊕ * .

4)
The adversary A sends the forged message * 1 = { * , * , * } to FA. Obviously, the forged message * 1 is easy to pass the authentication of FA. And since the forged message * 1 = { * , * , * } is indistinguishable from the real 1 , MU can also pass the authentication of HA. Moreover, The time cost of this attack is only ℎ . Therefore, Madhusudhan et al.' s scheme [25] cannot resist mobile user impersonation attack.

Session key disclosure attack
Suppose that the adversary A can extract the parameters { }} over public channel. Moreover, according to Section , A has been able to get . Then A can figure out the established session key by performing the following steps: Therefore, the adversary can easily get the session key without the private key of HA in Madhusudhan et al.'s scheme [25] .
}} over public channel. According to Section , A has been able to get . In order to impersonate the legitimate foreign agent FA, A must forge a real respond message to the mobile user. Accordingly, A can take the following steps:  [25] is vulnerable to foreign agent impersonation attack. A can get the parameters {   , , , , , ℎ()} of the smart card and the message ,

Suppose that the adversary
}} over public channel. In order to impersonate the legitimate foreign agent FA, A must forge a real respond message to the mobile user. Accordingly, A can take the following steps: ). Accordingly, A Xu et al. [21] Lack of mobile user untraceability Offline password guessing attack No perfect forward secrecy Mobile user impersonation attack Mobile user anonymity and untraceability [29] Anti offline password guessing [27] Perfect forward secrecy [27] Public key technology [27] Gupta et al. [23] offline password guessing attack Anti Offline password guessing [27] Madhusudhan et al. [25] Lack of mobile user untraceability Offline password guessing attack No perfect forward secrecy Mobile user anonymity and untraceability [29] Anti offline password guessing [27] Perfect forward secrecy [27] Public key technology [27] gets mod .

3)
A chooses a random number * .  [25] is also vulnerable to foreign agent impersonation attack.

No perfect forward secrecy
In Madhusudhan et al.' s scheme [25] , we suppose that A can extract the parameters {  [25] . Accordingly, Madhusudhan et al.' s scheme [25] cannot provide perfect forward secrecy.

THE DESIGN PRINCIPLES OF AUTHENTICATION SCHEME IN GLOMONET
Although a lot of work has been done to study the security flaws of existing protocols, there are relatively few studies to analyze the flaws of existing protocols from the perspective of the protocol design principles for GLOMONET, so the same common mistakes are repeated again and again. In fact, many security flaws of Xu et al. [21] ' s, Gupta et al. [23] ' s and Madhusudhan et al. [25] ' s schemes, that are pointed out in this paper, are caused by violating the basic design principles of the authentication schemes in GLOMONET (the details are summarized in Table 2). In fact, there are many security flaws in existing protocols because they violate the following four design principles proposed in this paper (see Table 3). Therefore, the four design principles proposed for authentication schemes summarized in this paper provide a reference for researchers to design secure and effective two-factor authentication protocols for GLOMONET.

PKTP: Public key technology principle
Public key technology principle means that public key cryptosystem (eg., RSA, ECC and quadratic residue.) is used in the proposed authentication scheme. In order to improve the security and efficiency of authentication in global mobility networks, Lee et al. [19] propose a new authentication protocol. But the protocol only uses private key cryptography primitives (such as hash operation and XOR operation), and it is vulnerable to offline password guessing attack, and it also cannot provide perfect forward secrecy. Moreover, Ma et al. [27] also

PFSP: Perfect forward secrecy principle
The meaning of perfect forward security is to ensure that the previously established session key is still secure when one or more long-term private keys are leaked. In 2000, Park et al. [48] researched the perfect forward secrecy principle of authentication and key agreement scheme for the first time. In 2014, Ma et al. [27] further points out that for the purpose of achieving perfect forward security, the two-factor authentication and key agreement scheme protocol must satisfy two basic conditions: (1) using public key cryptography; (2) at least two public key cryptography operations are required at the server side. This just explains the failure of forward security of Xu et al. [21] 's and Madhusudhan et al. [25] ' s schemes.
In order to achieve perfect forward security, authentication protocols can take advantage of the difficulty of factorization of large integers, computational Diffie-Hellman problems on elliptic curves and chaotic maps, and lattice cryptography for compatibility with quantum resistance. Based on the balance between security and practicability, the designer can make a reasonable choice of public key cryptography technology according to the actual application requirements in GLOMONET.

MUAUP: Mobile users anonymity and untraceability principle
In GLOMONET, mobile users anonymity and untraceability is one of the most basic security properties. In actual mobile application scenarios, such as mobile electronic payment and mental health online consultation, mobile users may not want strangers to know their user names and communication traces.
In 2014, Wang et al. [49] proposed the anonymity public key principle for the two-factor protocol for wireless sensor network environment. Based on the work of Halevi et al. [50] and Impagliazzo et al. [51] , Wang et al. strictly proved that it is infeasible to use symmetric key technology to realize user anonymity. Moreover, Wang et al. [49] also pointed out that the anonymity principle is universal and can be applied to other mobile application scenarios. Therefore, Xu et al. [21] ' s and Madhusudhan et al. [25] ' s protocols only use symmetric cryptography primitives such as hash function and XOR operation, which cannot realize user anonymity and untraceability. Specifically, in Xu et al.' s scheme [21] , a fixed parameter is transmitted by the mobile user on the common channel, which causes the adversary to track the mobile user' s communication behavior. In Madhusudhan et al.' s scheme [25] , the adversary can directly figure out the identity of mobile user. In the final analysis, the reason why provides anonymity and untraceability failure is that these parameters are not well protected by public key cryptography.

AOLPGP: Anti offline password guessing principle
Any authentication protocol in GLOMONET should be able to guarantee the security of password. If the password of mobile user can be guessed offline in polynomial time, it indicates that the protocol is vulnerable to offline password guessing attacks. Moreover, in this case, the security of the authentication protocol is completely collapsed. In Xu et al.' s scheme [21] , the adversary can guess the mobile user' s password and identity in three ways. Gupta et al.' s scheme [23] suffers from offline password guessing attack of two ways. Madhusudhan et al.' s scheme [25] is also vulnerable to offline password guessing attack.
In order to achieve "local password security update", Xu et al.' s scheme and Madhusudhan et al.' s scheme store password verification parameters in smart cards, which makes them convenient for offline password guessing, that is, there is a "security vs. usability" balance problem proposed by Huang et al. [52] . Fortunately, combining "Fuzzy-Verifiers" technology [33] with "Honeywords" technology in the field of system security, Wang et al. [26] successfully solves the problems left over in [52] , achieves a better balance of "security vs. usability", and achieves security beyond the traditional upper limit.
We can observe that Gupta et al.'s scheme uses "Fuzzy-Verifiers" technology [33] and "Honeywords" technology to provide local password verification, however, these parameters 3 , 5 are constructed improperly in public channel, so that the adversary can use them to perform offline guessing attacks. In addition to offline guessing attacks, there are online guessing attacks. However, online guessing attack is easy to be detected, and can also be dealt with by setting the number of online wrong logins.

CONCLUSION
This paper analyzes the security of three representative anonymous authentication protocols in GLOMONET environment, highlights some serious security threats against these protocols, and gives the specific attack methods that attackers may take, which will provide better reference for the analysis and design of such protocols in GLOMONET. Specifically, this paper first points out that Xu et al.' s scheme [21] is vulnerable to three kinds of offline password guessing attacks and suffers from mobile user impersonation attack. Moreover, Xu et al.' s scheme [21] cannot also achieve perfect forward secrecy and user anonymity and untraceability. Next, it shows that Gupta et al.'s scheme [23] cannot resist two kinds of offline password guessing attacks and sessionspecific temporary information attack. Then, it is pointed out that Madhusudhan et al.' s scheme [25] is vulnerable to offline password guessing attacks , replay attack, mobile user impersonation attack, seesion key disclosure attack and two kinds of foreign agent impersonation attack, and cannot achieve mobile user anonymity and perfect forward secrecy.
It is pointed out that the above protocols [21,23,25] fail to resist offline password guessing attack and achieve anonymity and forward secrecy because it violates four basic principles of two-factor authentication protocol design: public key cryptography technology principle, perfect forward security principle, user anonymity & untraceability principle and anti offline password guessing principle. According to the basic design principles of authentication schemes, designing efficient and usability secure anonymous two-factor authentication protocols for roaming service in GLOMONET is worth studying in the next step.


Imperceptible Adversarial Examples by Spatial Chroma-Shift

Deep Neural Networks have been shown to be vulnerable to various kinds of adversarial perturbations. In addition to widely studied additive noise based perturbations, adversarial examples can also be created by applying a per pixel spatial drift on input images. While spatial transformation based adversarial examples look more natural to human observers due to absence of additive noise, they still possess visible distortions caused by spatial transformations. Since the human vision is more sensitive to the distortions in the luminance compared to those in chrominance channels, which is one of the main ideas behind the lossy visual multimedia compression standards, we propose a spatial transformation based perturbation method to create adversarial examples by only modifying the color components of an input image. While having competitive fooling rates on CIFAR-10 and NIPS2017 Adversarial Learning Challenge datasets, examples created with the proposed method have better scores with regards to various perceptual quality metrics. Human visual perception studies validate that the examples are more natural looking and often indistinguishable from their original counterparts.

INTRODUCTION
Deep Neural Networks have shown impressive performance on several domains such as image classification [13], object detection [30], speech recognition [3] and natural language understanding [6]. However, they are vulnerable to intentionally crafted perturbations called adversarial examples [37] and various methods have been proposed to generate such examples, such as Fast Gradient Sign Method (FGSM) [11], Carlini&Wagner Attack [5] and DeepFool [26].
Although these examples range from adding a crafted noise to the input to changing the overall image brightness, hue or structure by applying a specified function [21], a common approach is to use norm based metrics to limit the perturbation and impose imperceptibility. Although the use of based norms are debatable and neither necessary nor sufficient criteria for human perception [33], they are still effective as long as the examples are kept in small -norm distances to the original images in the pixel space. However, this brings significant restrictions on perturbations that are not crafted in pixel space. Specifically, spatial transformations, even the most simple ones like translation or rotation, result in high values, while the resulting distortions are hardly perceivable by humans.
Xiao et al. have proposed an alternative approach to perturbation in pixel space, Spatially Transformed Adversarial Examples [39], to generate adversarial examples. This approach is based on altering the pixel positions. This is done by applying a flow field regularized by Total Variation based smoothness term, followed by differentiable bilinear interpolation. The prospect that deep networks can be fooled only by pixel shifts in the image has opened a new research path in generation of adversarial examples. The vulnerability of neural networks to such flow field based spatial transformations raises a natural research question: How far the spatially transformed examples can be optimized for better human perception?
It is known that the human vision is more sensitive to information loss in the luminance than the chrominance [19] and the distortions in the pixel brightness would be more noticeable by humans than the distortions in the color. This has been one of the main motivations behind visual media compression standards [24] where the images are first converted into YUV domain, where Y channel is the grayscale image (luminance) and U and V channels (chrominance) hold the color information, and, taking human vision into account, a more lossy compression is applied to the U and V components to reduce the data size without compromising perceptual quality. Sample images for a visual comparison of luminance channel distortions and chroma channel distortions are provided in Figure 2. Our method succeeded to generate spatially transformed adversarial examples while not producing geometric distortions.
We hypothesize that spatial transformations should not alter the luminance of the input image pixels as a precondition and the grayscale image data should be kept intact. On the other hand, while suppressing the grayscale perturbation increases perceptual quality of generated examples, the unnatural colors and color transitions in adjacent pixels still generates examples which are unnaturally looking and distorted [1].
Motivated by these ideas, we propose a method that perturbs the input image by spatial transformations only in the colorspace. We show that the method is still able to successfully create adversarial examples both in CIFAR-10 and NIPS2017 Targeted Adversarial Attack Challenge datasets with competitive fooling rates under

Benign
UV Only Y Only YUV Figure 2: Visual comparison of spatially transformed adversarial examples applied on different channels using the identical parameters with exaggerated perturbations for better illustration of the effects (i.e., higher than necessary to fool the model).
white-box settings while creating much less visible distortion in comparison to both pixel-value based and spatially transformed examples. In addition, the method alleviates the need for smoothness or norm constraints.

RELATED WORK 2.1 Adversarial Attacks
The existence of adversarial examples has been formally demonstrated by Szegedy et al. along with a constrained optimizationbased LBFGS attack [37]. Following that work, the widely used single-step adversarial example generation algorithm Fast Gradient Sign Method (FGSM) [11] was introduced, which was further improved by adding several regularization and optimization methods while remaining a one-step attack [31,38]. Various iterative versions of FGSM were also introduced, which applied the same gradient ascent optimization in multiple steps [9,20,23]. Another widely used attack is DeepFool, which aims to find the smallest perturbation by carrying the benign example to the closest decision boundary [26]. This algorithm was then modified to learn a single perturbation that can force misclassification on any input image [25]. Carlini

Color-based Adversarial Attacks
Hosseini et al. proposed shifting the hue and saturation components of the HSV color space representation of an input image [14]. Another approach is to optimize the perturbations such that the perceptual color distance, which is reported to be a good representation of the human color difference perception. For that, the norm term of the C&W objective function is replaced by CIEDE2000 perceptual color distance [22,42]. cAdv attack works in CIELAB color space and clusters the image in AB channel representation, then perturbs the clusters whose entropy values are relatively high [4]. Similarly, ColorFool uses image semantic segmentation to find the color sensitive regions of an image, and shifts the pixel values in the respective LAB color space representation [32]. On the other hand, ReColorAdv aims to learn a color transformation function that minimizes the adversarial loss, and applies the same function to each pixel of the input image [21]. Aksoy et al. proposed an algorithm in YUV space, where the noise in UV channels are iteratively suppressed. Then Gaussian filters are applied to all the channels to further reduce the 2 distance between the benign and adversarial images [1].

Transformation-based Adversarial Examples
While the most common adversarial attacks are done in the pixel domain, i.e. by directly adding noise to individual pixels, deep classifiers have also been shown to have vulnerabilities against spatial deformations and transformations [10]. Xiao et al. proposed an algorithm that learns a smooth flow field for the pixel values to shift and produces adversarial examples with less disturbing noise [39]. Similarly, ADef attack applies spatial perturbations to the image in an iterative fashion [2]. Zhao et al. proposed an algorithm that takes advantage of both spatial and pixel space distortions [41].

METHODOLOGY 3.1 Flow field Calculation
A flow field consists of a two dimensional vector ( , ) = Δ , Δ for each pixel location ( , ) in the adversarial image x . For each pixel in the adversarial image x ( , ) , a pixel from location ( + Δ , + Δ ) is sampled from the input image. Since the sampled location is not an integer value, 4 neighboring pixels around the location ( + Δ , + Δ ) are (bilinear) interpolated [43] using Eq.

Spatial chroma shift
We aim to generate adversarial examples having little or no visible distortions. To accomplish this, we first convert the input image into YUV representation , , according to BT.470 System M [17], with the linear transformation given in Eq. 1.
Then, we apply a flow field to , to obtain , while keeping intact. After applying the flow field to the chrominance channels, we concatenate the input image luminance and adversarial spatially transformed chrominance , channels to obtain the adversarial example , , . The method is illustrated in Figure  3. Since the process is end-to-end differentiable, the flow field can be optimized by gradient based optimizers. Application of flow field is as explained in [39]. While a local smoothness constraint is applied in [39] to enforce adjacent pixels to move in coherence, our method alleviates the need for such constraint and eliminates the need for optimization of this constraint. It also eliminates the need for norm constraints, commonly used in pixel-based perturbation methods, to limit the perturbation.
In addition, the search space can be further reduced to apply subpixel color changes by parameterizing [27] the applied flow field and optimizing instead, ′ to minimize adversarial loss. This limits the applied flow field vector size to the range (−1, 1) which further reduces visual artifacts of adversarial perturbation without adding any regularization term to the optimized loss value, with a minor decrease in fooling rate.

EXPERIMENTAL EVALUATION
In this section we first quantitatively analyze, and comparatively evaluate the proposed method using learning-based perceptual distance metrics. Then we perform a human perceptual evaluation study for a subjective assessment of the generated adversarial examples. We also evaluate our method against existing defense mechanisms, and compare the results with those of widely known adversarial attacks.
Datasets. We used CIFAR-10 for targeted and NIPS2017 Adversarial Learning Challenge datasets for the untargeted attack evaluations. As the proposed method is designed to work on chroma channels, we excluded the images which are not colorful (i.e., which have a colorfulness measure below the threshold of 15) according to the colorfulness measuring technique in [12]. This resulted in exclusion of 1467 not-colorful images in CIFAR-10 dataset, leaving 8533 images and in exclusion of 97 not-colorful images in NIPS2017 dataset, leaving 903 images.
Attacks. We compare the proposed method against several widely known methods: FGSM [11], Projected Gradient Descent (PGD) attack [23], Carlini & Wagner attack [5], DeepFool [26] and stAdv [39]. We used Foolbox [29] implementations for FGSM, PGD, C&W, DeepFool attacks and re-implemented the stAdv attack. We made the implementation publicly available at https: //github.com/ayberkydn/stadv-torch alongside the implementation of the proposed method. Models and Parameters. We used ResNet50 [13] for the CI-FAR10 test set and Inception-v3 [36] for NIPS2017 dataset against all attack types. We set the number of iterations to 1000 and color threshold to 15 for all attack types to test different attacks on an identical ground. We used the default parameters for the existing attacks implemented in the Foolbox. For the proposed methods, we set the learning rate as 0.005 for the CIFAR10 testset and 0.01 for the NIPS2017 dataset.
Evaluation. Regarding that norm distances are both insufficient for perceptual similarity and not applicable to spatially transformed adversarial examples [33,39], we have tested perceptual loss of our attack with two learning-based perceptual metrics that are applicable to spatially transformed adversarial examples: Learned Perceptual Image Patch Similarity (LPIPS) metric [40] and Deep Image Structure and Texture Similarity (DISTS) index [7]. LPIPS is a technique that measures Euclidean distance of deep representations (i.e. VGG network [34]) calibrated by human perception. LPIPS has been already used on spatially transformed adversarial example studies [18,21]. DISTS is a method that combines texture similarity with structure similarity (i.e., feature maps) using deep networks with the optimization of human perception. We used the implementation of Ding et al. for the both perceptual metrics [8].
Performance Evaluation with Respect to the Objective Metrics. Table 1 shows that the proposed method in subpixel mode has the lowest LPIPS and DISTS perceptual loss results, 0.006 and 0.009 respectively, in comparison to FGSM, PGD, DeepFool, C&W and stAdv attacks on CIFAR10 test set and Table 2 shows our subpixel attack has the lowest LPIPS and DISTS results on NIPS2017 dataset in comparison to stAdv and C&W targeted attacks, 0.008 and 0.008 respectively. In addition to the objective evaluation, we also conducted a human perception experiment, since no perceptual distance metric can accurately represent the human visual perception [5].
Human Perceptual Study. The experiment has been performed for a subjective evaluation, i.e. whether the chroma-shift based adversarial images are, in most cases, indistinguishable from their original counterparts.
Our claim is that the perturbations in chroma-shift based adversarial images not only keep the image in the perceptually realistic domain but also go beyond this by making the differences almost imperceptible to human eye. We aim to test this claim and compare the results with those of stAdv, which was already shown to keep the images in perceptually realistic domain [39]. In our human perception study, we follow a similar protocol to this work with a minor difference that there is no time limit for the participants and images stay on the screen until the participants make their choices. This allows participants to analyze the images in more detail for any visible distortions.
For the experiments, we used the first 10 images from the NIPS 2017 competition dataset. For each original image, we prepared an adversarial images using stAdv and another using the proposed chroma-shift based method, an example image and their adversarial counterparts are shown in Figure 5. The participants are presented with two images, one of which is always the original image and the other one shown alongside is either the same image again, stAdv image, or chroma-shifted image in a random order. The participants are asked the question "Is the image on the right the same with the one on the left or a slightly distorted version of it?" for each pair. This results in 30 annotations per participant.
A total of 77 participants participated in the study, providing a total of 1860 annotations. As shown in Figure 4, the chroma-shift based adversarial examples are perceived frequently the same as the original images. Note that in some cases the original images (i.e., control group) were perceived differently and 87.01% of the total annotations the original vs. original were annotated the same. While this reduced down to 77.53% for the original vs. chroma-shift images, most were still indistinguishable by the participants. For the original vs. stAdv images, the participants were able to spot the distortion with more ease and 50.51% of the total annotations stated that they were the same.
Performance Against Robust Models. We also measured the performance of our attack against adversarially trained ResNet50 models. To test the robustness of our attack against adversarially trained models, we have used 3 robust models that are trained with FGSM, 20-step PGD and ensemble adversarial training methods. The models were attacked in an ∞ setting where budget is set to be 8/255 (in 0-1 scale). Ensemble adversarial training [38] was performed on ResNet50, where the generator models were set to be DenseNet121 [16], GoogLeNet [35] and MobileNet [15]. Table 3 shows the fooling rates obtained by attacking these robust models using different adversarial attacks. Chroma-shift attack, when run in unrestricted mode, has on par or better than C&W attack, it also outperformed stAdv attack against each augmentation based adversarial defense mechanism. Adversarial examples applied only on image colors (i.e., preserving the luminance content and modifying only the chroma channels) are expected to be not robust to conversion to grayscale as mentioned in the study of Laidlaw et al. [21]. As the proposed method is applied to only chrominance (UV) channels, it could be argued that converting the input images from RGB to grayscale could be used as a defence. However, conversion to grayscale may be counterproductive as it has an undesired effect of decreasing the overall accuracy on both benign and adversarial examples. When grayscale versions of the input images are used, CIFAR10 testset accuracy decreases down to 86.3% from 93.7% and NIPS2017 dataset accuracy decreases to 79.5% from 92.6%.

DISCUSSIONS
Our hypothesis was that it is possible to create adversarial examples that contain visually imperceptible perturbations by only shifting the chrominance values of the original image. The results obtained with targeted and untargeted attacks show that our hypothesis is valid, there is a decrease in the perceptual distances with regards to the LPIPS and DISTS metrics with competitive fooling rates. Furthermore, the common expectation would be in the way that 'adversarial energy' are mostly accumulated in the luminance channel, which is where the perturbations naturally appear when adversarial attacks are applied in the RGB space [28]. While this assumption is still valid, our findings suggest that chrominance channels contain sufficient adversarial energy so that the network can be fooled by only perturbing the UV channels. On the other hand, our attack is still disadvantageous in fooling rate compared to C&W attack, which is a pixel space attack whereas our algorithm does optimization in the vector space.
We also validated our hypothesis by performing a human visual study, as despite both perceptual metrics we have used have been shown to be correlated with human vision, no perceptual metric is a precise measure for human vision [5]. Results shown in Figure 4, signify that our algorithm not only makes improvement on quantitative visual metrics, but also performs notably well in practice and it is robust against widely used adversarial training based defense mechanisms.

CONCLUSIONS
In this work, we have shown that small spatial transformations exclusively in the color space can yield adversarial examples that are often indistinguishable from their original counterparts. We compared the proposed attack with widely used attacks, both for standard and adversarially trained models, using different perceptual difference distances. Furthermore, we have evaluated the visual quality of our adversarial examples with a subjective human experiment; which showed that chroma-shifted adversarial examples are indistinguishable from the original images in most cases. While conversion to grayscale at the input could be used as a defence against the proposed attack, it has to be noted that this also results in an overall decrease in the standard accuracy. In the future, to further weaken such defense, the attack could be combined with traditional norm attacks such as the method proposed in [18]. As another alternative, some limited amounts of perturbations in Y channel could be included in addition to the perturbations in UV channels, similar to the implementation of Aksoy et al. [1].


extfieldsofstudy:
Engineering
Computer Science

text:
A Receding Horizon Approach for Simultaneous Active Learning and Control using Gaussian Processes

This paper proposes a receding horizon active learning and control problem for dynamical systems in which Gaussian Processes (GPs) are utilized to model the system dynamics. The active learning objective in the optimization problem is presented by the exact conditional differential entropy of GP predictions at multiple steps ahead, which is equivalent to the log determinant of the GP posterior covariance matrix. The resulting non-convex and complex optimization problem is solved by the Sequential Convex Programming algorithm that exploits the first-order approximations of non-convex functions. Simulation results of an autonomous racing car example verify that using the proposed method can significantly improve data quality for model learning while solving time is highly promising for real-time applications.

Abstract-This paper proposes a receding horizon active learning and control problem for dynamical systems in which Gaussian Processes (GPs) are utilized to model the system dynamics. The active learning objective in the optimization problem is presented by the exact conditional differential entropy of GP predictions at multiple steps ahead, which is equivalent to the log determinant of the GP posterior covariance matrix. The resulting non-convex and complex optimization problem is solved by the Sequential Convex Programming algorithm that exploits the first-order approximations of nonconvex functions. Simulation results of an autonomous racing car example verify that using the proposed method can significantly improve data quality for model learning while solving time is highly promising for real-time applications.

I. INRODUCTION
Modeling the system dynamics play a pivotal role in the performance of model-based control techniques such as Receding Horizon Control (RHC, also known as Model Predictive Control). Nevertheless, for many complex dynamical systems, obtained mathematical models are often insufficiently accurate due to the existence of uncertainties and ignored dynamical parts. This challenge motivates learning-based models for control, which leverages Machine Learning (ML) techniques to model dynamical systems from data and certain prior knowledge. For example, Gaussian Processes (GPs) [1] have been applied for dynamics and control recently [2]. A fundamental but challenging problems in learning the system dynamics using GPs is how to obtain a training dataset such that the learned models can efficiently capture the actual dynamics. This is because, in most control applications, the experiments for data collection are limited by time, cost, or constraints of environment, whereas using only historical data is not suitable due to the lack of input excitation. For instance, consider a motivating example of an autonomous racing car. In this example, the experiments are constrained by narrow and sharp racing tracks, while the historical data obtained from manual control or simple automatic control techniques do not have sufficient excitation. A better approach would be simultaneous learning and control where the car explores the state space for learning as quickly as possible while satisfying other control objectives such as maintaining safety and tracking a racing trajectory. The learning objective requires that the system dynamics are sampled at the states associated with informative GP inputs. Given current models, the goal can be achieved by driving the system to the state where the information from collected data can minimize prediction uncertainty at the region of interest. Once the system is controlled to the new state, the GP inputs and outputs are collected, then the model is retrained. This repetitive procedure is referred to as active learning for dynamical systems.
The general active learning framework has been applied to various domains (see [3] for a survey) to address the problem pertaining to optimal training data collection. However, the active learning problem for dynamical systems has only been studied recently [4]- [7]. In [4], a greedy scheme for choosing the next data point in Optimal Experiment Design (OED) was proposed by exploiting the maximum variance and information gain methods. The models learned from the experiment was then used for an RHC problem. Meanwhile, active learning using multi-step look-ahead was considered [5]- [7] and shows better performance than single-step approach. Particularly, in [5], based on the conditional differential entropy of multi-step GP predictions, the authors first determined the most informative data point within the region of interest by a greedy algorithm, then steering the system towards that state using RHC. The paper [6] considered an RHC problem for dual control (i.e., simultaneous learning and control) in which the objective function consists of a control objective and a knowledge-gain (information-gain) objective for learning the dynamics. The knowledge-gain objective was obtained using concepts from information theory, such as mutual information or relative entropy. In [7], an RHC problem for OED including both active learning objective represented by the differential entropy of GP predictions and dynamic constraints was formulated. However, to limit the computational burden, only an upper or lower bound of the differential entropy in [7] and the estimated knowledge gain in [6] of the multi-step GP predictions was utilized. That is, the information-gain metrics for multi-step GP predictions were replaced by the sum of individual metrics for step-wise predictions over the horizon. As a result, the problems can be solved in continuous domain by nonlinear programming solvers instead of grid-based methods. However, computation time for solving the optimization problems was not reported in those papers to judge whether the used methods are suitable for real-time control.
In this paper, we presents a Receding Horizon Active Learning and Control (RHALC) problem for dynamical systems using the GP regressions. The presented problem formulation covers both the dual control problem in [6] and the problem for experiment design in [7]. However, instead of approximate information-gain metrics, we take the exact conditional differential entropy of multi-step GP predictions into account. The resulting optimization problem involving the GP dynamics and the log determinants of posterior covariance matrices is thus non-convex and highly complex that may prevent the success of any non-linear programming solvers. To overcome this challenge, we apply the Sequential Convex Programming (SCP) method [8] to efficiently address the problem by performing the first-order approximations of the GP means and the GP posterior covariance matrices. The effectiveness of the proposed method is validated by simulations of an autonomous racing car example. The results on trajectory tracking control performance of the racing car and prediction accuracy of the learned models show that the active learning can improve data quality for model learning in both offline learning (experiment design) and simultaneous learning and control. In addition, the reported fast computation time of the SCP algorithm demonstrates its capability for real-time implementation.
The remainder of this paper is organized as follows. The RHALC problem formulation is introduced in Section II. Section III provides SCP algorithm, while the simulation results are reported and discussed in Section IV. Finally, Section V concludes the paper.

II. PROBLEM FORMULATION
In this section, we introduce a general Receding Horizon Active Learning and Control (RHALC) formulation in which the GPs are employed to model the system dynamics while the exact conditional differential entropy is used as an optimization metric in active learning.

A. Gaussian Process Regression
Consider a latent function f : R n → R and N noisy observations y (i) of it, y (i) = f x (i) + ǫ (i) , at inputs x (i) ∈ R n and with i.i.d. Gaussian noise ǫ (i) ∼ N 0, σ 2 n , for i = 1, . . . , N . We will use X = [x (1) , . . . , x (N ) ] ∈ R n×N to denote the collection of all input vectors and Y = [y (1) , . . . , y (N ) ] ∈ R N to denote the collection of the corresponding observed outputs. Let D = (X, Y) be the set of observation data of f . A GP of f , which will be denoted by G f , is a probability distribution over all possible realizations of f and can be formally defined as a collection of random variables, any finite number of which have a joint Gaussian distribution [1]. It is fully specified by a covariance function k(x, x ′ ; θ) and a mean function m(x; θ). The mean function is employed to include prior knowledge about the unknown function. In this paper, the mean function is assumed to be zero without loss of generality. At , in which the predictive mean vector µ ⋆ and the M ×M posterior covariance matrix Σ ⋆ are computed as follows where σ 2 n is the Gaussian noise variance, I is an identity matrix of appropriate dimensions, K ⋆⋆ ∈ R M×M is the covariance matrix at x ⋆ , K ⋆ ∈ R M×N is the cross-covariance matrix between x ⋆ and X, K ∈ R N ×N is the covariance matrix at X, in which the elements K ij of each matrix is computed by K ij = k(x (i) , x (j) ). Note that in this paper, we only utilize the GP means without uncertainty propagation to represent the predicted values of the nonlinear dynamics. For more details on GPs and its usage in controls, readers are referred to [1], [2].

B. Receding Horizon Active Learning and Control with Gaussian Process
We define the control input vector as u ∈ R nu , the vector of GP output variables y ∈ R ny and the vector of non-GP variables z ∈ R nz . For any variable , where is y, z, or u, let k denote its value at time step k. The state of the system at time step k comprises y k and z k . We denote the GP dynamic model as y k ∼ G(x k ). The input vector x k , or features, of the GP is formed from current and past values of the control inputs u τ and non-GP states z τ , for τ ≤ k, as well as from past GP outputs y τ , for τ < k. To simplify the notation and formulation, we assume that y is scalar; however, our results can be readily extended to multivariate y. Given an input x k , letȳ k = µ(x k ) denote the predictive mean of the GP model G(·) at x k .
Let H > 0 be the length of the control horizon, t be the current time step and I t = {t, . . . , t+H −1} be the set of all time steps in the control horizon at time step t. DenoteȲ t = {ȳ k |k ∈ I t }, Z t = {z k |k ∈ I t }, U t = {u k |k ∈ I t }, and X t = {x k |k ∈ I t } as the sets collecting the predictive GP output means, the non-GP states, the control inputs, and GP inputs over the control horizon. To simplify the mathematical notations, we will use [X] to denote the vector concatenation of all vectors in a set X (e.g., . Given a GP model trained on the data generated up to the current time step, the most informative GP regressor vectors in the next horizon X t can be determined by maximizing the conditional differential entropy of GP predictions at these vectors that is computed by the log determinant of the covariance matrix [9], leading to the following optimization problem where Σ([X t ]) is a H × H posterior covariance matrix of GP predictions at H input vectors in the set X t and can be computed by (1b). The RHALC problem is thus formulated as follows where H(X t ) = log det Σ([X t ]) is an active learning term and J(Ȳ t , U t , Z t ) is an control objective function, g j (Ȳ t , U t , Z t ) ≤ 0 and h j (Ȳ t , U t , Z t ) = 0 are inequality and equality constraints while J ieq and J eq are the sets of inequality and equality constraint indices, respectively. In the problem (3), γ is a positive constant representing a tradeoff between learning and control objectives. Similar to [10], we make the following assumption about the problem (3).
Assumption 1: Suppose that J is convex, each g j is convex, and each h i is affine in the optimization variables U t and Z t . In other words, the non-convexity of the problem (3) results from the GP dynamics and the log determinant of the GP predictive covariance matrix.

III. SEQUENTIAL CONVEX PROGRAMMING FOR RECEDING HORIZON ACTIVE LEARNING AND CONTROL
The problem (3) is highly nonconvex due to the active learning objective and the GP dynamics. Moreover, the complexity of the objective function involving the log determinant of the GP predictive covariance matrix makes the problem (3) computationally intractable. In this section, we employ the Successive Convex Programming (SCP) approach [8] to effectively address this problem.
Suppose that nominal feasible control inputs are given in We then simulate the GP model G(·) over the RHC horizon to obtain the nominal output means can be obtained from these values. Consider small perturbations to the nominal control inputs u k = u ⋆ k + ∆u k , which are collected in ∆U t = {∆u k | k ∈ I t }. They will cause perturbations to the predictive output means and regressor vectors during the MPC horizon as: Using these perturbation variables, the RHALC (3) can be approximated locally around the nominal values by replacing the GP predictive meansȳ k = µ(x k ) and the log determinant of GP predictive covariance matrix with their first order approximations as follows.
Defineỹ k as the first-order approximation of y k around the nominal solution x ⋆ k , which can be computed as follows where from (1a) we have being the gradient of k with respect to the first argument. We also defineỸ t as a collection of y k for k ∈ I t . Meanwhile, the first-order approximation of Note that the derivative of the log determinant of the GP predictive covariance matrix with respect to each element ν j of a vector ν can be computed by With a slight abuse of notations, we will write U t = U ⋆ t + ∆U t . We now obtain the convexified RHALC problem, as stated below.
Constraints (6c) specify a trust region in which the local convexified subproblem is valid. To avoid the artificial infeasibility [8] of the problem due to the approximations (4) and (5), the inequality and equality constraints in (6) are encoded into the objective function by the exact penalty functions [8] leading to the following penalized convex problem minimize ∆Ut,Zt where τ j , ∀j ∈ J ieq and λ j , ∀j ∈ J eq are large positive penalty weights. Under Assumption 1, the penalized subproblem (7) is convex and can be solved efficiently by convex solvers. The SCP algorithm for solving the RHALC problem is outlined in Algorithm 1. For further details on the SCP method and its application to the RHC problem using the GPs, the readers are referred to [8] and [10], respectively.

IV. SIMULATION
This section validates the advantages of the RHALC problem in two scenarios, experiment design and simultaneous learning and control problem, as well as the effectiveness of the SCP algorithm in solving the problems in real-time by a numerical simulation of an autonomous racing car example.

A. Autonomous racing car example
We revisit the example of an autonomous racing car mentioned in Section I. The simulation consists of two phases: learning phase and racing phase. During the learning phase, starting with initial GP models trained on a few historical data, the controller collects and adds new data points and retrain the GP models. Once sufficient data for learning accurate models have been obtained, the learning phase is disabled and the obtained models are utilized to perform tracking control task in the racing phase. Form convex subproblem (7) by using (4) and (5) 5:

14:
else 15: Similar to [11], the kinematic bicycle model of the vehicle is used in the simulation while the following discrete-time dynamics with a sampling time ∆T > 0 are utilized to design the control problem where (x, y) is the position vector of the vehicle, θ is the heading angle, v is the speed of the vehicle, and a and α are respectively the linear acceleration and steering angle of the vehicle. The continuous-time kinematic bicycle model and parameters of the racing car can be found in [11]. The nonlinear components in (8) are learned by three GP models: ∆x k ∼ G x (x p,k ), ∆y k ∼ G y (x p,k ), and ∆θ k ∼ G θ (x a,k ), in which the vectors of GP inputs The GP models result in the following GP dynamical equations The RHALC formulation for this example is given by subject to (8) and ( where the constraints hold for all k ∈ I t , (10c) are velocity bound constraints, (10d) are bound constraints on the control actions, and (10e) are affine constraints representing collision avoidance to the border of the racing track. The objective of this problem consists of two parts: trajectory tracking control and active learning for dynamics. The control objective function J is given by where r k+1 denotes the reference at time step k + 1. Given a vector ν and a positive semidefinite matrix M, we define ν 2 M = ν T Mν. The active learning goals for the GP models are captured by where x p,t+1:t+H and x p,t+1:t+H denotes the concatenated vector of GP inputs from time t + 1 to t + H. To avoid the collision with the track borders, we employed a scheme presented in [12] that limit the movement of the car at each time step in the horizon to lie within two parallel halfplanes. As a result, the collision avoidance scheme can be represented as a set of linear inequality constraints in (10e). All the penalty weights in (7) are chosen to be 10 6 . The parameters of the SCP algorithm are: ρ (0) = 0.1, β fail = 0.5, β succ = 2.0, r 0 = 0.01, r 1 = 0.1, r 2 = 0.3, j max = 100, ǫ = 10 −4 . Two virtual racing tracks developed by the University of California, Berkeley (UCB) [13] and the University of Pennsylvania (UPenn) [14] are used in the simulations.
In what follows, we present two scenarios for the racing car example considered in this paper: Offline learning and Simultaneous Learning and Control (Online Learning).
1) Offline learning: Assume that a large area without obstacles is available for experiments. We design an experiment to obtain an optimal dataset for training the GP dynamics prior to the race. This procedure is also referred to as Optimal Experiment Design (OED) [7]. In the OED problem, only the active learning term is included in the objective function while the control objective J is removed. The constraints (10e) are simplified to bound constraints on x and y to ensure that the car lies within the experimental space.
We compare the optimal experiment design using RHALC with a randomized experiment design where random control inputs that satisfy all the input constraints are applied to the system. In the optimal experiment design, three simple GP models with 25 initial data points are utilized, then the RHALC problem is applied in 50 time steps to collect new data points. The models learned from experimental data are used to perform a reference tracking control task. The receding horizon reference tracking problem can be derived by removing the active learning term in (10).
2) Simultaneous Learning and Control: In the application where a free area for experiments is not available, it is required to perform Simultaneous Active Learning and Control, or Online Learning, in the learning phase. The vehicle is expected to efficiently collect online data to update the GP dynamics while following the racing track and avoiding collision to the borders. Hence, this scenario covers the dual control problem in [6].
To validate the benefits of active learning, we consider two simulations depending on the effect of the active learning term. In the first simulation, the active learning objective is involved to whereas in the second simulation, the active learning is disabled and the vehicle only tracks to the reference and collects online data along the racing track. In this scenrio, three initial GP models trained on 50 initial data points are adopted to control the system at the beginning, then 100 new data samples are collected online. The simultaneous learning and control simulation is conducted using the UPenn track, while the UCB track is utilized for a testing race given the model learned from the previous race.

B. Results and discussions
The trajectories of the autonomous racing car in the offline learning simulation are shown in Fig. 1. Using the GP models from the optimal experiment, the receding horizon tracking controller can accurately track the reference and prevent the collision to the borders. In contrast, the GP models from the randomized experiment apparently are not accurate since the car cannot complete one racing lap without crashing into the borders. Particularly, the car initially can track well to the reference but collision to the borders happens when the car needs to turn sharply. Meanwhile, Fig. 2 shows the trajectories of the car in the Simultaneous Learning and Control simulation. As can be seen from the figure, with the active learning, the vehicle initially fluctuates around the reference to explore the informative states, hence the car does not perfectly track the reference in the first 100 time instants but the safety condition is guaranteed. However, once completing the learning phase, the obtained GP models are accurate so that the vehicle is able to track the reference trajectory in the rest of the racing track and in a new testing track. Meanwhile, without active learning, at the beginning of the race where the racing track is relatively simple, then the car can track to the reference. However, since the learned models do not have enough excitation, the car deviates to the reference trajectory when the track becomes more complicated leading to the crashes to the borders.
Furthermore, the prediction accuracy of the GP models obtained from all simulations are shown in Table I. We compare 4 types of GP models, OE, RE, AL and Non-AL, which are correspondingly obtained from the offline optimal and randomized experiments, and the learning and control simulations with and without active learning. Two validation metrics including the root mean squared errors (RMSEs), and the maximum absolute errors (MAEs) are considered. These validation metrics are computed using the GP predictions on a grid of GP inputs in which 20 linearly spaced points in the region of interest for each input and the corresponding latent non-linear functions. According to the table, with the same number of training data points, three GP models generated from the optimal experiment show better performance in prediction accuracy than those from the randomized experiment. Likewise, based on the metrics for AL and non-AL models, it is obvious that active learning can improve GP precision in simultaneous learning and control.
Regarding the computation time, in the optimal experiment, SCP algorithm takes 0.069 s on average for each time instant, while in the race, it takes 0.041 s and 0.040 s in UCB and UPenn racing tracks, respectively. Meanwhile, in online learning simulation that includes active leaning, SCP  algorithm averagely takes 0.104 s, whereas in control phase, it takes 0.058 s for each time step. Note that all simulations in this work are performed on a DELL computer with a 3.0 GHz Intel Core i5 CPU and 8 Gb RAM, and the Julia programming language is used for the implementation.

V. CONCLUSION
In this paper, a receding horizon active learning and control problem for dynamical systems using Gaussian Processes (GPs) was considered. We first developed a problem formulation subjecting to the GP dynamics while the exact conditional differential entropy was employed as a metric for active learning. The resulting complex and non-convex problem was solved by the Sequential Convex Programming algorithm. The proposed method was validated by numerical simulations of an autonomous racing car example. Not only guarantee the tracking performance in both offline and online learning scenarios, the control algorithm but also can be executed in a reasonable amount of time, which promises its potential practicality for real-time implementation.


extfieldsofstudy:
Computer Science
Engineering
Mathematics

text:
RIS-aided D2D Communication Design for URLLC Packet Delivery

In this paper, we consider a smart factory scenario where a set of actuators receive critical control signals from an access point (AP) with reliability and low latency requirements. We investigate jointly active beamforming at the AP and passive phase shifting at the reconfigurable intelligent surface (RIS) for successfully delivering the control signals from the AP to the actuators within a required time duration. The transmission follows a two-stage design. In the first stage, each actuator can both receive the direct signal from AP and the reflected signal from the RIS. In the second stage, the actuators with successful reception in the first stage, relay the message through the D2D network to the actuators with failed receptions. We formulate a non-convex optimization problem where we first obtain an equivalent but more tractable form by addressing the problem with discrete indicator functions. Then, Frobenius inner product based equality is applied for decoupling the optimization variables. Further, we adopt a penalty-based approach to resolve the rank-one constraints. Finally, we deal with the $\ell_0$-norm by $\ell_1$-norm approximation and add an extra term $\ell_1-\ell_2$ for sparsity. Numerical results reveal that the proposed two-stage RIS-aided D2D communication protocol is effective for enabling reliable communication with latency requirements.

I. INTRODUCTION
Ultra-reliable and low-latency communication (URLLC) is one of the pillar technologies in 5G and 6G mobile communication networks. It is an enabler for a wide range of missioncritical applications including factory automation, smart grid, tactile internet, intelligent transportation system, telesurgery, and virtual/augmented reality (VR/AR) [1]. These applications pose great challenges for end-to-end packet delivery. More specifically, a URLLC packet has to be delivered to the destination successfully within sub-millisecond latency and extremely high reliability. This calls for new technologies and transmission mechanism design to satisfy such strict qualityof-service (QoS) requirements.
One promising technology is the reconfigurable intelligent surface (RIS) which has attracted interest recently. RIS, a metasurface composed of massive low-cost and passive reflecting elements, is able to reshape and reconfigure the wireless environment by intelligently adjusting the amplitudes This work was supported in part by the Swedish Research Council (VR), ELLIIT, and CENIIT, in part supported by the the National Key R&D Program of China (2020YFB1806604), the Fundamental Research Funds for the Central Universities +2020JBZD005, the NSFC, China (61871027, 62031008, U1834210) and the China Scholarship Council (CSC) Grant #202007090174. and/or phase shifts of reflecting elements [2][3][4]. As a result, integrating RIS into a communication system can improve the desired signal at the destination and facilitate reliable end-toend communication. Moreover, as compared to active relays, RIS can work in a full-duplex mode without causing selfinterference and thermal noise and its energy consumption is low owing to its passive nature [5]. Hence, the deployment of cost-effective RIS is important for enabling URLLC packet delivery. The authors in [6] investigated the resource allocation design problem of a multicell RIS-aided OFDMA system for weighted sum throughput maximization subject to all URLLC users' QoS requirements.
Another potential technology is device-to-device (D2D) communication. In most mission-critical scenarios, devices (e.g. sensors, actuators, machines) are in close proximity to each other. Thus, the channel between devices is significantly more reliable than that between the access point (AP) and the devices [7]. By exploiting the benefits of a D2D network, successful URLLC packet delivery can be enabled. In [8], a D2D-based two-phase transmission protocol was proposed for industrial automation scenarios to achieve URLLC.
Recently, works considering the combination of both RIS and D2D technologies for system performance optimization have appeared. The authors in [9] found that by integrating RIS into a D2D network as well as optimizing the transmit power and discrete phase shifts of RIS elements, the interference of D2D networks can be significantly decreased. In [10], authors investigated a problem of joint power control of D2D users and passive beamforming for an RIS-aided D2D communication network to maximize energy efficiency. Unfortunately, the existing RIS-aided D2D communication design cannot be applied to URLLC packet transmission directly since these studies were based on Shannon's capacity with assumptions of infinite blocklength and zero error probability. Thus, the delay performance will be underestimated and the reliability performance will be overestimated [11]. This necessitates the URLLC packet transmission design in an RIS-assisted D2D network under the finite blocklength regime.
In this paper, we consider an RIS-aided D2D network to achieve downlink multiusers' URLLC packet delivery in smart factory scenarios. The system involves two stages. In the first stage, each actuator's received signal consists of two parts -a direct signal from the AP and a reflected signal from the RIS. In the second stage, actuators with successful reception in the first stage help relay critical messages to the ones who failed to receive via the D2D network and they use maximum-ratio combining (MRC) to jointly decode the message received in two stages. The main contributions of this paper are summarized as follows.
• To the best of our knowledge, this paper is the first to consider the two-stage RIS-aided D2D communication protocol for URLLC packet delivery. To enable all actuators' reliable communication, a joint active and passive beamforming optimization problem is formulated. • Instead of the common alternating optimization method, we first transform the original problem into an equivalent but more tractable form. Then we leverage the Frobenius inner product based equality for product decomposition and penalty-based method for tackling rank-one constraints. Eventually, we address the 0 -norm by 1 -norm approximation plus 1 − 2 for sparsity. • A low-complexity penalized successive convex approximation (SCA) based iterative algorithm is proposed to obtain suboptimal solutions. Simulation results demonstrate the superiority of the proposed two-stage RIS-aided D2D communication protocol.

II. SYSTEM MODEL
We consider a downlink communication system in a smart factory with one AP equipped with N t antennas and K single-antenna actuators indexed by k = {1, · · · , K}. All the actuators must successfully receive critical control messages in terms of short packets within a time window of τ seconds. As shown in Fig. 1, there are M reflecting elements on the RIS. The phase shift matrix of RIS is denoted by Φ Φ Φ = diag(φ 1 , · · · , φ M ), where φ m = e jθm and θ m ∈ [0, 2π] is the phase shift of the m-th reflecting element. Assume that the phase shifts of all elements are calculated by the AP and then fed back to the RIS controller through the dedicated RIS control link.  Two stages are involved in the RIS-assisted D2D cooperative communication system. We assume that the two stages have the same transmission duration, i.e., τ 1 = τ 2 = 0.5τ . In the first stage with duration of τ 1 seconds, each actuator can receive the direct signal from AP and the reflected signal from the RIS. In the second stage with duration of τ 2 seconds, the actuators who have successfully decoded the signal in the first stage can act as relays to forward the signal to the actuators with failed recepetion via the D2D network, and they jointly decode the message received in two stages by MRC. The twostage RIS-aided D2D communication protocol is described in detail as below.

A. RIS-aided first-stage transmission
In the first stage with duration of τ 1 seconds, the transmitted signal at the AP is given by where w ∈ C Nt×1 is the beamforming vector for broadcasting at the AP, and s denotes the combined symbol intended for all actuators. Without loss of generality, we assume E{|s| 2 } = 1.
The received signal at the actuator k can be expressed as where h k ∈ C Nt×1 , F ∈ C M ×Nt , g k ∈ C M ×1 denote the channels from the AP to the actuator k, from the AP to the RIS, and from the RIS to the actuator k, respectively, z is the additive white Gaussian noise (AWGN). By substituting (1) into (2), we can obtain the signal-tonoise ratio (SNR) of the k-th actuator in the first stage as γ Based on the normal approximation of the achievable rate for short packet communication under quasi-static AWGN channel conditions [12], we can characterize the maximum achievable rate of actuator k over L 1 = τ 1 B channel uses in the first stage by where B is the bandwidth, D is the number of data bits, This asymptotic expression characterizes the relationship between the achievable rate, transmission time (channel uses) and decoding error probability. From (4), the decoding error probability can be written as (5) Let ε th denote the maximum allowed packet error probability (PEP). Note that f γ In order to indicate which actuator successfully decodes the message in the first stage, we define an indicator as follows

B. Second-stage D2D transmission
In the second stage with duration of τ 2 seconds, the actuators with successful decoding in the first stage help to relay the signal to others via the D2D network. Here, we assume that the actuators acting as relays simply forward the signal at their full power, i.e., power control is not performed since the global CSI of the D2D network is not available. The transmit signal of the actuator k in the second stage can be given by where P is the common transmit power for all actuators. Then, in the second stage, if the actuator k cannot decode the message successfully in the first stage, its received signal can be expressed as where d jk ∈ C is the channel from the actuator j to actuator k, z At the end of this stage, the actuator k jointly decodes the message based on the signals received in both two stages by using MRC. Accordingly, the received SNR at actuator k can be written as whered jk = d jk σ k . Then, the maximum achievable rate of actuator k over L 2 = τ 2 B channel uses in the second stage can be characterized by where ε (II) k is the decoding error probability of actuator k in the second stage. Likewise, based on equation (10), ε (II) k can be represented as (11) Similar to our observation from (5), it is easy to obtain that ε k . Hence, the maximum PEP threshold ε th corresponds to the minimum SNR γ

III. PROBLEM FORMULATION
In this paper, we aim to jointly design the active and passive beamforming at the AP and RIS respectively to maximize the number of actuators that can receive the critical control signals successfully under the two-stage RIS-aided D2D communication protocol. Mathematically, the joint optimization problem can be formulated as Constraint (13b) is the transmit power budget at the AP. The constraint (13c) refers to the unit-modulus constraint for each RIS element. Our ultimate goal is to guarantee that all actuators can successfully receive the mission-critical control signals from AP after the two-stage transmission, i.e., K k=1 a Note that the larger the number of actuators with successful reception in the first stage is, the higher the probability of the remaining actuators having successful reception in the second stage with the aid of D2D network will be. This is mainly because of the high SNR of the direct D2D link between two actuators in proximity. Moreover, actuators which act as relays in the second stage are determined by the optimization results in the first stage. Thus, the original problem can be transformed into an alternative problem that to maximize the number of successful decoding actuators in the first stage. As a result, fewer remaining actuators are required to reach the SNR threshold γ (II) th for reliable reception in the second stage. Therefore, problem P1 can be reformulated as

IV. SUBOPTIMAL JOINT ACTIVE AND PASSIVE BEAMFORMING DESIGN
In this section, the suboptimal joint active and passive beamforming design for problem P2 is given.

A. Problem Transformation
Note that in problem P2, a (I) k is a discrete function of γ (I) k . The discrete nature makes it difficult to apply the standard optimization techniques. In view of this, we introduce a set of auxiliary variables q = [q 1 , · · · , q k ] T to transform problem P2 into an equivalent but more tractable form, which can be written as P3 : min (13b), (13c).
Here, nonzero element q k means the actuator k cannot decode the critical signal successfully. Consequently, maximizing the number of successful actuators in the first stage is equivalent to minimizing the number of nonzero elements in vector q. Now, our focus turns to solve problem P3. Its difficulty mainly includes 0 -norm in the objective function, the nonconvex constraint (15b) due to the coupling of w and Φ Φ Φ in the form of a product, and unit-modulus constraint (13c).
First, to address the coupling problem, let where W = ww H , V = vv H . Here, W and V have to satisfy the positive semidefinite constraints W 0, V 0, and the rank-one constraints rank(W) = 1, rank(V) = 1.
It is necessary to remark that W and V remain coupled together in the form of a product in (16). However, we can leverage the following equality to decompose the product, which is given by where A, B F = Tr(A H B) is the Frobenius inner product.
Here, W and R k VR H k are both N t -by-N t matrices. By utilizing the Hermitian property of W, i.e., W = W H , we can exercise the above equality for product decomposition. Therefore, γ (I) k can be equivalently written as k is not joint concave with respect to W, V, based on the first-order condition, we can obtain the lower bound of the convex function W + R k VR H k 2 F , which is given by where W (i) , V (i) are feasible points in the i-th iteration. Thus, γ (I) k can be approximated as Lemma 1. For any A ∈ H n , the constraint that A is rank-one is equivalent to Proof. For any A ∈ H n , the inequality holds where σ i is the i-th singular value of matrix A. And the equality holds if and only if A is rank-one. Thus, the implicit constraint of Hermitian matrix A, i.e., A * − A 2 ≥ 0 functions simultaneously with the constraint (21), which requires that A * − A 2 = 0, namely A is a rank-one matrix. This completes the proof.
According to Lemma 1, rank-one constraints of W, V can be equivalently reformulated as Then we adopt the penalty-based method by moving the constraint (22) into the objective function, thereby resulting in the following optimization problem P4 : min where α, β > 0 are penalty factors penalizing the violation of the constraint (22). Since penalty terms are in a form of difference-of-convex (DC) function, we apply the first-order Taylor series approximation to W 2 and V 2 , which are given by where λ λ λ max (W) denotes the eigenvector corresponding to the largest eigenvalue of matrix W, and W (i) , V (i) are feasible points in the i-th iteration. Hence, the penalty terms related to W, V can be represented as Finally, let us deal with the 0 -norm in the objective function. We first utilize q 1 to approximate q 0 . Note that this approximation may sometimes be loose. In view of this, we add a penalty term η( q 1 − q 2 ) to make the number of zero elements in vector q as many as possible, where η > 0 is a penalty parameter. In general, q 1 ≥ q 2 always holds true where the equality holds when vector q has at most one nonzero element. The two terms q 1 and q 1 − q 2 function simultaneously. As a result, the number of zero elements in vector q can be maximized while minimizing the value of nonzero elements (if exists) so that the SNR requirement in the second stage can be easier to reach. Note that q 1 − q 2 is in a form of DC function, we exploit the first-order Taylor series approximation of q 2 . Thus, the penalty term can be approximated as where q (i) is a feasible point in the i-th iteration. Therefore, we have the following penalized convex optimization problem which can be solved by standard convex tools like CVX.

B. Algorithm Design
According to the preceding analysis, we propose a penalized SCA based algorithm to solve problem P2, which is summarized in Algorithm 1. By iteratively solving a sequence of problem P5 at feasible points, we can obtain a stationary-point solution of problem P2 after convergence [14].

V. NUMERICAL RESULTS
In this section, we evaluate the performance of the proposed two-stage RIS-aided D2D communication protocol. The latency requirement is set as τ = 1 ms, each stage with 0.5 ms. We consider the AP equipped with N t = 4 antennas is located at (0 m, 0 m) transmitting critical control signals to K = 10 actuators. All actuators are randomly and uniformly distributed on a circle centered at (80 m, 0 m) with radius 20 m. The RIS with M = 16 reflecting elements is deployed at (50 m, 10 m).
The channel h k form AP to actuator k is modeled as where L 0 = (λ c /4π) 2 is a constant with λ c being the wavelength of the carrier frequency (2.4 GHz), d a,k is the distance form AP to actuator k, α a,k is the corresponding path loss exponent, β a,k is a Rician factor. h LoS k and h NLoS k represent the deterministic LoS and Rayleigh fading components, respectively. The channels from AP to RIS and from RIS to actuators are also generated in a similar manner. The path loss exponents for AP-actuator, AP-RIS, RIS-actuator channels are α a,k = 4, α a,r = α r,k = 2. The Rician factors for AP-actuator, AP-RIS, RIS-actuator channels are β a,k = 0, β a,r = β r,k = 4. The large-scale fading of the D2D link is 35.3 + 37.6 log 10 (d) in dB and the corresponding small-scale fading is modeled as Rician fading with Rician factor 4. Other parameters are listed in Table I.  Fig. 2, we can easily find that when RIS is integrated into the communication system, all actuators can achieve reliable communication within a larger range of data bits (i.e. 700 bits in transmission schemes 1 and 2, whereas without assistance of RIS, all actuators can only communicate reliably within the range of 500 bits in transmission scheme 3 and 400 bits in scheme 4. This demonstrates that RIS plays an important role in promoting reliable communication and URLLC-related applications. Moreover, it is observed that MRC has a more significant effect on the no RIS-assisted transmission schemes 3 and 4 than that on the RIS-aided schemes 1 and 2. This is because RIS can greatly improve the received SNR of the desired signal at the actuator. More specifically, for problem P5, RIS aims at maximizing the number of actuators with successful decoding in the first stage while minimizing the gap of failed actuators' received SNR and SNR threshold γ  To further understand the two-stage RIS-assisted D2D communication protocol, we plot the average number of actuators with successful decoding in stage I and II under RIS and no RIS-assisted transmission schemes for different D in Fig.  3. As can be seen from Fig. 3 (a), when the number of data bits is small (i.e. not larger than 400 bits), all actuators can just rely on the first-stage RIS-aided transmission to achieve reliable communication. As the number of data bits D increases, second-stage D2D communication starts to function for message relaying towards the failed actuators in the first stage through strong D2D links. As a contrast, actuators have to depend on the D2D network for reliable communication from D = 300 bits under the no RIS-assisted scheme, which is shown in Fig. 3 (b). Thus, whether the RIS is introduced or not, when the number of data bits is large, the second-stage D2D-assisted transmission is indispensable. But with RIS, the burden of D2D network is lower and it will be easier to enable reliable communication, namely  Finally, in Fig. 4, we plot the average transmit power required at the AP under RIS and no RIS-assisted schemes for different number of data bits. As expected, for both schemes, the required transmit power of AP increases as the number of data bits increases, and when D is relatively large, AP serves all actuators with full power for reliable communication. Notably, as compared to the no RIS-assisted scheme, the transmit power of AP can be considerably reduced by integrating the RIS into the communication system when the number of data bits is not larger than 600. This shows another superiority of introducing RIS other than improving the quality of the received signal.

VI. CONCLUSION
In this work, we proposed a two-stage RIS-aided D2D communication protocol to enable that all actuators in the smart factory can receive the critical control signals from AP successfully. Towards this end, the active beamforming at the AP and the passive phase shifts at the RIS needed to be jointly optimized. By tackling the indicator functions, we first obtained an alternative continuous form. Different from the common alternating optimization technique, we utilized the Frobenius inner product based equality to decouple the optimization variables. Then, we exploited the penalty-based approach for solving rank-one constraints. In addition, 1norm was applied to approximate 0 norm and we added an extra term 1 − 2 for sparsity. Numerical results validated the efficacy of the proposed protocol. Specifically, it can achieve reliable communication in a wider range of data bits as compared to the no RIS-assisted D2D communication protocol. Also, the proposed protocol can considerably reduce the transmit power consumption at the AP. The case of corresponding imperfect design will be studied in the future work.