text:
Network Security Roadmap

Users may already have some perception of provided security based on experience with earlier generations. To maintain the stability and coherent integration of 5G services, it is imperative that security and privacy features prevalent in earlier generations are also present in 5G. However, it is not sufficient just to provide the same security features as in the legacy systems due to the new threat model introduced by the integration of new technologies like SDN, virtualization and SBA. 5G systems are expected to be more service-oriented. This suggests there will be an additional emphasis on security and privacy requirements that spawn from the new dimension of service-oriented security architecture.

INTRODUCTION
The fifth generation network, popularly referred to as 5G, has the potential of enabling multiple vertical industries [1,2] like vehicle networks [3,4], IoT [5,2], VR/AR [6], smart cities [7,8,9] and smart healthcare [10,11,12]. The millimeter wave technology that 5G employs offers 100x lower latency and 100x more bandwidth [13] than the existing 4G technology. As a result of this, real-time security requirement is a prime concern for the deployment of 5G applications. When it comes to networks involving the Internet of Things (IoT), multiple layers of the stack are vulnerable as potential targets of security attacks [14], including the service and application layer, nodes, platforms, and the network/transport layer. For example, a unified framework of vulnerability detection in an IoT system is described in [9]. The increased speed and bandwidth enabled by 5G will eventually lead to new threat vectors and increased sophistication of cyber attacks.
Another growing concern is the acceleration of existing cyber attacks in the 5G framework. Today's cyber attacks have already targeted networks' security: both at the IP layer and signalling layer. Thus, simply making legacy security mechanisms operate faster is not an adequate measure. The Service Based Architecture (SBA) of the core 5G network provides multiple access points to third-party service providers, thus expanding the attack surface. Legacy approaches that depend on disparate security elements will neither scale nor will be able to adequately prevent successful attacks across the SBA of 5G networks [15]. New encryption schemes and authentication protocols like 5G AKA [16] are being designed for this purpose. System designs composed of individually secure subsystems are not provably secure. 5G radio network deployments include substantial augmentations of small cells connecting over insecure networks, edge communications, and edge nodes communicating with multiple cells concurrently [17]. Moreover, 5G comes with its new authentication protocols and encryption primitives, which have not received a sufficient amount of cryptanalysts' scrutiny yet. This evolution broadens the threat landscape by expanding the number of intrusion points. With billions of connected devices and critical enterprise applications relying on 5G networks, network operators and applications should preemptively defend against possible security breaches. Some effective measures for ensuring comprehensive end-to-end (E2E) security in the 5G framework may include: • Multiple factors of mutual authentication across all layers of the network including application, control and data planes [18,15,19].
• Cloud-based threat analysis and vulnerability management, enabled by machine learning(ML) algorithms, can be used across multiple platforms to provide immediate response to known and unknown threats in real time [20,21,22]. • The explosion in the number of end devices requires real-time quarantining of infected devices using data-driven approaches.
• Multi-layered encryption of software-defined network (SDN) data planes, enabled by data classification is required for confidentiality and integrity protection.
A primary reason for the necessity of stricter security measures in emerging networks is the change of the trust model as depicted in Fig. 1.
Security for 5G should be taken into consideration at the initial stages of the system design (Security by Design approach). Augmenting security features at a later stage is less efficient and often more expensive in the long run. In long term, security is a driving factor for service and network evolution. due to the integration of multiple 5G enabling technologies that alter the traditional architecture of 4G networks [17,24]. The enhanced mobile broadband and high reliability come with the introduction of flexible mobility and networks. Programmable networks, network function virtualization (NFV) and network slicing are some of the technologies that make the 5G dream possible [25,26]. However, the integration of these features increase the attack surface. The SBA of 5G has multiple third-party service providers sharing resources for efficient functioning of the network. This makes every entity a potential point of attack, unlike the existing systems where most services are provided by the network provider. Hence, 5G security is not just an extension of the current security measures.
The primary changes in the 5G Core from the existing 4G Evolved Packet Core manifest in the form of NFV, Multi-access Edge Computing (MEC) [27,28], Radio Access Network (RAN) [29] and SDN [30,31,32]. NFV decouples software from hardware by replacing various network functions such as firewalls, load balancers and routers with virtualized instances. Network slicing enables the implementation of NFV by allowing multiple logical networks to run on top of a shared physical resource. This provides increased flexibility across networks. SDN is a complimentary technology to NFV for achieving a unified network abstraction to expand data flow control in the form of virtual switches [33].

PROPOSED WORK
There have been preliminary studies on the emerging vulnerabilities in the individual components of the 5G network [34,14,35,36,37]. However, there has not been a comprehensive analysis of the vulnerabilities emerging due to integration of these various components yet. We aim to study such vulnerabilities. There are some attack classes that can be more effectively defended against in the 5G framework. We wish to study such scenarios as well. We also aim to study system level (OS and kernel) and application level vulnerabilities, their correlation to network vulnerabilities as well as the emergence of new threats using artificial intelligence methods.

Pillars of the methodology
Newer technologies integrated in 5G require the formulation of: • New stream ciphers: As mentioned earlier, the 5G ecosystem will enable multiple vertical industries like IoT. This involves the existence of a heterogeneous amalgamation of resource-constrained edge devices. More energy-efficient encryption algorithms and authentication methods are required for efficiency of these devices. Some good candidates for efficient 5G radio communications with constrained resources are mentioned in [38,19]. However, in-depth cryptanalysis is yet to demonstrate the strength of such ciphers.
• New security protocols: Due to the change in the architecture of existing 4G networks, new protocols have been designed for better attack resistance of 5G radio access networks. Random access procedure based on tunable puzzles is an example introducing generalization and improved usability in existing security puzzles [39,40].
On another hand, 5G network is composed of many subsystems functioning together. The security of each of these subsystems as stand-alone disparate systems is necessary but not sufficient to ensure the security of the overall system. The study of the security challenges arising due to the integration of the following subsystems is still in its infancy. Some of the major 5G security subsystems are: 1) Application layer security: Application layer security refers to ways of protecting the application layer from malicious attacks and the exploit of inherent software vulnerabilities. Application layer provides the largest threat surface to the attackers due to accessibility and the huge diversity and minimal abstraction from the user. Application layer vulnerabilities may lead to performance and stability degradation, loss of privacy, and the network being taken down [41]. Application layer threats include buffer overflows, race conditions, SQL injections and privilege escalation, amongst others. 2) Edge security: The edge of the network generally comprises the computing nodes, like smartphones, sensors, RFID tags and smart controllers. Edge devices often process and store confidential information which must be protected. In a smart system like an autonomous vehicle or an IoT ecosystem [42,5], an edge device (namely the microcode, kernel and operating system layers) are prone to security ex-ploits. It is essential to detect such exploits in realtime. Edge security poses many different threats, a major one being a DDoS attack launched by hijacked edge nodes. 3) Cloud RAN (C-RAN) security: C-RAN is a centralized cloud-based radio access network architecture that has multiple advantages over traditional radio access networks. C-RAN has been adopted by multiple operators for the 5G infrastructure [43]. Due to its widespread integration, C-RAN security has received a lot of attention recently. The traditional C-RAN threat vectors include eavesdropping attacks, impersonation attacks, jamming attacks, primary user emulation attacks and wireless channel attacks.
Most of these attacks can be evaded using existing cybersecurity measures but the potential emergence of novel threat vectors due to its integration to the disparate new technologies have not been explored yet [44,45]. 4) SDN security: Traditionally, network components like routers and switches have the rules for packet forwarding hard-coded into the devices. This inhibits the flexibility of the devices and may lead to inefficient packet routes. SDN separates the control plane and the data plane, allowing a software defined controller to dynamically determine the packet forwarding routes. This increases efficiency. Many 5G operators are looking forward to integrate SDN in the 5G framework. However, the SDN controllers are stand-alone components which need to be protected from attackers [46,47]. Other security aspects includes establishing trust between SDN controllers and the data plane (via authentication), creating a robust policy framework and conducting forensic tests to detect and remediate a compromised controller. Various attacks include DoS attacks on the controller and the data planes, topology poisoning attacks and man-in-the-middle (MiTM) attacks. These attacks are described in detail in section 4.2.2. 5) Proactive security analysis: The 5G infrastructure is a service based architecture. Such an architecture is controlled by smart controllers. Hence, future updates in 5G will be software updates. Because of the cyber vulnerabilities of software, we need a proactive real-time security analyzer to detect and alleviate the threats before an attacker exploits it [48,49]. The dramatic increase in bandwidth in 5G gives rise to additional attack vectors. Short range, small-cell antennae physically deployed all across the country become the new entry-points for attackers, expanding the attack surface. Functionally, these antennae will use 5G's Dynamic Spectrum Sharing capability in which multiple streams of information share the bandwidth in so-called "slices"-each slice with its own varying degree of cyber risk. Such dynamic modes of operation require dynamic cybersecurity solutions, rather than uniform rigid defences. 6) Hypervisor security: Hypervisors enable the mapping of various network functions to underlying logical instances and hardware in a virtualized 5G network. A hypervisor can create and execute multiple guest operating systems and performs the necessary resource allocation for those systems. This makes the hypervisor a potential point of entry to multiple networks. Hypervisors are susceptible to DoS attacks on virtual machines (VMs), VM hopping attacks and network slice attacks via host operating systems. Some common hypervisor vulnerabilities include software bugs and network attacks like network intrusion and DoS attacks. 7) Faster authentication: With the facilitation of multiple vertical industries like autonomous vehicles, IoT and smart healthcare, the number of edge devices will face an exponential growth. The number of virtual machines will also increase due to the integration of services like SDN and NFV. All these individual components need to authenticate themselves to one another to ensure secure interaction between them [50,51]. Faster and reliable authentication mechanisms and policies are required for effective operation of the 5G infrastructure. 8) NFV security: NFV utilizes virtualization to allow multiple networks to run on shared resources. NFV is an integral part of the 5G ecosystem and securing it is a fundamental concern. While some of the issues related to NFV security like hypervisor security and SDN security have been discussed before, there are certain unique threats to NFV which must be considered during secure 5G system design [52,53,54,55]. These vulnerabilities include software bugs, insider attacks (like Trojans), outsider attacks (like malicious third-party operator) and attacks on shared resources (like cache poisoning attacks) [56]. An advantage of NFV is that it allows the virtualization of network security functions like intrusion detection systems. This enables security-as-a-service infrastructure, which is integral to the service based architecture of 5G.

Results on WhatsApp
We consider WhatsApp as our secure chat, voice and video application running on a 5G network for our security evaluations. We propose to apply cyber-analysis to WhatsApp as a standalone application which implements several intransit and at-rest security countermeasures. We will also analyze WhatsApp in the context of 5G networks. WhatsApp is a secure instant messaging application with over 1.5 billion users in over 180 countries. The What-sApp messenger uses XMPP protocol and enforces E2E encryption. E2E encryption claims that no one other than the end users can decrypt messages in transit (including proxy and authentication servers). WhatApp E2E encryption is built on top of the security protocol used by the highly secure messaging application Signal. However, there are certain security measures that have been omitted by What-sApp for higher efficiency. As a stand-alone application, WhatsApp is highly secure. Morever, security vulnerabilities at different layers like the network, IP or transport layer can be exploited to compromise the security of WhatsApp. We aggregate all the known attack vectors on WhatsApp across the application layer, network layer, authentication interfaces and the edge device [57,58]. Till now, we have constructed a DAG of 5 classes of vulnerabilities. The DAG consists of 23 nodes currently. We will keep on augmenting our DAG as new vulnerabilities are exposed. The various threat vectors are: 1) Image file-jacking: This attack vector is launched by a malware or rootkit on the smartphone which has access to the shared memory. Sandboxing allows applications like WhatsApp to store its messages in a secure partition of the memory which is inaccessible to any other third-party application. However, the images and other media contents received via the messenger are stored in the shared external memory (/storage/emulated/0/Whatsapp/media). There is usually a delay between receiving the media file in memory and loading it in the chat user interface. The malware can potentially alter the media file during this time and the attack will go undetected. This is also known as the Man-in-the-Disk attack and can be used to manipulate images, payment information and audio files. The Control Flow Graph (CFG) of this attack is shown in Fig. 2. 2) Insecure storage of keys: The WhatsApp encryption keys are stored on the device in clear text.
It is assumed that sandboxing prevents any other application from accessing it [58]. However, root access to the phone and memory corruption attacks, enables access to key chains and renders encryption keys vulnerable to confidentiality and integrity attacks respectively. Besides these, there are methods to decrypt messages on an older smartphone after migrating to an existing WhatsApp account on a new device. These methods can be exploited to extract the private key. 3) SS7 network protocol exploit: WhatsApp verifies a user's identity with a one-factor authentication, namely the phone number used for registration, by calling the user's registered phone number or by sending him/her a verification code via SMS. The network protocol Signalling System 7 (SS7), which is used for communication by a large number of providers, is vulnerable and prone to session hijacking. A malicious adversary can hijack the authentication call (or SMS) and verify himself as the user, thus impersonating the user in future conversations. The CFG of this attack is shown in Fig. 3. 4) Voicemail exploit: As mentioned before, WhatsApp uses voice calls to verify a new account. If the user doesn't receive the verification call, the verification code is sent to the user's voicemail. Voicemail accounts can be hacked easily with brute-force attacks on voicemail account passcodes. Hacking the voicemail gives access to the verification code which can be exploited by the attacker to impersonate the user in future conversations. The CFG of the voicemail exploit is depicted in Fig. 4. 5) Insecurity of messages blocked in transit: This security feature is intentionally disabled by What-sApp as a trade-off for higher efficiency. The attack scenario involves Alice sending a message to Bob. Bob doesn't receive the message immediately because he is offline. In the meantime, the adversary Eve steals Bob's SIM card and uses it to verify herself as Bob on another device. Now, Eve receives the message from Alice which was meant for Bob [59]. The CFG of this attack is shown in Fig. 5.
These attack vectors demonstrate that, although What-sApp is highly secure as a stand-alone application, multiple implementation mechanisms on different platforms make it vulnerable to various attacks.

Attacks on SDN framework
The SDN stack in the 5G network architecture has introduced an entirely new attack surface with numerous threat vectors. The role of SDNs in virtualizing the network functions for greater flexibility and performance is pivotal for the realization of the promises made by 5G developers. Even at its nascent stage, it has already been adopted in multiple real-world systems like the Google datacenters.
Multiple network operating systems (NOS), also referred to as controllers, are available for the implementation of a SDN framework. The most popular amongst them are OpenDaylight [60], Floodlight [61] and POX [62]. All of them implement the most widely used OpenFlow [63] protocol for SDN communication. It has been shown that these NOSs are vulnerable to multiple classes of attacks [64]. A major concern is the absence of encryption of the messages exchanged between the controller and network switches.
We have considered the existing vulnerabilities of SDN and constructed an attack DAG that includes the CFGs of all such attacks. Then we implement cyber-analysis on this DAG to obtain new attacks on the SDN framework. The attacks that comprise the SDN attack DAG are: 1) Packet-In flooding: The network hosts and switches send PacketIn messages to the controller when it encounters a packet with unknown flow (a flow is the rule which instructs the switch where to forward the packet). The controller then sends an appropriate flow to the switch and the switch follows the specified flow rule in the future. If compromised switches send multiple PacketIn messages to the controller, the controller may exhaust all its resources in catering to them. This would lead the controller to an unpredictable state which may cause the network to fail. This is a classic DoS attack on the SDN controller. The CFG of this attack is shown in Fig. 6. 2) Switch The CFG of this attack is shown in Fig. 7. 3) Switch identification spoofing: Due to weak authentication of the switches in the SDN protocol, a malicious switch may pose as a legitimate switch and request connection access with the controller. This causes the controller to disconnect the legitimate switch and connect with the malicious switch. The CFG of this attack is shown in Fig. 8. 4) Malformed control message injection: OpenFlow messages between the switches and the controller contains a header. An adversary can modify the contents of the header or the message itself to crash the SDN controller. In such a scenario, the controller might disconnect the switch. The CFG of this attack is shown in Fig. 9. 5) System time manipulation: Many SDN applications depend on the system time and other system variables. Modifying the system time may lead to disconnection of the switch which tries to perform time-sensitive operations. The CFG of this attack is shown in Fig. 10. 6) Topology poisoning: In order to achieve superior performance to traditional networks, an overall view of the entire network is available to the controller for efficient routing. The overall configuration of the network is implemented via host tracking services and link discovery services. These services can be exploited by multiple techniques (like node spoofing)to alter the topology of the network in the controller database. This lead to various attacks like eavesdropping attack and MiTM attack. The CFG of this attack is shown in Fig. 11. 7) Arbitrary system termination: A malicious SDN application can call the system exit function. This will lead to the termination of the entire controller. All applications will be killed and all entries of the switch table will be lost. This shall be devastating for the data plane elements like the switches and hosts. The CFG of this attack is shown in Fig. 12   Combining all the CFGs mentioned above, we construct an attack CFG for SDN as shown in Fig. 25.

Future Work
Using ML on the system level operations has showed significant preliminary success [65,2]. We plan on extending this methodology to various other platforms and applications.

Representation of attacks
We will delve deep into each of the security categories mentioned above (along with other categories) and document all the possible attacks in each category. Then we decompose each attack into its constituent assembly-level instructions [2]. These assembly-level instructions are represented in the form of a CFG. We merge the CFGs of all the categories to form a unified DAG which encompasses all the known threats in the 5G ecosystem. Some of the detailed threats in 5G are documented in Tables

Machine learning on DAG
After the construction of the DAG, we characterize each node of the DAG with multiple binary-valued features. We train multiple ML models like SVM, K-nearest neighbors and decision trees with the node features. The ML model is trained to predict new possible branches in the DAG, giving rise to new attacks. Deep learning methods cannot be used in this scenario due to the insufficient data.

Implementation on different frameworks
This framework is a general approach which can be executed on any application. The algorithm remains the same but the DAG changes depending on the features and spec-ifications of the framework. Our immediate focus is to implement this on a secure instant messaging application, like Signal or Whatsapp. We aim to test its effectiveness on different platforms like Android, iOS, Unix, etc. This analysis will help to comprehend the underlying dependencies of application layer security on the network and hardware stacks.

Evaluation of the 5G stack
5G network security is a challenging research problem as its infrastructure is entirely different from its predecessor. Many unknown vulnerabilities potentially exist across multiple levels in the hardware, software and network stacks of  implementation. We aim to discover these weaknesses using our algorithm to make the 5G infrastructure safer and more secure. There are certain classes of attacks that can be easily evaded in 5G due to the separation of the data plane and the control plane. Multiple attack and defence measures are expected to be accelerated in the 5G ecosystem. We expect to get novel insights into the overall security framework of 5G by probing into these intricacies with ML.

Reinforcement Learning for security
Reinforcement learning (RL) [66] can be utilized to provide real-time defence solutions. Similarly, it can also be utilized to come up with real-time attack vectors. We aim to use RL to study the implications of real-time attack scenarios in the 5G framework. A defensive or attacking agent can be deployed in a real-world scenario to learn the inconsistencies in the operations. Multi-agent RL may also prove to be useful in such scenarios. Counterfactual regret (CFR) [67] minimization games have shown better results than RL in certain classes of games (where the environment is not fully observable to the player) like poker. We would like to investigate the potential impact of CFR minimization games in 5G security.

Year 1
• Develop a unified framework for analyzing the interconnected security vulnerabilities between the hard-ware, software (microcode, kernel, OS and application layers) and network (including, cryptographic protocols, virtualization layers, and others) stacks with 5G network architectures.
• Study the dependencies of application layer security on the network layer security in the 5G framework.
• Analyze the security of cryptographic protocols and primitives proposed as part of the 5G framework.
• Analyze the security of a mobile application (which is secure at the application layer) running on 5G. This will give a deeper insight into the applicability and usability of the developed tool.
• Utilize deep learning techniques to analyze graphs representing various attacks and threat models. Recent graphical deep learning analysis tools like Graph2Vec [68] enables us to use deep learning to analyze graphs. This has a potential of generating new methods of analyzing threat models.

Year 2
• Utilize the developed framework to analyze the security of multiple 5G-specific applications like autonomous vehicles, Internet-of-Things and smart healthcare.
• Extend the existing framework to other broader areas of security like application security, OS security, etc.

Year 3
• Integrate state-of-the-art methods of ML like RL to deploy real-time vulnerability detection systems.

•
With the advent of 5G, the amount of data being transferred will increase manifold. We wish to study the implications of the magnanimous amount of data transfer on user privacy.


Privacy-Preserved In-Cabin Monitoring System for Autonomous Vehicles

Fully autonomous vehicles (FAVs) lack monitoring inside the cabin. Therefore, an in-cabin monitoring system (IMS) is required for surveilling people causing irregular or abnormal situations. However, monitoring in the public domain allows disclosure of an individual's face, which goes against privacy preservation. Furthermore, there is a contrary demand for privacy in the IMS of AVs. Therefore, an intelligent IMS must simultaneously satisfy the contrary requirements of personal privacy protection and person identification during abnormal situations. In this study, we proposed a privacy-preserved IMS, which can reidentify anonymized virtual individual faces in an abnormal situation. This IMS includes a step for extracting facial features, which is accomplished by the edge device (onboard unit) of the AV. This device anonymizes an individual's facial identity before transmitting the video frames to a data server. We created different abnormal scenarios in the vehicle cabin. Further, we reidentified the involved person by using the anonymized virtual face and the reserved feature vectors extracted from the suspected individual. Overall, the proposed approach preserves personal privacy while maintaining security in surveillance systems, such as for in-cabin monitoring of FAVs.

Introduction
Intelligent monitoring and surveillance systems are widely used to ensure safety and security. Popular applications of monitoring in public are video surveillance cameras (closedcircuit television); monitoring in intelligent transportation systems, including in-cabin monitoring and road traffic monitoring; and video monitoring for data generation and navigational tasks around city centers, airports, and public roads [1]. Driving automation also requires public visual information for multiple tasks [2]. e Society of Automotive Engineers defined six levels of autonomy in driving automation in 2014 (from no automation (level 0) to full automation (level 5)) [2][3][4]. Level 4 autonomous vehicles (AVs) are highly automated and capable of performing all driving tasks under certain conditions without human intervention. However, the driver (human) may control such AVs as and when required. In particular, fully autonomous vehicles (FAVs) (level 5 AVs) have no drivers; all occupants are passengers only [3,4]. erefore, no one oversees such AVs. In addition, in public and shared vehicles (such as ridesharing, carsharing, and car-full services in AVs), the passengers do not know each other. erefore, it is important to ensure the security and safety of all occupants sitting in the cabin of such AVs. Furthermore, the vehicle should be protected from any malicious behavior of the occupants and/or external threats. erefore, FAVs essentially require a multipronged in-cabin monitoring task in real time [5]. However, many countries have imposed a ban or severe restrictions on facial recognition techniques to secure personal information [6][7][8][9][10][11][12][13][14][15][16]. ere are legal and ethical issues that impose various restrictions on public monitoring and surveillance systems [16][17][18][19]. Furthermore, identification of the accused is also important in abnormal (irregular) situations. is study was motivated by the fact that facial monitoring is important for safety; however, it poses a threat to individual privacy. In this study, we focused on the following two problems associated with incabin monitoring systems (IMSes): (i) Protection of facial privacy. (ii) Evidence of the accused in abnormal situations. erefore, a robust solution is required to provide privacy-preserved monitoring in public [20]. Moreover, it should be capable of identifying the concerned person when required. Figure 1 shows the dilemma of intelligent monitoring systems.
As illustrated in the above figure, an anonymous face protects personal information during in-cabin monitoring of an FAV. However, in certain irregular situations, personal identity is required to identify the accused person. An example of an abnormal incident or irregular situation can be an occupant of the FAV acting violently or attempting vandalism against the other occupants or toward the FAV itself. In such cases, it is important to identify the concerned person. Furthermore, this is an abnormal situation; however, in-cabin monitoring with real faces is not a solution to this problem. e breach of facial information leads to multiple consequences, such as misuse of facial data and banking and financial fraud [1,6,7,13,14]. One of our motivations for this work was to provide an approach that can protect against such problems in public monitoring systems, particularly the IMS. In-cabin monitoring with facial anonymization has security issues, while those with facial identity have privacy issues. erefore, it creates a contradiction between privacy and security.

In-Cabin Monitoring.
In-cabin monitoring is important in level 4 and beyond AVs [5]. It provides safety and security to the occupants. Simultaneously, it provides safety to the vehicle itself in an irregular situation. Past research works include in-cabin monitoring in various situations [21]. Incabin monitoring for violence detection inside a FAV was reviewed in [22]. Bell et al. performed in-cabin monitoring to detect harsh vehicle maneuvers and risky driving behaviors [23]. Szawarski et al. patented the idea of in-cabin monitoring for a monitoring vehicle seat, occupants inside a vehicle, and the orientation of both the occupants and the vehicle seat [24]. Safety and cleaning problems of in-cabin monitoring of a vehicle were presented in [25]. However, a monitoring system should protect against any breach of personal privacy (facial identity) with the simultaneous ability to identify an actual person in case of irregular situations.

Facial Privacy versus Facial Recognition in Monitoring
Applications. Real-time monitoring is essential in multiple monitoring applications. However, privacy in the public domain is an important concern in real-time monitoring tasks [26][27][28][29][30]. Facial anonymization is a common practice for preserving personal privacy. Recently, generative adversarial network-(GAN-) based deep learning (DL) models have been widely used for face swapping and anonymization [31][32][33][34]. In our previous study [31], we demonstrated a robust approach to preserving the facial identity of the occupants in a FAV cabin. It incorporated the facial swapping and reenactment technique to maintain privacy in in-cabin monitoring. However, in ab abnormal situation, the anonymized face of the occupants made it difficult to identify the concerned person [20].

Our Key Research Highlights.
In this study, we propose an intelligent IMS. It is an efficient approach for identifying a person, even with an anonymized face. is method resolves both privacy and security issues. Accordingly, we can identify the person who causes an irregular situation, even with their anonymized face. In this approach, we preserved the key facial information of the occupants and stored these identity features on the cloud. ese key features help in recognition of the person involved in the irregular situation. e highlights of this study are as follows: (i) e concept of having an appropriate source face for each target face enhances puppeteering and reenactment of facial emotion and behavior. It helps in event and behavior detection in intelligent monitoring and surveillance systems in the public domain. (ii) e involvement of the two-dimensional (2D) landmark position in the reenactment generator and separate segmentations of face and hair in the segmentation generator with inpainting and blending generators enhances the facial anonymization and reenactment operations. (iii) e 128D identity feature is a key marker for accurate facial identification in an anonymized domain. e concept of storing a pair of IDs (original and anonymized) leads to reidentification without any privacy threat. It is not possible to know the original face with only 128D identity features. For reidentification, both the original visual input and the ID are required. In the cloud, the anonymized visual image with the original ID is stored. erefore, there is no threat of privacy breach, even though the IDs are stored in the cloud. (iv) erefore, the proposed approach augments the facial identity feature information to locate the involved person in any abnormal situation without any personal privacy breach.
is approach pioneers a newer method of monitoring and surveillance to avoid any legal or ethical issues. erefore, a monitoring database can be created in the anonymized domain, thereby facilitating further research on events and behavior monitoring in the public domain.

Materials and Methods
Personal privacy with identification is a challenge as well as a demand in real-time monitoring applications [20]. In this study, we developed a privacy-preserved IMS with the reidentification capability that can identify the accused person. e framework of the proposed method is shown in Figure 2. e proposed system operates in three stages. In stage 1, facial anonymization was performed to ensure personal privacy. It was performed using the onboard device of the 2 Computational Intelligence and Neuroscience AV. In stage 2, a pair of identity features (IDs) was generated for each face before and after anonymization (ID R and ID A ). Further, the anonymized video along with the IDs was fed to the cloud. e pairs of IDs were kept in the cloud for person reidentification when required. e anonymized video frames were sent to the data center for further processing (monitoring and surveillance). In stage 3, the IDs were matched to search the accused (person involved in an    irregular situation (ID A_AS )). During the investigation, the similarity between IDs ensured the identification of the concerned person (ID). Further, during the investigation, this approach was verified by matching the IDs of the suspect face (ID inv ) at the time of investigation with the accused person's ID. e dilemma between monitoring requirements and legal and ethical issues is also resolved through this approach. e details of the proposed approach are discussed thoroughly in Section 2.2.
is approach is suitable for creating a monitoring and surveillance database with legitimation.
2.1. Materials. Many research works have been published on personal privacy and person identification considering these two issues as separate research problems. In this study, we briefly surveyed the related works and developments on both face anonymization and person identification.

Face Anonymization.
Face deidentification preserves privacy-sensitive information. It alters the original face to hide privacy-sensitive information. Anonymization of faces is an easier and more robust solution to personal privacyrelated threats in the digital domain [35]. Blurring, masking faces, or creating a patch over faces is slightly easier than any other face anonymization approach; however, those methods suffer from significant loss of facial information [32,36]. erefore, face swapping has attracted significant attention for facial anonymization purposes. e morphable model-based facial exchange approach is considered a pioneering work in face swapping [37]. Bitouk et al. demonstrated automatic face replacement in their work [38]. Machine-learning-based face swapping was suggested in [39]. A convolutional neural network (CNN) was used for face segmentation and swapping in [40]. GAN-based deep models have become popular for virtual human face generation [33,34]. erefore, along with autoencoders, GANbased face swapping has gained considerable attention among researchers for seamless end-to-end face anonymization [33,34,41]. Face swapping-based automatic generation and editing of faces was showcased in [42]. It used a region-separative GAN (RSGAN). An autoencoder-based algorithm for face swapping was presented to detect fake videos [43]. In [44], a GAN-based encoder-decoder network was suggested to swap human faces. Collateral privacy issues have also been resolved using the face swapping method [45]. Nirkin et al. suggested a face swapping GAN (FSGAN) in [46]. It provided subject agnostic face swapping and reenactment between a pair of faces. Naruniec et al. presented a fully automatic neural face swapping method in [47]. Sun et al. proposed a hybrid model for face anonymization [36]. Hukkelas et al. introduced a GAN-based DeepPrivacy architecture for face deidentification to remove all privacy-sensitive information [34].

Person Identification.
Facial recognition has multipurpose objectives, such as recognition, classification, and discrimination. Urbanization and smart cities demand widespread applications for face recognition [48][49][50][51][52]. erefore, various face recognition approaches involving person identification have been demonstrated by past researchers. Face recognition approaches are classified into three categories: local, holistic, and hybrid approaches [52]. Local approaches involve only partial facial features (such as eyes, mouth, and nose) to recognize a face, whereas holistic approaches involve complete facial features, including background for facial recognition. Hybrid approaches, as the name suggests, involve both local and holistic approaches. In holistic approaches, popular algorithms involve independent component analysis, linear discriminative analysis, and principal component analysis [53,54]. e development of artificial intelligence (AI) incorporating DL and CNNs has boosted the performance of facial recognition algorithms. Taigman et al. presented a deep neural network-based face recognition system, DeepFace [55]. Furthermore, many other extended versions of DeepFace have been demonstrated in multiple studies [56][57][58][59]. Adjabi et al. thoroughly reviewed face recognition techniques and their comparisons and future scope in their study [51]. Kortli et al. surveyed popular face recognition techniques in all three categories, that is, local, holistic, and hybrid approaches, in their study [52]. ey compared these techniques in terms of accuracy, complexity, and robustness. ey also discussed the advantages and disadvantages of the respective approaches. Wang et al. efficiently surveyed DL-based face recognition techniques in their study [60]. ey exhaustively reviewed various popular DL-based approaches, including autoencoder-based, CNN-based, and GAN-based techniques. ey also enumerated the key features, advantages, and disadvantages of these techniques. Furthermore, they summarized some of the commonly used datasets for deep face recognition. Moreover, they indexed the emerging realworld issues and major technical key challenges in deep facial recognition.
However, an application involving person identification must address important privacy concerns [61]. In particular, facial identification in the public domain must tackle individual freedom and ethics-related issues [51,62]. erefore, the state-of-the-art research problem in face recognition is the reidentification of an individual on anonymized data. Rocher et al. demonstrated the likelihood of correctly reidentifying a specific individual, even with the anonymized dataset [30]. ey suggested a generative graphical model that can be trained on incomplete data to accurately identify individuals. Rooijen et al. suggested 2D video tracking for the reidentification of individuals in an anonymized dataset [20]. ey suggested that the real facial information of a person is not necessary for reidentification. Luo et al. suggested effective training tricks for person reidentification [63]. A residual learning framework using the residual network (ResNet) model was suggested in [64] for visual recognition tasks. is facilitated the easier and more efficient training of a substantially deeper network. Schroff et al. suggested unified embedding using only 128 bytes per face for efficient face recognition [65]. ey developed their network by incorporating the batch input 4 Computational Intelligence and Neuroscience layer and deep CNN, followed by normalization. ey used triplet loss to minimize the training errors. e world's simplest face recognition library (Dlib face recognition) is a popular and efficient tool for extracting facial landmarks [66]. It is a cross-platform open-source machine-learning toolkit that supports the development of machine-learning algorithms. It helps in recognizing and manipulating faces. Intent and behavior have been successfully detected using various techniques. Facial gesture sensing is performed using virtual reality (VR) and augmented reality (AR) devices, respectively in [67,68]. AR/VR devices provide sensor responses to detect the intent or behavior of the user. However, FAV in-cabin monitoring requires intent or behavior detection using visual (computer vision (CV)-based) monitoring approaches.

Method.
In this study, we proposed a representation learning-based approach to generate the identity signature of occupants. is signature is capable of deidentifying a person concerned with an irregular situation in the cabin of level 4 and beyond AVs. We proposed facial anonymization and reidentification system to provide countermeasures in case of an irregular situation. erefore, this method provides personal information security with traces of the concerned person in case of any abnormality. e proposed method includes four main tasks. First, face anonymization with reenrollment.
is is performed by using the face agnostic face swapping technique. It uses a set of GANs.
ese GANs are used for three purposes: facial reenactment and segmentation, facial inpainting, and facial blending. After accomplishing face anonymization, the second task is to extract the facial identity features of the occupant's faces in pairs (before and after anonymization, i.e., ID R and ID A ) using the ResNet-based model. ese IDs are stored in the cloud, and the anonymized video frames of in-cabin monitoring are transferred to the data center via the cloud for further processing. e third task of the proposed approach is to identify the accused by identity feature matching. Similarity matching of the ID of the accused obtained at the data center with the IDs of the occupants stored in the cloud ensures the identification of the concerned person (ID A ). However, it is the ID of the anonymized face of the accused. e Euclidean distance metric was used for similarity matching. Similarly, using the stored pairs of IDs (IDR and ID A ), we can obtain the real face identity feature of the accused (ID R ). Finally, in the fourth task, the evidence of the accused is obtained by matching the similarities between the IDs of the suspects with the ID of the accused during an investigation. Further details of the proposed method are provided in the following sections.

Facial Identity Feature Vector.
e facial identity feature is (128, 1)-dimensional encoding of a facial image. It contains the encoded landmarks of the face using the ResNet model. e FaceNet-based CNN model and Facedlib face recognition library are used to extract the 128D identity features (ID) from the faces. Additionally, 128D is optimal embedding, which results in appropriate features required for reidentification or measuring the similarity between two faces. It has already been validated in the "FaceNet" architecture that fewer than 128D identity features deteriorate the identification performance; however, increasing the dimension only unnecessarily increases the number of parameters. is is the main reason for adopting the 128D identity features for recognizing faces. Furthermore, a distance-based classifier compares the 128D features to identify the person involved in an irregular situation. It represents the difference between two feature vectors in Euclidean space. Suppose that image (R) represents the person. Image (C) is the stored image (copy) of the same person on the cloud, and image (D) is an image of another occupant. Further, f (x) represents the 128D encoding of the image f(x). e similarity (S) in the vector space is measured by the following equation: It guarantees that images (R) and (C) are of the same occupant and are different from image (D), which is the image of another occupant.

Source Image Generation.
A source image was required for face swapping in facial anonymization. It is used to replace the face appearing in the target image. is replacement, that is, swapping, should produce a realistic result that seamlessly reenacts the anonymized face that is similar to the target face. Our recommendation is to use a nonreal face as the source image. It mitigates any chaos/ conflicts that may occur by using any real face as the source image. erefore, in our proposed method, we used GAN-Computational Intelligence and Neuroscience generated virtual human faces as the source image. We have considered generating appropriate source faces that can effectively render the original emotions or behaviors performed by the occupants. It helps in further event and behavior-monitoring tasks. Figure 4 shows the proposed source image generation process. We applied the concept of similarity matching in vector space to select a similar source face for each target face from the set of virtual human faces (nonreal face as the source image). Similarity matching between source and target faces facilitates reciprocating similar emotions and intents, which is necessary for further monitoring applications. Figure 4 shows the source image generation process. e face detector detects the faces (target faces) of the occupants (from the in-cabin visual input). e identity feature extractor extracts the IDs (128D identity features) of faces (target faces) and matches the similarity of the target faces with the set of virtual human faces (source faces) to find the most appropriate source face.
is similarity matching is in the vector space (Euclidean distance matching between the extracted face ID and IDs of the set of virtual human faces).

Facial Anonymization.
Facial anonymization requires exactitude in the anonymized faces to mitigate errors in further processing. erefore, swapping should be performed efficiently to provide unaltered expressions and emotions over the anonymized face. We used the concept of FSGAN for facial anonymization to provide personal privacy during in-cabin monitoring of irregular situations. is requires perfection in the following three tasks: (i) Facial Reenactment and Segmentation. To obtain proper facial swapping, we must estimate the proper reenacted face.
is is performed by the proper segmentation of the face and hair segments of the target image. Proper facial reenactment requires separate face and hair segmentations with the mapping of 2D facial landmark positions. erefore, the stepwise loss function is considered as the objective function for implementing facial reenactment. For i th layer feature map (F i ∈ R C i ×H i ×W i ), the perceptual loss (L perc ) between pairs of images (x, y) is expressed as follows: e reconstruction loss (L rec ) between a pair of images (x, y) is expressed as follows: L rec (x, y) � λ perc × L perc (x, y) + λ pixel × L pixel (x, y), (4) where "λ" is the corresponding hyperparameter (λ perc � 1; λ pixel � 0.1; λ adv � 0.001; λ SG � 0.1; λ rec � 1; λ stepwise � 1) and λ reenactment is linearly increased from 0 to 1 during training. Pixelwise loss (L pixel ) between a pair of images (x, y) is calculated as (L pixel (x, y) � ‖x − y‖). We have used the multiscale discriminator adversarial loss objective function to improve the realism of the generated images. e adversarial loss (L adv ) between the generator and discriminator (G, D) is expressed as follows: where "E (x,y) " is the expected value over all real data instances. "E (x) " is the expected value over all random inputs to the generator. e reenactment generator loss (L RG ) is given by the following equation:  Computational Intelligence and Neuroscience e perpetual loss is used to estimate the errors in capturing fine facial details, and the reconstruction loss is used to evaluate pixelwise color inaccuracy. Adversarial loss improves the generated images and provides a realistic look. e standard cross-entropy loss (L CE ) is defined as (for truth label "t i " and the "SoftMax" probability "P i " for i th class) Further, segmentation generator loss (L SG ) is obtained by the following equation: (ii) Facial Inpainting. is method estimates the missing portions of the reenacted face based on the face and hair segmentation of the target image. e inpainting generator loss (L IP ) was calculated using the following equation: (iii) Facial Blending. It blends the completely reenacted face such that the swapped face matches the background environment like the original target face. e loss function (L B ) for facial blending is obtained using the following equation: e identity signature is generated corresponding to each occupant (a pair of identity signatures for real and anonymized faces) in the FAV. After facial anonymization, the video frames are transmitted to the cloud along with a pair of identity signatures of the occupants.

Anonymized Person Reidentification in Abnormal
Situations.
e proposed IMS facilitates the reidentification of the person involved in an abnormal situation. In our algorithm, in-cabin facial anonymization for preserving identity before transmitting the video frames to the cloud was achieved through the following pseudocode. e identity signature is generated corresponding to each occupant in the FAV. It is a vector of size 1 × 128. erefore, for each occupant, we have a pair of identity signatures corresponding to the original and anonymized faces. Each pair is stored in the cloud. In any irregular situation, the concerned person is back-traced by matching the identity signature and anonymized face. e following is Pseudocode 1 of our proposed approach for obtaining the identity features (ID) of the person involved in an abnormal situation.
We considered virtual human face generation for the source faces. ese faces are used to swap the target face in the captured visual in-cabin dataset. e source faces are generated depending on the similarity of the target face in the vector space. A similar source face provides the exactitude in replaying the facial gestures. is facilitates better reenactment performances. e concept of virtual human face generation for the source face protects any chaos or risk of threatening others' identities. Furthermore, we generated the facial identity signatures of the original and anonymized faces. ese identity signatures help backtrack the concerned person in the event of an irregular situation. e identity signature is only vectored information. In other words, the identity signature in our proposed approach is extracted from a face that is used to reidentify the face. However, a face cannot be recreated using this information. erefore, personal identity is not revealed through the identity signature. Our proposed approach provides proof or evidence that confirms the identity of the concerned person. e  Computational Intelligence and Neuroscience following is Pseudocode 2 of our proposed approach for evidence of the person involved in an abnormal situation.
In the case of proof or evidence, our method determines who is the concerned person. e returned identity feature (real face ID R (k)) in Pseudocode 1 refers to the crucial identity parameter of the person involved in an abnormal situation. Matching the identity feature at the time of investigation with the obtained ID (real face ID R (k)) confirms the person involved in an abnormal situation. erefore, this approach easily locates the person involved in an irregular situation without any breach of others' identities.

Results and Discussion
In our experiment, we first anonymized the occupants of the FAV to secure their privacy in the public domain. Further, we applied the concept of vector space similarity to match the representation learning-based identity features for face recognition to locate the person involved in an irregular situation. e augmentation of the representation-learning-based identity feature introduces a new domain in reidentification.
e proposed system was introduced to maintain personal privacy during the monitoring. We examined our proposed system for the in-cabin monitoring task of the FAV. We captured our database for in-cabin monitoring in abnormal situations. e similarity measure (S i,j ) is calculated by the Euclidean distance (ED) metric that is expressed as follows: where f (i) and f (j) represent the 128D encoding of images i and j, respectively. erefore, the similarity measure identifies the distance (Euclidean distance) between two pairs of IDs (128D encoding). e lesser the distance is, the closer the faces are. (1) for i � 1 to range of the occupant: At datacenter: monitor event and behavior for AS: if occupant j is involved in AS, then: (xi) generate (ID A_AS (j)) #ID of j th occupant in abnormal situation (x) match ID: for ID from 1 to range of the ID:  Computational Intelligence and Neuroscience appropriate source face is obtained by matching their similarity in the vector space. We considered various scenarios to assess the efficacy of our proposed approach, including single and multiple faces in the input image frame. Figure 5 shows the complete set of the considered source faces in our experiment. We considered a set of 24 source faces (shown below). All these faces were not real (AIgenerated). e source faces were used to swap the target face in the facial anonymization process. ese faces are nonreal virtual human faces. Generated Photos provides GAN-generated faces, which are human faces of nonreal humans.
is has the benefit of further augmentation in anonymization. We considered various scenarios in our experiments. Examples include images with a single face only (for both males and females), multiple faces for males only and females only, and multiple faces for both males and females. ese are in-cabin images obtained from the public domain (through an image search on the web) and are shown in Figure 6. We considered different scenarios for the occupants in the cabin. erefore, in F1 and F2, there is only a single person in-cabin (F1: male and F2: female). In other scenarios, we considered more than one person in the cabin (only males, only females, and both males and females). Finally, we considered a family with children. ere are four most appropriate source faces (S1 to S4) chosen for face anonymization. Table 1 presents the similarities (in vector space) between the source and target faces, as shown in Figures 5 and  6.
ese values follow the facial similarities of the source and target faces. ese values measure the distance between the identity features of the source and target faces. e lower the values are, the more similar the faces are. e values in the green boxes represent the minimum Euclidean distances.
ese minimum values indicate appropriate source faces for anonymization. We can observe that the male target faces have lesser distances for male source faces than for female source faces. Interestingly, the distance values follow the similarity in looks as well. e eastern looks target faces have a lesser distance for eastern source faces than for the western source face, and vice versa. Female source faces have a lesser distance than the identity features of children's target faces.

Privacy Preservation during In-Cabin Monitoring.
Facial anonymization is performed after deciding the appropriate source face using FSGAN-based face swapping and reenactment. Figure 7 depicts the reenacted anonymization of the target faces. Here, the first row (F1 to F8) and the third row (F9 to F23) show the original in-cabin visual inputs, and the corresponding anonymized output is represented in the second row (A1-A8) and fourth row (A9-A23). We chose four source faces (S1 to S4 shown in Figure 6) to swap the target faces (F1 to F23).
It is evident from this result that perfect reenactments are achieved even in the anonymized domain. us, it discerns the preservation of personal privacy during monitoring and surveillance operations. Furthermore, this appropriate reenactment supports the detection of abnormal or irregular situations in real time. To examine abnormality detection in the anonymized domain, we have experimented by considering vandalism as an irregular situation inside the vehicle cabin. We created our database for a similar situation. Snippets of the vandalism inside the vehicle are shown in Figure 8. We created a situation wherein occupants in the back seat of the vehicle started fighting with the occupants in the front seat. Four scenes were captured in our experiment. Shoulder shaking is shown in scene #1. Scene #2 shows a slapping scenario. Head shaking is discerned in scene #3, and scene #4 represents a neck choking incident inside the cabin of the vehicle. e identity features (IDs) of each occupant were calculated for normal and irregular situations. It is clearly observed that O3 (in the green box) is responsible for the irregular situation (in-cabin vandalism of the vehicle shown in the red box). Table 2 presents the similarities of the anonymized identity feature (ID A ) with the anonymized facial identity feature of occupant #3 (ID A_IS ). Here, ID A_IS is the anonymized identity feature of the occupant who is involved in an irregular situation calculated at the data center, and ID A is the anonymized identity feature of the occupant stored in the cloud.

Person Reidentification in Abnormal Situations.
e values in the green boxes represent the minimum Euclidean distances. ese minimum differences between the IDs indicate the involved person. e original ID of this person is stored in the cloud. erefore, by mapping the ID, we can easily identify the real person. Reidentification was performed by backtracking the ID obtained from the cloud and pictures of the occupants taken during the investigation.
e ID of the person involved in an abnormal situation from the cloud (ID R ) needs to be matched with the IDs of the occupants inside the vehicle for facial identification of the person. is approach provides proof or evidence confirming the identity of the concerned person. For assurance of the person involved in the abnormal situation, we took pictures of the occupants (during an investigation). e images are shown in Figure 9. Now, the identity feature of each occupant is extracted to match the concerned person ID (ID R ) (as per Pseudocode 2). First, we compared the similarity between the faces of the occupants inside the vehicle with those of the other faces captured during the investigation. is is required to ensure that the occupants are the same. Table 3 presents the similarity measures between the occupants' IDs extracted during an investigation and their IDs extracted from the in-cabin images. e minimum Euclidean distances are represented by the green boxes. Here, minima indicate that the occupants O and O′' are the same. ereafter, assurance of the involved person is performed by matching the identity feature of the occupants extracted from the in-cabin image of the vehicle with the ID of the person involved in an abnormal situation (stored in cloud ID R ). Table 4 presents the similarity measures between the occupants' IDs extracted from the incabin image with the obtained ID of the person involved in an abnormal situation (stored in cloud ID R ).
Computational Intelligence and Neuroscience  . ese virtual human faces are obtained from Generated Photos. It provides AIgenerated images that are free from any copyrights, distribution rights, and infringement claims (source: Generated Photos (https:// generated.photos/)).

10
Computational Intelligence and Neuroscience e zero value in the green box indicates that the occupant (O3) is the person involved in an abnormal situation. Overall, this approach focuses on in-cabin monitoring with personal privacy preservation to avoid abnormal situations.
Personal privacy preservation is achieved by using the concept of event and behavior monitoring in an anonymized domain. e person's reidentification is only for providing evidence in cases where the involved person is denying it.

Conclusions
Identity feature augmentation in anonymization is a potential solution for providing privacy in public domain monitoring. Identification of the involved person is crucial, especially in abnormal situations. e proposed intelligent IMS augments the security features with privacy. is method is suitable for creating a monitoring database without any restrictions or legalities. We performed various scenarios to assess the efficacy of the proposed system. It provided an efficient algorithm to perform monitoring tasks in the public domain without any threat to the personal identity of a person. is helped in reidentification, even with an anonymized face. In the future, this algorithm can be implemented on various public domain monitoring platforms, such as transportation systems, shopping centers, theaters, hospitals, highways, fuel refilling stations, smart city applications, and toll plazas.

Data Availability
e image data used to support the findings of this study are included in this paper.

Conflicts of Interest
e authors declare that there are no conflicts of interest regarding the publication of this paper.

Authors' Contributions
Ashutosh Mishra and Shiho Kim contributed to the conceptualization. Ashutosh Mishra, Jaekwang Cha, and Shiho Kim developed the methodology, performed formal analysis, and investigated. Ashutosh Mishra reviewed and edited the paper. Ashutosh Mishra and Jaekwang Cha provided the software and performed validation, visualization, and data curation. Ashutosh Mishra provided the resources and preparation. Shiho Kim contributed to supervision, project administration, and funding acquisition. All authors have read and agreed to the published version of the paper.

Acknowledgments
is study was partially supported by the Brain Pool Program through the National Research Foundation of Korea  Table 4: Identity feature matching between ID R stored in the cloud with other occupant's IDs extracted from the in-cabin of the vehicle.


extfieldsofstudy:
Computer Science
Engineering

text:
Domain Adapting Deep Reinforcement Learning for Real-world Speech Emotion Recognition

Computers can understand and then engage with people in an emotionally intelligent way thanks to speech-emotion recognition (SER). However, the performance of SER in cross-corpus and real-world live data feed scenarios can be significantly improved. The inability to adapt an existing model to a new domain is one of the shortcomings of SER methods. To address this challenge, researchers have developed domain adaptation techniques that transfer knowledge learnt by a model across the domain. Although existing domain adaptation techniques have improved performances across domains, they can be improved to adapt to a real-world live data feed situation where a model can self-tune while deployed. In this paper, we present a deep reinforcement learning-based strategy (RL-DA) for adapting a pre-trained model to a real-world live data feed setting while interacting with the environment and collecting continual feedback. RL-DA is evaluated on SER tasks, including cross-corpus and cross-language domain adaption schema. Evaluation results show that in a live data feed setting, RL-DA outperforms a baseline strategy by 11% and 14% in cross-corpus and cross-language scenarios, respectively.

INTRODUCTION
E Motion-aware interaction can significantly improve human-computer interaction. Consequently, there has been much research on automatic speech emotion recognition (SER). While SER within the same corpus is possible with reasonable accuracy, cross-corpus SER performance needs significant improvement [1], [2], [3]. Cross-corpus SER is crucial to achieving emotion-aware interactions in real-life applications, as speech signals obtained from different devices, recording background, spoken language, and acoustic signal conditions can be different with the training dataset and real-world implementation [1].
Researchers have explored domain adaptation techniques to improve the cross-corpus SER performance [4], [5], [6], [7]. Domain adaptation is a transfer learning approach where a trained model undergoes training aiming to optimise for a different distribution other than the previously trained distribution. The key limitation of the existing domain adaptation methods is that they cannot be adapted to the dynamic nature of the real-life setting. In such a setting, it will be beneficial if an intelligent agent interacting with a customer/user could dynamically update itself when it wrongly categorises speech emotion. In this paper, we address this challenge. We develop a domain adaptation technique based on reinforcement learning (RL), where we exploit the dynamically updating feature of RL to develop a cross-corpus SER technique that can be dynamically up- Rajapakshe  dated. Reinforcement Learning has been used to optimise realworld interactions with robots and machines after training in a simulated environment [8], [9], however, it has not been used in the field of SER. In a typical RL scenario, an agent takes actions in an environment, which is interpreted into a reward and a representation of the state, which are fed back into the agent. This does not trivially match a domain adaptation scenario of SER. To the best of our knowledge, we are the first to develop and evaluate techniques for using RL for this task.
A possible use case of the proposed technique is shown in Figure 1. A base deep learning model can be pre-trained with a labelled source dataset before starting Reinforcement Learning based Domain Adaptation (RL-DA). Then, the pretrained model is optimised to the target domain with RL. During this optimisation, the data feed can come from a live data source, where the model gets feedback from the customer regarding its interpretation of emotion in the customer's voice. Feedback can, for example, be implemented by a thumbs up or down button on the customer console. An example application demonstrating the above use case can be found online 1 . This application consumes recorded audio and infers the emotion from that audio utterance while users are able to provide feedback on the inferred emotion. This feedback is used for RL optimisation.
Using widely used speech corpora, we evaluate the model in cross-corpus and cross-language scenarios and show that we achieve better SER performance than fully supervised benchmark models. Moreover, we simulate a real-life (live) data feed scenario and show that our proposed RL model significantly outperforms fully supervised benchmark models.  [10], [11], [12], recommendation systems [13], and robotics [14]. However, the use of RL for speech-based applications is slowly getting traction [15].
In this section, we discuss the existing literature in two groups: one group combines literature on Domain Adaptation in SER, and the other group consolidates the previous work done with RL for domain adaptation.

Domain Adaptation
Domain adaptation was introduced to overcome the "corpus bias" issue. Since deep learning methods are powerful in extracting non-linear features in the input, domain adaptation can easily be implemented on deep learning-based platforms, yielding more robust and performing models [16], [17].
Mao et al. experimented on a two-class task for domain adaptation by sharing priors between source and target domains. They pre-trained a two-layer neural network with unsupervised learning and then shared the common classifier parameters between source and target [4]. Our approach follows the RL paradigm for domain adaptation instead of unsupervised learning, and we pre-train the source model with a labelled source dataset.
Gharib et al. proposed unsupervised adversarial domain adaptation and pre-trained the model with two conditional sets achieving nearly 10 % increase in accuracy [24].
Ahn et al. proposed a Few-shot learning methodology in unsupervised domain adaptation for cross-corpus SER tasks [23]. They used multiple corpora to optimise the emotion recognition robustness to unseen samples.
Abdelwahab and Busso studied the best adaptation technique for speech emotion recognition models out of supervised domain adaptation and online training. They also studied how the size of the labelled data affects the performance of the resulting model [25].
proposed using adversarial training techniques to improve cross-lingual domain adaption [29]. This is an unsupervised learning method that utilises two distinct auto-encoders for source and target, as well as a single discriminator, to determine whether an audio utterance is fake or real. Although we focus on cross-lingual domain adaptation, our main target is enabling domain adaptation in real-world applications.

Reinforcement Learning for Domain Adaptation
The RL agent gains knowledge via experiences, and these experiences are obtained by exploring and exploiting. The RL agent uses the policies to choose the best action given the present state -the method is called exploitation. Since the RL agent does not have enough experience to update the policy in the initial phase, it explores the environment. Exploration is done by randomly selecting actions without considering the policy, and the experience is stored in the memory buffer with a reward. The main challenge in this exploration is the unsafe choice of actions in some states [22].
Prior knowledge can be fed to the RL agent before releasing it to the real world in many ways. Demonstration based learning [30], [31], and pre-training and domain adaptation [22] are some of the methods to nourish the RL agent with knowledge. This reduces the amount of exploration time and the risk of unsafe exploration. Previous studies also have shown that pre-training has reduced the training time to achieve higher performance [32], [33], [34].
Koo et al. consider the feature extractor for the target domain as a generator where the parameters are similar to the latent feature set in the source and are trained with adversarial training loss. They proposed this to enhance the coherent training of the target domain feature extractor [21]. Though their research proposes an RL-based approach in domain adaptation, they have not studied speech emotion recognition.
Leonetti et al. have introduced a new RL algorithm, "DARLING", which can adapt to the new environment and improve the reliability of the decisions [18].
Hazara and Kyrki proposed a transfer approach that captures a simulated system's core features and rationalises the dynamics combined with incremental learning. They demonstrated their approach by applying it to a robot with a basketball task and showed the target model has improved task generalisation capability over direct usage [35]. In summary: 1) Literature on domain adaptation for SER is very rich, many approaches involve deep learning techniques. 2) Very few studies involve RL for speech emotion recognition tasks such as EmoRL [19]. But there is lack of research in the literature that uses RL for domain adaptation for SER. 3) Table 1 provides a brief overview of the literature on RL for domain adaptation and speech emotion classification, which attests that there has been a lack of literature that combines RL with DA for the problem of SER, which is the focus of this paper.  [4] Carr et al, 2018 [12] Lakomkin et al, 2018 [19] Hossain and Muhammad, 2019 [20] Koo et al, 2019 [21] Arndt et al, 2020 [22] Ahn et al, 2021 [23] This paper

Background
Reinforcement Learning follows the principle of behaviourist psychology and learns similar to a child who learns to perform a new task. There are two main components in an RL problem: the Environment and the Agent. The agent performs actions on an environment, and the environment returns a reward regarding the action performed along with the next state of the environment. Figure 2 shows the concise architecture of general RL methodology.
A Markov Decision Process can interpret an RL problem, which has the parameters state space S, action space A, and reward R. A state s ∈ S is the representation of the current environment, action a ∈ A is a single task that can be performed on the environment, and reward r ∈ R is the feedback value returned from the environment after executing the task a for the state s. RL agent learns a policy π(a, s) that represents a probability of taking action a for a specific state s. The Q-value or quality value represents the expected reward when action a is performed on an environment with state s. The optimum Q-value Q * (a, s) can be written as in Equation 1; Q-values are estimated using Q-tables in q-learning, while a deep neural network called Deep Q Network (DQN) is used to estimate the Q values in deep q-learning. The loss function in Equation 2 is used to calculate the loss and Q target can be as of Equation 3.
Combining the two equations 2 and 3, the loss function can be re-written as in Equation 4.
The loss L should be minimised in the training phase of RL.

Domain Adapting Reinforcement Learning for Speech Emotion Recognition
In this study, RL is used as an emotion recognising game where an RL agent attempts to recognise the correct emotion (action a) for a given audio utterance (state s). A reward r is returned by the environment. The reward value is obtained either by the feedback from the user in a real-world scenario or by comparing the ground truth and inferred emotion by the RL agent when training with labelled datasets. A policy π is learnt by the RL agent to maximise the reward gained at each episode. The state space S is defined as the distribution of speech audio, the Action space A is defined as the discrete emotion classes, and action selection is the inference of an emotion to a given audio utterance. According to Equation 4 to minimise L, Q(s t , a t ) should be maximised whereas the Reward R t+1 should be maximal at L * . Pre-training the DQN model prior to utilising an RL agent has proven to increase the performance of the model [33], [34]. Here, RL is used to optimise the pre-trained model to the target domain. The RL agent with a pre-trained DQN model interacts with the environment and tries to optimise the DQN to adapt to the target domain.

Baseline
Though RL methodologies do not report performance based on accuracy since there is no labeled data associated with RL experiments, we are using a supervised learning (SL) approach as the baseline to compare the performance in our methodology. We use a DNN architecture similar to the DNN used in the RL agent (Deep Q Network in Figure 2). We first train the model with the Source Dataset to be aligned with the pre-training in the RL approach and obtain a Base Model as in Figure 1. Then, the Base Model is trained with the Target dataset to match with the RL-DA. This allows us to compare the two approaches RL-DA and Supervised Learning-based Domain Adaptation, with similar parameters. We use four labelled datasets, and section 4.1 describes the datasets used in this study. A fraction of 20 % of each dataset is reserved as a testing dataset, and the remaining subsection is used for either pretraining or RL domain optimisation. Section 4 describes the different kinds of experiments carried out within this study to see the effectiveness of our approach.

Datasets
To measure the performance of the proposed RL based Domain Adaptation (RL-DA) approach, we use four widely used publicly available datasets for our experiments: MSP-IMPROV [36], IEMOCAP [37], ESD [38] and EmoDB [39]. The selection of datasets covers the ability to use them for Cross-Corpus (CC) experiments as well as Cross-Language (CL) experiments. Since the datasets do not contain a balanced number of utterances under each emotion class, we use an audio augmentation technique to generate audio utterances. We use Vocal Tract Length Perturbation (VLTP) [40] to augment audio utterances balancing utterances under each emotion class when preparing the testing subsets. All the four speech emotion datasets we use contain categorical emotional labels, and we use four emotions widely used in the literature: 'happiness', 'sadness', 'anger', and 'neutral'.

IEMOCAP (IEM)
IEM is a popular dataset containing 12 hours of multispeaker acted audio-visual data. The IEMOCAP dataset contains dyadic sessions with both improvised and scripted scenarios. It contains categorical emotion labels happiness, sadness, anger, and neutrality as well as dimensional labels valence, dominance, and activation [37]. This study uses the audio data modality from improvised scenarios and categorical labels.

MSP-IMPROV (MSP)
MSP is an acted audio-visual database that is popular in multi-modal speech emotion recognition research. The database was initially developed for a study in audiovisual emotional perception, but this dataset has also been used in speech emotion recognition studies [23], [41], [42], [43] which makes it a candidate for our study as well. The dataset was recorded in a controlled environment and consists of 20 determined scripts covering the primary four emotions: happiness, sadness, anger, and neutral [36].

Emotional Speech Dataset (ESD)
ESD is a public database of speech data that was initially developed for speech synthesis and voice conversion. It contains utterances from 20 speakers in two languages (10 Mandarin and 10 English) categorically labelled into five emotion classes: happy, surprise, neutral, angry, and sad [38]. We use utterances from 10 English speakers under the four emotions happiness, sadness, anger, and neutral.

Berlin EmoDB (EmoDB)
EmoDB is an actor recorded database of emotional speech utterances in the German language. 10 actors (5 male and 5 female) in the age range between 21 years and 35 years speak 10 scripted texts in 7 different emotions such as anger, boredom, disgust, fear, happiness, sadness, and neutral [39]. We use EmoDB as the target dataset for cross-language experiments. We choose the utterances of the four emotion classes: anger, happiness, sadness, and neutral.

Diverse Environments Multichannel Acoustic Noise Database (DEMAND)
The DEMAND dataset is a popular real-world background noise dataset. The database contains background noises from 18 different environments under six categories. An array of 16 microphones is used to record the audio in each environment stored in 16 channels. The DEMAND dataset contains versions of both 16 kHz and 48 kHz sampling rates, and we use 48 kHz and a down-sampled to 22 kHz version to match with the other datasets used in this study.
A summary of the datasets used in this study is shown in Table 2

Input feature
We use Mel Frequency Cepstral Coefficients (MFCC) as the input features [44], [45]. We use 2, 048 as the frame length and the hop length as 512 and extract 40 MFCCs using the Librosa python library for audio and music analysis [46].

Model configuration
The functionality of the DNN model is to classify the emotion embedded in speech utterances, and the study uses MFCCs of speech utterances as the input for the model. We use a Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) combined model since it motivates the capability to learn both frequency and temporal components in the speech signal [34], [47]. We stack CNN, LSTM, and fully connected layers, respectively, for the model to learn the discriminative features [45]. As shown in Figure 2, the feature matrix is fed into two layers of 2D convolution layers of filter sizes 5 and 3, and the output of the first 2D convolution layer is batch normalised. Hyperparameter batch size is set to 128. The output from the second 2D convolution layer is passed to the LSTM layer of 16 cells after flattening and then to a fully connected layer of 256 units. A dropout rate of 0.3 is applied before the last Dense output layer. The number of units in the output layer is four, equal to the number of emotions to be classified. The linear activation function of the output layer is used when the model is used in the RL agent since it outputs Q-values for a specific state, and they should not be normalised. The input shape of the model is 40 × 87, where 40 MFCCs are used in the input and 87 is the number of MFCC frames.
The model is optimised using the Adam Optimiser with a learning rate of 2.5 × 10 −4 . We use the popular Deep Learning API Keras [48] with Tensorflow [49] (version 2.1.0) as the back-end for modelling and training purposes in this study.

Experiments
We design and run several experiments to evaluate the proposed RL based domain adaptation approach. For evaluation, we develop three scenarios. (1) Pre-training with a source dataset and domain optimisation with a target dataset separately; (2) Pre-training with a subset of the source dataset and domain optimisation with a mix of the remaining source and target datasets; (3) Domain adaptation in a recreated real-world scenario. Details of each scenario and results are explained in Sections 5.1, 5.2, and 5.3 respectively.
The Unweighted Average Recall rate (UAR %) is one of the popular metrics used by researchers to measure the performance of speech based machine learning models. The recall is calculated for each label of the classification problem, and the UAR is calculated by taking the unweighted average of the recall values of each label The Python code repository of the experiments and data preparation is hosted in public GitHub repository "RL-DomainAdaptation" 2

EVALUATIONS
This section reports the results and description obtained through the experiments in section 4. We use the IEMO-CAP and MSP-Improv datasets as source datasets, while 2. https://github.com/jayaneetha/RL-DomainAdaptation the ESD dataset is used as the target dataset in the crosscorpus schema, and EmoDB is used as the target dataset in the cross-language schema. We use Max Boltzman Policy (Max.B) [50] as the RL policy in the RL agent.

Experiment with separate datasets for source and target
Here, we evaluate the proposed RL-DA approach for separate source and target datasets. We do this for both the crosscorpus and cross-language schemas. Table 3 and Figure 3 (a) show the datasets used for each schema. First, the model is pre-trained using the training subset of the source dataset, obtaining the Base model. Then, the pre-trained base model parameters are transferred onto the DQN of the RL agent. Afterwards, the RL deep Q learning algorithm is executed to optimise the DQN model applying the target dataset in the environment which outputs the domain optimised DNN model. To measure the performance of the domain optimised model, we use a Supervised Learning approach. We take the testing subset of the Target dataset and infer the emotion of each testing utterance and record the inference emotion with the labelled emotion for the utterance. The UAR value for each experiment is calculated using the recorded data about inference emotions and ground truth.
Supervised Learning approach based performance is also measured to compare the performance of our methodology as the baseline (SL-DA -sep). We create a DNN model that is similar in architecture as used in DQN (the Deep Q Network in Figure 2) and is pre-trained by the Source dataset. Then, the same model is trained by the training subset of the Target dataset. Subsequently, the resulting model is tested with the testing subset of the Target dataset to obtain the testing accuracy.
The UAR of the baseline (SL-DA -sep) model after training with supervised learning and the UAR of the model after training with the RL-DA approach (RL-DA) are reported in Table 4. We find that the accuracy of RL-DA surpasses the baseline UAR by 0.45 %, 2.92 %, and 8.83 %, respectively, except for in the IEM-EmoDB combination. Additionally, while comparing the SL-DA method and the RL-DA method, the cross-language schema's standard deviation has decreased from 8.26 to 0.71, respectively, demonstrating that the model performances are more consistent.
This outcome is noteworthy because, unlike SL approaches, which feed labelled data to train the model, the RL methods only provide feedback rewards that show the accuracy of inference made during the RL optimisation phase. So, achieving on par SL accuracy also means that  We calculate UAR to compare the performance of our models. We use the model architecture, source, and the target dataset as the parameters of similarity in the literature on cross-corpus speech emotion recognition to compare our results. The results are represented in Table 5.

Experiment mixing the source and target datasets
This experiment is carried out to evaluate our proposed methodology while including a subset of the source dataset in the RL optimisation phase. The aim of such mixing is to preserve generalisable features of the source dataset. This experiment is performed similar to the previous experiment mentioned in 5.1, but with the following modifications: Only a subset of the training set of the source dataset is used in pre-training, and a mix of the remaining training set of the source dataset and the training set of the target dataset is used in the RL optimisation. Table 6 and Figure 3 (b) show how the datasets are used in this experiment. We measure the performance by using SL as a baseline (SL-DA -mix) to compare with our proposed approach. Results of the experiment by mixing source data into the target dataset as mentioned above are presented in Table 7. The accuracy of the RL-DA approach is compared with the Supervised Learning approach -SL-DA -mix. The RL-DA approach outperforms SL-DA -mix at least by 8 % in each experiment. Since the target dataset includes components from the source dataset, the representations of the model learnt during the pre-training stage are kept while the model adapts to the new domain.
We compare the performance of RL-DA in two scenarios where the target dataset is separated (S) with the source dataset (Section 5.1), and the target dataset is mixed (M) with utterances from the source dataset (Section 5.2) and visualised in Figure 4.
From the above visualisation, we note that the mixing of source and target datasets yields better performance than experiments with separate datasets except for the experiment for MSP-ESD combination. The mixing of datasets increases the diversity of the training utterances which leads the model to learn generalised feature representations when training.  8 Testing Accuracy of SL-DA -rw and the proposed RL-DA approach of each experiment mimicking a real-world scenario for each cross-corpus (CC) and cross-language (CL) schema with (w/) and without (w/o) background noise. Background Noise is added to simulate real-world conditions.

Experiments simulating a live data feed scenario
We experiment to quantify the performance difference between SL and RL optimisation in a simulated live data feed scenario. Testing in a real live data feed scenario would require integrating the system with a call centre or similar setting, which is problematic. However, we closely simulate the live scenario by continually pushing the data from the target dataset. RL-DA uses this data to continually update itself, whereas the SL-DA method uses the static model, which is trained using the source dataset. The methodology of the RL optimisation is similar to the experiment described in 5.1. The baseline model (SL-DA -live) is first trained with a training subset of the source dataset, and then, the testing accuracy is measured with testing subsets of the target dataset. We also measure the performance of RL-DA with realworld audio. We mix background audio with our target datasets to create a new variation of the dataset with background sound. Audio from the kitchen environment of the DEMAND dataset is used as the background sound to mix with target datasets (ESD and EmoDB). We use -5 db as the Signal-to-Noise ratio when mixing the background audio to the speech utterances.
The testing accuracy of the models trained in three different ways (SL-DA -live, RL-DA -target dataset without background noise (RL-DA w/o noise), and RL-DA -target dataset with background noise (RL-DA w/ noise)) in a realworld scenario mentioned in 5.3 are tabulated in Table 8.
The ∆ column shows the difference in accuracy between our approach and the baseline, where one has to use a model pre-trained only in the source dataset.
Observing the ∆ column in Table 8, it is clear that RLbased domain adaptation (RL-DA) outperforms the baseline SL strategy in the live data feed situation (SL-DA -live). Supervised learning methodologies do not have the possibility to adapt to the deviation of a domain distribution without a labelled dataset to retrain during inference time. On the other hand, RL based learning methodologies constantly receive feedback from the environment and optimise the agent model for the domain deviations on the go. One can argue that for the supervised method, one can retrain the model. But this requires manual intervention, whereas applying RL, this can be reached without any manual intervention. Therefore, we suggest that RL is more suitable in domain adaptation to be used in real-world scenarios because of the adaptability during the optimisation phase.  Figure 5 shows the comparison between the UAR of the Baseline, RL-DA without noise, and RL-DA with noise after executing the 'simulated live data feed' experiments. We observe that both RL-DA experiments (with and without background audio) show better results than that of the baseline approach.

CONCLUSION
This study investigated the possibility of using RL for domain adaptation in SER. While several strategies have been used for domain adaptation for SER in the past, we were the first to propose RL-technique for domain adaptation herein. Evaluations of this research suggest that using RL can improve domain adaptability in a real-world scenario compared to a SL approach. An accuracy increment of 11 % in cross-corpus and 14 % in cross-language schema was observable in the results when using the proposed RL-DA approach compared to the baseline strategy.
We discussed that the main reason for the improved domain adaptability in a real-world setting is the ability of the RL agent to continually incorporate feedback. On the other hand, supervised learning does not have such a feedback mechanism unless the model is retrained using a manual process.
The outcome of this study is likely to lead the community in this new direction and help eventuate real-world emotion-aware applications that can adapt to the users' emotional expression under a new domain. Sara Khalifa is currently a senior research scientist at the Distributed Sensing Systems research group, Data61-CSIRO. She is also an honorary adjunct lecturer at the University of Queensland and conjoint lecturer at the University of New South Wales. Her research interests rotate around the broad aspects of mobile and ubiquitous computing, mobile sensing, and Internet of Things