text:
MagicChem: a MR system based on needs theory for chemical experiments

Real chemical experiments may be dangerous or pollute the environment; meanwhile, the preparation of drugs and reagents is time-consuming. Due to the above-mentioned reasons, few experiments can be actually operated by students, which is not conducive to the chemistry learning and the phenomena principle understanding. Recently, due to the impact of Covid-19, many schools adopt online teaching, which is even more detrimental to students’ learning of chemistry. Fortunately, MR(mixed reality) technology provides us with the possibility of solving the safety issues and breaking the space-time constraints, while the theory of human needs (Maslow’s hierarchical needs) provides us with a way to design a comfortable and stimulant MR system with realistic visual presentation and interaction. The paper combines with the theory of human needs to propose a new needs model for virtual experiment. Based on this needs model, we design and develop a comprehensive MR system called MagicChem, which offers a robust 6-DoF interactive and illumination consistent experimental space with virtual-real occlusion, supporting realistic visual interaction, tangible interaction, gesture interaction with touching, voice interaction, temperature interaction, olfactory interaction and virtual human interaction. User study shows that MagicChem satisfies the needs model better than other MR experimental environments that partially meet the needs model. In addition, we explore the application of the needs model in VR environment. Supplementary Information The online version contains supplementary material available at 10.1007/s10055-021-00560-z.

Introduction
Maslow's hierarchical needs (MHN) theory believes that human needs are composed of five layers: physiological needs, security needs, belongingness needs, esteem needs, and self-actualization needs. They are the power to inspire and guide individual behavior McLeod (2007). The human needs theory is also applicable to the field of experimental teaching, so that students can get a better experience and motivation to obtain a good learning effect.
Experiments occupy an important position in chemistry learning Bernath et al. (2005). However, in the real experiments, limited to time, space and resource constraints students have fewer opportunities to repeat experiments. Real chemical experiments may be dangerous, especially with flames, corrosion, toxic gases, etc. Guex et al. (2017), which will be difficult to meet the security needs in MHN theory, causing users to have psychological pressure Luo et al. (2020). In addition, in many schools, due to the lack 1 3 of teacher resources Ángel (2015), Martin-Villalba et al. (2012), students often encounter problems hard to be solved, which is difficult to meet the needs of self-actualization. Even with teacher's guidance, students who often make mistakes in operation will be criticized. It will violate esteem needs and frustrate students' learning motivation.
Guided by needs analysis to design the experiment, the virtual experiment system can make up for the problems of real experiments Duan et al. (2020), Hodges et al. (2018), and improve the interest and efficiency of students in learning chemical experiments Ullah et al. (2016). However, the current researches on virtual experimental systems are mainly focused on the single demand point such as experiment simulation Verkuyl et al. (2017), education and teaching Wang (2018), interactive experience Bozzelli et al. (2019), Luo et al. (2018) and interest de Souza Silva et al. (2017). There are few systematic researches on needs model of human-oriented virtual experiments, especially in MR experiments.
The theory of human needs provides a new perspective for the system design of virtual experiments. The basic purpose of the virtual experiment system is to simulate the real experiment. However, in order for students to obtain better learning effects and enhance the learning interests, it is more important to make the experience of the interactive environment close to the real experiment while having a safe, comfortable, not alone(layers 1-3 in MHN theory) and encouraging (layers 4 and 5 in MHN theory) environment. Compared with VR, MR provides users with a more natural interface that combines the real and virtual world. However, in terms of visual presentation, most MR experimental systems lack the consistency of virtual-real occlusion and illumination, which will cause visual confusion Ellis and Menges (1998), Kalkofen et al. (2007) and make users feel unreal and not easy to immerse the interaction Jacobs and Loscos (2006). In terms of interactive design, most of the MR experimental systems are based on gesture interaction with specific actions or hand-held markers in single camera environments. These systems do not have the operational feeling and tactile feeling as real experiments do. These special shortcomings of the MR environment put forward new requirements for the needs model.
The main contributions of this paper are as follows: • Based on the theory of human needs and the special needs of virtual experiment system, this paper proposes a new needs model to guide the design of virtual experiment system. • To effectively meet the needs model, we develop a MR system called MagicChem which supports the consistency of virtual-real occlusion and illumination in terms of vision, and supports 7 multi-modal interaction methods in terms of interaction. • We develop 3 ablation virtual experimental systems for grouped user study, and explore the effectiveness of Mag-icChem in terms of usability, user experience, and learning effects.

Related work
There are many researches on needs-oriented experimental systems. These experimental systems for education can indeed promote students' learning Arici et al. (2019), Akçayır and Akçayır (2017), Chang and Hwang (2018).

MR experimental systems with mobile environments
The interactive methods of MR on mobile terminals are mainly divided into two versions. One is based on 2D gesture Gan et al. (2018), Boletsis and McCallum (2013), Tuli and Mantri (2015), and the other is on hand-held markers Nachairit and SrisawasdI (2015), Matsutomo et al. (2017). The advantage of this kind of interactive method is that users will be more familiar with the interactive methods of mobile phones. The disadvantage is unnatural to use with one hand holding the mobile phone. In addition, some researchers develop a method for exploring 3D gesture interaction with touching in the mobile environment , but the results are not ideal. The author analyzes the reasons: lacking touch feeling and without correct relationship of virtual-real occlusion.

MR experimental systems with glasses
A lot of MR applications Müller et al. (2018), Strzys et al. (2017), Zhu et al. (2018), Sapargaliyev (2015 use optical see-through glasses, and achieve good results, but this kind of environments is difficult to achieve correct virtual-real occlusion, since virtual objects are projected on the atomic lens, and the user sees the real scene through the lens, which cause virtual objects cannot completely occlude real objects and show a translucent blending effect of the virtual and real scene in visual. Since it is difficult to judge the distance between the hands and the virtual object, the main interaction mode of this kind of MR system is gesture interaction with specific actions rather than touching virtual objects directly. Mapes Mapes and Moshell (1995) believe that gesture interaction with touching is easier to learn, more natural and reliable than specific actions. Moreover, in terms of tracking algorithms, many MR systems use foreground threshold segmentation, then recognize fingertips and palms based on curvature and distance, finally implement click interaction through fingertips Bai et al. (2013) or gesture 1 3 interaction with specific actions Hürst and Van Wezel (2013). These methods cannot adapt to complex scenes with tangible objects and flexible gestures.

MR systems with tabletop environments
For virtual experiments, the most appropriate way is based on tabletop, because real experiments are mainly carried out on the tabletop. The main interaction models of the systems are tangible interaction and gesture interaction.
In recent years, many MR experimental applications Benko et al. (2012), Oswald et al. (2014), Roo et al. (2017) use optical projection and tangible interaction. The advantage is that this kind of systems has a natural viewpoint and touch feeling, but the disadvantage is that it requires a projection medium, such as a tangible object' surface, which usually makes tangible objects not leave the tabletop and lack 6-DoF interactive space. In addition, this kind of systems generally requires a dark lighting environment which is inconvenient for tracking using color information due to the influence of projected light.
Some researchers Maier and Klinker (2013), Seo and Lee (2013) stick the markers on the tangible devices for tangible interaction. However, these systems are based on a single camera. In flexible tangible interaction, MR registration often has errors due to the large angle or distance between the marker and the camera. Controlling the orientation of the marker to face the camera will affect the comfort of the users. Moreover, most of the existed MR environments lack systematic human-oriented research and module design. Benware and Deci Benware and Deci (1984) find that students who use actively orientated learning have higher internal motivation than those who use passively orientated learning. Ryan and Deci Ryan and Deci (2000) prove the importance of intrinsic motivation based on contemporary research and theories, and it reflects the natural human propensity to learn and assimilate knowledge. The mainstream motivation theory is MHN theory Healy (2016). It believes that everyone has five layers of needs. In terms of motivation, each need layer must be substantively met before the next goal can be activated. At the same time, once the needs of a certain layer are substantially met, it no longer has a stimulating effect, and the next layer will become the dominant need.

Needs theory of human
MHN theory has been effectively applied in many fields. For example, in management, the research results of Shoura and Singh Shoura and Singh (1999) show that it is desirable to use MHN theory to improve employees' work consciousness and enhance their personality and sense of mission. In terms of career choice, Goel et al. (2018) find that the motivational factors for students to choose medical study are equivalent to MHN theory. In the medical industry, Benson and Dundis Benson and Dundis (2003) apply MHN model to motivate people to face the growing demands of the healthcare industry. In terms of learning, Kiel (1999) propose through this theory that we need an open and broad structure to better realize ourselves and carry out lifelong learning. In terms of mental health, Lester et al. (1983) summarize and develop a questionnaire to measure people's satisfaction with the five basic needs described by Maslow, and propose that the level of satisfaction with basic needs is related to the scores of neuroticism and belief in an internal locus of control. In terms of long-term immersive VR systems, Guo et al. (2019) propose the needs based on MHN theory in VE to guide the design of VR office. But they only consider three layers of MHN theory. MHN theory has inherent logic and is universally recognized. We believe that based on the needs theory of human we can effectively design an experiment system that is comfortable, close to the real experiment experience and stimulate users' learning motivation.

Needs model for virtual experiments
The goal of the virtual experiment is to let students gain motivation and interest in a comfortable, safe, and realistic virtual environment in order to obtain good learning results. To achieve this goal, as shown in Fig. 1, this paper proposes a human-oriented and systematic needs model for virtual experiments based on the MHN theory.

Basic needs
In the first layer of MHN theory, physiological needs are the most basic needs for maintaining human life. But in the virtual experimental environment, food, water and clothing, etc. do not constitute a problem. Therefore, in combination with the application conditions of virtual experiment, we expand the physiological needs to the basic needs, including sensory needs and system needs. Because the sensory experience is the basis of all perceptions of experiments for users, system needs ensure that users use the system effectively.
The real experiment is to fully mobilize the students' multi-channel sensory experience, so that the students can master and understand the experiment faster. Therefore, we needs to simulate the multi-sensory experience of experiments in the MR experimental environment, including visual experience needs, tactile experience needs, action experience needs, listening and speaking experience needs, temperature experience needs and olfactory experience needs.
In the MR experiment environment, spatial presence needs and the presence of virtual objects are important for visual experience needs. Spatial presence is the cognitive 1 3 sensation of being in a place Skarbez et al. (2017), with which users can accurately judge the position and distance in the virtual space Balakrishnan andSundar (2011), Pollock et al. (2012), Luo et al. (2019). For MR experiments, it requires the designed system to have accurate, multilayer virtual-real occlusion functions. If the rendering of the lighting and shadow of a virtual object is incompatible with the real scene, the designed system is visually not conducive to the user's immersion in the experimental interaction Chamilothori et al. (2019). It requires the system to have illumination consistency function.
System needs include basic equipment needs, experimental content needs, usability needs and saving needs. The basic equipment needs include the experimental devices, hardware and software systems, effective input and output systems. Experimental content needs require the system to have experimental procedures and experimental simulation phenomena conform to objective laws. Usability needs require the system to be complete, robust and easy to learn and use. An important purpose of virtual experiments is to reduce the cost. Since the virtual experiment system generally requires a computer, if it is based on HMD, additional costs are also required. In a virtual experiment environment with tangible interaction, the cost of tangible objects also needs to be taken into consideration. It requires the design of the system to balance saving cost with other needs.

Security and comfort needs
The second layer of MHN theory is security needs. We extend this layer to security and comfort needs, including safety needs, physical comfort needs and psychological comfort needs. A safe and psychological relaxed virtual environment prevents users from worrying about explosions, corrosion, flames, getting tired, etc., so they can focus more on experimental operations and principles. To meet safety needs and psychological comfort needs, how to design the presenting ways of (virtual/real) objects in MR system is what we need to consider. For physical comfort needs, studies show that compared to not wearing HMD, HMD will cause greater physical stress to the users Chihara and Seo (2018) and more likely to produce simulator disease Sharples et al. (2008), which is more obvious in the VR environment. In addition, in the virtual experiments, the user's perspective should focus on the tabletop, and there is no need to turn the head to see the surrounding environment. Therefore, our system does not use HMD.

Belongingness needs
The third layer of MHN theory is the belongingness needs. The core of this needs is to reduce the loneliness of users. In the real experiment classes, the social interaction between teachers and students is an important factor that increases the emotional belongingness of students. In addition, students are accustomed to listening to the teacher's explanation of the principles during the experiment, and they accept this consistent audio-visual learning method. Some studies Hacker et al. (2009), Machidon et al. (2018 show that virtual humans can make the user interface rich in expressiveness of emotional interaction, and bring users more sense of participation and enjoyable experience. This paper uses technology of virtual human to create a virtual teacher to meet this needs. The design allows students to reduce loneliness and enter the state of learning and experiment faster.

Esteem needs and self-actualization needs
The high-layer needs (layers 4 and 5) of MHN theory are esteem needs and self-actualization needs. Among them, esteem needs are divided into inner esteem needs and external esteem needs. Self-actualization needs are divided into interest needs and encouragement needs. The purpose of the two layers' needs is to make users believe in their own power and let users feel the progress they have made step by step through hard learning, so as to achieve the effect of motivation. Therefore, we need to take certain incentive measures in the virtual experiment to stimulate more learning motivation of students. We believe that more positive feedback (praise voice, applause and other encouraging behaviors) and less negative feedback (criticism or blame voice) of teacher helps students to establish their esteem and selfactualization. In this paper, the virtual teacher's voice and behavior combine interactive method to meet the two needs.

System design
The environment of MagicChem is shown in Fig. 2a, including three cameras, a smell generator and a temperature generator. Users can freely interact with virtual and real devices. We use one RGB-D camera as the main camera and two RGB cameras as the auxiliary cameras. The main camera is placed in front of the human eye, parallel to the direction of the human eye toward the desktop to simulate the eyes' angle. We take an interactive scene shown in Fig. 2b as an example to introduce MagicChem. One hand of the user holds the tangible alcohol lamp, and the other hand uses the virtual match to light the alcohol lamp.
We design four modules (user interface module, tracking module, rendering module, interaction module) for Magic-Chem to meet the needs model. The relationship between the modules is shown in Fig. 3.

User interface module
This module is used to design the experimental environment of MagicChem, including the presentation form of the user interface, selection of tangible and virtual objects. In order to meet the visual experience needs and physical comfort needs, the user interface we designed is MR environment without HMD. In order to meet the security and comfort needs, the tactile experience needs and the usability needs, we design the tangible and virtual form for experimental devices. For devices that are cheap and suitable in size and weight, as well as easy to attach markers, we directly use real devices and put markers on them for tracking. For experimental devices that are expensive, big or heavy, we substitute these devices with some similar shape and lightweight materials of tangible substitutes (meeting tactile experience needs, saving needs), in order to reduce the physical pressure (meeting physical comfort needs). For devices that are inconvenient to attach markers, we use the form of virtual objects. For dangerous or polluting objects such as flames, corrosive liquids, toxic gases, we use the form of virtual objects (meeting safety needs).

Tracking module
The design of this module is to achieve robust tracking, registration, scene segmentation and illumination estimation to meet the security and comfort needs and paves the way for the functions of rendering module and interaction module. In the traditional single-camera MR environment, the marker needs to be kept facing to the camera, when the angle or distance between the marker and the camera is too large, 3D registration will fail, as shown in Fig. 4a. So that the user needs to focus on controlling the orientation of the marker, thereby reducing the comfort. Therefore, we make multiple cameras cooperate with each other to solve this problem robustly. In this paper, we have one main camera and two auxiliary cameras to cover the user's interactive space in multiple directions. In the offline stage, we calibrated the multiple cameras Zhang (2000). In the online phase, multiple cameras track markers' 6-DoF pose Lepetit et al. (2009) in real time. Finally, the pose parameters are converted to the main camera coordinate system. Figure 4b shows even if the marker on the alcohol lamp is not facing to the main camera, the virtual flame is correctly registered on the tangible alcohol lamp when user uses a virtual match to ignite the tangible alcohol lamp.
The module is also used to segment multiple foreground objects to provide the basis for virtual-real occlusion of the rendering module. We create a dataset of the foreground objects used in the experimental environment, and introduce the Unet network Long et al. (2015) with convolutional block attention module Woo et al. (2018) to conduct the training and the real-time pixels segmentation.
This module is also used for the detection of light sources in real scenes, as a basis for the consistency of illumination of the rendering module. Since this study is in the indoor experimental environment with controlled illumination, we estimate the parameters of multiple light sources in the offline stage. We use a general source model Zhou and Kambhamettu (2008) to estimate multiple illuminants of different types. We use a plastic sphere to probe both diffuse and specular reflections, using specular reflections to estimate geometric parameters and using diffuse reflections to estimate photometric parameters of the illuminants.
The module also tracks the poses of the hand joints and registers them to the virtual scene, providing the basis for the gesture interaction with touching of the interaction module. We tried the popular gesture tracking device Leap Motion, but the robustness was not ideal due to occlusion and confusion with tangible devices. Therefore, we design a quick and robust way to implement gesture interaction in this system, attaching red and blue markers (these colors seldom appear in tangible objects) to the most important thumb and index finger of the hand for tracking, and reconstructing joints into virtual scenes for collision detection. Figure 4c is the MR scene in which the left hand takes the virtual gas cylinder and the right hand takes the virtual dropper. Figure 4d is a virtual scene corresponding to Fig. 4c, where four different colored spheres represent markers on four fingers.

Rendering module
The design of this module is to meet the visual experience needs. This module is mainly used for rendering the virtual and real fusion of MagicChem, including the realistic rendering of the virtual-real occlusion (spatial presence needs) and illumination (presence needs of virtual objects).
The traditional virtual-real occlusion method is the original depth data of real scene to compare with depth data of virtual scene Yang et al. (2019). Generally, there are be many errors due to low-quality depth data. We correct the depth map of the real scene through the accurate pixel area of each type object segmented by the tracking module. Then, we complete the depth test by sampling the depth map of virtual scene and the corrected depth map of real scene. Finally, opaque objects and semi-transparent objects are rendered according to light sources' parameters calculated by the tracking module. Figure 5a shows an interactive scene without virtual-real occlusion where the right hand holding the virtual dropper, and Fig. 5b is the MR image rendered by this module. It can be seen that our method can handle multilayer virtual-real occlusion. Figure 5c is an interactive scene without illumination consistency where the left hand holding the virtual stalactite, and Fig. 5d is the MR image rendered by this module. It can be seen that our method can make MR experiments more realistic.

Interaction module
Multi-modal interaction is more similar to the multi-sensory experience that humans encounter in the physical world Dumas et al. (2009). This module is designed to meet the sensory needs, belongingness needs, esteem needs and selfactualization needs.

Visual interaction
Visual interaction is used to meet the user's visual experience needs in the experiments. This interactive mode mainly relies on the visual effects of MR to bring users a good visual experience. It depends on the consistency of virtual-real occlusion and illumination provided by the rendering module. Moreover, we simulate experimental phenomena and experimental details to bring users a realistic visual experience. Figure 6a shows the soft deformation of the dropper when the user squeezes the dropper and the ripples caused by the liquid dripping into the water surface.

Gesture interaction
For the interaction with virtual devices, we use gesture interaction with touching to meet the user's action experience needs rather than specific actions that do not match the real 1 3 Fig. 5 The rendering module. a No virtual-real occlusion. b Our virtual-real mutual occlusion method. c No illumination consistency. d Our illumination consistency method Fig. 6 Multi-modal interaction in MagicChem. a Gesture interaction by touching. b Tangible interaction. c Voice interaction and virtual-human interaction. d Action interaction of virtual teacher. e Temperature Interaction. f Olfactory Interaction experimental operation. Since it is easy to judge the positional relationship between the finger and the virtual object, the MR environment with virtual-real occlusion provides the possibility for gestures interaction with touching. For gesture interaction, we use the finger joints registered by the tracking module to detect collisions with virtual devices. In order to prevent accidental pick-up caused by accidental touch, the paradigm of gesture interaction is: when two finger joints of a hand are in contact with two sides of the virtual device at the same time, it is regarded as picking up the virtual device. If the distance between the two finger joints is too large to meet the picking conditions, the virtual device will fall onto the desktop. As shown in Fig. 6a, the real hand interacts with the virtual dropper like real experiment.

Tangible interaction
Tangible interaction is used to meet the user's tactile experience needs. Based on tracking module, we render virtual objects like drugs, reagents, flame, etc., on tangible devices robustly. Moreover, we call some handmade or 3D printed objects that are similar in shape to the target devices as tangible substitutes. For these tangible substitutes, we render them as target devices as needed in MR images. As shown in Fig. 6b, the left hand holds the tangible alcohol lamp emitting a virtual flame, and the right hand holds the tangible temperature probe for temperature measurement. The highprecision thermometer (rendered by tangible substitute) is placed on the desktop.

Voice interaction
Voice interaction is used to meet the user's listening and speaking experience needs.
In terms of listening, we use virtual teachers to provide users with guidance on experimental process and principles. Figure 6c shows the virtual teacher explaining the principle of the reaction of sodium and water. It's worth mentioning that, instead of a machine-generated voice, the voice of the virtual teacher is from a chemistry major student, which is more friendly, persuasive and more enjoyable Stern et al. (2006), Cabral et al. (2017).
In terms of speaking, we calibrate the desktop by placing a marker on the desktop and used it as a "devices warehouse" to generate and destroy virtual devices. The virtual devices are generated on the "devices warehouse" through the user's voice instructions. Moreover, if the user's voice is monitored and recognized throughout the entire experiment, the real-time performance and recognition rate of the system will be reduced. We design a virtual microphone (Fig. 6c) as an interaction constraint to overcome this problem. The user needs to use the thumb and index finger of one hand to touch the virtual microphone for starting to monitor user's voice. When two fingers leave the microphone, the monitoring is ended.

Virtual-human interaction
In order to meet the needs of belongingness, esteem and self-actualization, we set up a virtual teacher to provide users with virtual human's behavior and voice interaction. The virtual teacher is an optional auxiliary virtual person according to the user's preferences, which can be called out through the user's specific voice to assist the user in experiments. When the user completes an experiment step correctly, the virtual teacher will give encouraging behavior and voice feedback. On the contrary, the virtual teacher will give actions of shaking head or crossing hands, and then re-guiding this step. As shown in Fig. 6d, virtual teacher gives behavior feedback for the user's operation.
Compared with the common monologue teaching, the virtual role dialogue teaching method can enhance the positive emotional factors of the learners Gratch et al. (2016). In addition to the voice interaction mentioned in Sect. 4.4.4, we design a test section with voice interaction to simulate the questions-answers of teachers and students in real classrooms.
The interactive mode of virtual teacher not only allows users to enter the state of learning and experiment faster, reducing their loneliness (meeting the belongingness needs), but also allows users to experience the process of completing the experiment step by step (meeting the self-actualization needs). Even if users make mistakes, they will be encouraged instead of criticized (meeting the esteem needs) to improve users' learning motivation.

Temperature interaction
We made a temperature generator to meet the user's temperature experience needs in the experiments. The temperature generator uses a PTC heating body as heat source, and uses a PT100 thermistor sensor to measure temperature, and combines a PID control algorithm to achieve temperature control. For example, when the tangible alcohol lamp in Fig. 6b is ignited to emit a virtual flame, the temperature generating device will emit hot gas to simulate heat release (Fig. 6e).

Olfactory interaction
We made a smell generator to meet the user's olfactory experience needs in the experimental environment. By high-frequency resonance of ceramic atomizer, the high-frequency oscillation of the electron is used to break the liquid water molecular structure to produce a natural and elegant water mist. For example, when sodium reacts with water to generate hydrogen in Fig. 6c, the olfactory device will spray water vapor into the air to simulate the smell (Fig. 6f).

Details of system
In this paper, we use Intel Realsense D435i depth camera as the main camera and one Logitech C270i camera and one Gucee HD92 camera as auxiliary cameras. The tracking module and the interaction module are mainly developed through the C ♯ language on Unity3d. The rendering module is developed through cg/hlsl shader on Unity3d. The FPS of MagicChem is stable at about 25 frames/s.

The design of user study
In order to study the experience and learn effect of Magic-Chem, we design three ablation groups of independent variables as shown in Table 1. Among them, only the parts listed in E1, E2, E3 of Table 1 are different from E4(MagicChem). For user experience and learning effects compared with E4(MagicChem), the function of E1 is to test the impact of rendering module and the function of E2 is to test the impact of interactive modules. E3 uses Leap Motion for gesture tracking. We apply our needs model to the E3 to compare with E4 to explore whether a first-person perspective, virtual-real fusion and tangible interaction have an impact on user experience and learning effects. We do not choose real experiments as the ablation group because there may be dangers in the experiment process and difficult to control. Table 2 illustrates three tasks that each user needs performing. T1 aims to make students learn the operating steps of the experiment "Reaction of sodium and water" and analyze the principle based on the reaction phenomenon (sodium floating on the water, sodium melting into metal balls, sodium swimming around, making a "hissing" sound, and the reaction product turns the color of solution to red). The specific operation steps of T1 include taking a piece of sodium from the bottle, taking a sink filled with water, putting sodium in the sink, and using a dropper to absorb the phenolphthalein reagent from the bottle and drop it into the water tank after the reaction is complete. T2 aims to make students learn to use temperature measuring device and explore the temperature difference between the inner and outer flames. The specific operation steps of T2 include igniting a match with a matchbox, igniting an alcohol lamp with a match, and detecting the temperature of the inner flame and outer flame of the alcohol lamp with a temperature probe. T3 aims to make students learn the operating steps of the experiment "Corrosion of stalactites", observe the reaction phenomenon of stalactites with different types and different concentrations of acid, and then explore the principles of the phenomenon. The specific operation steps of T3 include putting hydrochloric acid/sulfuric acid/nitric acid in the sink, taking a stalactite from the stalactite samples and putting it into the sink, adjusting the concentration of each type's acid.
Each participant takes 3 minutes to practice the interactive method of his group and then complete T1, T2, T3 in order. After each user completes all tasks, we use SUS Brooke (1996) to evaluate the degree of satisfaction on system needs, and a subjective questionnaire (Table 3) to evaluate the user experience. Table 3 includes questions of Nasa-TLX(Q1-Q6) Hart (2006) and some other questions (Q7-Q10) based on our needs model. The questionnaire uses a five-point Likert scale (1 to 5, where 1 means lowest and 5 means highest). After completing the questionnaire, users need to make further oral comments on the questionnaires. Finally, we ask users to describe the experimental steps, experimental phenomena and principle of T1 as detailed as possible within 2 minutes to evaluate the learning effect of   Physical pressure(comfort needs) Q3 Time pressure (comfort needs) Q4 Own Performance(needs of esteem and self-actualization) Q5 Effort (comfort needs) Q6 Frustration(needs of esteem and self-actualization) Q7 Visual fidelity (visual experience needs) Q8 Natural operation (sensory needs) Q9 Loneliness (belongingness needs) Q10 Interest (interest needs) the experiment. The purpose of this arrangement is to get a long interval between the operation of T1 and the description of T1, so as to better evaluate the effects of learning and memory. On average, the entire user study takes about 45 minutes per participant.

Experiments preparation
We recruit 56 students (30 males and 26 females) as users of the experiments. These students from the first grade of senior middle school in a local middle school, and they have not studied the experiments of T1. Students are between the ages of 14-16. All the 56 students have experience in doing real chemical experiments. We divide them into 4 groups (E1-E4) equally according to gender, academic performance, etc. Each group has 14 students. 5 of them have VR interface experiences from science and technology museum and VR experience museum, and none of them have experience of MR user interfaces. We briefly introduce the concept of VR/MR, and explain in detail how to use the system of the corresponding group.

Balance of experiments
We assign a staff member to supervise each experimental environment. All computers in environments are unified. In E2, since there is no guidance and principle explanation of virtual teacher, we use the virtual text-UI method instead. Moreover, E2 has no voice interaction, we set up a 3D-Menu. The users can click the target button on the 3D-Menu to generate virtual device through gesture interaction. Moreover, we ask users to concentrate as much as possible.
6 Results and discussion

Analysis of system usability
The SUS score reflects the overall availability of each environment. Figure 7a shows the average SUS score of each experimental environment, and the error bar represents the standard deviation. We summarize the ten questions of the usability scale into three indicators, which are functional indicators (the integration of functions, the complexity of the system, the inconsistent of interactive behavior and results, and the cumbersome operation), mastery indicators (easy to learn, easy to use, requiring professional help, and requiring training) and application indicators (use in future and use with confidence). Figure 7b shows the score of three indicators (each question is the converted score Brooke (1996) of 0-4). Table 4 shows the results of post hoc test for MagicChem and each ablation environment. Among them, E4&E1 represent two sets of users' data of E4 and E1 for post hoc test, and so on. S represents the significance. Y represents yes, and N represents no. ↑(E4&Ei) represents that the average value of E4 is greater than the Ei(1⩽ i ⩽ 3), and ↓ is the opposite. The result shows that the total SUS score of E4 is  significantly higher than E1 and E2. The difference in the total SUS score of E4 and E3 is not significant. E4 is significant higher than E1 in terms of all three types of indicators, higher than E2 in function indicators and application indicators, but lower than E2 in mastery indicators, and higher than E3 in application indicators. Through users' feedback and our analysis, we obtain the reasons for the significant difference. In E1, where there is no consistency of virtual-real occlusion relationship and the lighting and shadows, virtual objects are "floating" on the real scene to occlude the real scene. This visual confusion and falsehood greatly affects user interaction, especially gesture interaction with touching. Since the position of virtual objects can only be judged by the position of tangible objects in real scene, when there are few tangible devices in the scene, it is hard for user to interact with virtual devices. For example, when a user needs to pick up the virtual dropper, he/she is difficult to use his thumb and forefinger to touch the surface of the dropper accurately. It results in lower scores in the three types of SUS indicators for E1.
E2 has only one interactive method of gestures with touching, and the way to generate virtual devices is to click target button on the 3D-menu by gestures with touching. If the finger accidentally touches other buttons, unexpected virtual device will be generated and the user needs to find and select target button again. E4 generates virtual devices by voice interaction. The difference causes the score of E2's functional indicators to be lower than that of E4. In addition, E4 has multiple interaction methods, and E2 only has single interaction method, which results in E2's application indicators being lower than E4. However, in terms of mastery indicators, E4 causes users to spend more effort to learn multi-modal interaction than E2. Finally, we find that E4 has higher application indicators than E3, which proves that users are more willing to use MagicChem.

Analysis of user experience
To evaluate the quantitative data collected from these questionnaires, we use ANOVA with post hoc test to analyze the significant differences between groups. Figure 8 shows the mean score of questionnaire and error bars which indicate standard errors of the means. Table 5 provides detailed values in post hoc comparison test.
The results show that E4 is significantly lower than E1 in psychological pressure, physical pressure, time pressure, effort and frustration, and is significantly higher than E1 in visual fidelity, natural operation and interest. The reason is that E1 lacks the realistic 3D feeling visually. Due to the lack of occlusion relationship, the user may need to go through many attempts to correctly take a virtual object, which requires more psychological pressure, physical pressure, effort, time pressure and frustration. The feedback of E1 is generally poor, such as "The virtual object floating on the real, I cannot judge the position of the virtual object, I can only guess and try," "Very bad experience, it makes my sense of position confusing." In E4, there is no negative feedback about spatial relationship and interactions. It proves that our rendering module satisfies the need for the presence of virtual objects. It also proves that the visual experience needs in sensory needs are very important to the experience.  E4 is significantly lower than E2 in physical pressure, own performance, effort, and loneliness, and is higher than E2 in natural operation and interest. Compared to E4, E2 lacks interaction in touch, temperature, smell, hearing and speaking, which is not as natural as E4. In addition, we find that the reason for the greater physical pressure and effort in E2 is that the way E2 generates virtual device needs more effort to find and click the target button, especially by accidentally touching. Due to the use of silent text-UI guidance, E2 lacks encouraging voice and action interaction from virtual teachers. The virtual environment is more "cold," making students feel lonely (not meeting belongingness needs) and less progress (not meeting esteem and self-actualization needs) than E4. Moreover, although all the experimental devices in E2 are virtual, it does not lead to a significant difference in visual fidelity between the E4 and E2.
E4 is significantly lower than E3 in terms of physical pressure. For visual realism, natural operation and interest, E4 is significantly higher than E3. Completely virtual environment is the reason that E3 is lower than E4 in terms of visual realism. In terms of interaction, more operations are tangible in E4 than in E3. The tactile sense of taking objects is important to the user's sense of interaction, which is the reason that interaction of E3 is not as natural as that of E4. In addition, the reason why the physical pressure of E3 is greater than that of E4 is that wearing HMD causes a certain amount of pressure on users. Many users in the E3 group give the feedbacks such as "HMD is heavy," "It's not comfortable wearing HMD." According to communicate with users, the users in E4 group do not feel that MagicChem's perspective is unnatural during the experiment. It also proves compared with the first-person VR environment, the design of E4 can better meet the physical comfort needs of users.

Evaluation of learning effect
We design a scoring standard with a total of 15 points, with the correctness of 5 steps (placing the sink, using a medicine spoon to dig out the sodium in the jar, putting the sodium into the sink, putting the dropper in the jar after reaction, absorbing and dropping phenolphthalein), with the correctness of 5 experimental phenomena (sodium floating on the water, sodium melting into metal balls, sodium swimming around, making a "hissing" sound, and the reaction product turns the color of solution to red), with the correctness of 5 experimental principles corresponding to phenomena (the density of sodium is less than water, the melting point of sodium is low and the reaction will exothermic, the generated hydrogen pushes sodium to move, the gas rushes out of the liquid surface to make the sound, reaction generates NaOH). Record 1 point for each item. Figure 9 shows the mean score of learning effect and error bars indicating standard errors of the means.
The results of ANOVA and post hoc tests show that the learning effect of E4 is significantly better than E1 (p < 0.001) and E2 (p < 0.001), but there is no significant difference with E3 (p = 0.445). Through analysis, we believe that due to the lack of consistency of virtual-real occlusion and illumination in E1, the visual unreality and poor interaction experience affect the user's focus of experimental phenomena and the principle learning. However, due to the lack of a virtual multi-modal interactive environment, E2 only presents knowledge through text guidance in visual rather than multi-sensory, it is not as impressive as E4. Although E3 is inferior to E4 in some experience indicators, E3 and E4 have no significant difference in the learning effect. It shows that when the virtual experiment environment meets the needs model we proposed, the learning effect of the experiments is affected by the way of presentation and perspective little.

Discussions and future work
The results of user study show that, in terms of usability, experience, and learning effects, MagicChem (E4) is superior to the two traditional MR environments (E1 and E2) that partially meet needs models. E1 received poor user reviews, which shows that students are very concerned about the visual experience in interactive MR scene. So in the design of the MR experimental system, if the presence needs of space and virtual objects can be met, the system' practicability, the user experience and user's learning effect can be greatly improved. Moreover, the more the user's sensory experience needs are met, the better the experience and learning effect for students. In terms of application indicators, physical pressure, visual fidelity, interactive naturalness and interest, MagicChem is superior to the VR experiment environment (E3) that meets the needs models. However, there is no significant difference in the learning effect between MagicChem (E4) and the VR experiment environment (E3). It shows that when the virtual experiment environment meets the Fig. 9 The mean score of learning effect needs model, the learning effect of the experiments will be hardly affected by the way of presentation and perspective. In addition, MagicChem has received good feedback and evaluation from students, which is also reflected in the average score of the scales in terms of system usability, experience, and learning effect. The results of these test indicators also achieve our purpose of designing a human-oriented virtual experiment needs model: to guide the design of a virtual experiment system that is close to real experiments and have multi-sensory, comfortable and stimulating experience.
Based on the proposed needs model and the results of user studies, we propose the following design guidelines of the virtual experimental system: The bottom-level needs (basic needs, security and comfort needs) of the needs model proposed in the paper are the "material" basis to ensure the stable operation of the experimental system and the good experience of users. First, the needs require the design of a virtual experiment system to be as practical as possible, including good system usability and complete experiment simulation, which can solve some shortcomings of real experiments (danger, pollution, chemical consumption, etc.). Secondly, the system is required to have as many multi-sensory simulations as possible, including vision, tactile, listening/speaking, temperature, olfactory, etc. that exist in real experiments. Among them, the visual needs are the most important, which greatly affects the users' sense of reality experience. In particular, for the MR environment, visual needs depend on the consistency of illumination and the consistency of occlusion in virtual-real spaces. After satisfying the bottom-level needs, the high-level needs (belongingness needs, esteem needs, and self-actualization needs) are the "spiritual" assistance that improves the pleasure, satisfaction, and learning effect of users who use the experimental system. The needs require the design of the virtual experiment system to consider the social interaction of teachers and students, how to motivate students to learn, etc. In this way, the human-computer interaction environment will become "warm" and students will like to use the system to do experiments.
Although the paper proposes a general needs model for virtual experiments and develops a corresponding system, only a few typical chemical experiment examples have been researched. In the future, we will develop more experiments in different fields (physics, biology, etc.) based on this system framework, and conduct research to improve this needs model and explore how to make experiments in more fields benefit from the needs model. In addition, we will also explore multi-users collaborative virtual experiments (student-student social needs) to expand the belongingness needs of the needs model.

Conclusion
Combining the human needs theory and the special characteristics of virtual experiments, the paper proposes a human-oriented needs model for virtual experiment. In order to verify the effect of the needs model on students, we design an MR experiment system MagicChem based on this needs model and conduct user study. With different aspects of user studies, we find that MagicChem is better than the virtual experimental environment of unsatisfied or incompletely satisfying the needs models. This shows that the needs model we proposed can effectively improve experience and learning effects of the student users.


Defending Label Inference and Backdoor Attacks in Vertical Federated Learning

Since there are multiple parties in collaborative learning settings like federated learning, curious parities might be honest but are attempted to infer other parties’ private data through inference attacks while malicious parties might manipulate the learning process for their own purposes through backdoor attacks. However, most existing works only consider the federated learning scenario where data are partitioned by samples (HFL). The feature-partitioned federated learning (VFL) can be another important scenario in many real-world applications. Attacks and defenses in such scenarios are especially challenging when the attackers and the defenders are not able to access the features or model parameters of other participants. Previous work have only shown that private labels can be reconstructed from communication of per-sample gradients. In this paper, we first show that private labels can be reconstructed even when only batch-averaged gradients instead of sample-level gradients are revealed. It is a common presumption that batch-averaged information is safe to share, therefore batch label inference attack is a severe challenge to VFL. In addition, we show that a passive party (a party without labels) in VFL can even replace its corresponding labels in the active party with a target label through a gradientreplacement attack. To defend against batch label inference attack, we introduce a novel technique termed confusional autoencoder (CoAE), based on autoencoder and entropy regularization. We demonstrate that label inference attacks can be successfully blocked by this technique without hurting as much main task accuracy as compared to existing methods. Our CoAE technique is also effective in defending the gradient-replacement backdoor attack, making it an universal and practical defense strategy with no change to the original VFL protocol. We demonstrate the effectiveness of our approaches under both two-party and multi-party VFL settings. To the best of our knowledge, this is the first systematic study to deal with label inference and backdoor attacks in the feature-partitioned federated learning framework.

Introduction
Federated Learning (FL) (McMahan et al. 2016;Yang et al. 2019;Kairouz et al. 2019) is a collaborative learning framework for training deep learning models with data privacy protection. In the original proposal of cross-device FL (McMahan et al. 2016), data samples are distributed among different participants, thus these works can be regarded as sample-partitioned federated learning, or horizontal federated learning (HFL) . However, featurepartitioned federated learning, or vertical federated learning (VFL) Hu et al. 2019b;Liu et al. 2019;Liu, Chen, and Yang 2018;Cheng et al. 2021;He, Annavaram, and Avestimehr 2020) is another important scenario for many real-world applications. For example, when a bank and an E-commerce company collaboratively train a credit risk model, different features of the same group of users are partitioned among different parties. It is important to safeguard the VFL framework from attacks and data leakages.
Because FL is a collaboration system that requires parties to exchange gradient or model level information, it has been of great research interest to study the potential information leakage from gradients. Previous work (Zhu, Liu, and Han 2019;Zhao, Mopuri, and Bilen 2020;Yin et al. 2021) have shown that it is possible to recover pixel-level (deep leakage) raw data from transmitted gradients information. Previous studies on label leakage of VFL (Li et al. 2021) have made the assumption that per-sample gradient is communicated and revealed in VFL, therefore the value of the gradients can be used to induce the label. However, sample-level information is not necessarily available in VFL with Homomorphic Encryption (HE) or Multi-Party Secure Computation (MPC) protection , where the encrypted intermediate results are communicated, and both data and local model parameters are kept secret. Therefore attacks and defenses in this scenario are both challenging since only batch-leveled local gradients are available. In addition, FL opens doors to new backdoor attacks. A typical example is backdoor attacks with model poisoning. Many recent works (Bagdasaryan et al. 2018;Bhagoji et al. 2019) have shown that FL is vulnerable to model poisoning where the attacker manipulates the model's performance on an attacker-chosen backdoor task while maintaining the performance of the main task. These attacks only apply to HFL where model parameters are exposed.
In this paper, we first demonstrate that private labels can be reconstructed when only local batch-averaged gradients are revealed, which is against the common presumption that batch-averaged information is safe to share in VFL. Furthermore, we show that a passive party (a party without labels) in VFL can replace its corresponding labels in the active party with a target label through performing a gradientreplacement attack. Experiments on multiple datasets show that accuracy on both tasks can achieve over 90%. Finally, we introduce a novel technique termed confusional autoencoder (CoAE), based on autoencoder and entropy regularization to defend against label inference attack. We demonstrate that our technique can successfully block both label inference attack and gradient-replacement attack without hurting as much main task accuracy as existing methods like differential privacy (DP) and gradient sparsification(GS), making it an universal and practical defense strategy without changing the original VFL protocol.

Related Work
Recently, deep neural networks have been adopted to learn sensitive information from the exposed gradients in FL (Zhu, Liu, and Han 2019;Zhao, Mopuri, and Bilen 2020;Yin et al. 2021). Differential Privacy (DP) (Bagdasaryan et al. 2018;Xie et al. 2020) and gradient sparsification (GS) (Lin et al. 2017) are techniques commonly adopted to preserve privacy in these scenarios. A data augmentation approach (Gao et al. 2020) is also recently proposed to defend the gradientbased information reconstruction attacks. Bagdasaryan et al. (2018) introduced a backdoor attack to federated learning by replacing the global model with a targeted poisoning model and discussed effectiveness of possible defense strategies. Bhagoji et al. (2019) carried out stealthy model poisoning attack to federated learning by alternatively optimizing for stealth and the adversarial objectives. Xie et al. (2020) introduced distributed backdoor attacks to federated learning. Sun et al. (2019) studied backdoor and defense strategies in federated learning and show that norm clipping and "weak" differential privacy mitigate the attacks. All the works above consider the HFL scenario.
VFL frameworks have been developed for models including trees (Cheng et al. 2021), linear and logistic regression (Gratton et al. 2018;Kikuchi et al. 2017;Hu et al. 2019a), and neural networks (Liu, Chen, and Yang 2018;Hu et al. 2019b). Privacy-preserving techniques such as Homomorphic Encryption (HE) (Rivest, Adleman, and Dertouzos 1978;Acar et al. 2018), Secure Multi-party Computation (SMPC) (Yao 1982) and Differential Privacy (DP) (Dwork 2006) are typically applied in these frameworks to preserve user privacy and data confidentiality. Recently, label leakage and protection for the vertical federated learning (VFL) framework is studied in (Li et al. 2021). This study is limited to two-class scenario and plain-text communication so per-sample gradient information is available. In our work, we attack and defense a more challenging system, where we assume the communicated messages are also protected (e.g. by HE), and only the final local model gradients are accessible to attackers. A Privacy-Preserving Feature-partitioned collaborative learning framework In a typical VFL system , K data owners collaboratively train a machine learning model based on a set of data {x i , y i } N i=1 and only one party has labels. This is a reasonable assumption in cross-organizational collaborative learning scenarios, because in reality labels (such as users' credit scores, patients' diagnosis etc) are expensive to obtain and only exist in one or few of the organizations. Suppose that the feature vector x i can be further decomposed into K blocks {x k i } K k=1 , where each block belongs to one owner. Without loss of generality, assume that the labels are located in party K. Then the collaborative training problem can be formulated as where θ k denote the training parameters of the kth party; Θ = [θ 1 ; . . . ; θ K ]; N denote the total number of training samples; f (·) and γ(·) denote the prediction function and regularizer and λ is the hyperparameter; Following previous work, we assume each party adopts a sub-model G k which generates local predictions, i. e, local latent representations H k i and the final prediction is made by merging H k with an nonlinear operation, such as softmax function. That is, where G k can adopt a wide range of models such as linear and logistic regression, support vector machines, neural networks etc. Let H i = K k=1 H k i , then the gradient function has the form We refer the party having the labels the active party, and the rest passive parties. In a feature-partitioned collaborative learning protocol, each passive party sends {H k i } to active party, and the active party calculates { ∂ ∂Hi } and sends it back to passive parties for gradient update. To further protect information leakage from intermediate results, privacy-preserving techniques such as HE, denoted as [[·]], are applied to protect sample-level information {H k i }, and the gradients calculations are performed under encryption. The computed encrypted gradients are then sent to a Trusted Third Party (TTP) for decryption. This process ensures that only batch-averaged gradients rather than sample-level gradients are available to the passive party. See Algorithm 1 and Figure 1(A).

Attacks Threat Model
Based on Algorithm 1, we consider the following attack model: (i) the attacker has access to the training data of one or more data parties and it controls the training procedure and the local model of attacked parties. (ii) The attacker only receives batched-averaged local gradients in plain text from TTP, but does not receive encrypted per-sample intermediate results. (iii) The attacker can modify local updates such as training weight and gradients before sending transmitted data to other parties. (iii) The attacker does not control any benign party's training nor does the attacker control the global communication and aggregation protocols. (iv) Contrary to traditional data poisoning attacks which focus on poisoning training data, the attacker in collaborative learning focus on learning sensitive information from other parties or poisoning gradients and model updates that are communicated among parties in the protocol, similar to previous works (Steinhardt, Koh, and Liang 2017). The fundamental difference between our threat model and the model adopted in previous HFL works is the following. First, for all kinds of attacks, under previous HFL settings, the attacker has access to the entire feature and label space information and the entire set of model parameters, while in our scenario, the attacker only has access to a portion of the feature space and the model parameters of parts that it controls, and it may not have access to labels if it does not control the active party; Secondly, for backdoor attacks, under HFL settings, the attacker controls only a portion of the entire dataset and any model updates it sends get averaged in the server. While under our VFL setting, the backdoor will survive and propagate throughout the communication protocols. Due to these differences, the attacks and defense strategies for VFL are fundamentally different from those for HFL. The differences between our model and the VFL framework considered in (Li et al. 2021) is that our threat model further assumes that the communicated messages between parties are private and encrypted, so that per-sample gradient-related information is not available for inference, therefore our attack is on the batch or population level.

Adversarial Objectives
The attacker in our setting aims to train a model which achieves high performance on both the original task and the attacking task. Label inference attack. The label inference task is aimed to infer the information about labels at a passive party (the party that does not obtain the label information in VFL setting). Backdoor attack. Unlike Byzantine attacks (Muñoz-González, Co, and Lupu 2019), preventing convergence is not the attacker's goal under our setting. The backdoor task is to assign an attacker-chosen label to input data with a specific pattern (i. e. , a trigger), as shown in Figure 2. We will elaborate our discussion on both attacks and how these attacks are inter-connected in the following parts.

Label Inference Attack
Inspired by the attack performed by (Zhu, Liu, and Han 2019), we adopted a deep learning approach to accomplish this attack, see Figure 1(A, B) and Algorithm 2. Specifically, the passive party (attacker) p set up an internal model which tries to guess the labels and communicated intermediate results H a for every sample in a batch so that the simulated local gradients match the observed ones. First, for each sample in each batch B it randomly initializes y and H a , then computes its gradients using: Then the final objective is to match the observed and simulated gradients: We show experimentally that this reconstruction can be quite powerful, achieving an over 90% label reconstruction rate. Therefore, we conclude that batch averaging is not sufficient to protect the label information in VFL setting. In the next section, we propose a novel defense technique to protect labels.

Gradient-Replacement Backdoor
With the backdoor adversarial objectives in mind, it is easy to see from Figure 2 that the active party who has labels is able to replace the loss function of the original model with a new loss function to inject backdoor attacks: where o K i is defined in equation 1, D cln ,N cln ,D poi ,N poi denote the clean dataset, number of samples in clean dataset, poisoned dataset and number of samples in poisoned dataset, respectively. r K i denotes the target label. Since the active party has labels, it can directly modify the label of the poisoned dataset without affecting the training protocol. During training, the active party just needs to send { ∂L b ∂Hi }instead of { ∂L ∂Hi } to successfully inject the attack. In this work, we focus on the more challenging problem, injecting backdoor from passive parties who have no access to either labels or other passive parties' contributions at every iteration. From algorithm 1, the only information the passive parties obtain is the batch-averaged gradients. Therefore the main challenge is for the passive parties to assign targeted labels to specific input data with trigger features without direct access to labels.
This backdoor task is built on the label inference attack since once a passive party learns the corresponding true labels of the each sample, it can carefully design the communicated messages to replace the true label with a target label, shown as the following. In equation 4, if f represents a softmax function and represents a cross-entropy loss, then g i H = ∂ ∂Hi is a m-dimension vector where m is the number of labels with the jth element being: where S j is the softmax function S j = e h j i e h i over H i . Here we abuse the notation y to denote the index of the true label. If g i H is revealed, the label information is known because the yth element of g i H will have opposite sign as compared to others. To inject the attack, the passive party only needs to replace g i H with Where τ is the target label. Note this injection can be performed under HE encryption without knowing the plain-text value of S j due to the additive property of HE.
, j = y Therefore if we can recover per-sample label information, which is exactly what batch label inference attack aims to achieve, we can further exploit the recovered labels to perform backdoor attacks. However, such a backdoor attack requires the label inference step as a prerequisite, which requires running and observing the VFL process separately first. In the following, we propose another backdoor approach which does not require prior knowledge about the true labels of every sample to be poisoned. Instead, we only assume that the passive attacker knows a few clean samples which have the same label as the targeted label of the backdoor task. We mark this sample set D target . We use this assumption because it's usually not too difficult to get the label of a few samples in practice. With this assumption, we propose to inject a gradientreplacement backdoor to the learning process directly as follows: • In the forward propagation, the attacker performs local computations of H k i , but for each poisoned sample i, it randomly selects one sample j from D ∂H k i ∂θ k according to equation 5 where ∂ ∂Hj is replaced by g j ; Here γ is an amplify rate that we adjust to control the level of backdoor. This can also be understood as an identity steal strategy where the backdoor sample steals a target's identity. By using such a strategy, the passive party will obtain the corresponding gradients with respect to the targeted label instead of its own therefore its local backdoor updates will be successful. To further prevent the active party from learning a reasonable mapping of the labels and intermediate results H i of the poisoned samples, the passive party can instead output a random-valued vector to the active party for each poisoned sample during training. See Algorithm 3 and Figure 1(A, C)

Defense
In the previous sections, we have shown that label leakage and backdoor attacks are both challenging problems to VFL. In this section, we discuss possible routes for protection. Especially, the active party needs to establish a stronger defense mechanism. The common defense mechanisms previously adopted include differential privacy and gradient sparsification. However these techniques usually suffer from accuracy loss, which might be unacceptable in certain scenarios. Another line of work is to exploit the distribution differences of per-sample gradient values (Li et al. 2021), but they are not always accessible. Here we propose a simple yet effective label disguise technique where the active party learns to transform the original labels to a set of "soft fake labels" so that 1) these soft fake labels contrast with the original ones, i.e. the fake labels and the original ones are likely not the same after the transformation; 2) the original labels can be reconstructed almost losslessly; 3) the fake labels should introduce as much confusion as possible to prevent the passive party to infer the true labels. For example, a simple mapping function (e.g., a function that assigns label "dog" to "cat" and "cat" to "dog") would satisfy the first two conditions but not the last one, since if a party knows the true label of one sample, it would know the true labels of all samples belonging to the same class. To increase confusion, we propose learning a confusional autoencoder (CoAE) to establish a mapping such that one label will be transformed into a soft label with higher probability for each alternative class. For example, a dog is mapped to [0.5,0.5] probability of dog and cat, respectively. Whereas autoencoder is simple and effective for hiding true labels, confusion is important in changing the probability distribution of classes, making backdoor attacks harder to succeed.

Algorithm 4 Label Disguise via training CoAE
Input: the learning rate η, number of classes c, batch size N , the maximum number of epochs E. Output: trained W e , W d . 1: for i ← 1 to E do 2: Randomly generate one-hot labels: y ∈ R N×c 3: Generate fake and reconstructed labels by eq. (9); 4: Compute total loss L by eq. (11) and update parameters: W e ← W e − η · ∂L/∂W e W d ← W d − η · ∂L/∂W d 5: end for 6: return W e and W d Algorithm 4 and Figure 1 (D) describe the training procedure. The encoder takes as input ground-truth labels y and outputs fake labelsỹ, while the decoder takes as input fake labelsỹ and reconstruct original true labelsŷ. That is: Where W e and W d are the parameters for encoder and decoder,respectively. To satisfy our conditions, we introduce a contrastive loss and an entropy loss, respectively: Where CE(·) is the cross-entropy loss. Then, we form the final learning objective function as: L = L contra − λ 2 L entropy (11) Here L contra is the contrastive loss that enables the CoAE to reconstruct true labels from fake ones while forcing the fake labels to be different from the original labels; L entropy is the entropy loss that maps each true label to multiple alternative labels ("confusion"); λ s , s ∈ {1, 2} are loss weights. After trained, the active party can leverage the CoAE to produce fake labels and use these fakes labels to compute gradients in VFL, thereby preventing label leakage and backdoor attacks (Algorithm 5). When performing inference, the active party transforms the predicted labels back to true labels using decoder.
Note that the assumption for such a defense mechanism to work is not different from that of the VFL attacks. That is, we assume the passive party does not know the model structure or parameters of the active party. The passive party may only know a few clean labels at most, which may be enough to inject backdoor attacks but not enough to learn a good mapping between the true labels and fake labels. By replacing true labels with soft fake labels locally, the active party does not need to make any changes to the protocol, preserving both label privacy and protocol integrity.

Experiments
In our experiments, we evaluate the effectiveness of our proposed attacks: batch label inference attack and gradientreplacement backdoor at the passive party, as well as our proposed CoAE defense at the active party in a VFL training protocol.

Models and Dataset
NUS-WIDE dataset In this dataset, each sample has 634 image features and 1000 text features. We partition the data into an active party with image features, and the passive party with text features. The backdoor trigger is that the last text feature equals 1. The data samples that satisfy this feature are less that 1% in both training set and testing set, thus it's difficult to detect this backdoor using a validation set. In the following experiments, we choose five labels: ['buildings', 'grass', 'animal', 'water', 'person'], except for the batch label inference attack, where all 81 labels are used to study the impact of number of classes. The target label is set to class 0 and ten target samples are randomly chosen. A 2layer MLP model is used for this dataset with 32 neurons in the middle layer. MNIST and CIFAR dataset For these two datasets, We evenly split each image into two halves and assign them to the active and passive party, respectively. Following (Gu, Dolan-Gavitt, and Garg 2017), we inject trigger 255 at pixel positions [25,27], [27,25], [26,26] and [27,27] for the MNIST dataset. We randomly select 600 samples from the 60000 training samples and 100 samples from the 10000 testing samples and mark them with this trigger. As for the CIFAR dataset, pixels [29,31] and [30,30] are set to [0,255,0] along the channels, and pixels [31,29] and [31,31] are set to [255,0,255] (see Figure 2). We randomly mark 100 samples from 10000 training data and 20 samples from 2000 test data with this trigger. (D backdoor ). The target label is randomly chosen. Then 10 target samples are randomly chosen from that class. A 2-layer MLP model is used for MNIST dataset with 32 neurons in the middle layer. Resnet18 is used as the backbone model for CIFAR dataset.

Attacks at Passive Party
Label Inference Attack To test the effectiveness of the label inference attack, we use recovery accuracy as our metric. If the predicted label matches the true label, the recovery is successful for that sample. The recovery accuracy is then the ratio of successfully reconstructed samples. The samples and the classes are randomly selected for each experiment and each run is repeated 30 times. The results are shown in Figure 3 with different batch size and different number of classes. It can be seen that the batch label inference attack can be quite successfully for various batch size and number of classes, despite the fact the only clue is the batchaveraged gradients. As the batch size increases, the recovery rate gradually drops, indicating the inference task is more difficult. Another interesting finding is that as the number of classes increases, the recovery rate increases at first, likely due to the fact that the diversity of the labels reduces the possible combinations in the search space. This is consistent with (Yin et al. 2021), where the assumption for successfully guessing the labels in a batch is that the number of classes exceeds the batch size. As the number of classes continues to grow (e.g. to 81 in NUS-WIDE), we also observe a drop in recovery rate, likely due to increasing difficulty for solving the optimization problem.

Gradient-Replacement Backdoor
The left column of Figure 4 shows the backdoor task accuracy at various levels of backdoor, indicated by the amplify rate γ. Some of which are more than 90%, meaning that the backdoor task has succeeded. The right column of Figure 4 shows the main task accuracy, which stays high for various attack levels. When γ is set too large the backdoor task succeeded while the main task failed. In the following experiments, we set γ to 10, at which the main task and the backdoor task both work well.

Defenses at Active Party
We evaluate our proposed label defense mechanism on both batch label inference attack and gradient replacement backdoor attack, and compared our methods with three baselines: Differential Privacy (DP), Gradient Sparsification (GS) and optiMized perturbAtion to pReVEnt Label Leakage (MARVELL) following previous work (Li et al. 2021). Both encoder and decoder of the CoAE have the architecture: fc((6c) 2 )-relu-fc(c)-softmax, where fc denotes fully- connected layer and c denotes the class number. We evaluate the performance of CoAE with various λ 2 from 0.05 to 2.0. For DP, gaussian differential private mechanism (DP-G) and Laplacian (DP-L) differential private mechanism are employed to defense backdoor attack. Gradients were 2norm clipped with 0.2. Then, a Gaussian or Laplacian noise was added. The standard deviation of the applied noise ranges from 0.001 to 0.1. The form of added noise are shown below: For GS, we evaluate the defense performance of gradient sparsification strategy with various drop rates, from 99.0 to 99.9.
For MARVELL, as it only supports binary classification and is designed to defend label inference attack, we evaluate its defense performance against our method, CoAE, separately using only the data from two classes ('clouds' and 'person') since these data form a relative balanced dataset. The power constraint hyperparameter various from 1 to 10 times the norm of gradients.
The defence experiment results will be reported separately on batch label inference attack and backdoor attack in the following paragraphs.
Batch Label Inference Attack The results of batch label inference defense of several strategies are shown in Figure  5. The left column of Figure 5 shows the results of defense under multi-class classification task (10, 5, 20 classes for MNIST, NUSWIDE and CIFAR separately), while the right column shows the results of binary classification task. In each figure, the upper left indicates high attack performance and low main task accuracy, so a better defense should be towards the lower right. Compared to other approaches, we see that our CoAE defense consistently achieves better trade-off performance, with high main task accuracy and low attack task accuracy for multi-class classification task, and relative the same main task accuracy and low attack accuracy for binary classification task. The above proves that CoAE is superior in defending batch label inference attack. Also, for this task, GS has a better performance than DP-based meth-ods. And a change in the power constraint hyperparameter for MARVELL do not lead to a significant variation in the result.
Gradient-Replacement Backdoor After demonstrating that our method CoAE can successfully defend the label inference attack, we want to show that our defense mechanism is still effective when faced with another kind of attack, namely gradient-replacement backdoor attack. We also evaluate our CoAE defense and several other baseline defending methods against gradient-replacement backdoor attack. The results are shown in Figure 6. This experiment is carried out under multi-class classification task setting (10, 5, 20 classes for MNIST, NUSWIDE and CIFAR separately). Same as the batch label inference attack, a better defense should be towards the lower right in each figure. It's clear to see from Figure 6 that results of gradient sparsification (GS) are in the upper left corner, indicating a high backdoor accuracy and low main task accuracy. This suggests that GS is inferior to defend our backdoor attack. Comparing to DP and GS mechanism, our CoAE method can defend backdoor attack to the same level, while retaining a high accuracy of the main task.
We further evaluate our defense strategy under distributed backdoor attack settings as the one proposed in previous work (Xie et al. 2020). We compare our method, CoAE, with three baseline methods on CIFAR100 dataset astill using only images from 20 classes. In this experiment, the feature of data is equally partitioned into four parties with only one active party owning the labels and three other passive parties without label information. Similar as previous work (Xie et al. 2020), the three passive parties work together and conduct gradient-replacement-backdoor attack to the active party. Each attacker has their one-pixel trigger at the lower right corner of each data sample of their own which together forming a three-pixel trigger. Notice that trigger is smaller than the four-pixel trigger we use in previous experiments mentioned above. The result is shown in Figure 6(d). It's clear that our method beats all the baseline methods, achieving a low backdoor task accuracy at a high main task accuracy. Also, comparing with Figure6(c), although distributed backdoor attack is much stronger than single-party backdoor attack (comparing the two black square in each plot which demonstrate the pure attacking results without defense), our defense strategy as well as the baseline strategies can achieve similar defending results.
Impact of Defense strategies In both batch label inference attack and gradient-replacement backdoor attack, as the noise level of DP or the drop rate of GS increases, both label recovery rate and backdoor accuracy decreases at the expense of significant drop in main task accuracy.
Notice the opposite trends in the performance of CoAE in label recovery and gradient-replacement backdoor. In defending batch label inference attacks, as the confusion coefficient increases, the label recovery rate gradually goes up, because higher confusion means higher probability for the soft fake labels to be mapped into the true labels. For the backdoor attacks, the backdoor accuracy decreases as the confusion level grows, indicating confusion is important for defending the attack. Without confusion (λ 2 = 0), the backdoor attack will still succeed since then the autoencoder's sole function is to switch labels among classes and the transformation would work the same for the backdoor samples and other samples. Figure 7 depicts the probability distribution (PD) over labels restored by the passive party conducting batch label inference attack (Algo 2) on CIFAR10, when the active party is performing CoAE protection (Algo 5). Specifically, each element in the PD matrix is computed by Cij j Cij , where C ij denotes the number of samples having ground-truth label i but is restored with label j. Figure 7(b) and Figure 7(a) depict the probability distribution when the CoAE is trained with (λ 2 = 1) and without (λ 2 = 0) entropy loss L entropy , respectively. Without entropy loss, the PD matrix is sparse so the attacker can learn all samples having the same label belong to the same class. With confusion, samples belonging to the same class are restored as multiple alternative labels, demonstrating that the passive party cannot classify its samples based on restored labels.

Conclusion and Future Work
We systematically study the batch label inference attack and gradient-replacement backdoor attack for featurepartitioned federated learning (VFL) problem and show that both these attacks can be conducted at passive party without knowing the labels. Then, we defined confusional autoencoder (CoAE) based on autoencoder and entropy regularization and used it as a novel defending strategy against these two attacks. Our experiments testify that when implementing our CoAE defense mechanism, both attacks can be prevented with superior performance over existing methods regardless of the number of classes for classification and the number of attack parties in VFL setting. All these demonstrate the effectiveness and universality of our CoAE defending method in preserving the privacy in VFL scenario.