text:
GadgetArm—Automatic Grasp Generation and Manipulation of 4-DOF Robot Arm for Arbitrary Objects Through Reinforcement Learning

Automatic robot gripper system which involves the automated object recognition of work-in-process in production line is the key technology of the upcoming manufacturing facility achieving Industry 4.0. Automatic robot gripper enables the manufacturing system to be autonomous, self-recognized, and adaptable by using artificial intelligence of robot programming dealing with arbitrary shapes of work-in-processes. This paper specifically explores the chain of key technologies, such as 3D object recognition with CAD and point cloud data, reinforcement learning of robot arm, and customized 3D printed gripper, in order to enhance the intelligence of the robot controller system. And it also proposes the integration with 3D point cloud based object recognition and game-engine based reinforcement learning. The result of the prototype of the intelligent robot gripping system developed by the proposed method with a 4 degree-of-freedom robot arm is explained in this paper.

Introduction
Due to the acceleration to mass customization, automatic bin-picking has became crucial capability of modern manufacturing system. Lot size "one" production is upcoming market demands, as consumers' needs vary, and so manufacturer started to customise their products and therefore requires ability to quickly manufacture various products. In particular, automatic robot gripper system which involves the automated object recognition of work-in-process in the production line is the key technology. Automatic robot gripper enables manufacturing system to be autonomous, self-recognized and adaptable by using artificial intelligence of tool-path generation dealing with arbitrary shapes of work-in-processes. But existing programming systems, such as Pick-n-Place, are not suitable for frequent change of the shape to be handled.
In terms of automation, Pick-n-Place task is one of the major important activity in the manufacturing system. Pick-n-Place of only a single object is relatively easy task because its object geometry is always identical in mass production line. But in the case of lot size "one" production [1], the Pick-n-Place becomes a difficult task, since it has to adapt many varieties of different shapes every time of grasping task. Most of manufacturing Pick-n-Place facilities are suitable just to standard shape of product geometry, which causes difficulty dealing with customised products.
It is not straightforward for robots to completely adapt their control program of Pick-n-Place tasks to arbitrary product shapes without humans' inputs. For this reason, many factories still produce their robot arm paths by on-line programming of material handling. Most well known method of this way is using 'teach pendant'. It is efficient for picking identical shape by manual programming because the robot program does not change after a robot expert made it once. But, it is an inefficient method when responding to lot size "one" production having to reprogram the robot path every single time. And also the quality of the path generation could vary according to the worker. Implementing intelligence and flexibility that makes the robot smart, could solve the above-stated problems with reasonable investment which will lead to profitability. This paper suggests using RL (reinforcement learning) for automated robot arm path generation and deep learning for object classification to grasp various arbitrary object. Legacy commercial system of automatic bin-picking uses 2D based image detection algorithm to detect the orientation and location of target product and predefined robot path after long and large training are applied. But the authors propose the integration with PointNet object recognition algorithm and game-engine (Unity) in RL. Automatic object recognition, reinforcement learning of robot arm are the main technologies of the proposed method. The proposed method reduces the effort to prepare training data and the expertise to establish predefined program as Figure 1. By implementing this method, difficult Pick-n-Place task could be easily solved with low cost device. This paper specifically explores the chain of key technologies, such as 3D object recognition with point cloud data, RL of robot arm and gripper, and a customised 3D printed gripper, in order to enhance the intelligence of the robot controller system which can lead to highly mass customised production line. In Section 2, related and existing models are introduced, such as fundamental researches to realize autonomous robot programming. In Section 3, describes the overall intelligent robot controller architecture of automatic robot gripper with more detailed explanation composed with object detection, automatic tool-path generation, and robot programming. Then in Section 4 the result of the prototype of the intelligence robot gripping system, developed by the proposed method with a 4 degree-of-freedom robot arm is explained. Section 5 discusses the comparison and improvement with legacy robot programming(teach pendant) method and proposes intelligent gripper system method "GadgetArm". Conclusions are briefly given in Section 6.

Literature Survey
In this section, previous related methodologies for robot arm grasping are introduced by dividing them in three parts. First, a method for robot arm to recognize object recognition by 2D data format to 3D data format is presented. Then various ways of training the robot arm to grasp the object properly, automatically and combining them to the robot programming are introduced. Finally a summary and opportunities of the related work are discussed.

Object Recognition
In the past there were many approaches to deal with robot arm's limitation, grasping unknown object reliance to humans vision system with online programming. Traditionally used 2D contour image data for object recognition in grasping [2][3][4].
Development of a camera that captures depth data made it convenient to earn 3D data format, which would provide more precise information compared to 2D data format. Also, improvement of DL (deep learning) made it possible to use 3D data as input in object recognition tasks by various DL algorithms. Most well known and used 3D data formats are Voxel and PCD (Point Cloud Data). 3D ShapeNet [5] and VoxNet [6] uses Voxel data format to classify object. But since voxel data format increases the data size and obscure recognizing the essence of the shape, PointNet [7] used PCD for object classification. The advantage of the PCD is that since it is the raw data of the object, it is accurate and has acceptable data size. In the same way, Wang et al. proposed a novel global PCD descriptor to improve robot grasping operation by using object recognition and pose estimation [8]. And Ji et al. combined real-time grasp control with vision data in order to achieve reliable and accurate object grasping even though operation in a cluttered scene [9].

Existing Platforms of Reinforcement Learning
Using RL in various fields has been successful. This is due to well made simulation software tools and well described by Erez et al. [10]. These tools are widely used especially in robotics. Johns et al. used Dart simulation physics engine with CNN image processing to grasp object with parallel gripper [11]. Also with most popular physics engine Mujoco integrated with OpenAI Gym, Plappert et al. tested robotic task such as reaching, pushing, sliding and Pick-n-Place task [12] and Wang, C. et al. trained Husky Dual UR5 robot for mobile manipulation by RL integrated with camera for object pose estimation [13]. But these simulation software tools are difficult to build complex simulation environment. So for alternative, games based simulation engine Unity is used for RL task, since it is more convenient building complex simulation environment. Mousavian et al. used game engine Unity, integrated with physics engine Flex to generate successful grasp for training data [14]. Advantages of Unity simulation platform are easy to learn and have sufficient physical complexity, also provide compelling cognitive challenges, and support dynamic multi-agent interaction [15].

Automatic Robot Programming in Smart Factory
Demands of gripping arbitrary objects in an industrial environment have also been discussed in papers for the flexibility of smart factories. Reddy et al. stressed the importance of universal gripper, more like a human hand, to avoid changing the gripper for each part. The human-like hand would make the gripper commonly suited for use in arbitrary areas from household chores to variable industrial tasks [16]. Birglen and Schlicht aimed at presenting characteristics of commonly used impactive grippers to pneumatic and parallel finger movement grippers in terms of performance index referred to as a C-factor which including stroke, force, and weight specifications. With these statistical analysis data, they could figure out that even though the variety of grippers on the market is large, almost all of them share similar characteristics that are used in the industrial environment [17]. Zhang et al. pointed out the complication of existing self-calibration accompanied by camera calibration, corner detection and laser tracking. To solve the problems, they introduced inertial measurement unit and position sensor to obtain the robot poses by Quaternion algorithm and Kalman filters. After the estimation, extended Kalman filter was used to carry out kinematics identification to improve the accuracy of robot gripping [18]. In the same way, automatic grasp generation has been developed by several researchers to improve the intelligence of robot programming. Kraft et al. suggested methods of automatic grasp generation to replace manual grasp selection with a bin-picking process to reduce cost [19]. To reduce the cycle time of the process planning, off-line programming method was implemented by calculating inverse kinematic automatically [20]. Weigand et al. applied a Runge-Kutta neural network as a neural adaptive control in order to reduce gearbox errors on industrial robots [21]. Also, recently, to avoid slip Muthusamy et al. proposed the robot system to grasp an object with slip detection by using computer vision though real-time direct robot programming [22].

Summary and Opportunities
The conclusions we can draw after reviewing the literature are: 1.
The increase of the rate of object detection has possibility to replace the capability of human observation which is the guide of robot position in legacy robot programming.

2.
Traditional programming methods, such as legacy on/off-line programming, are limited in relying on human expertise rather than improving the automation of mass customization. 3.
Lot size "one" needs the high flexibility to handle all different shapes of work-in-processes.
From these conclusions it appears that there is opportunity to motivate the autonomous robot control of different shapes. As well as there being a clear need for alternative technological solution in this area, the series of intelligent robot programming will ensure that the new development will provide benefits in the personalised material handling.

Overall Architecture
Typically legacy robot gripping applies without automatic grasp generation. Humans can detect a product and think of a grasp strategy without difficulties while a robot needs programming task before picking objects. Typical robot programming is the iteration of eye-observation and path-adjustment on teach pendant. Simple picking is reasonable to manual handling because of small benefit of robot gripping but the high cost of robot systems. Human expertise is required even though using off-line programming finds an optimised path among the alternatives generated automatically. This means that gripping task depends on the knowledge and the experience of human operators in both methods.
For this reason, automatic grasp generation proposed in this paper is a low-cost system dealing with arbitrary shapes. The main considerations of automatic grasp generation are (1) pre-training from CAD data and (2) one-shot handling without programming iteration like teach pendant. Pre-training is finished before work-in-process arrives in the production line. All try-and-error finding optimal path are finished in virtual reality. Object detection helps for searching the optimal path of the object arrived in production line. By pre-training and object detection, one-shot handling without any iterations can be realized. And the automatic grasp generation keeps seamless performance of grasping accuracy even though human operator replaces to another operator.
To develop the automatic grasp generation system, the overall architecture consists of object recognition, automatic positioning, and integrated robotic system. This chain of components reduces manual operation and allows labour to be used elsewhere. Whenever the product shape manipulated by robot arm is changed, either iterative programming or robot program needs to generate new robot path again.
An automatic robot gripper system for autonomous manufacturing systems requires two technologies which are automated object recognition, path-finding and grasping for a full automation system. This paper suggests an innovative way to satisfy the stated requirement by using a depth camera for 3D scanning and a video game engine (Unity) for RL. The overall procedure is explained briefly in Figure 2.

1.
STEP1: After the product designer uploads new CAD file of new part shape on manufacturing database, the file is imported into the virtual reality of RL. The virtual reality consists of two training environments of two-step RL. First step is to find the gripping position. Second step is to find the path of the robot arm to reach the gripping position. Two-step RL and virtual reality were made by Unity ml-agents. Each gripping position and robot path optimised are transmitted into manufacturing database via SQL query. Then, CAD data is also transmitted into production line. Factory starts to produce the new part as the physical object of new part. 2.
STEP2: After part production of a new part shape is laid down on the working table of a material handling facility surrounded by a 3D scanner. 3D scanner traces the reflection curves of scanning laser and converts the signals into point cloud file. The PCD is input data of 3d object detection. PCD is stored in the file system of PointNet algorithm. 3.
STEP3: When 3D scanning is done, PCD is utilised to find the part type which just arrived by using PointNet, one of 3D object detection algorithms. Based on the PCD, the part shape placed on working table is detected by classification algorithm of PointNet. The output of this step is classifying which type of part shape is arrived in production line. The part type found is sent into next step. 4.
STEP4: The part type classified by PointNet and the gripping position returned from the database are the input of this step. First task of this step is to send a query of part type detected by PointNet to manufacturing database where gripping positions of all part shapes are stored. Robot controller manipulates the gripper along the gripping position returned from database. The chain of these steps makes the robot control adaptive to arbitrary part shape.

Recognition by Point Cloud
An object recognition task is automated by using 3D data in order to improve classification accuracy and data handling. Proposed architecture apply PointNet [7] architecture ( Figure 3) to point cloud classification task with CAD data and 3D scanned data. Object recognition tasks were traditionally done by methods such as RFID tag or simply by the human eye. Tagging RFID object could be the easiest way for automating object recognition. But tag needs to be attached to each object and this causes operation cost which could not be ignored.
The most primitive way, recognized by the human eye, is used in a teach pendant. RFID tag and human recognition are expensive methods in the case of mass customization. Automatic process is required to follow market change.
To make it automatic, the authors propose the integration of a 3D scanner and a deep learning algorithm. Using 3D data improves classification accuracy and data handling. Improvement of 3D camera makes it possible to collect depth data and extract point cloud data. Using 3D data improves the accuracy of the classification of arbitrary objects rather than using 2D image. Only by the 2D data format, it is hard to recognize a 3D geometry accurately due to limited viewpoint. Most known 3D data used for classification task are Voxel format and Point cloud format. But when using voxel data format, data becomes unnecessarily large because it approximates cubic representation of the object. Earning richness of geometry information and avoiding complexities that make the calculation faster is possible by directly taking point cloud as input.
In this paper, accuracy for classification task was considered as an important factor to enhance the performance of the system. For this reason, choosing PCD as an input format was necessary, since it could provide both enough geometry data and reasonable data size for the overall process. The authors used PointNet architecture for point cloud classification task. It is a simple and robust deep learning algorithm for classifying and segmenting object using directly PCD as input. As PointNet architecture receives PCD as its input, it is used as the main algorithm for classifying object.
There are mainly two methods for collecting PCD.
The first way of collecting PCD is from the CAD file. Even though there is tolerance between design shape and manufactured shape, the existence of CAD data is reliable input of object detection. CAD file is shared by product data management system commonly used by most manufacturing industry. By using CAD file, for example .obj format, it can easily be changed to point cloud format and extract point cloud. Also it is possible to change the density of the point cloud. High density of points means extracting high number of points from the object. Extracting high density of point can give more accurate geometry information, but too much can cause degradation of computing speed of classification algorithm. To make the algorithm fast enough, but still has enough geometry information, 2048 points are chosen from the point with poisson-sampling method for point sampling in order to sample uniformly.
The second way is 3D scanning an object using depth camera. 3D scanning takes longer time compared to a fixed depth camera in case of collecting data. But when the camera is fixed, it could collect imperfect data, only about half of the geometrical data, since the opposite part is not being shown. The object detection task by 3D scanning finishes in a short time which is allowable in production line. The proposed system also uses only single algorithm for object detection. Using an another algorithm makes the system more complicated and reduces the speed for classification task.
A test dataset is generated from CAD file and 3D scanning. CAD file is labelled by the designer and converted into training data of PointNet. 3D scanning data is used for the input during operation. PointNet compares 3D scanning data with the training data and returns the label indicating product model.

Reinforcement Learning in Unity Ml-Agents
The proposed automatic robot programming system employs the RL in order to generate the gripping position of each part shape. The virtual reality of RL consists of robot arm, gripper and part shape to be manipulated. It provides the environment where a machine learning algorithm attempts all actions that robot arm can do. For this paper ml-agents [23] was used, an RL toolkit provided by Unity Technologies, to make robot arm learn how to grasp the object.
A high DOF (Degree of freedom) of RL has a relatively high possibility of going in the wrong direction during training. The target robot arm has 6 behaviours which are 4-DOF of robot arm, gripping action, and application of gravity. Before the behaviour of the gripper gets stable, part shape might be hit by unexpected gripper movement during training mode. Hitting objects cause the part shape to fly away and lead to rapid decline of the learning rate. Limiting DOF in time is required for avoiding an unexpected learning direction.
For this reason, training environments are separated into two pieces. The proposed RL is composed of gripping training and robot arm training. Separating the training into two parts reduces the running time to achieve finding optimised gripping position. Gripping training has 2-DOF of gripping action and gravity application. And robot arm training controls 4-DOF rotations.
In this work, the PPO (Proximal Policy Optimization) [24] algorithm was used. PPO is one of the policy gradient method algorithms. PPO is easier to implement and has more ease of sample complexity than the previous policy gradient algorithm TPRO (Trust Region Policy Optimisation) [25]. PPO has also displayed the best performance on continuous control tasks. One of the advantages of ml-agents is making multi-agents at the same time like in Figure 4. This function saves time for RL since it could test various grasp positions simultaneously. This way we can get the gripper's position where the gripper can grasp the object successfully. Then the gripper's position is sent to the robot offline programming software to move the robot arm at grasp position in real. Basic components of RL are agent, observation, action and reward. In this RL, the agent is the gripper and observation, action, reward is generated by the Unity environment shown in Figure 5. First, gripper moves along X,Y,Z axis. During RL the agent observes distance of gripper to bottle as Vector3 value(x,y,z) frame by frame. Next, gripper runs grasping motion and after applying gravity to the object to check if it has grasp well or not. If an object is dropped, the agent gets minus value reward, and if gripped correctly it gets plus value reward.

Robot Programming & Gripper
Robot programming methods can be divided into two categories-online programming, commonly used with teach pendant connected to the robot arm via a cable or lead through method, and off-line programming. Although online programming is easily suitable for all types, improving off-line programming is necessary to realize industrial 4.0 for the reason that the suitability of online programming decreases as the industrial processes become complex. Furthermore, from an economical perspective, off-line programming only could be justified for large volumes of production in big enterprises. However, by cooperating with RL in Unity ml-agents, this method can break through the limitations compared to commonly used robot programming using augmented reality which combines the features of online and off-line programming to make off-line programming more interactive and flexible.

Target Objects
This section is mainly focused on introducing the prototype of the GadgetArm and how it is implemented. As mentioned above, this paper is suggesting a simple and reasonable method for automatic grasping. Any object that can be classified by PointNet can be our target object. In this work, a bottle was chosen as the main target object because it could be easily obtained and has reasonable size considering the size of the gripper. For the test environment, the gripper was made using a 3D printer, which modified the open source design to fit the robot. Also modified gripper was controlled by Arduino-UNO board for grasping and connected with IgUS Robolink RL-DCi-4-DOF. The robot arm could reach 510 mm and payload 1000 g with 4 Degree of freedom. For vision task to extract PCD, Intel RealSense D435 was used as fixed depth camera and for 3D scanning task, Shining 3D scanner was used. Ideal prototype simulation environment are shown in Figure 6. Proposed architecture was implemented within the test environment where Windows 10 PC equips NVIDIA GTX 1060 GPU and 16 GB memory.

Implementation of Pointnet
The process of the recognition task is explained in Figure 7. As mentioned above, a bottle, vase and cup were selected as the target object. If there exists a CAD file, it could be easily converted to PCD format by various open software. In case, where no CAD file exist, it could be scanned by 3D scanner to get the PCD. For experiment, even though existed an actual size CAD file, printed with 3D printer and 3D scanned the actual bottle for testing. For both two cases, CAD file converted to PCD and 3d scanned PCD, 2048 points were sampled by poisson sampling method using PCU library [26]. Training PCD set was based on ModelNet40 [5] dataset. For custom target object, in this case bottle, vase and cup, CAD data was earned from web crawling which is 1 to 1 scale to the real object. To test 3D scanning method, printed the object with the 3D printer and after then performed scanning as Figure 8. Shining 3D EinScan-Se was used for scanning. Another important process left is scaling the size of the PCD by zero-mean and normalizing into a unit sphere for effective learning. Finally, by the algorithm based on PointNet which classifies up to 40 objects, classifies which class it belongs to. To test the performance of the PointNet algorithm, PCD earned from both CAD data and 3D scanning were used for each object type. The algorithm classified well on both cases on bottle and vase. But the cup PCD from CAD data was not able to classify correctly, even though PCD from 3D scanning classified perfectly. According to the classification result, the data of the object (CAD data) information is delivered to the Unity environment to start RL.

Reinforcement Learning
Among the objects in ModelNet40, we selected three objects for RL-bottle, vase, and cup shown in Figure 9 as the ones that the gripper can grasp in reality. The version of RL environment is Python 3.7, Tensorflow 2.3.0, ml-agents 0.18.1. And we used the following PPO hyper-parameters Table 1.  Training model was trained on a continuous environment and the gripper was set able to move XYZ axis and rotate only by B axis since the tested robot was 4-DOF. Final gripping position derived from gripping training is the input of robot path training in two-step RL. Robot path generators are also provided by on-line/off-line program system of commercial robot system. This section explains the detail of gripping training because robot path training is replaceable to legacy robot system.
For reward progress it was divided into 3-steps-(1) Gripper movement stage, (2) Gripping stage, (3) Apply gravity. Reward was given at each stage for a step-wise learning. The gripper observes the distance to the object and learns to get closer, otherwise it gets negative reward. Colliders which can detect collision with others were added to both sides of the gripper and got positive reward only if the object and the both side of gripper were tagged at the same time. After the gripping stage, gravity was applied to the object to find out whether the object was grasped properly. To train the gripper to keep holding the object, two box colliders were added, one at the bottom floor and one at the above the gripper, so in case of object drops or object flies away it ends episode and is set to get negative reward. The data flow of gripping training is depicted in Figure 5.
The reward and cumulative reward graphs are shown in Figure 10. We trained three objects for 500,000 steps and repeated it 10 times independently. The gripper is trained more stable in bottle than other objects. This result is because of constant cylinder shape of bottle. Round shape fits well with the shape of our gripper and gripper can grasp anywhere more successfully than a cup which is asymmetrical. With the consideration of industrial applications, we added two turning machining examples as the industrial experiments. Industrial parts are shown in Figure 11. All the turning examples are tested with standing position. because typical production line brings the parts on pallet system. The cumulative reward graphs of GadgetArm reinforcement learning are shown in Figure 12. We trained two objects in the same conditions to the above experiment. Two graphs always have a value greater than 0 and keep high average rate. It means learning results are stable rapidly and do not decline easily. Even though simple shape parts, industrial robots need to fine control to have stable gripping program of robot arm. The proposed methods let operators not to consider about detail of robot motion after the parts are located in fixed pallet. This means robot arm can learn well to grasp industrial parts too.

Robot Programming
Most robot arms have their own programming language based on their software and users can improve by themselves to drive the robot arms. Igus Robolink robot arm was used with Commonplace CPRog programming environment to drive the robot arm, and XML files were used to control it wirelessly through TCP/IP protocol. Therefore whenever creating an optimal path in Unity, a 3D simulation environment, it was necessary to fit this data into an XML format. It was automatically converted by Unity to XML format and the experiment was conducted by using the paths.
Robot programming to grasp the object used the optimal coordinates obtained by RL in Unity. This coordinate was used in the Igus Robolink software to move the robot arm to the coordinate in reality. However, in Unity, the robot arm joints deflection due to the mechanical vibration and the influence of gravity is not considered, so the process of matching the reference using the program was additionally required. Therefore, in order to minimize vibration, the speed of movement of the robot arm was slowed down to optimize, and the amount of change in the z-axis (vertical axis of ground plate) was considered as the optimized coordinate through multiple iterations.

Discussion and Conclude Remarks
The most well known and used method for robot arm grasp is teach pendant and bin picking. The comparison of the overall process between them is explained in Figure 13. In the term of automation, RL has much more advantages than other methods. If unknown object is added, teach pendant needs trained worker and bin picking needs to be programmed by programmer manually. But RL can grasp unknown object only by adding CAD file and be trained. Repeatable work done by human could be replaced by our automatic robot gripper system. This system is easily adaptable according to many situation. (1) Object detection: object recognition task is implemented by using PCD based algorithm. 3D data based approach is the way to utilise more geometric information than 2D images used by 2D image recognition algorithm. From the implementation of the proposed object detection, followings are found. First, in preparation for dataset, if new part comes, several hundred of image data is needed in 2D image recognition. Capturing images is time-consuming and difficult to be automated. PCD is extracted from CAD data which is pre-defined before production line starts to produce the parts. That is CAD data is existed before the part arrived in material handling robot. Simple sampling iteration of point cloud is required for the data collection and is fully automated without additional works for building dataset. Second, in generate variations of part geometry, another benefit to use PCD data is to make many variations of part shapes composed of training dataset. CAD solid kernel has the geometric functions of Boolean operation, extrusion, lofting, and so forth. These functions are used for modifying original CAD shape and building the dataset of its variations. Third, capturing point cloud is the challenges of PCD object detection, 3D camera observes the only one side of part shape which is not sufficient for comparing with whole shape of CAD file. This paper showed the solution of 360-degree rotation table is applied to resolve the issue of capturing PCD in this paper. As for the further possible issue related to capture PCD, other cases to consider are when the object is tilt and when the object shape is complicated.
(2) Reinforcement learning environment: Even though the gripper reaches the optimal grasp position, the time spent searching for the optimal position varies according to the object. For example, in the case of bottle and vase, gripper reaches optimal position easily to bottle rather than vase. It could tell that object's shape complexity matters and since the bottle has the more constant shape of a cylinder, the gripper arrived at the optimal position earlier. For industrial parts, we should implement more experiments to find which shapes are hard to grasp and other aided methods to increase utility. Due to the limitation of the proposed learning model, reinforcement was conducted step-wise instead of one-step to reach the final goal in a more stable manner. Not using the entire robot arm and using only a gripper for training prevented the learning model from being complex. But it was effective enough for grasping an object in a reasonable position with fine-tuned gripping behaviour. Since there are also many errors in the final gripping operation even in actual robot control, these functions are expected to work effectively in the field as well. For the RL algorithm, PPO is only one of the policy gradient based algorithm. We used SAC which is hybrid algorithm that combines the advantages of policy gradient and Q-Learning but in our case it did not work well. The direction of future studies is changing RL algorithm, for example DQN [27], DDPG [28] and tuning hyper-parameter of PPO and SAC for higher performance.
(3) Robot Programming: Since the 4-DOF robot arm was used in the current implementation, there were some physical limitations. If a robot arm could move, when a higher DOF is used, diverse paths can be formed for a single target coordinate. Therefore, even if there are obstacles interfering to reach the target coordinate, it is possible to avoid these and achieve the goal by using multi paths. However, 4-DOF robot arm can operate perfectly only when there are not any objects around the target due to the physical limitations. In future studies, higher DOF robot arm will be employed to apply RL in more diverse environments. And we should consider precision of robot manipulation. One of the methods for increasing precision at industry 4.0 can be online calibration.
Finally, we should consider applying the GadgetArm System to factories. For example, worker safety (Human-Robot Interaction), change of gripper to a more universal gripper, and the validity of the importing GadgetArm system instead of the current system.

Conclusions
In this paper, simply combining systems or technology from various fields, it was able to build a system for an automatic robot gripper. Rather than using a passive teach pendant, GadgetArm suggests a method for getting one step closer to achieving Industry 4.0. PointNet for arbitrary object recognition, Unity ml-agents RL for finding optimal grasp position, control the robot control system with 3D printed gripper and connecting these independent processes with one system. As shown in Figure 14, in the range of the quantity appropriate to an automatic process, by adapting GadgetArm, it could lower the cost per item compared to the legacy grasping method. This process can be easily modified according to diverse industry, not only for manufacturing facility but every field that suffers from Pick-n-Place task, or even in the household.


extfieldsofstudy:
Computer Science
Mathematics

text:
An Analysis of SVD for Deep Rotation Estimation

Symmetric orthogonalization via SVD, and closely related procedures, are well-known techniques for projecting matrices onto $O(n)$ or $SO(n)$. These tools have long been used for applications in computer vision, for example optimal 3D alignment problems solved by orthogonal Procrustes, rotation averaging, or Essential matrix decomposition. Despite its utility in different settings, SVD orthogonalization as a procedure for producing rotation matrices is typically overlooked in deep learning models, where the preferences tend toward classic representations like unit quaternions, Euler angles, and axis-angle, or more recently-introduced methods. Despite the importance of 3D rotations in computer vision and robotics, a single universally effective representation is still missing. Here, we explore the viability of SVD orthogonalization for 3D rotations in neural networks. We present a theoretical analysis that shows SVD is the natural choice for projecting onto the rotation group. Our extensive quantitative analysis shows simply replacing existing representations with the SVD orthogonalization procedure obtains state of the art performance in many deep learning applications covering both supervised and unsupervised training.

Introduction
There are many ways to represent a 3D rotation matrix. But what is the ideal representation to predict 3D rotations in a deep learning framework? The goal of this paper is to explore this seemingly lowlevel but practically impactful question, as currently the answer appears to be ambiguous.
In this paper we present a systematic study on estimating rotations in neural networks. We identify that the classic technique of SVD orthogonalization, widely used in other contexts but rarely in the estimation of 3D rotations in deep networks, is ideally suited for this task with strong empirical and theoretical support.
A design choice common to all of these models is selecting a representation for 3D rotations. The most frequent choices are classic representations including unit quaternion, Euler angles, and axisangle. Despite being such a well-studied problem, there is no universally effective representation or regression architecture due to performance variations across different applications.
A natural alternative to these classic representations is symmetric orthogonalization, a long-known technique which projects matrices onto the orthogonal group O(3) [23,35]. Simple variations can restrict the projections onto the special orthogonal (rotation) group SO(3) [13,18,44]. This procedure, when executed by Singular Value Decomposition (SVD [9]), has found many applications in computer vision, for example at the core of the Procrustes problem [2,35] for point set alignment, as well as single rotation averaging [11]. A nearly identical procedure is used for factorizing Essential matrices [12].
Despite its adoption in these related contexts, orthogonalization via SVD has not taken hold as a procedure for generating 3D rotations in deep learning: it is rarely used when implementing a model (e.g. overlooked in [21,5,32,26]), nor is it considered a benchmark when evaluating new representations [22,50].
In light of this, this paper explores the viability of SVD orthogonalization for estimating rotations in deep neural networks. Note, we do not claim to be the first to introduce this tool to deep learning, rather our focus is on providing a comprehensive study of the technique specifically for estimating rotations. Our contributions include • A theoretically motivated analysis of rotation estimation via SVD orthogonalization in the context of neural networks, and in comparison to the recently proposed Gram-Schmidt procedure [50]. One main result is that SVD improves over Gram-Schmidt by a factor of two for reconstruction, thus supporting SVD as the preferred orthogonalization procedure.
• An extensive quantitative evaluation of SVD orthogonalization spanning four diverse application environments: point cloud alignment, object pose from images, inverse kinematics, and depth prediction from images, across supervised and unsupervised settings, and benchmarked against classic and recently introduced rotation representations.
Our results show that rotation estimation via SVD orthogonalization achieves state of the art performance in almost all application settings, and is the best performing method among those that can be applied in both supervised and unsupervised settings. This is an important result given the prevalence of deep learning frameworks that utilize rotations, as well as for benchmarking future research into new representations.

Related Work
Optimization on SO (3), and more generally on Riemannian manifolds, is a well-studied problem. Peculiarities arise since SO (3) is not topologically homeomorphic to any subset of 4D Euclidean space, so any parameterization in four or fewer dimensions will be discontinuous (this applies to all classic representations-Euler angles, axis-angle, and unit quaternions). Discontinuities and singularities are a particular nuisance for classic gradient-based optimization on the manifold [37,41].
Early deep learning models treated Euler angle estimation as a classification task [43,38], by discretizing the angles into bins and using softmax to predict the angles. This idea was extended to hybrid approaches that combine classification and regression. In [20], discrete distributions over angles are mapped to continuous angles via expectation and [25] combines classification over quantized rotations with the regression of a continuous offset. In [22] it is shown that typical activations used in classification models (e.g. softmax) lead to more stable training compared to the unconstrained setting of regression. The authors introduce a "spherical exponential" mapping to bridge the gap and improve training stability for regression to n-spheres. All methods that discretize suffer from the problem of increased dimensionality and expressivity. All of the above methods require supervision on the classification objective, which makes them unsuitable for unsupervised settings.
Probabilistic representations have been introduced for modeling orientation with uncertainty [34,8], with the von Mises and Bingham distributions respectively. While these are best suited for multimodal and ambiguous data, such approaches do not reach state of the art in tasks where a single precise rotation must be predicted.
The closest approach to SVD orthogonalization is the recent work of [50] which makes a strong connection between the discontinuities of a representation and their effect in neural networks. In search of continuous representations, they propose the idea of continuous overparameterizations of SO(3), followed by continuous projections onto SO (3). Their 6D representation is mapped onto SO(3) via a partial Gram-Schmidt procedure. This is similar in spirit to SVD orthogonalization which will map a continuous 9D representation onto SO(3) with SVD. We leave the deeper comparison of these two methods to the following sections, where we show that SVD provides a more natural projection onto SO(3) in several respects (Section 3).
SVD derivatives have been presented in [7,30], and there exist multiple works that build neural nets with structured layers depending on components of SVD or Eigendecomposition [16,33,15,3,28]. Close to our setting are [40] which applies the orthogonal Procrustes problem to 3D point set alignment within a neural network, and [17] which proposes singular value clipping to regularize networks' weight matrices.
SVD is amenable for learning via backpropagation. Its derivatives have been presented in [7,30], and there exist multiple works that build neural nets with structured layers depending on SVD or Eigendecomposition [16,33,15,3]. The closest to our setting is [40] which applies the orthogonal Procrustes problem to 3D point set alignment within a neural network, and [17] which proposes singular value clipping to regularize networks' weight matrices. We discuss the stability of SVD orthogonalization in neural networks in the following section.

Analysis
In this section we present our theoretically motivated analysis of SVD orthogonalization for rotation estimation. We start here defining the procedures and introducing well-known results regarding their least-squares optimality before presenting the analysis.
Given a square matrix M with SVD U ΣV T , we consider the orthogonalization and special orthogonalization SVDO is orientation-preserving, while SVDO + maps to SO(n). Orthogonalization of a matrix via SVD is also known as symmetric orthogonalization [23]. It is well known that symmetric orthogonalization is optimal in the least-squares sense [2,13,35]: This property has made symmetric orthogonalizations useful in a variety of applications [44,35,40].
To reiterate, the procedure for deep rotation estimation we will evaluate experimentally in Section 4 is SVDO + (M ) in 3D, which takes a 9-dimensional network output (interpreted as a 3 × 3 matrix M ), and projects it onto SO(3) following Eq. 2.

SVDO(M ) and SVDO + (M ) are maximum likelihood estimates
In this section we show that SVD orthogonalization maximizes the likelihood and minimizes the expected error in the presence of Gaussian noise. Let M = R µ + σN, M ∈ R n×n represent an observation of R µ ∈ SO(n), corrupted by noise N with entries n ij ∼ N (0, 1). Following from the pdf of the matrix normal distribution [10] the likelihood function is The solution is given by SVDO + (M ) (Eq. 3). Similarly, SVDO(M ) will minimize Eq. 4 when R µ ∈ O(n).

Gradients
In this section we analyze the behaviour of the gradients of a network with an SVDO + layer, and show that they are generally well behaved. Specifically, we can consider ∂L ∂M for some a loss function L(M, R) = SVDO + (M ) − R 2 F . We will first analyze ∂L ∂M for SVDO(M ). Letting • denote the Hadamard product, from [16,42] we have T V , we see that X, Y are antisymmetric and X = −Y (this is a result of the loss function depending on U V T ). We can reduce ∂L ∂M = U ZV T where the elements of Z are See Appendix C for the detailed derivation. For SVDO(M ) Eq. 8 tells us ∂L ∂M is undefined whenever two singular values are both zero and large when their sum is very near zero.
For SVDO + (M ), if det(M ) > 0 then the analysis is the same as above. If det(M ) < 0, the extra factor D = diag(1, 1, . . . , −1) effectively changes the smallest singular value s n to −s n . The derivation is otherwise unchanged. In particular the denominator in equation (8) is now s j − s n or s n − s i if either i or j is n. Thus, ∂L ∂M is undefined if the smallest singular value occurs with multiplicity greater than 1. It is large if the two smallest singular values are close to each other, or if they are close to 0.

Error analysis
In this section we approximate the expected error in SVDO(M ) and Gram-Schmidt orthogonalization (denoted as GS(M )) in the presence of Gaussian noise, and observe that the error is twice as large for GS as for SVDO. If M is a matrix with QR decomposition M = QR, define: We consider M = R 0 + σN , a noisy observation of a rotation matrix R 0 ∈ SO(n), where N has i.i.d. Gaussian entries n ij ∼ N (0, 1) and σ is small. Since det(M ) > 0 as σ → 0, the analysis is identical for SVDO + and GS + . The analysis is also independent of R 0 (Appendix A), so for simplicity we set R 0 = I. First we calculate the SVD and QR decompositions of M to first order for N an arbitrary (non-random) matrix.

Proposition 1
The SVD and QR decompositions of M = I + σN are as follows: 1. (SVD) Let N = S + A be the decomposition of N into symmetric and antisymmetric parts. Then, to first order, an SVD of M is given by This corollary is by straightforward calculation since each entry of A and L − L T is Gaussian (see appendix A for the proof). Notably, Gram-Schmidt produces twice the error in expectation (and indeed deviates 1.5 times further from the observation M itself). The same holds for SVDO + and GS + . This difference in performance can be traced to the fact that Gram-Schmidt is essentially "greedy" with respect to the starting matrix, whereas the SVD approach is coordinate-independent.
Although i.i.d. Gaussian noise is not necessarily reflective of a neural network's predictions, it does provide meaningful insight into the relationship between SVDO + and GS + and their relative performance observed in practice. See Appendix A for further remarks.
Proof of Proposition (sketch; see Appendix A for the full proof).
(1) Let M have SVD M = U SV T for some orthogonal matrices U, V and diagonal matrix S ≥ 0.
To first order in σ, we can write (This is using the fact that the antisymmetric matrices give the tangent space to the orthogonal matrices.) The claims come from breaking this equation into symmetric and antisymmetric parts; note if X is (anti-)symmetric and Q is orthogonal, then QXQ T is again (anti-)symmetric. Simplify to get SVDO(M ) = I + σA + O(σ 2 ).
For (2) the proof is similar: we expand the QR decomposition M = QR to first order as Q 0 orthogonal, Q 1 antisymmetric, and R 1 , R 0 upper triangular; examine the upper and lower parts.

Continuity for special orthogonalization
The calculation above shows SVDO(M ) and SVDO + (M ) are continuous and differentiable, at least at M = I. In fact SVDO(M ) is smooth, as is SVDO + except for a discontinuity 1 if (and only if) det(M ) = 0 or det(M ) < 0 and its smallest singular value has multiplicity greater than 1. In fact the optimization problem (3) is degenerate in this case. For example, the 2 × 2 matrix M = diag(1, −1) is equidistant from every rotation matrix; perturbations of M may special-orthogonalize to any R ∈ SO(2). GS + is continuous on a slightly larger domain -det(M ) = 0 -because it makes a uniform choice, negating the n-th column of M if necessary, at the cost of significantly greater error in expectation. This reflects the fact that SVD orthogonalization is coordinate-independent and GS, GS + are not: and similarly for SVDO + . GS and GS + are rotation-equivariant on only one side: is not a function of R 2 and GS(M ); likewise for GS + . See Appendix B for a proof of smoothness and further discussion.

Summary
The results above illustrate a number of desirable properties of SVD orthogonalization. SVDO + is optimal as a projection in the least squares sense as well as in the presence of Gaussian noise (MLE). Viewed through the lens of matrix reconstruction, the approximation errors are half that of the Gram-Schmidt procedure. Finally we show the conditions that lead to large gradient norms (conditions that are rare for small matrices). In the following, we support this theoretical analysis with extensive quantitative evaluations.

Experiments
This section presents the experimental analysis for SVD orthogonalization and numerous baseline methods. To recap, the SVD orthogonalization procedure SVDO + (M ) takes a 9D network output (interpreted as a 3×3 matrix), and projects it onto SO(3) via Eq. 2. The procedure can easily be used in popular deep learning libraries (e.g. PyTorch [31] and TensorFlow [1] both provide differentiable SVD ops), and it does not affect efficiency (3 × 3 SVD adds negligible computational overhead in both forward and backward passes).  For SVD, 6D, 5D, and the classic representations, the loss is (3) this is related to geodesic angle error θ as L(R, R t ) = 2 − 2 cos(θ). All other methods require an additional classification loss. See Appendix D for additional experimental details (architectures, implementations, etc).

3D point cloud alignment
The first experiment is the point cloud alignment benchmark from [50]. Given two shape point clouds the network is asked to predict the 3D rotation that best aligns them. The rotations in the dataset are sampled uniformly from SO(3) (no rotation bias in the data). Table 1 (left) shows geodesic error statistics (mean, median, std) on the test set. We omit the maximum error as it is approx. 180 • for all methods, a reflection of the symmetries in the data rather than limitations of the methods. SVD-Train outperforms all the baselines, and even SVD-Inference performs on par with the best baseline (6D). Interestingly, the hybrid approaches 3D-RCNN and M G underperform the top regression baselines, a point we will return to later. Table 1 (middle) shows the mean errors on the test set as training progresses. The best performing methods at the end of training (SVD variations, and 6D) also show fast convergence. The errors at different percentiles are shown in Table 1 (right).

3D Pose estimation from 2D images
The second experiment follows the benchmark set forth in [22]. Images are rendered from Model-Net10 [45] objects from arbitrary viewpoints. Given a 2D image, the network must predict the object orientation. We used MobileNet [14] to generate image features, followed by fully connected regression layers. Rather than averaging over all 10 ModelNet categories as in [22], we focus on chair and sofa which are the two categories which exhibit the least rotational symmetries in the dataset. Results are shown in Tables 2 and 3. Interestingly, SVD-Inference also performs similarly to SVD-Train on final metrics with faster convergence, indicating short pretraining with SVD-Inference could improve convergence rates.
3D-RCNN and M G are again underperforming the best methods. These hybrid methods have shown state of the art performance on predicting 3D pose from images [20,25], but in those benchmarks the 3D rotations exhibit strong bias (camera viewpoints are not evenly distributed over SO(3)) which is

Pascal3D+
[46] is a standard benchmark for object pose estimation from single images. The dataset is composed of real images covering 12 categories. For training we discard occluded or truncated objects [25,38] and augment with rendered images from [38]. The model architecture is the same as Section 4.2. Table 4 shows results on two categories and the mean over all categories (see Appendix D.7 for results on each of the 12 categories). The individual metrics we report are the median error as well as accuracies at 10 • , 15 • , and 20 • .
The best performing method is clearly S 2 -Reg. As expected, the hybrid method 3D-RCNN performs well on this task, but SVD-Inference and SVD-Train are on par. The SVD variations are also the best performing of the regression methods (those that only train with a rotation loss). Interestingly, SVD-Inference slightly outperforms SVD-Train, which suggests in this scenario where viewpoints have a non-uniform prior, training a network to regress directly to the desired rotation can work well.

Unsupervised rotations
So far we have considered supervised rotation estimation. Given the growing attention to self-or unsupervised 3D learning [48, 26,19,50], it is important to understand how different representations fare without direct rotation supervision. We omit 3D-RCNN, M G , and S 2 -Reg from the experiments below as they require explicit supervision of classification terms, as well as SVD-Inference as it does not produce outputs on SO(3) while training.

Self-supervised 3D point cloud alignment
To begin, we devise a simple variation of the point cloud alignment experiment from Section 4.1. Given two point clouds, the network still predicts the relative rotation. However, now the only loss is L2 on the point cloud registration after applying the predicted rotation. All other experiment details remain the same. From Table 5, SVD-Train performs significantly better than the next closest baseline, 6D.

Inverse kinematics
Our second unsupervised experiment is the human pose inverse kinematics experiment presented in [50]. A network is given 3D joint positions and is asked to predict the rotations from a canonical "T-pose" to the input pose. Predicted rotations are transformed back to joint positions via forward kinematics, and the training loss is on the reconstructed joint positions. We use the code provided by [50]. Table 6 presents the results. SVD-Train shows the best overall performance while 6D is closer than in other experiments.

Unsupervised depth estimation
The final experiment considers self-supervised learning of depth and ego-motion from videos [48]. Given a target image and source images, the model predicts a depth map for the target and camera poses from the target to sources. Source images are warped to the target view using predicted poses, and reconstruction loss on the warped image supervises training. In [48] the rotational component is parameterized by Euler angles. Following [48], we report the single-view depth estimation results  Table 7). The error metrics are in meters while accuracy metrics are percentages up to a distance threshold in meters (see [4] for a description).
Observe that the difference between the best and second best method in each metric is small. This is not surprising since the camera pose is a small (albeit important) part of a complex deep architecture. Nonetheless, SVD-Train performs best for 4 out of the 7 metrics, and second best in another two. For driving data the motion is likely to be mostly planar for which axis-angle is well suited. Finally, it is worth noting that carefully selecting the rotation representation is important even in more complex models -the default selection of Euler angles in [48] is outperformed in every metric.

Conclusion
The results of the previous sections are broad and conclusive: a continuous 9D unconstrained representation followed by an SVD projection onto SO (3) is consistently an effective, and often the state-of-the-art, representation for 3D rotations in neural networks. It is usable in a variety of application settings including without supervision. The strong empirical evidence is supported by a theoretical analysis that supports SVD as the preferred projection onto SO(3).
[  Note that since SVDO(RM ) = R · SVDO(M ) and GS(RM ) = R · GS(M ) for any orthogonal matrix R, and likewise for SVDO + , GS + if R is special orthogonal. Therefore the error analyses are the same for all matrices M : since orthogonal matrices preserve Frobenius norm and R −1 N has the same distribution as N since N was assumed isotropic.
with U 0 , V 0 orthogonal, U 1 , V 1 antisymmetric and Σ 0 , Σ 1 ≥ 0 diagonal. This is using the fact that the antisymmetric matrices give the tangent space to the orthogonal matrices. Similarly, the tangent space to the diagonal matrices is given again by the diagonal matrices. This gives an overall expression for M as Setting σ = 0 we see I = U 0 Σ 0 V T 0 , which implies Σ 0 = I and U 0 = V 0 . Next, collecting the first-order σ terms gives If a matrix X is (anti-)symmetric and Q is orthogonal, then QXQ T is again (anti-)symmetric. So, the symmetric and antisymmetric parts of the equation are Note that the first equation is an SVD of the symmetric part of N , while the second equation shows that U 1 and V 1 satisfy U 1 + V T 1 = U T 0 AU 0 . Finally, dropping the Σ 0 + σΣ 1 factor from Eq. (19) and expanding out shows that SVDO(I + σN ) = I + σA + O(σ 2 ).
(2) Let M = QR, where Q is orthogonal and R is upper-triangular with positive diagonal. As above, by expanding to first order in σ we have with Q 0 orthogonal, Q 1 antisymmetric, and R 1 , R 0 upper triangular. Setting σ = 0, we see I = Q 0 R 0 and so Q 0 = R 0 = I. For the σ terms, we split N into its upper, lower and diagonal parts to get which by comparing parts gives Q 1 = L−L T and R 1 = U +D+L T . Then GS(M ) = I +σ(L−L T ) by simple algebra.
We now prove Corollary 1. N (0, 1), then with error of order O(σ 3 ), Proof. Simplifying the error expressions using the first-order calculations in the Proposition gives with notation for S, A, U, D, L as in the proposition. Thus each expression is σ 2 times the Frobenius norm of the corresponding matrix. Each entry of A, L−L T , S and U +D+L T is a linear combination of the entries of N , hence is Gaussian since N has i.i.d. Gaussian entries n ij ∼ N (0, 1). The expectations are the sums of the entrywise expectations of these matrices. For example, A = 1 2 (N − N T ) has six nonzero entries of the form 1 2 (n ij − n ji ), each having variance 1 2 , so E[ A 2 F ] = 3. For L − L T , the above diagonal entries are −n ji and the below-diagonal entries are n ij , and the diagonal is 0, so the total variance is 6. The other two calculations are similar (the entries do not all have the same variances).
Remark. The tangent space to the identity matrix along the orthogonal matrices is the space of antisymmetric matrices. Both of the calculations above can be thought of as giving orthogonal approximations of the form where A ′ is a choice of antisymmetric matrix that depends on the approximation method. The fact that SVDO(M ) produces the approximation A ′ = A = 1 2 (N − N T ) means it corresponds to the natural projection of N onto the orthogonal tangent space. By contrast, GS(M ) produces A ′ = L − L T , essentially a "greedy" choice with respect to the starting matrix (minimizing the change to the leftmost columns). For certain matrices GS can have smaller error: for example if N happens to be upper-triangular, GS(M ) = I and the error is zero. For isotropic noise, however, the SVD approximation is the most efficient in expectation.

A.1 Accuracy of error estimates as σ increases
From Corollary 1 (Sec 3.3) we see special-orthogonalization with Gram-Schmidt (GS + ) produces twice the error in expectation as SVD (SVDO + ) for SO(3) reconstruction when inputs are perturbed by Gaussian noise. We compare these derived errors with numerical simulations. See Figure 1. We calculate the derivative with respect to Q ∈ O(n): for A an antisymmetric matrix, If this vanishes for every A, then M T Q is symmetric, that is, (M, Q) is a root of the function g(M, Q) = M T Q − Q T M . Let M 0 be a fixed matrix. As discussed above, the optimal solution to this equation is given by an SVD, To show that Q is a smooth function of M , it suffices by the Implicit Function Theorem to show that the Jacobian matrix ∂g ∂Q is full-rank at (M 0 , Q 0 ). To see this, we differentiate it again: where A is antisymmetric. Some algebra shows that this is, equivalently, To see that this is an invertible transformation of A, note that conjugating by V 0 is invertible since V 0 is orthogonal. So it is equivalent to show that the function is invertible. This function just rescales the entry a ij to (s i + s j )a ij . Since the singular values are positive this is invertible as desired.

Proposition 3 The special symmetric orthogonalization is a smooth function of M if either of the following is true:
• det(M ) > 0, • det(M ) < 0 and the smallest singular value of M has multiplicity one.
Proof sketch. The analysis is identical to the main proof, except that if det(M ) < 0, S 0 is effectively altered so that the last entry is changed from s n to −s n . Thus the function A → S 0 A + AS 0 now sends the entry a ij to (±s i ± s j )a ij , with negative signs at i = n and/or j = n. If s n occurred with multiplicity one, the result is still invertible since s i − s n = 0 for i = n and for i = j = n the coefficient is −2s n . Otherwise, however, s n−1 = s n and the operation sends a n−1,n to (s n−1 − s n )a n−1,n = 0; likewise for a n,n−1 . In this case there are many optimal special orthogonalizations of M 0 , and the operation is not even continuous in a neighborhood of M 0 .

C Gradients
Here we provide the detailed derivation sketched out in Sec 3.2. We will first analyze ∂L ∂M for SVDO(M ). With • denoting the Hadamard product, from [16,42] we have Inspecting the individual elements of (F • X)Σ and Σ(F • X)) we have For SVDO(M ) Eq. 40 tells us ∂L ∂M is undefined whenever two singular values are both zero and large when their sum is very near zero.
For SVDO + (M ), if det(M ) > 0 then the analysis is the same as above. If det(M ) < 0, the extra factor D = diag(1, 1, . . . , −1) effectively changes the smallest singular value s n to −s n . The derivation is otherwise unchanged. In particular the denominator in equation (40) is now s j − s n or s n − s i if either i or j is n.

C.1 Gradients observed during training
In Figure 2 (left) we see the gradient norms observed while training for point cloud alignment (Section 4.1). SVD-Train has the same profile as for 6D (GS + ). SVD-Train converges quickly (relative to all other methods) in all of our experiments, indicating no instabilities due to large gradients.
On the right of Figure 2 we profile the gradients for the scenario where we begin training with the SVD-Inference loss and switch to SVD-Train after 100K steps (after roughly 4% of training iterations). SVD-Inf trains the network to produce outputs that are close to SO(3), which eliminates some conditions of instability in Eq. 40. This is confirmed by seeing much smaller gradient norms after switching to SVD-Train at 100K steps. Note, this approach was never used (or needed) in our experiments.

D.1 Additional baseline implementation details
• 6D is the partial Gram-Schmidt method which computes GS + (M ). Our implementation follows exactly the code provided by [50]. • 5D maps outputs in R 5 onto SO(3) as described in [50].
• Spherical Regression (S 2 -Reg) regresses to n-spheres. Following [22], we use regression to S 1 for Pascal3D+ and S 3 regression (quaternions) for their ModelNet experiments (section 4.2). The method combines abs. value regression with sign classification. Our implementation of the final regression layers follows the provided code. We select the hyperparameter that balances the classification and regression losses by a simple line search in the neighborhood of the default provided in [22].  . We found this version of Geodesic-Bin-and-Delta outperformed the One-delta-network-per-pose-bin variation in these experiments. For Pascal3D+ we follow the reference and use K-means with K = 200. This method also requires both classification and regression losses and we were unable to train successfully in the unsupervised setting. • Quaternion, Euler angles, and axis-angle are the classic parameterizations. In each case they are converted to matrix form before the loss is applied. In our experiments we did not filter any outputs from the network representing angles (e.g. clipping values or applying activations such as tanh). We found this gave the best results overall.

D.2 Learning rate decay
An observation from the point cloud registration results is that the curves for mean test errors as training progresses do not decay smoothly as one might expect for any method (Table 1, middle, in the main paper). This is in part due to the training code from [50] does not utilize a learning rate decay for this experiment. It is reasonable to observe the variance in evaluation when a decay is introduced as would be common in practice. Table 8 (left) shows the curves when the learning rate is exponentially decayed (decay rate of 0.95, decay steps of 35K). The evaluation over time is smoother but the results are consistent with those presented in the main paper.

D.3 Geodesic loss
In [50] it was shown that geodesic loss for training did not alter the results much, and we have the same observation. The model architecture follows exactly the architecture described in [50]. Point clouds are embedded with simplified PointNet (4-layer MLP) ending with a global max-pooling. Three dense layers  The regression layers are the same as above. The image embeddings are generated with Mo-bileNet [14]. We found no difference in performance between MobileNet and VGG16 [36] (not surprising given the comparison in [14]).

D.5 Inverse kinematics (Sec. 4.4.2) and unsupervised depth estimation (Sec. 4.4.3)
For these experiments the training and evaluation code is provided by [50] and [48], respectively. We simply change the rotation representation layer.

D.6 Remark on timings
We noticed no measurable difference in training time with SVDO + as expected, since the forward and backward pass for small, e.g. 3 × 3, SVD is efficient.

D.7 Pascal3D+ full results.
Here in Table 9 we show the results for all 12 categories in the Pascal3D+.


Towards Hypervisor Support for Enhancing the Performance of Virtual Machine Introspection

Virtual machine introspection (VMI) is the process of external monitoring of virtual machines. Previous work has demonstrated that VMI can contribute to the security of cloud environments and distributed systems, as it enables, for example, stealthy intrusion detection. One of the biggest challenges for applying VMI in production environments is the performance overhead that certain tracing operations impose on the monitored virtual machine. In this paper, we show how this performance overhead can be significantly minimized by incorporating minor extensions for VMI operations into the hypervisor. In a proof-of-concept implementation, we demonstrate that the pre-processing of VMI events in the Xen hypervisor reduces the monitoring overhead for the use case of VMI-based process-bound monitoring by a factor of 18.

Introduction
Virtual machine introspection (VMI) is the process of analyzing the state of a virtual machine from outside, i.e., the perspective of the hypervisor [5]. Based on the external view, VMI-based monitoring has certain properties that make it appealing for many application scenarios, including solutions that aim to enhance the security in cloud environments and distributed systems [6,8]. Hence, it is not surprising that the first application for VMI was an intrusion detection system [5]. Those properties are: isolation between the monitoring and the monitored system, an untampered view on the system state, and the stealthiness of monitoring.
In the following we use the term production virtual machine for the virtual machine that is monitored via VMI and is performing the normal operations. The term monitoring virtual machine is used for the virtual machine that analyzes the production virtual machine using VMI [18]. The focus of this paper is on the Xen architecture. Hence, the term monitoring virtual machine refers to either the Dom0 of Xen or any other virtual machine with the permissions to perform VMI-based operations. The VMI application that performs the analysis does not run directly in the hypervisor.
There are two forms of virtual machine introspection: asynchronous and synchronous. The main difference between these operation modes is how the analysis is started [8]. We speak of asynchronous virtual machine introspection if the analysis is started by external events, such as timers that periodically retrieve the system state. This is useful to regularly observe the system state and retrieve information that is in memory for a longer time frame, such as the list of running processes. Synchronous virtual machine introspection is triggered by sensitive operations in the control flow of the monitored system, e.g., software breakpoints. This can be required to retrieve ephemeral information that resides in main memory only for a short period, such as the parameters of function calls.
In this paper we use the term performance impact to quantify the overhead induced on the execution in the production virtual machine due to VMI-based operations. The overhead is mainly caused by the fact that the production virtual machine is paused for the analysis, which has three different reasons. The first one concerns the case when the production virtual machine and monitoring virtual machine share the same physical CPU core. In this case, the production virtual machine gets paused when the monitoring system is running. This overhead factor can be reduced by using an additional CPU core or by minimizing the amount of time required for the analysis.
The second reason is that the production virtual machine must be paused to prevent that the analyzed data structure (e.g., the process list) is modified during the analysis. If the analysis were performed on a running system, memory contents changing concurrently to the analysis could result in incorrect information. In order to obtain reliable information, it is, therefore, necessary to perform the analysis in a non-modifying state in order to retrieve a consistent view of the system state. A common approach to address this problem is to take a snapshot of the contents in main memory and then run the analysis on it. However, taking a snapshot of the main memory may also require to pause the production virtual machine. Klemperer et al. [9] addressed this problem by implementing a snapshot mechanism that uses a copy-on-write strategy.
The third reason is that the production virtual machine is paused due to synchronous VMI-based mechanisms, e.g., when a breakpoint is invoked and the control flow of the production virtual machine traps to the monitoring virtual machine. The impact on the performance of the production virtual machine, in this case, depends on how often the monitored virtual machine is interrupted and the time how long the monitored system is paused for the analysis.
While the first two causes of performance overhead have adequate solutions, the third reason is an open problem that severely limits the applicability of synchronous VMI-based monitoring in practical use cases. In this paper, we focus on how to minimize the performance overhead in this third case and discuss mechanisms to enable efficient VMI-based monitoring. In detail, we discuss an approach that reduces the overhead of process-bound system call monitoring. Minimizing the overhead of synchronous VMI-based monitoring is important to implement VMI-based security solutions in production environments that tolerate only a minimal performance impact.
The contributions of this paper are: -a presentation of different approaches that aim to minimize the performance impact of synchronous virtual machine introspection; -a proof-of-concept implementation that minimizes the performance impact of process-bound monitoring; -the evaluation of the proof-of-concept implementation.
The outline of the paper is as follows: Sect. 2 introduces the most important technologies used by virtual machine introspection and the most common mechanisms of synchronous virtual machine introspection. Section 3 discusses approaches on how to minimize the overhead of synchronous VMI-based monitoring. In Sect. 4, we discuss a prototype implementation that aims at minimizing the monitoring overhead of process-bound system call tracing. In Sect. 5, we measure the performance gain of the prototype. Section 6 discusses related approaches and Sect. 7 concludes the paper.

Virtual Machine Introspection
This section describes the technologies that are used for virtual machine introspection and synchronous monitoring.

Hardware Requirements
Synchronous virtual machine introspection requires that the CPU is able to trap to the hypervisor in order to monitor the execution of a virtual machine. For example, the hardware virtualization instruction set of Intel processors (Intel VT-x) allows trapping to the hypervisor (VM-Exit) when certain sensitive instructions are invoked in a virtual machine [7,20]. The virtual-machine control structure (VMCS) defines for each virtual machine in which case it should trap to the hypervisor (VMX root mode). While the hypervisor handles the event, the virtual machine that invoked the sensitive operation is paused.

Hypervisor Support
The hypervisor has an important role for virtual machine introspection. For asynchronous introspection, the hypervisor must give the monitoring virtual machine the permissions to access the main memory of the production virtual machine. This can be done for example by mapping the memory pages from the production virtual machine into the address space of the monitoring virtual machine. For synchronous monitoring, the hypervisor needs to forward the information about traps to the monitoring virtual machine so that it can run analysis operations. The concepts presented in this paper are mostly hypervisor independent. However, the focus of this paper is on the design of the Xen architecture.
Xen [1] is a bare-metal hypervisor that supports the hardware virtualization instruction set of Intel. The relevant components for VMI-based operations using The numbered items illustrate that synchronous tracing with breakpoints requires two iterations, the first to intercept the breakpoint and singlestep the original instruction, the second to re-insert the breakpoint and resume the production VM.
Xen are depicted in Fig. 1. The Dom0 is the virtual machine with the most permissions and can access the main memory of all virtual machines running on this hypervisor. Thus, most VMI-based approaches implement the VMI application in Dom0. However, with the Xen security modules (XSM) it is possible to grant other virtual machines than the Dom0 the permissions required to perform VMIbased operations [17,18]. This has certain advantages and can be useful in cloud computing when it is necessary to restrict the VMI permissions of a monitoring virtual machine to the production virtual machine of a specific user.
Asynchronous VMI-based operations can be implemented with Xen by mapping the relevant memory pages of a virtual machine to the monitoring virtual machine, e.g., the Dom0. The functions that map memory pages are implemented in the libxencontrol (libxc) library and use Xen hypercalls to instruct the hypervisor. For synchronous VMI-based operations, sensitive operations in the production virtual machine cause a trap to the hypervisor. Then, the monitoring module of the hypervisor becomes active and sends the corresponding VMI event via the event channel to the VMI application in the Dom0/monitoring virtual machine. While the event is processed in the monitoring virtual machine, the production virtual machine is paused.

VMI Applications
LibVMI [13,14] is the defacto standard library that helps to build VMI-based monitoring applications. LibVMI provides the API to perform VMI-based operations on Xen and KVM. This includes, for example, functions for reading and writing content from the main memory and functions for translating virtual and physical addresses. In addition, LibVMI supports synchronous monitoring with Xen. For this purpose, it provides primitives for listening on VMI-events generated by the Xen hypervisor.

Basic Synchronous VMI Methods
The main advantages of synchronous virtual machine introspection are that it can be used to monitor the control flow of virtual machines and to extract ephemeral information such as parameters of function calls. The mechanisms of synchronous monitoring utilize the concepts of hardware virtualization that allow trapping to the hypervisor under certain conditions, e.g., when sensitive instructions are invoked. In the following, we explain the most common VMIbased monitoring techniques: breakpoints, the monitoring of the write access to CPU control registers, and system call tracing.

Breakpoints.
A common approach to monitoring the control flow using virtual machine introspection is to insert software breakpoints, by using the INT3 instructions that trap to the hypervisor when they are invoked. One approach to implement a software breakpoint mechanism for virtual machine introspection requires the following eight steps. First, the software breakpoint is inserted at the function that should be monitored, either by changing the content in memory or by manipulating the page tables 1 . Second, the production virtual machine invokes the function and the CPU traps to the hypervisor. Third, the hypervisor handles the trap caused by the INT3 instruction, creates a VMI monitoring event and sends it via the Xen event channel to the virtual machine handling the event. Fourth, the monitoring virtual machine gets active and the VMI application receives the new event. Fifth, the VMI application analysis the state of the production virtual machine and re-inserts the original instruction. The VMI application finishes the investigation and tells the hypervisor that the production virtual machine should perform a single-step and execute the original instruction. Sixth, the CPU is configured to run the original instruction in the context of the production virtual machine in single-step mode. Afterward, the CPU traps again to the hypervisor. Seventh, steps four and five are repeated with the difference that the VMI application inserts the original breakpoint. Eight, the VMI application tells the hypervisor to resume the production virtual machine.
In total, this procedure requires switching twice between the monitoring virtual machine and the production virtual machine. First, to handle the breakpoint, and second, to reinsert it after the original instruction is executed in single-step mode. Since a software breakpoint actually traps first to the Xen hypervisor and is then handled by a monitoring virtual machine, eight VM Exit/Entry-based context switches are required to handle a single breakpoint (see Fig. 1).
Also, since the VMI application is usually implemented as a userspace application in monitoring virtual machine, additional context switches between the kernel of monitoring virtual machine and the VMI applications are required. The actual number of required context switches depends on many factors, such as the scheduler's decision and how many processes are active in the monitoring virtual machine.
Access to Control Registers. Modifications to the control registers of a CPU can change the general behavior of the system. Hence, observing the changes is a valuable mechanism of virtual machine introspection. The most common example is the monitoring of modifications of the CR3 register. In the Intel architecture, the CR3 register holds a pointer to the directory table base (DTB) of a process. Whenever a process is dispatched by the scheduler of the operating system, it updates the contents of the CR3 register to point to the DTB of the next process. Thus, monitoring changes of the CR3 register can be used to be informed whenever a new process becomes active.

System Call Tracing
Software breakpoints can be used to monitor the invocation of system calls. To do so, the breakpoint is set on the first instruction of the system call handler function. Since the operating system kernel is used by all processes, this means that all invocations of system calls of all processes are monitored when a breakpoint is set on a system call handler. Thus, if only the system calls of one process should be monitored, this approach can have unnecessary high overhead.
There are two strategies to obtain the system call invocations of a specific process. The first one is to use post-processing. This means that the monitoring application itself filters the events in order to obtain only those system calls that belong to a process with a specific process identifier (PID). This, of course, does not have any positive impact on the performance at run-time. The second strategy is to additionally monitor modifications of the CR3 register, which indicate that a different process in the production virtual machine is being dispatched. Every time a new process is dispatched, the breakpoints can be inserted/removed from main memory so that the tracing is only active when specific processes are running. Sentanoe et al. [16] call this approach process-bound monitoring.

Improving the Performance
The methods of synchronous virtual machine introspection have a severe performance impact on the production virtual machine. One big part of that problem is the long chain of subsequent operations that are required to handle a VMI event in the hypervisor and the monitoring virtual machine while the production virtual machine is paused. Those steps are processed for each VMI event, even if the VMI application just disregards the event because it does not match a certain filter.
The processing of a breakpoint in the VMI application can be organized into three processing layers. On the lowest layer, the breakpoint is handled with the steps explained in Sect. 2.4. Then on the layer above, the VMI-events of a breakpoint are filtered and processed in the VMI application, e.g., when only breakpoints of a specific process should be monitored VMI events can be filtered based on the value stored in the CR3 register. On the highest layer, additional data from the production virtual machine, e.g., the parameters passed to a function are extracted.
In the following, we discuss possible approaches that aim at minimizing the VMI performance overhead. All of these approaches achieve this goal by reducing the time needed by the data extraction routine and the time during which the monitored virtual machine is paused.
Pre-filtering of Events: In some cases, only a few of the VMI events are necessary for a specific use case. For example, in the case of process-bound monitoring, many events are generated by changes in the CR3 register. However, only events for changes that either write the value of the DTB of the process to be monitored to CR3 or replace that value with a different value are required to enable/disable the breakpoints. All other CR3-based events (e.g., when a context switch from a not monitored to another not monitored process occurs) don't need to be observed. Nevertheless, in the usual approach to monitor CR3 events, a context switch to the monitoring virtual machine is required in all cases, even if the VMI application simply ignores irrelevant events.
One approach to solve this problem is to filter events at the hypervisor level. For example, CR3 events that are captured because of the production virtual machine changing the register from oldDTB to newDTB could be forwarded to the userspace VMI application only if either oldDTB or newDTB match the DTB of the process to be monitored.
Breakpoint Mechanism: A fast and efficient breakpoint mechanism helps to minimize the tracing overhead. Since breakpoints can occur frequently, small optimizations can make a big difference. Hence, the implementation of breakpoints should be as optimized as possible so that it requires as little time as possible.
Moreover, the breakpoint mechanism should be implemented into the hypervisor. This eliminates the need to switch to the monitoring virtual machine after a single-step operation was executed as the restoration of the original instruction is independent of the VMI application logic. So instead of switching to the  Pre-processing of Events: Another approach to reducing the number of context switches of VMI-based monitoring is to integrate parts of the analysis into the hypervisor so that it is not necessary to always switch to the monitoring virtual machine. However, hypervisors should be designed to be very minimal and contain only the functions to manage virtual machines. Integrating additional VMI code would violate this principle. Thus, we propose to run only very basic information extraction routines that are necessary to access ephemeral data in the context of the hypervisor. Complex analysis of this data should still occur in the context of the monitoring virtual machine. With that approach, it should be possible to minimize the time when production virtual machine must be paused, i.e., the time to extract ephemeral data. The extracted data must then be sent to the VMI-application in the monitoring virtual machine so that it can process the data asynchronous to the execution of the production virtual machine.

Prototype
In the following, we describe how the performance impact of process-bound VMI-based monitoring can be minimized by implementing basic pre-filtering primitives into the hypervisor. The implemented architecture is depicted in Fig. 2. This goal of process-bound monitoring is to decrease the monitoring overhead, especially, when the production virtual machine runs many applications that are not relevant to trace for the analysis. As described in Sect. 2.4, one way to implement process-bound monitoring is to intercept write access to the CR3 register and en-/disable the breakpoints based on the fact which process becomes active. The problem of this approach is that the operating system updates the CR3 registers frequently. Hence, depending on how often the operating system changes the active process, many VMI-events must be processed by the VMI application. However, most of those events are irrelevant, because the VMIapplication does not need to be informed about context switches between two processes that are not monitored.
To tackle that problem, our prototype implements a basic pre-filtering mechanism in the Xen hypervisor. The VMI application can configure the filter so that only CR3 events are forwarded to the VMI application when the current or the new value of the CR3 register matches the DTB of the monitored process. The filter can be configured at run-time using hypercalls from the monitoring virtual machine. To achieve that, we extend the hypercall API of Xen to activate or disable the monitoring and set values of CR3 events that should be forwarded. With this approach, we eliminate the need for many context switches to the monitoring virtual machine.
The changes to the Xen hypervisor are essentially the extension of the hypercall and the filtering mechanism. Both modifications require only a few lines of code in the Xen hypervisor. Additionally, the libxc used by the VMI application must be extended to support the introduced parameter of the hypercall. Finally, the VMI application must be extended to use the new CR3 filtering.

Evaluation
In this section, we evaluate the performance improvement of our prototype implementation for process-bound monitoring. In our evaluation, we examine whether our prototype that pre-filters VMI events in the hypervisor can help to reduce the performance overhead. For that purpose, we use a similar use case as Sentanoe et al. for the Sarracenia Honeypot [16], which aims to reconstruct the inputs and outputs of an attacker in a bash terminal session. Hence, for this use case, it is sufficient to monitor only the read, write and exec system calls invoked by the main bash process of an SSH session.
We obtain the performance impact of tracing by measuring the time that is consumed by extracting the zip file of the jansson library 2 and compiling the source code. Both the extracting of the zip file and the compilation process require the invocation of many system calls. Most of them are not important for monitoring the attacker's behavior as they belong to the compilation process. Thus, process-bound monitoring of the main bash process that only starts the extraction and compilation should help to reduce the monitoring cost. In our evaluation, we run this use case while monitoring with four different mechanisms: The measurements are executed on an HP Elitebook 820 G4 with an Intel(R) Core(TM) i5-7200U CPU @ 2.50 GHz processor. The production virtual machine has one CPU core and 256 MB of main memory. The VMI application is running in the Dom0 and is implemented to monitor the read, write, open, close and exec system call of the bash process of an SSH session that runs the extraction and compilation. The monitoring of the system calls is implemented by placing a breakpoint on the corresponding system call handler function.
The results of our measurements are depicted in Table 1. The second row shows the average execution time of the compilation process that was executed ten times. The execution time is measured in the production virtual machine using the Linux time command. We use the elapsed real-time to quantify the run-time of the unpacking and compilation process over all ten iterations. The third and fourth rows provide the number of monitored system calls and CR3 events over all ten iterations.
At first glance, the approach of process-bound monitoring of system calls appears to be beneficial in order to reduce the impact of monitoring. However, the measurements of monitoring method three show that the required monitoring of write access to the CR3 register in the VMI application almost annihilates this positive effect. This is because the CR3 events are more frequent than system call events. Nevertheless, we measure a slight improvement in performance. The results of the method four show that implementing basic analysis mechanisms into the hypervisor can significantly reduce the impact of VMI-based monitoring. Based on method four we can estimate the overhead for a single CR3 event by dividing the overhead through the number of CR3 events. From this calculation, we get an overhead of about 11 ms per CR3 event. This long time period can be explained with the different layers in which the event is processed (see Fig. 1).
To sum up, the process of pre-filtering CR3 events decreases the run-time of the execution with process bound tracing from 148.92 s to 8.31 s, which is about 18 times faster. Additionally, these measurements show that integrating basic VMI primitives into the hypervisor can reduce the performance overhead and help to make VMI applicable in production environments. Hence, there is ample room for performance improvement in the current Xen mechanisms for VMI that needs to be addressed by future research. To further improve the performance of synchronous VMI-based monitoring, we, therefore, suggest that more VMI monitoring primitives should be implemented in the Xen hypervisor. Nevertheless, by adding VMI-based functionality to the hypervisor, an additional access control mechanism should be added to the hypervisor. Additionally, if code is loaded to the hypervisor, it must be verified that it does not affect the reliability and security of the overall system.

Related Work
The field of dynamic control flow instrumentation has a long history and in the last decades has been many different approaches that tackle the performance overhead. In the following, we want to discuss the most important related approaches. DTrace [3] is a tracing framework for the Solaris operating system. One of the main objectives of DTrace is to have no impact on the performance when the tracing is not implemented. This is implemented by inserting nooperation (nop) instruction at possible probing points. When the monitoring is activated, the nop instructions are replaced by jumps to the analysis routines. The monitoring actions can be implemented by users in the D programming language.
Our prototype approach follows the same approach as the eBPF filters in the Linux kernel [4,12], which have a similar concept as DTrace. The core concept of eBPF is to run application-specific tracing code in the high privileged Linux kernel in order to monitor functions in user and kernel space. This approach helps to minimize (synchronous) context switches to the user space that stalls the execution. The features of eBPF filters are very advanced and the Linux kernel is already able to run small programs, whereas our prototype for VMI-based tracing currently only allows pre-filtering tracing events in the Xen hypervisor.
Tuzel et al. [19] analyzed the performance impact of VMI-based monitoring techniques in detail. The focus of their paper is to analyze to which extent VMI-based monitoring techniques can be detected from the production virtual machine. The result of their conducted study is that VMI-based monitoring in their setup is not stealthy to applications running in the production virtual machine. The focus of our paper is to decrease the monitoring overhead to make it applicable in production environments but not to make VMI-based monitoring completely stealthy.
Klemperer [9] proposed to use copy-on-write-based snapshots in order to perform the analysis on a non-changing system state. For this purpose, they extended the KVM hypervisor and hook instructions that alter memory pages during the normal execution. If a page is modified they copy the original version of the page for the snapshot. After the analysis is finished, the snapshot gets deleted and changes to memory pages are not monitored anymore. This approach is probably most effective when larger parts of main memory must be analyzed. For the extraction of function call parameters when a breakpoint is invoked this approach is too expensive.
Drakvuf [11] is a VMI framework that uses libvmi and Xen and is mainly designed for dynamic analysis of Windows malware. It uses the altp2m approach [10] to implement software breakpoints. The advantage of this approach is that instead of replacing the original instruction with the INT3 instruction in memory, it creates a new memory page with the software breakpoint. Depending on whether a process in the production virtual machine is reading from that page or executing an instruction the memory page is swapped in the page tables. The rVMI framework designed by Pfoh and Vogl [15] is similar to Drakvuf. However, it uses a patched version of KVM and Qemu instead of Xen 3 . The communication between the VMI application and Qemu is established via the QMP interface of Qemu.
Westphal et al. [21] define a VMI monitoring language that supports the most common methods for VMI. Their prototype is implemented for the VMware KVM hypervisor. Similarly to our proposed approach, they execute VMI analysis scripts in the domain of the KVM hypervisor.
Bushouse et al. [2] implement VMI-based monitoring into Linux containers running in the Dom0 of Xen instead of running the VMI-application in a monitoring virtual machine. This should not have any positive beneficial impact on the performance.

Conclusions
By implementing a pre-filtering approach in the hypervisor, we significantly reduce the impact of VMI-based monitoring on the production virtual machine. Our prototype reduces the monitoring overhead for process-bound monitoring by a factor of 18. We, therefore, conclude that more complex event processing in the hypervisor, such as the implementation of a breakpoint mechanism, can help to solve the problem of minimizing the performance impact of VMI-based tracing.