A Parallel Clustering Analysis Based on Hadoop Multi-Node and Apache Mahout

The conventional procedures of clustering algorithms are incapable of overcoming the difficulty of managing and analyzing the rapid growth of generated data from different sources. Using the concept of parallel clustering is one of the robust solutions to this problem. Apache Hadoop architecture is one of the assortment ecosystems that provide the capability to store and process the data in a distributed and parallel fashion. In this paper, a parallel model is designed to process the k-means clustering algorithm in the Apache Hadoop ecosystem by connecting three nodes, one is for server (name) nodes and the other two are for clients (data) nodes. The aim is to speed up the time of managing the massive scale of healthcare insurance dataset with the size of 11 GB and also using machine learning algorithms, which are provided by the Mahout Framework. The experimental results depict that the proposed model can efficiently process large datasets. The parallel kmeans algorithm outperforms the sequential k-means algorithm based on the execution time of the algorithm, where the required time to execute a data size of 11 GB is around 1.847 hours using the parallel k-means algorithm, while it equals 68.567 hours using the sequential k-means algorithm. As a result, we deduce that when the nodes number in the parallel system increases, the computation time of the proposed algorithm decreases.

1.
Introduction Big data is a combination of large amount, substantial, and multiple formation data created from varied and separated data sources. Researchers and scientists think that big data is one of the most important subjects in computer sciences nowadays [1]. Sensors, sites of social media, hospital annals, and several new foundations are beyond the phenomenon of big Data [2]. A data warehouse cannot deal with the whole dataset because of its vast size [3]. Also, conventional algorithms are incapable of dealing with such enormous amounts of data, so they are not efficient for analysing them [4]. The traditional k-means clustering algorithm [5,6] is not sufficient to manipulate the massive amount of data. Hadoop and Map Reduce tools can be used for dealing with such data [7].
Apache Hadoop Ecosystem is a technology that makes it possible to capture vast amounts of data about each client member and store these data over a considerable time scale [8]. It introduced the concepts of automatic parallelism and distribution. The system of parallel clustering can reduce the large size of data by gathering data that have the same characteristics over the task of data separation to the nodes that are connected in parallel [9]. They are used to speed up the clustering process for large scale datasets. Some researchers [10] suggested a parallel k-means clustering algorithm by using the mahout Application Programming Interface (API) to speed up the clustering process for big datasets. They defined a virtual parallel system with 8 GB of RAM and four threads. The results showed that the proposed implementation improved the computation complexity of each iteration from O (nk) for traditional k-means to O (nk /p) by separating the complete dataset into (p) subsets. The contribution in this paper is to deal with big data by proposing a parallel k-means clustering algorithm, through designing a real server-client model with a Hadoop multi-node system using Intel® Xeon ® CPU E7-286022.27 GHz x6 processor with llvm pipe (LLVM 7.0,128 bits) graphics, 20 GB of memory, and 133.7 GB, Java1.8.11-JDK, Ubuntu-64-bit-18.04.2 disk. This is assigned as a Name Node. Pentium ® Dual-core CPU E5300@2.60 GHz x2 processors, each with a 101.6 GB disk, 4 GB of memory, and a Ubuntu-64 bit-18.04.2 disk, are allocated as Data Nodes. The nodes are connected using a Transmission Control Protocol (TCP) based protocol. Also, we used Apache Hadoop-2.5.3 (64-bit) version with JDK 1.8.11 to compile the source code.

Related Works
The continued growth of the size of datasets has led to traditional clustering methods, but their limits resulted in parallel clustering. In this section, we show some associated clustering algorithms and Hadoop"s performance prediction.
In an earlier work [11], the authors introduced an improved method for Parallel K-means called IPKMeans that uses the k-d tree for the parallel pre-processing stage. Both IPKMeans and k-d trees are run using MapReduce and Hadoop. IPKMeans is faster than PKMeans because it works with one Map and more reducers in parallel, where the I/O overhead is decreased to 2/3. This was applied to a large dataset and an initial centroid; many reducers caused high speed while keeping very close accuracy. In another article [12], the authors introduced a new system called iiHadoop (incremental iterative) to increase calculations on the small segment of data that is pretentious by vicissitudes instead of all the data. This system recovers the presentation for the Map and reduces responsibilities. In another study [13], researchers depended on a parallel framework for the algorithm of the Partitioning Around Medoids (PAM), which is one of the clustering algorithms. This algorithm is divided into sub-tasks that run in parallel. The outcomes appeared to significantly reduce the computational complexities of the algorithm with an excess in the processing cores number. In another article [14], the authors developed a two-factor clustering-based system to process massive amounts of data. The first factor was an attempt to build a random pattern from specified cluster numbers to be used as a centroid in individual clusters. The second factor was included in the parallel schema of the clustering algorithm for increased speed with high stability and quality results. In a later work [15], the authors used tools such as Hadoop and k-means algorithms to analyze and predict data, which is about

Sugheer and Yousif
Iraqi Journal of Science, 2021, Vol. 62, No. 7, pp: 2431-2444 2433 26 GB of editions from the sixteenth-century to 2014. They found the age group of authors who wrote most of the books in a certain amount of time. They published more than 3,000 books, which is the number of books published in 2000 by authors who were more than 60 years old. In another study [16], the author proposed a method of robust Hadoop Cloudera, which depended on data tubes to analyse any type and scale of data. This method can be used to detect stocks that are likely to increase in value. The US stocks chosen for predicting profits depend on Yahoo economics data. The researchers used Hadoop to store and process big Datasets and Spark for predicting the stocks. Depending on Yahoo Finance historical data, they found that the Coefficients in a trouble are 0.0817,0.1282,0.0,0.086,0.0,0.0675,0.11,0.0438,0.0046,0.0,0.0, and 0.0, while the regression model intercept value is -0.097938396250894261. They concluded that the linear regression model is not appropriate to predict stocks returning profits from the high dimensionality data because the Average Mean Error equals to 1.95%.

HDFS and Mahout framework
Hadoop file system is an essential part of the Hadoop Ecosystem, and it is a file method intended for storing massive datasets through the manifold node of the product hardware [17]. Where the data files are stored on the HDFS as blocks, the version of Hadoop used in this model was 2.x. Therefore, the default size of each block on HDFS was 128 MB. The Mahout is an exposed machine learning programs group, which is fundamentally a Java archive set [18]. The defined Mahout"s core algorithms are counting clustering, classification algorithms [19], design mining and text removal, regression, dimensionality reduction, and batch based cooperative sifting track on the topmost of the Hadoop stage over the Map-Reduce frame [20].

Dataset
The used dataset is a group of files in a Comma Separated Values (CSV) file format. The dataset comprises a large amount of data on the healthcare plans available through health insurance. The sizes of these files were different. The purpose of analysing health insurance data was to help the insured persons to choose what is financially appropriate for them in terms of medical and surgical expenses. These files can be loaded from local systems to the Hadoop file system (HDFS). The "local system" in this model means a personal computer that contains a large dataset.

Proposed Model and Work Flow
The large quantity of data requires high cost for computational operations, while when applying the clustering techniques on the scale size of data, high quality was achieved. The clustering is helpful for analyzing the data. As a step of preprocessing for some learning tasks, using clustering can improve

Sugheer and Yousif
Iraqi Journal of Science, 2021, Vol. 62, No. 7, pp: 2431-2444 2434 the accuracy of prediction during obtaining much information about data [21]. Figure-1 explains the whole system for analyzing big data.

Design Stage
The design stage is a crucial stage in the proposed model, which includes the hardware components represented by the master node and two slave nodes and their configurations. The software requirements were set as installing Hadoop single node clustering, Hadoop multi-node clustering, and Mahout and Ubuntu installation. Finally, the connection between the master and slave nodes is performed.

CSV to Vectors Stage
Before processing the data in Mahout, the data must be uploaded to HDFS. The following steps were applied: 1. Start running the Hadoop server in the Master node. The overview of the master node "noor-Virtual Machine-9000" shows that it is active now. Other details of Hadoop windows are shown in Figure-3. 2. Make directories to store the csv-files in the HDFS file system. 3. Copy the data file from the Linux file system (Ubuntu) to the HDFS file system by using the copy command bin/$hdfs dfs -put /…path to file in local system/ /…path to Hadoop file system. Figure-4 shows an example of how data is saved as blocks in HDFS, where a Rate_BUF.csv is one of the healthcare dataset files with the size of 842 MB; therefore, the number of blocks to store in HDFS is seven, as described below: 842 MB ÷ 128MB = 6.578 The division result shows that six out of all the seven blocks stored 128 MB, while the last block stored the remaining (74 MB).

Sugheer and Yousif
Iraqi Journal of Science, 2021, Vol. 62, No. 7, pp: 2431-2444 2436   The process of writing the generated blocks into HDFS multi-nodes, where the size of the dataset is 11 GB stored into more than one file of different sizes, is shown in Figure-6. According to the size of each file, the number of generated blocks will be 107 blocks (from Block0 to Block106), and it will be saved randomly in the HDFS of the two defined data nodes, as shown in the following steps: A. The client node sends a request to the name node (noor-Virtual-Machine). The name node will provide the map where data is and where data should be saved in the HDFS. The name node has saved only the metadata of all data nodes. B. The name node checks the ready data nodes and returns the IP addresses of the available data nodes. C. The client node will then check if the specified data node is ready to write blocks randomly on the specified IP addresses of the data nodes (noor-slave01, noor-slave02). The replication factor is one, so only one copy of the data will be saved on the ready data nodes. This is due to the proposed model in which one rack with one name node and two data nodes is introduced. D. Finally, an acknowledgment message is sent to the client node by the data node, which tells that the writing process is completed, and in turn, an acknowledgment message is sent to the name node by the client node, which tells that the blocks are written successfully. Finally, the metadata of the name node will be updated. The process of reading the saved blocks from the HDFS multi-node for processing steps is presented in Figure-7. The number of blocks that will be read is 107 blocks (from Block0 to Block106) that are saved randomly in the HDFS of the two defined data nodes. The following steps are applied for the mechanism of reading blocks: A. The client node sends a read request to the name node (noor-Virtual-Machine). The name node will provide the map of where data is written in the HDFS. B. The name node checks the data nodes that have the required blocks and returns the IP addresses of the available data nodes. C.
The client node will then check if the specified data node is ready on the data nodes' (noor-slave01, noor-slave02) for the specified IP addresses. D. Finally, the data node sends the required blocks to the client node and then sends an acknowledgment message to the client node, telling that the reading process is completed, and in turn, the client node sends an acknowledgment message to the name node, telling that the blocks are read successfully. After that, the metadata of the name node will be updated. The job of Hadoop is to change the data from CSV format to the Hadoop sequence-file format. The Hadoop sequence-file is a structure of flat-file data containing (key, value) couples in binary. The mapper of Hadoop service is the input reader for parsing input key and value. As the default format of the Hadoop input reader is a text input anywhere, each text line signifies a record. However, this is not appropriate for the CSV format because technical provision requests to cross many appearances. Thus, a reader and partitioner of the custom input record are required for the proposed process. The reader of the input record gathers text from the input file until it reaches the identified record sign end. The mapper needs "key, value", where the value is a text resulting from the reader of an input record, while the key is adjusted to the location in the record"s file in order to extract and pass them to the reducer. Finally, the reducer obtains these (key, value) couples and writes them into a Hadoop sequence file format to be processed using Mahout Commands. The following is the mahout command to convert the data into the sequence file: $mahout seqdirectory -i /path to Dataset file in hdfs/ -o /path to output file in hdfs/ -ow

Analysis Stage (parallel k-Means clustering algorithm)
When the data are loaded to the Hadoop file system, one of the denominations below must be selected to improve the speed and the scalability of big data analysis: 1.
Clustering algorithm of the single machine (Shared Memory System): it is operated in one computer and used only its sources, and includes the sample-based and dimension-reduction techniques.

2.
Clustering algorithm of the multi-machine (Distributed system): it is operated in multi (computer/server) and can run in one computer and access other sources. It includes the parallel clustering and Map-Reduce based clustering. This technique deals with a large dataset. In this paper, a multi-machine is used for implementing the clustering algorithm. The k-means algorithm can be implemented in two modes: Serial mode: When the k-means algorithm is implemented in the serial mode, the initial value of k centers of clusters is chosen randomly; after that, the data files are loaded from HDFS. Then the algorithm computes the distance between all the data points and the initial centers of clusters, and it allocates the data point to the cluster that has the nearest center to it. When the first round finishes recomputing the new centers of the clusters, the algorithm repeats the process until it reaches the number of iteration specified by the user, or to no change in the value of clusters centers. Finally, all the generated clusters are written to HDFS. Algorithm-1 below lists the work of serial k-means clustering. Algorithm 1: serial k-means clustering

Input:
Sequence files Sequence.vec; initial centers K; and Threshold X Output: Group of clusters Step19: Until X || K no change; \\ repeat until satisfying threshold or stable K reached End Parallel mode: There are two categories for the parallel algorithm, which are the control-parallel and data-parallel. In data-parallel, the extensive dataset is divided into small datasets, and parallel computing is conducted on each set of data. In the control parallel, diverse computing is conducted with each small dataset [10]. In this paper, data paralleling is used for implementing the parallel kmeans algorithm because it has good scalability and it is appropriate for processing large datasets. First, the algorithm splits the large dataset into the small datasets after reading them from HDFS. For each dataset, it is processed in a single data node which is used to compute the distances between the clustering center and each data point. When all data points are allocated, the new clustering centers are computed in the master (name node). Then new centroids are re-assigned to the data nodes. This algorithm finishes when the specified number of repetitions has been met or no change in the centroid value. Furthermore, the number of the data node matches the subset"s number, and the master node communicates to the data nodes. This communication plays a significant part in parallel mode. Algorithm-2 below represents the steps of the parallel k-means clustering algorithm implemented using Java code. Algorithm 2: parallel k-means clustering Input: sequence files sequence.vec; Canopy centers K; and Threshold X. Output: K clustering centers with dictionary file (ID, term).  Step27 : Until X|| no change K; \\ repeat until satisfying the condition End Finally, the cluster dumper is a utility provided by Mahout and used to read the contents of the resulted files of the k-means algorithm and store it as a text file. The following command shows how we can read the outcomes of the algorithm by using the dictionary file: $mahout clusterdump -i /path to/clusters-*-final -dt sequencefile -d /path to dictionary file -of TEXT -p /path to clusterdpoint -o /path to the output file in the local system -b -n Where -i means input file path, -d is the dictionary file path, -p is the points of cluster, -o is the output file path, -b is the substring to print, -n is the number of words in top terms.

6.
Results and Discussion The experiments are based on the initial number of clusters and the run time for both parallel and sequential k-means clustering. The defaulting k-means algorithm is run to find four clusters. Throughout our experiments, we set bounds to several active nodes. For sequential execution, the whole algorithm is tracked as a single processor on a single node. For parallel execution, the number of the active processor is restricted by the number of nodes. Figure-8 shows the execution time when loading the data from the local system to the Hadoop system for both the sequential and parallel implementations with different data sizes. In a single node, the time spent to load 5 GB was 11.9 minutes, while the three-nodes required 0.98 minutes. Moreover, when loading 11 GB in multi-node, the time spent was 2.02 minutes, whereas 27.3 minutes were spent at the single node. The time was increased in both cases because of the increases in the size of the dataset, but time at the single node was increased more than ten times as much as that at the multinode.   Figure-9 show the total time for clustering the datasets that have 11GB with all details for both sequential and parallel implementations.   There are several attempts to choose the best k (number of clusters) for analysing 11 GB of data. In this paper, we chose the best k, depending on the resulting clusters. This means that when we select a number for k, the cluster outputs should be accurate and can be analysed. Hence, when the number of clusters equals two, as in Figure-10.b, there are some undesired words, such as y, 0, and x. When k = 7, good topic terms were discovered ( Figure-10.a). However, when k = 10 and higher, undesired words arose ( Figure-10.c), and the clusters were inaccurate. Whereas, n means the number of words that appear in the text file. c. the output of k-means when the k=10,n=10 Figure-11 shows the performance of the K-means algorithm for the clustering of 11 GB of data. In both serial and parallel modes, the time required was increased when the number of clusters was increased. This is because the increased distance computations between the centers of clusters and their elements. When we selected k = 2, the time spent to run the algorithm mentioned above in the parallel mode was 1.7 hours, while in the sequential mode it was 65.367 hours. Also, when we selected k = 4, the time required in the sequential mode was 68.567 hours, while in the parallel mode it was 1.846 hours. In the same way, when we selected k = 10, the time required in the parallel mode was 4.556 hours, while in the sequential mode it was 81.683 hours. Figure 11-Performance of parallel and sequential k-means algorithm with changing clusters number Table-3 shows the seven clusters resulted after applying the parallel k-means clustering algorithm with some of their contents (some attributes are selected manually from the files mentioned in Table-1). The one cluster contains several features as those in the other clusters, but different that is distinguishing each cluster. These features can be adopted by individuals and small businesses to choose a suitable health insurance offer. Families or individuals with excellent income are classified within the cluster1, while those with limited income are classified within the cluster5 or cluster7. For example, cluster1 provides benefits such those for dental accidents, dialysis, denture adjustment, and disease management programs. Also, it is appropriate for people who do not prefer smoking and have no conditions for their ages, as well as, for example, when the metal used for dental treatment is gold. Furthermore, the participants can have insurance as a primary subscriber with one dependent (adult or child). While, cluster3 provides benefits such as acupuncture, root canal therapy for children, and blood test services. Also, it is appropriate for persons who are tobacco smokers, their ages range between 20-40 years, and when the metal used for the dental treatment is silver. Furthermore, the participants can have insurance as a couple and with one dependent (only one child).


Fruit tree disease classification system using generative adversarial networks

Received Jul 3, 2020 Revised Dec 10, 2020 Accepted Dec 26, 2020 Smart farm refers to a farm that can remotely and automatically maintain proper growth and management of crops and livestock by integrating technology with agriculture. Currently, smart farms are concentrated in the field of smart horticulture, and although spreading research is being conducted in limited spaces. In addition, it is difficult to obtain a sufficient amount of data to be used for learning, and there is a problem that data imbalance occurs because it is difficult to obtain a similar amount for each class. In this paper, we propose a method to amplify a small amount of data and to solve the problems of imbalance data by using a feature that can learn to mimic the data of a generative adversarial network. The proposed method can create dataset of various crops and also show high hit rate. Dataset generated from crops would be used to solve problems of data imbalance by learning.

INTRODUCTION
Smart farming is evolving with the application of the 4 th industrial revolution technology. It refers to a technology that enables remote control as well as services such as automation and intelligence by integrating new technologies such as ICT, IoT, big data, cloud, and AI into the growth and environment of crops or livestock [1][2][3]. The need for smart farms is deeply linked to global climate change and food shortages as populations grow. Food demand is on the rise due to an increase in the world's population, but there is a growing shortage of people who will grow because of the shrinking and aging population. In other words, as the population increases, urbanization reduces the area of crop cultivation, and farmers on the production site are aging. Even in Korea, as of 2017, the average age of farmers' chief executives is 67, and this average age is only increasing with time. As a result, smart farms are becoming increasingly important. However, smart farms are concentrated in the field of smart horticulture, and although they are recently spreading, research is being conducted in limited spaces, such as facilities. In addition, it is difficult to obtain a sufficient amount of data to be used for learning, and there is a problem that data imbalance occurs because it is difficult to obtain a similar amount for each class [2].
In this paper, we propose a method to amplify the amount of data through generative neural networks and to solve data imbalance between classes by generating data of various classes. The proposed technique improves the speed by reducing the amount of computation by preprocessing the data and highlights the features to help learning. The preprocessed data uses a generative neural network to generate new data and filters out the low quality data through filtering for data integrity. In case of using the proposed technique, it is possible to amplify a small amount of data to generate a sufficient amount of data for learning, and to apply it to an existing system as it is. Based on these advantages, the proposed method can be used to compensate for the limitations of the recent smart farm.

RELATED RESEARCH
In this chapter, we analyze the region of interest extraction method and the generative neural network used in preprocessing and the problems that can be caused by data imbalance.

Generative adversarial networks (GAN)
GAN is a paper written by Ian Goodfellow in 2014 and various studies are being conducted based on it [4][5][6][7]. The generative hostile neural network learns and produces the result through the competition of two neural networks as the name of the hostile neural network. In (1) represents the formula of the generative host neural network In the above formula, x~Pdata (x) represents an image generated by the probability distribution of the actual image, and x~Px (z) represents an image generated using noise. D (x) is a discriminator, with a value between 0 and 1 indicating the probability that the image is real. In the case of D (G (z)), the image generated by the creator is distinguished through the discriminator, which also has a probability between 0 and 1 that the image is real. To maximize the above equation, the value of D (G (z)) should be close to 1 and the value of D (x) should be close to 1. In this way, constructors and discriminators learn by solving the Minmax problem.

Crop region of interest (RoI)
The region of interest means an area of interest on the image. When the object is detected or detected by processing an image [8][9][10], the detected area may be referred to as a region of interest [11]. There are three main reasons for specifying a Region of Interest. The first is to remove unnecessary images of the area around the object, and the second is to reduce the amount of computation and resources. Finally, the accuracy of learning can be improved. By eliminating unnecessary information in advance, only necessary parts are used for learning, which increases accuracy.
The most representative ones in crop region of interest may be referred to as RGB separation and contour extraction. In the case of RGB separation, a region of interest can be obtained by specifying a range of channels and outputting pixel values within a range by using properties having different values for each channel of an image. Contour extraction means extracting a point where the brightness of the image changes from a low value to a high value or vice versa. Contour detection is a technique to find the pixels corresponding to the contour, and calculates the slope based on the calculation of the partial differential operator [12][13].

Data imbalance
If there is a large difference in the amount of data that each class has in the data, it is said that there is a class imbalance. Most of the data in the real world has this data imbalance problem, and it rebalances the classes by sampling again before training the model. Undersampling is selecting parts of a lot of data, such as the image on the left, and aligning them toward the smaller data. Undersampling can reduce execution time by reducing the size of the data set, but it is also possible that the useful data is not extracted or biased to one side [14][15][16][17]. Oversampling involves copying less data and fitting it toward more data. In the case of oversampling, the data set has more oversampling, so learning is better. In other words, it performs better than undersampling. However, there is a possibility of overfitting when the same data is extracted repeatedly [18][19].

Problems and solutions
Problems between smart farms and data include: It is difficult to obtain enough data to be used for learning, and even the provided data has an adverse effect on learning due to unbalanced data per class [20][21][22][23]. Based on these problems, this paper proposes a method of amplifying the amount of data through a generative antagonistic neural network and generating data of various classes to solve the data imbalance between classes. Using a generative adversarial network not only amplifies a small amount of data to generate enough data for learning, but also has the advantage of using oversampling with less redundancy of the data used.

SYSTEM DESIGN 3.1. Overall system design
The proposed system can be divided into the data processing stage that manages data preprocessing and postprocessing, and the network stage that defines the model for generating data and classifying images. Figure 1 shows this structure. In the data processing stage, it is divided into preprocessing and postprocessing. In the preprocessing step, image size adjustment, contrast adjustment, and region of interest extraction are performed. Image resizing reduces the size of the image, lowering the memory footprint of the system and reducing computation. The size of the image is changed to the n power of 2 to adjust the size of the image so that it does not give a real value when training the model. Region of interest extraction reduces the amount of computation by reducing data outside the region of interest. Contrast makes the characteristics of the data stand out by making the contrast and hue clearer. Post-processing involves a data filter that either stores the results or filters the data. The data filter filters out low quality data from images generated by genetic adversarial networks to ensure the quality of the generated image. Data storage is responsible for sending data to the classification system when the user gathers more than a predetermined amount. At the network level, the generator is divided into a generator, a discriminator network, and a classification network that divides the image. Genetic adversarial networks take preprocessed data as input and generate new images based on it. In the classification system, the generated images are learned and the actual images are determined based on the generated images. In the image discrimination step, the validity of the proposed system is verified by comparing the proposed system with the existing system. Figure 2 shows the flow chart. The flow of the proposed system proceeds from the size control of the input data. By adjusting the size, it plays a role in speeding up the future work. The resized image is then extracted from the region of interest. This plays a role of improving the speed of learning by reducing the amount of computation later. The final preprocessing is to give the image a contrast. The image is then generated through genetic adversarial networks. The image you create is based on an uninfected image, which allows you to create an image similar to the diseased one from the original. The generated image is saved or discarded according to its quality and is repeated until the image reaches a certain number. When more than a certain number of images are created, the generation stops and the classification model is used to train the classification model.

Crop region of interest design
The region of interest was extracted using some Sobel mask method based on RGB extraction. Figure 3 shows the flow chart. First, the image is searched and the average value is calculated. The average value is calculated using a Sobel mask. Find the average and start checking every pixel. If the average value is not exceeded, the next pixel is checked. If the average value is exceeded, the next pixel is checked to see if the pixel is stored. If not, the pixel is stored and moved to the pixel. At this time, the inspection direction of the pixel proceeds constantly. If the pixel is stored, the search ends and a mask is created from the stored pixels to extract the point.

GAN design
The designed generative adversarial networks is designed based on DCGAN. The constructor uses the existing model of generative adversarial networks, but receives 64x64x3 variables which are the size of the image to be generated as input values [24][25]. Figure 4 shows the model of the discriminator network.

2511
The discriminator was modified according to the image given as input, which has a certain form of leaf, the disease occurring in the leaf, and the size of the image is 64x64x3. The size of the convolutional layer was changed from 5x5 to 3x3 to reduce the amount of computation and save memory. In addition, the activation function is designed to generate images of various classes using softmax rather than sigmoid.

Classification system design
The designed model was designed based on CNN. First, in the input phase, the image is changed to code and the data for each class is adjusted regularly. There are 5 convolutional layers, which change according to the size of the input image. The smaller the size of the image, the smaller the size of the feature, so there is no need to build a deeper convolutional layer. Therefore, the image in this paper is 64x64x3, so 5 convolutional layers and 1 fully-connected layer are used. The convolutional layer is a filter that is characterized by a 3x3 filter and a Max Pooling layer that ignores minor changes. Max Pooling creates a small output image by extracting only the main values from the filtered image. The Max Pooling size is used to reduce the size of the image by half every 2x2. In the fully-connected layer, the image which has repeatedly passed the convolutional layer is extracted only from the feature, and it changes the image into one dimension and distinguishes the image in order to transfer the extracted feature to the precombined layer.

SYSTEM IMPLEMENTATION
To implement the system, the Windows 10 Pro operating system was installed on a computer equipped with an i5-6500 processor, 16 G memory, and a GeForce GTX 1060 graphics card. We also installed CUDA 10.0 to use the GPU and implemented the system using Python3.5 in Jupyter Notebook.

Implement preprocess
The crop region of Interest is responsible for finding key parts of the image before the generative hostile neural network proceeds. First, we search the entire pixel to find the average value, and then search through the pixel values of the image as designed. After searching, it prints and saves only the image inside the newly created mask. Figure 5 shows the results of the region of interest.
In Figure 5, from left to right, the original, crop of interest extraction 1 time and interest region extraction 2 times. As can be seen from the Figure 5, it is necessary to proceed two times in order to see the extraction of the region of interest. This occurs when the average value is calculated. The first time, the region of interest is extracted by calculating the average value of the background, the shadow, and the leaves. Get it. In addition, the shadow is caused in the picture, but this problem occurs when the focus is blurred at the edge of the leaf. Therefore, in this paper, we implemented the region of interest twice using the crop region of interest to concentrate the region of interest. Figure 6 shows the preprocessed image.
In Figure 6, From the left, the original, the extracted region of interest, and the contrasted image. In the last image, you can see that the image of the affected area is more intense in color. The image is then generated through a generative hostile neural network. The image you create is based on an uninfected image, which allows you to create an image similar to the diseased one from the original. The generated image is saved or discarded according to its quality and is repeated until the image reaches a certain number. When more than a certain number of images are created, the generation is stopped and the classification model is used to train the classification model.

Implement GAN
Genetic adversarial networks play a role in learning preprocessed data and generating new images. The generator network has four convolution layers to normalize the output and activation values via batch normalization. The discriminator is implemented with three convolutional layers and one fully-connected layer, and sigmoid is used as an activation function to discriminate images of various classes. Figure 7 is an image created by applying the disease called Black Rot to the blueberry leaves. In Figure 7, the black rot leaves are well formed. In addition, by limiting the region of interest during the pretreatment process it can be seen that the location of the disease is properly seated on the leaves.

Implement proposed system
We implemented and compared the designed system with the existing system. Figure 8 shows the classification per epoch when sorting without preprocessing (system 1) and oversampling by increasing data through rotation (system 2). This graph shows the accuracy comparison.
The axis, epoch, represents 10 times. In Figures 8 and 9, system 1 took a long time to maximize accuracy due to lack of data. System 2 was maximized in the epoch similar to the proposed system, but the accuracy was confirmed to be low. In contrast, the proposed system reached its maximum at a faster rate than system 1 and showed an average 2% accuracy improvement over system 2. Table 1 shows the comparison of accuracy when classified using the test set. Experimental results show that system 1 has a 77.99% probability of lacking a data set, while system 2 has a 91.14% probability that some data sets are small. On the other hand, the proposed system increased the amount of data used for learning using generative antagonistic neural networks, and provided the same amount of data for each class, showing high accuracy of 98.17%.

CONCLUSION
Smart farm refers to a farm that can maintain and manage crop and livestock growth conditions appropriately remotely and automatically by using ICT in agriculture. In addition, it was possible to improve the productivity and quality of agricultural products by creating an optimal growth environment based on data on crop growth information and environmental information. However, crops to which smart farms are currently applied are limited to horticulture or house crops, and there is a problem in that a large cost is required to introduce smart farms. In addition, the provided data may show a big difference from the actual data, and since the amount of data is different, there is a problem that the data imbalance problem must be solved in order to learn.
In order to solve this problem, This paper proposes a method that learns the data through genetic adversarial networks and generates data similar to the actual data by using the input data. In order to reduce the resources used when learning data, this system proposed a preprocessing method to reduce the amount of computation and speed it up. We then used genetic adversarial networks to increase the amount of data used for learning and to resolve data imbalances by class. This enabled them to generate data for smart farms across a wide range of crops, increasing the accuracy of learning by 98 percent. Future research should generate data that can be used not only in the classification system used in this paper but also in logistics and automation systems. As a result, it is considered that various smart farms can be constructed.


extfieldsofstudy:
Computer Science
Medicine

text:
Consistent Two-Flow Network for Tele-Registration of Point Clouds

Rigid registration of partial observations is a fundamental problem in various applied fields. In computer graphics, special attention has been given to the registration between two partial point clouds generated by scanning devices. State-of-the-art registration techniques still struggle when the overlap region between the two point clouds is small, and completely fail if there is no overlap between the scan pairs. In this paper, we present a learning-based technique that alleviates this problem, and allows registration between point clouds, presented in arbitrary poses, and having little or even no overlap, a setting that has been referred to as tele-registration. Our technique is based on a novel neural network design that learns a prior of a class of shapes and can complete a partial shape. The key idea is combining the registration and completion tasks in a way that reinforces each other. In particular, we simultaneously train the registration network and completion network using two coupled flows, one that register-and-complete, and one that complete-and-register, and encourage the two flows to produce a consistent result. We show that, compared with each separate flow, this two-flow training leads to robust and reliable tele-registration, and hence to a better point cloud prediction that completes the registered scans. It is also worth mentioning that each of the components in our neural network outperforms state-of-the-art methods in both completion and registration. We further analyze our network with several ablation studies and demonstrate its performance on a large number of partial point clouds, both synthetic and real-world, that have only small or no overlap.

INTRODUCTION
Shape registration is a long-standing problem with a large variety of methods proposed over the last decades. The registration of partial shapes is significantly more challenging than complete shapes, particularly when the overlap between the parts is small. Popular methods, like RANSAC that match between three or four points, perform well when the overlapping base is large [1], [2], [3], [4], [5], but completely fail when there is no overlap between the two partial shapes. In the case of partial matching, the information theoretic explanation is that the lack of sufficient information in the scans leads to a family of plausible completions, which in turn results in failure of traditional rigid registration.
In recent years, neural networks for geometry processing have rapidly emerged and changed the landscape of 3D processing. One notable competence of neural networks is their ability to learn priors of a family of shapes, thus effectively capturing a distribution over possible shapes. Two examples are the reconstruction of a complete shape from a partial input, and the registration of two non-overlapping partial shapes.
We present a rigid registration technique for two partial scans presented in arbitrary initial poses and having little E-mail: cohenor@gmail.com • Ruizhen Hu is the corresponding author of this paper • Our code is available at https://github.com/Salingo/CTF-Net or no overlap at all. This non-overlapping setting has been referred to as tele-registration, and had been attempted in 2D [6] and 3D [7], based on a prescribed feature-conforming prior. In our work, we approach the tele-registration problem using learning tools, in particular, deep learning to encode shape priors. The idea is to jointly train two separate networks on the two tasks of shape completion and shape registration of non-overlapping partial shapes. In training, the networks learn proper priors that allow performing well on these two difficult tasks, rather than treating the tasks independently.
Our key observation is that registered shapes are easier to be completed than each one alone, and complete shapes are easier to be registered, since their overlap clearly increases. Hence, we combine the registration and completion tasks in a way that reinforces each other. In particular, we train the registration and completion networks simultaneously using two coupled flows. One network performs register-and-complete and the other complete-and-register, such that both registration and completion consistencies are maximized. Fig. 1 illustrates our consistent two-flow network (CTF-Net). Note that our completion network only generates information for the missing part, and hence the completions along the two branches of the network can be different, and require a dedicated consistency term to produce canonical completion.
Given two partial point cloud inputs with little or no overlap, our method transforms the partial shapes to canonical positions by learning the prior geometry of its class of shapes, and thus improves the state-of-the-art in terms of completion results on 3D model from the learned class. By comparing our two-flow method to each singleflow and other baselines, we validate that composing two

CTF-NET
Two partial scans Registration Completion Fig. 1: CTF-Net registers pairs of partial scans with little or no overlap. The network is designed to encourage the registration and completion network branches to mutually cooperate to be consistent, and thereby regularizes both the (global) registration and completion problems. Here, we show the result of the combined registration and completion of two partial scans, with little overlap, of a real scan of a toy airplane. flows together effectively strengthens each component. We evaluate our method on synthetic and real-world examples (e.g., RedWood and Pix3D), and demonstrate the superiority of the approach compared to state-of-the-art methods in both global registration (4PCS, DCP) and completion networks (TopNet, PF-Net). In summary, we present a method that addresses the problem of non-overlapping point cloud registration, based on a symmetric neural network that is designed to jointly perform registration and completion, in a way that reinforces each other to establish a new state-ofthe-art.

RELATED WORK
There has been extensive research on general registration problems and in three dimensions in particular. ICP [8] is a widely used algorithm for 3D registration. Some following ICP variants [3], [5], [9] aim at improving from different aspects. Recently, DCP [10] is proposed that revisits ICP from a deep learning perspective.
There is an abundance of SLAM works that deals with registrations and pose estimation at the scene level. For a broad survey, see [11]. An apparently similar idea to our is presented by Yang et al. [12]. Unlike our one-step technique, they refine the registration and completion modules iteratively. Yang et al. [13] also propose hybrid representations for relative pose estimation. These methods, however, match 3D RGB-D scenes rather than point clouds, thus require more information such as color, 360-image. Chen et al. [14] introduce a plane-based descriptor for the point cloud registration with a small overlap. However, for many shapes on object-level, like cars or lamps, it is hard to find a plane surface for matching. Brachmann et al. [15] propose a learning-based method for pose estimation, however, this method is mainly designed for camera localization and it's hard to directly adapt it for shape registration.
In the following, we discuss previous works that are most related to the specific tasks of paired shape registration and partial shape completion on the object level, focusing on deep learning techniques.

Paired shape registration
Recently, there have been research efforts to apply deep neural networks for the task of rigid [16] and non-rigid [17], [18] registration, to offer faster and more robust than classic techniques. Elbaz et al. [19] propose a method that focuses on localizing the close-proximity scanned point cloud in a large-scale point cloud scene. They use super-points to match the corresponding region, and a deep neural network to calculate the transformation between the local and global point cloud. Yew et al. [20] propose a weakly supervised deep learning framework to holistically learn a 3D feature detector and descriptor from GPS/INS tagged 3D point clouds. They use a Siamese architecture that learns to recognize if the given point clouds are from the same location. The correspondences between point clouds are obtained by a learned descriptor vector. Choy et al. [21] propose a framework for pairwise registration of real-world 3D scans. This method contains a 6-dimensional convolutional network for correspondence confidence prediction, and then the pose is estimated and further refined recursively.
Aoki et al. [22] propose PointNetLK, a modification to the classical LK(Lucas & Kanade) algorithm which circumvents the need for convolution on the PointNet representation. This framework for rigid registration is more robust to initialization and missing parts than classic ICP. Wang et al. [10] propose Deep Closest Point(DCP), that revisits ICP from a deep learning perspective. The ICP-style method consists of three parts, that learns the common features of the input point clouds to register them together. Similarly, Yew and Lee [23] propose RPM-Net, which is less sensitive to initialization alignment comparing to the original ICP method. However, RPM-Net assumes that the normal information is given in the point cloud data. Wong and Solomon [24] propose PRNet, a sequential decision-making framework to achieve point cloud registration iteratively. Unlike the methods mentioned above, this method is able to handle partial-to-partial registration, the key is to use a detector to find the points in common between partial views, and keypoint-to-keypoint correspondences. These deep-learning based methods are aiming at detecting key points in the input paired shape, then match and pair them to compute the alignment transformation. These methods assume that the two parts have a significant overlap region that contains a few key points. Huang et al. [7] present a field-guided algorithm that is able to automatically compose the 3D shape given several pieces of it, where the input pieces The architecture of CTF-Net. Given a pair of partial scans, CTF-Net simultaneously predicts the transformation parameters for registration and coordinates of points for completion. The prediction follows a mirrored manner, which performs register-and-complete(R-C) in one flow and complete-and-register(C-R) in another. The R-C flow and C-R flow are denoted by green and purple lines respectively. These two flows mutually reinforce each other by enhancing consistency on their outputs, which is represented by gray lines.
have no overlapping. In our work, we utilize a data-driven approach to enable the registration, without the prescribed feature prior. Given two partial point clouds with little or no overlapping, we do registration and completion at the same time, to allow the network to better align the input shapes.

Partial shape completion
There are an increasing number of works focusing on the partial to complete shape generation, many of which are applied on point cloud representation, since it is strongly related to realistic scenarios, where the point clouds are the raw data coming from 3D acquisition devices. PointNet [25] proposed a deep learning method for point cloud shape, which promotes several learning-based applications on the point cloud, including completion. Yuan et al. [26] proposed a method that generates point clouds in two stages, where the first stage is to use a fully-connected decoder to obtain a coarse resolution point cloud and the second stage generates the final output by a folding-based decoder. Tchapmi et al. [27] introduced a novel decoder for point cloud completion which generates arbitrarily structured point clouds without explicitly enforcing a specific structure. The proposed decoder generates point clouds according to a tree structure where each node of the tree represents a subset of the point cloud. Wang et al. [28] proposed a completion method that contains an up-sampling module that predicts denser results than other completion methods. Huang et al. [29] introduced a method that only completes the missing regions of the input shape and can preserve its details, and successfully addressed the blurring issues caused by the auto-encoder structure. However, they still require the input partial data to cover a significant portion of the surface region of the shape, while in many real cases only a small region of the shape is captured by a single scan.

OVERVIEW
Our tele-registration method consists of a two-flow network, as illustrated in Fig. 2. The idea is to simultaneously learn two networks, one for registration (colored in green in the figure) and one for completion (colored in purple in the figure). Taking a pair of partial shapes as input, the C-R flow branch first completes each partial shape, separately, and then registers the parts, which now have higher overlap, to produce an aligned shape. In the R-C flow branch, the input pair is first registered by the registration network, and then completed by the completion network.
The key of our method is to connect and couple the R-C and C-R flows with two losses: one is a registration consistency loss, which encourages the registration networks in the two flows to predict the same transformation parameters; the other is the completion consistency loss, which encourages the two flows to output similar reconstruction results. As we shall show, the two flows strengthen each other. It should be noted that the complexity of the completion and registration in two flows are different. For the completion network, the one trained in R-C flow is easier, since the input shape is already registered, and contains more overlapping geometric information than the one in C-R flow. Similarly, the task for the registration network in the C-R flow is easier because the input pair shapes are more complete than the one in R-C flow. Thus, our final output is the shape completion results from the R-C flow, and the registration parameters from the C-R flow.
In the following method section, we first describe the input and output of our method, then introduce the registration and completion networks separately, which form the proposed CTF-Net. Finally, we describe the loss functions that enable the two flows to reinforce each other.

Input and output
The input of our method is a pair of partial point clouds, denoted as P 1 and P 2 , with little overlap extracted from the same shape S. The point cloud pairs are centered at the origin point and randomly rotated in 3D. We achieve the goal  Fig. 3: The architecture of the completion network. For each partial shape P, the orientation module first predicts the parameter R o that rotates P into the canonical view and obtains P o . The following generation module predicts the missing part G o relative to P o by an encoder-decoder pair, and the completed shape P o ∪ G o is then rotated back to the original pose, by multiplying the inverse of R o , to obtain the final output S = P ∪ G.
of tele-registering the point cloud pairs and complete the missing part by making use of our key idea to combine the registration and completion tasks in a way that reinforces each other. Our outputs include the completion of those two parts in their original states denoted respectively as S 1 and S 2 , and the relative transformation between them with M 21 denoting the transformation that registers P 2 → P 1 and M 12 denoting the transformation that registers P 1 → P 2 . CTF-Net consists of two flows: C-R flow and R-C flow. Since each flow provides a set of outputs, we use superscripts to distinguish the two sets of outputs.
In the C-R flow, P 1 and P 2 are first passed through the completion network to get the completion results S C-R 1 = C(P 1 ) and S C-R 2 = C(P 2 ). Subsequently, the two completions are passed to the registration network to output their relative transformation M C-R 12 and M C-R 21 . In the R-C flow, P 1 and P 2 , which can come in arbitrary poses, are first passed through the registration network to get their relative transformations M R-C 12 and M R-C 21 . The aligned parts are first combined, before passing to the completion network to get the final results S R-C Our goal is to make sure that the completion and registration outputs of both flows are close to the ground-truth, denoted as S * 1 , S * 2 , M * 1 and M * 2 , and more importantly, making the two flows mutually consistent.

Completion Network
The completion network, as shown in Fig. 3, takes a partial shape P as input and outputs the completed shape S = C(P). Note that similar to [29], for the given partial shape P, we only generate the missing part G and thus the final completed shape is a union of those two S = P ∪ G. Moreover, different from most of the previous works on completion which assume that the input shapes are all wellaligned, our input partial shape P are given in an arbitrary orientation. Therefore, our completion network is composed of two modules: one orientation module and one generation module, where the orientation module rotates P into a canonical pose to facilitate the following generation module and then the generation module generates the missing part to complete the shape.
Specifically, the orientation module takes the partial shape P as input, and outputs the rotation transformation R o , which is then applied to P to obtain the oriented shape P o . The generation module is adapted from the PF-Net proposed in [29]. The original PF-Net assumes that the input partial shape covers a large portion of the whole shape, thus the number of the generated points is far less than the input. However, in our work, since the input is usually a much smaller part of the shape, we set the output point number to equal to the input point number. Note that directly modify the output points of the original PF-Net will highly increase the number of network parameters and lead to huge memory cost and slow training speed, so we modify the parameters of each layer to enlarge the output size gradually. In more detail, we first pass P o to a encoder that extracts the feature of dimension n c = 1920, which is then passed to a decoder that generates the missing part in three levels: primary G p , secondary G s and the final detailed output G o , with the points number of 128, 512, 2048, respectively. G o is then concatenated with P o to form the complete shape. Finally, we multiply G o ∪ P o with the inverse of R o to obtain the final complete shape S = C(P) = G ∪ P.

Registration Network
The registration network, as shown in Fig. 4, takes two shapes, P 1 and P 2 , either complete or partial, and outputs the relative transformation M 12 from P 1 → P 2 by taking P 2 as the anchor. We decompose the transformation M 12 into rotation R 12 and translation T 12 to reduce the complexity of 3D transformation, i.e., the network first rotates P 1 to make it have the same pose of P 2 , and then translates it to align with P 2 .
Specifically, we first pass the input pair through a Point-Net [25] encoder, to obtain the feature vector of dimension n r = 512 for P 1 and P 2 , respectively. The feature vectors are concatenated and passed to a decoder composed of several linear layers, which provides the quaternion parameters that represent the 3D rotation. The quaternion is then converted to rotation matrix R 12 and multiplied with P 1 to obtain the rotated part P 1R . Note that all the quaternions are normalized in each process. Afterwards, P 1R is passed to the PointNet encoder again to obtain the feature vector of dimension R nr (n r = 512). The feature is then concatenated again with that extracted from P 2 and passed to another set of linear layers to output the set of translation parameters (dx, dy, dz), which are then converted to the translation matrix T 12 and multiplied with P 1R to obtain the final

T-Net T-Net
PointNet Encoder

T-Net T-Net
Multiply Multiply Fig. 4: The architecture of the registration network. Taking two paired shapes P 1 and P 2 and considering P 2 as the anchor, the registration network first outputs parameters of a relative rotation R 12 from P 1 to P 2 , then, a relative translation T 12 from R 12 P 1 to P 2 is predicted. The complete transformation is then denoted as M 12 = T 12 R 12 .
registered part. The final transformation matrix is calculated by M 12 = T 12 * R 12 .
In both flows, our registration network also takes those two shapes with the other order and output the relative transformation M 21 from P 2 to P 1 . Furthermore, in the C-R flow, the registration network takes two completed shapes S 1 and S 2 as input. We further apply the transformation on the moving shape, say S 1 , and combine it with the anchor shape S 2 to get the registered full shapes. Note that since the points size of the registered point cloud is twice that of each input partial point cloud, we down-sample it to half of its original points size in order to keep all the point clouds in the flows to have the same size.

Loss Functions
To train this two-flow network, we design suitable loss functions to account for each type of output and also the consistency between two flows. We define the loss function of our CTF-network as: where L c and L r are the completion and registration loss against the ground-truth, and L s is the loss to ensure the consistency between two flows.

Completion loss.
Note that our completion network works for input parts with arbitrary orientation, and we rotate the parts into the canonical view first and then generate the missing part to complete the shape. Therefore, the completion loss is computed for outputs from the R-C and C-R flows against the respective ground truth missing part geometry and rotation transformation as where the weights ω c-r c , ω c-r o , ω r-c c , ω r-c o are set as 1, 3, 0.5, and 1.5, respectively. D emd is the distance measure between the generated part and the corresponding ground truth, which is defined as the mean of the earth mover's distance (EMD) [30] computed for all three generated levels(G p , G s , G o ), and D r is the distance measure between two rotations. In more detail, the rotation matrix is converted to a quaternion Q = q(R). Since a quaternion Q is equivalent to its minus −Q when representing a rotational transformation, we measure the distance as follows : Note that for the same part, for example P 1 , the ground truth missing part is different in two flows. In the R-C flow, P 1 will first be registered and combined with P 2 before completion, so the missing part would be smaller than the one in C-R flow. All the ground truth missing parts are extracted and subsampled from the corresponding ground truth complete shape S * .

Registration loss.
The registration loss is also computed for outputs from the C-R and R-C flows against the respective ground truth transformations as L r = ω c-r where D m is the distance measure between two transformations, which is defined as the D q between their quaternions plus the mean square error between the translations in the x, z, y axis. The weights ω c-r r and ω r-c r are set as 3 and 9, respectively.

Consistency loss.
The consistency loss consists of four components: where L O s , L C s and L R s are the consistency loss defined on the orientation correction, completion and registration in those two flows, while L T s is the consistency loss defined on the relative transformations between two shapes with either one as the anchor. In more detail: where I is the 4 × 4 identity matrix. The weights ω so , ω sc , ω sr and ω st are set as to 3, 1, 3 and 3, respectively by default. For the details about how we choose this set of weights, please refer to the supplementary material.

RESULTS AND EVALUATION
We first introduce the details of our training data generation in Section 5.1, then, we show results of pair registration and partial completion obtained with our CTF-Net and present a set of qualitative results to demonstrate the capabilities of our method in Section 5.2. A quantitative evaluation and ablation studies are described in Sections 5.3 and 5.4. We present results on both synthetic and real data.

Data preparation
Our dataset contains 31, 742 shapes in eight categories from ShapeNet v2 [31]. The training, validating and testing split is similar to the dataset of Tchapmi et al. [27]. Each shape in our dataset is represented as a point cloud which contains 16, 384 points, and all the shapes are normalized into a unit cube centered at the origin.
For each shape S * in the dataset, we crop two parts from it and apply random transformations to get the training pairs (P 1 , P 2 ). Specifically, we first randomly generate two spheres centered at the surface of the bounding sphere of each shape, with the radius sampled from [0.3, 1.3], and then use these two spheres to crop two parts out of the shape. We use this method to simulate the real scan cases when a camera is held and scan around the shape. Note that we set a minimum distance (0.3 in our setting) between each pair of cropped parts to avoid too much overlap between those two parts, where the distance is calculated by the Euclidean distance between the centers of those two parts. We also ensure that each cropped part and the remaining part both have more than N = 4096 points, so that they can be down-sampled to N = 2048 points. Each part is first

Qualitative results
We show results for synthetic shapes on eight categories. To verify the generality of our method, we also test our method on real scans. Fig. 5 shows visual examples of the registration and completion results. The input pair of parts are centered at the origin point, as shown in the first two columns. By taking either part as the anchor, our method is able to transfer the other part to make it align with the anchor well, as shown in the third and fourth columns respectively. We can see that even when there is little overlap between the input parts, like most of the examples in the results, CTF-Net can still register them correctly. Also, the union shapes after registration with different anchor parts are quite similar thanks to the effect of L T s . The fifth and sixth columns show the completed shape for each part, and the last two columns are the corresponding ground truth complete shape. We can see that the completion module in our CTF-Net can work well for parts given in two different orientations, and by solving registration and completion together, the parts do not need to be pre-aligned before completion. Note that we take registration results from the C-R flow and completion results from the R-C flow, mainly because each will improve the performance for the other, and the end output of each flow is more reliable even with the consistency loss. Fig. 7: The prediction results on RedWood (upper 4 rows) and Pix3D (lower 4 rows) datasets. The first column shows the photo of each real model, and the next two columns show the paired partial inputs. The fourth and fifth columns are the registered results with either part as the anchor, and the completion results are shown in the following two columns. The last column shows the corresponding reconstructed meshes provided by these two datasets.

Results on real data
To demonstrate the generality of our method, we use 3D scanners to manually scan several objects from all those eight categories for testing. Specifically, for small objects with detailed textures, in particular, Plane, Car, Watercraft, and Lamp, we place each object on a turnplate and use an Artec Spider scanner for scanning which directly produces the reconstructed model; for other larger objects, we use a Microsoft Kinect v2 scanner to do the scanning, then utilize bundle fusion [32] to obtain the reconstructed model. The reconstruction results are shown in the last column in Fig. 6, denoted as F.
To generate the input pairs, we first normalize the reconstructed full model into a unit box and put it in a virtual environment, and then randomly place two cameras pointing at the model to capture RGB-D images. Two partial point clouds are then obtained by back-projecting the depth images to the 3D reconstructed model, and those two singleview point clouds are taken as input to test our CTF-Net. Note that here we do not use the single-view point cloud directly from the scanner since there exist some affections such as reflect light which could make the single-view point cloud extremely sparse. Fig. 6 shows the results on our real scanned data. Note that the input partial parts can still be correctly aligned even the point clouds are noisy and non-uniformly distributed. For example, although the two parts of the couch shown on the fifth row are quite noisy and covers different region of the couch, our method is still able to successfully align them and complete the missing region in the front.
To further justify the generality of our method, we also tested our method on RedWood [33] and Pix3D [34] datasets, both of which provided the full reconstructed model. Similar to our real scanned data, the input pairs of these two datasets are generated by single-view scanning. Some example results are shown in Fig. 7. For each example, the full reconstructed model is shown in the last column, denote as F, for comparison. We see that although the single-view input data is quite different from the synthetic training data, the predicted registration is still quite accurate and the completion results are also quite reasonable. For example, in the fourth row, our method predicts the missing leg of the table, and in the fifth row, we can notice that the entire chair back is successfully reconstructed. Note that for each real scan example, we slightly rotated the view during visualization to show the single-view inputs more clearly.

Quantitative evaluation
We perform quantitative evaluation of both registration and completion networks by measuring the errors of predicted transformation parameters and reconstructed point clouds.
For registration, we calculate the error for the final output of the C-R flow. Rotational error E θ is calculated as the absolute error between the predicted and ground truth angle in degrees, and Translation error E t is calculated by the L2 distance between the predicted and ground truth translation (in normalized units): where θ denotes the angle derived from the relative quaternion from the predicted quaternion R C-R to the ground truth quaternion R C-R * , and t denotes the distance between the translation decomposed from the predicted transformation T C-R and the ground truth transformation T C-R * . For completion, we also calculate the error for the completion obtained from the C-R flow. Since our completion network generates the missing region without modifying the given part, we use two measures E g emd and E f emd to compute the errors in the generated region and full shape, respectively, where E g emd is calculated as the EMD between the reconstructed and corresponding ground truth point cloud of the missing region, and E f emd is calculated as the EMD between the reconstructed shape concatenate with the input part and ground truth full shape: (10) Table 1 shows the errors of all eight categories. Note that E t , E g emd and E f emd values have been scaled by 10 3 to amplify the error. In most categories, our CTF-Net is able to predict accurate registration together with completion. We observe that the registration error of Lamp is significant higher than other categories due to the ambiguity arising from its strong symmetry. For example, the stands of most lamps are cylindrical, which could lead to the rotational ambiguity. Furthermore, the variety and complexity of Lamp category are very large, e.g., a lamp with multi-fold stand, or a pendant lamp with cluttered accessories. The rotational error of other categories are all lower than 20 degrees; meanwhile, the full shape completion errors of these categories are lower than 3, meaning that the good reconstruction quality is achieved.
Overall, we achieve an average of 15.12 degrees rotational and 5.97 translation error in registration, and 2.40 completion error for the whole shape.

Comparison on registration
We compare the registration results of our method to five other options: 1) A classic registration method 4PCS [4], which can globally register complete or partial point sets; 2) A deep-learning based registration method PRNet [24], which focuses on partial-to-partial registration, with self-supervised learning; 3) A deep-learning based registration method DCP [10], can be seen as state-of-the-art; 4) Baseline registration network which predicts the rotation and translation parameters at once from a single decoder, denoted as BL-Regi. The detailed network structure can be found in the supplementary material; 5) Our registration network alone, denoted as Regi.
The geometry-based method 4PCS is directly tested on our dataset, while the remaining three learning-based methods are trained/tested on the same dataset as ours. The comparison of prediction error is reported in Table 2. We can see that the prediction error of 4PCS is the highest since it assumes that there are certain-level overlap between the input pair, while our data are mainly non-overlap.  The errors obtained using PRNet are lower than 4PCS but higher than others, meaning that PRNet can not works well on our data with little overlap between the input pairs. DCP performs better comparing to our BL-Regi method in rotational error, however, the translation error of BL-Regi is 23% lower. Comparing DCP to our single registration module Regi, we can see that the rotational error is quite close, but the translation error of Regi is 68% lower. The key idea of DCP is to find the correspondences between two point sets, which also fails on our dataset. The baseline registration method, which predicts the rotation and translation parameters simultaneously from the same decoder, has slightly higher rotation error to our registration network, however, the translation error is more than twice as much as ours. This proves that splitting the prediction of rotation and translation could achieve better performance. Last but not least, our method with two complete flows gets the best results comparing to all five other options, which shows the benefit of combining the registration and completion tasks.
In order to assess the advantage of our method on nonoverlap data, we further compare our method with 4PCS, PRNet and DCP on the data with different levels of overlap. To generate testing pairs with a certain overlap η, we modify our data preparation procedure slightly. In more detail, for each shape in our testing set, we randomly generate two spheres with fixed centers (0, 0.75, 0) and (0, −0.75, 0), respectively, and the radius randomly sampled from [0.3, 1.3] to crop two parts from a complete shape. We then calculate the IoU of the cropped parts and keep the ones with IoU in [0.9η, 1.1η]. Finally, we downsample each part to N = 2048 points.
We take the Plane category for testing, and quantitative comparisons of the rotation and translation errors are shown in Fig. 9. We can see that both the rotation and translation errors of almost all the methods keep increasing as the overlap region decreases. Regards to rotation error, 4PCS performs best when the overlap region is 80%, however, as the overlap decreases, the performance of 4PCS dropped significantly and the error is lager than 50 degrees from 40% overlap to 0%.
PRNet and DCP performs more stable than 4PCS, and the rotational error of DCP is slightly lower than ours at 80% and 60% overlap, but CTF-Net keeps the lowest error from 40% to 0%. For translation error, the four methods are quite similar at 80% overlap, however, the error of 4PCS increases rapidly when overlap region decreases. The error of DCP is slightly lower than PRNet but significantly higher than ours at 20% and 0%. PRNet is supposed to be able to deal with a partial-to-partial registration, however, it shows worse  performance than DCP in our setting. The main reason is that PRNet assumes a large overlap between the input pair data and requires a good initial alignment for the ensuing iteration, while for our input data with quite little overlap, they fail to find a sufficiently good initial alignment. We observe that the translation error of our method decreases a bit even the overlap region drop from 80% to 20%, implying that our network has higher capability in learning data with little overlap. Fig. 8 shows the visual comparison of our method to 4PCS, PRNet, and DCP. To highlight the differences in the overlap regions, we rotate all the input pairs and align their first part (shown in blue), and use a lighter color to show the overlap region between the input pair, shown in the second row. From the results, we can see that 4PCS fails at 20% and 0% overlap, since the algorithm assumes that two parts should have a large overlapped region. For the deeplearning based methods PRNet and DCP, we observe that the rotational error of PRNet is large at and 60%, 20% and 0% overlap. DCP produces more accurate rotation angle, however, the translation error is still high. Our method performs stable on different overlap data. We observe that even there is no overlap between the head and tail of the plane, our method is still able to align two parts in the correct position, with a certain margin in the middle. This experiment also shows that our method are robust when the area of the input data variants, which is helpful for the use of real scanned data.

Comparison on completion
We compare the completion results of our method to five other options: 1) TopNet [27], which direct reconstruct the whole shape using a structural decoder; 2) MSN [30], which predicts the whole shape in a coarse to fine manner; 3) The original PF-Net [29], which can be seen as state-ofthe-art; 4) Our completion network alone without orientation module, denoted as BL-Comp; 5) Our completion network alone, denoted as Comp; All the five methods are trained/tested on the same dataset as ours, TopNet and PF-Net are trained using chamfer distance (CD) as proposed in their paper, and the remaining methods are trained using EMD. For fair comparison, we add another two completion quality measures based on CD instead of EMD, denoted as E g cd and E f cd , by substituting D emd by D cd in Equation 9 and Equation 10. Specifically, Note that E g cd and E f cd values have been scaled by 10 4 to amplify the error. The reconstruction error is reported in Table 3. TopNet and MSN predict the whole shape directly, thus we only compute E f emd and E f cd for them. For all other methods, only points on the missing regions are generated, so E g emd and E g cd are also computed. We can see that TopNet gets the highest error E f emd since it doesn't keep the points from the input part and it is trained using CD. For comparison, MSN also predicts the whole shape, but it is trained using EMD, so both E f emd and E f cd are lower than TopNet. PF-Net achieves the lowest error in E g cd and E f cd , since it keeps the input region and is trained using CD. The main difference between BL-Comp and PF-Net is that BL-Comp is trained using EMD measure, thus both E g emd and E f emd are lower than PF-Net. To further improve the results on input part pairs with randomly 3D rotation, we added an orientation module to BL-Comp, denoted as Comp, and all the errors get lower comparing to BL-Comp. Our method obtains the best result in EMD measure, with 76%, 32%, and 51% drop on E f emd comparing to TopNet, MSN and PF-Net, thanks to the consistent two flow network. Fig. 10 shows two examples of different completion methods. We see that the prediction results of TopNet are quite blurred and lack fine details. Similar results can be seen in the fourth column, i.e., PF-Net keeps the original input but the predicted parts are sparse, leading to the inconsistent results. These two methods are trained using CD measure, which is not able to effectively constrain the sparseness of the point cloud. The results of MSN are obviously more compact, however, we can see that there are some outlier points distributed in the blank region. For example, there are some unnecessary points located between the back and the seat region in the second row. Our baseline completion network performs better than PF-Net, but incurs some distortions in the output. The completion method, which includes the orientation module to the baseline completion, predicts points more accurately. Our method achieves the best results comparing to others, and the generated points are evenly distributed in the missing regions. Note that here we take the completion results from C-R flow to compare with the other methods. Fig. 13: Visual example of the input point cloud with outliers. We show original scan, scan with 100 outliers, scan with 100 outliers processed by a filter, respectively.

Stress test
To better simulate the scenario of real scanning, we generate the data with different noise level to test the robustness of our CTF-Net. Specifically, to generate the data with noise level ζ, we randomly sample N × 3 values range in [0, ζ] as the noise, then add the sampled noise to the original input partial pair P 1 and P 2 . We set five different ζ to test our method, and the results are shown in Fig. 11. Note that our original data generated by virtual scanning have the noise level at 0.005. We observe that all the errors increase quite slightly as the noise level increases from 0.005 to 0.09, i.e., E θ , E t , E g emd and E f emd keep lower than 20, 10, 8 and 5 respectively. At noise level of 0.12 and 0.15, which rarely appear in real scanning, the errors increasing slightly faster, but the rotational error keeps lower than 27 degree. The results show that our method is able to maintain low prediction error given noisy data. Fig. 12 shows how the registration and completion errors change with the increase of noise level added to the input parts. The registration of our method is correct even the input partial data are quite noisy, as shown in the examples in the last row. Besides, the missing parts generated by our method are recognizable at all noise levels.
Other than points with random and small noise, outliers can also exist in the captured point cloud data, thus we perform another experiment to show how outliers affect the performance of our CTF-Net. Specifically, as shown in Fig. 13, for each testing point cloud data P, we randomly select K points and each of them is then translated by a random displacement to simulate the outliers. The direction of each displacement is uniformly sampled in SE(3) space, and the length of each displacement is uniformly sampled between [0.1, 0.5]. In this experiment, we set K to be 50 and 100, respectively, to test our method. We denote the processed data as P ol .
Furthermore, since automatic outlier removal algorithms are widely used in many works related to point cloud processing, we also test our CTF-Net on the outlier data after being processed by an outlier filter. Specifically, we utilize the radius outlier removal method from Zhou et al. [35], which removes points that have few neighbors in a given sphere around them, and we use the default parameters setting proposed in [35]. The processed data are denoted as P ol with filter.
The results are reported in Fig. 14. We observe that the errors are relatively high when the outliers are added to the input scans without filtering, especially for E t . Such amount of random outliers will confuse the network when distinguishing the part poses, thus lead to high errors. However, when we apply a filter to the data contaminated with outliers, the errors remain very close compared to those of the results on clean data, meaning that our method is able to perform well on the data cleaned up by a simple and automatic outlier removal filter.

Ablation studies
To justify the network structure and loss functions designed in our method, we perform several ablation studies for our CTF-Net. For network structure, we take single R-C flow and single C-R flow to perform the comparison, the results are shown in Table 4. Since both R-C flow and C-R flow can produce the transformation parameters and completed shapes, we use the same measure as described in Section 5.3. Since our CTF-Net contains twice parameters comparing to each single flow, for fair comparison, we slightly modify R-C flow and C-R flow to make them have comparable parameters to ours, denoted as R-C 1 and C-R 1 . Specifically, we double the output dimension of each layer (except for the last layer) of all the decoders, and then obtain the network with almost twice parameters as the original single flow network. In addition, the registration network in our CTF-Net is trained with both partial point clouds and (predicted) complete point clouds and the completion network in our CTF-Net is trained with both partial input point clouds and (predicted) registered point clouds, so we compare our method to each single flow which trained with augmented data. Specifically, for single R-C flow, we train the completion network with both input partial point cloud and combined point cloud after registration, denoted as R-C 2 , while for single C-R flow, we train the registration network with both input partial point cloud and predicted complete point cloud, denoted as C-R 2 .
From the results, we can see that for single R-C flow, the errors of both rotational and translation is significantly higher than our method, which shows that the direct registration of two non-overlap partial shapes is quite challenging. For the version with double parameters R-C 1 , the errors are quite similar to R-C flow. For R-C 2 , we observe that training the completion network with additional input can improve the results of completion comparing to single R-C flow, but the errors are still higher than our method. The errors of single C-R flow are also higher than ours, the registration error of C-R 1 is lower than C-R, but the completion error increases. For C-R 2 , the error of registration is  lower than C-R and C-R 1 , however, the errors are higher than CTF-Net. The experiments show that the performance of either single flow is not comparable to our CTF-Net, even with the doubled parameters, and only the combination of two flows with consistency loss can actually improve the results. We design four consistency losses after combining the R-C flow and C-R flow together: L O s , L C s , L R s , L T s . To show the effectiveness of our consistency losses, we perform comparisons of our method with and without each loss term, as shown in Table 5. By comparing the errors of the last five columns, we validate that the consistency losses are able to make the two flows strengthen each other, which leads to the lowest error. We can see that both registration and completion errors are higher without the reconstruction consistency loss. Removing the parameter consistency loss results in the registration between two parts being less consistent, and thus leads to larger errors in prediction.

DISCUSSION AND FUTURE WORK
We presented CTF-Net for tele-registration of two partial point clouds. Our method excels where the surfaces represented by point clouds have little or no overlap. The success of our method is attributed to the competence of neural network to learn a prior of a class of shapes, which allows predicting complete shapes from partial observations and registering non-overlapping parts. The key architectural design comes in the form of consistency between the registerand-complete and the complete-and-register networks.
The consistency of the two network flows encourages the two networks to predict reliably and performs surprisingly well. Given that parts P 1 and P 2 are disjoint, then their completions S 1 and S 2 should agree without conditioning. We expect S 1 to trivially agree with P 1 in the overlap regions, but the rest of S 1 is ambiguous. Here, we also expect S 1 to agree with P 2 in the overlapping region, which is a hard task. Thus, the completion task by itself is ambiguous, while the registration with no overlap is ill-posed. Solving both tasks jointly and consistently, however, makes the problems more well-posed and less ambiguous. Limitations of data-driven methods are naturally carried over to our method. The range by which our method works well is directly derived by the training data and the capacity of the network, which includes the variety of the expected input geometries and their mutual pre-disposition. As shown in Fig. 15, our CTF-Net may fail in registration prediction if the input pairs are sufficiently ambiguous. The upper row shows a table example, where the matching parts have very similar geometry and the ground truth degree of overlap between two input parts are ambiguous, which can cause different registration results. If we assume that the overlap region is larger, the two parts will be placed closer to each other as the result shown in the figure. For the chair shown in the second row, since the complete version of P 1 that captures the leg and seat information can be symmetric, so the back part (complete version) of P 2 can be put on any side to make it a valid chair, which introduce the ambiguity of registration between P 1 and P 2 . As shown in the figure, the back in placed on the right side instead of back side of the seat. Another limitation is that our dataset is generated by sphere cropping, since the ground truth for each network is obtained in this way. However, a more natural way to create training data might be by back-projecting depth images to 3D. Also, currently, our method is trained category-by-category, which may limit the generality. We are planning to enable cross-category training by improving the network structures and loss functions in the future. Further, our current implementation considers the coordinates of the point cloud, and ignores the additional attributes that can be associated with scanned data, like normals and colors. We leave this for future research.
Another direction is to leverage the tele-registration of our method to align the parts from different objects. For example, given a chair without legs and a partial cabinet, one may register them to form a new object that have both the functionality of chair and cabinet. This part-based modeling requires well designed dataset and network.
In the context of continuous scanning (e.g., using Kinect Fusion), data fusion works better in short sequences than on longer ones, where tracking and registration errors accumulate. In our tests, we found CTF-Net to complement scanning performance by successfully stitching shorter bursts of fused results. This emphasizes the importance of teleregistration for disjoint scans in the context of autonomous (or semi-autonomous) robots and drones.