\section{Introduction}

Cloud computing is an emerging computing paradigm that enables businesses and individuals to access computing resources as a service. These attractive features of cloud computing have gained significant attention from cyber attackers, and it is now extensively been exploited by adversaries for launching ``\textit{stealth}'' attacks or even setting up phishing Websites \cite{c15}. The study reported in this paper, alongside the security reports from various public cloud providers \cite{c2}, highlights that the cloud is continuously being \textit{weaponized} for launching attacks. According to the 2017 Microsoft Security Intelligence Report \cite{c2}, attackers can \enquote{weaponize} the cloud to create their own Virtual Machines (VMs) or gain access to or compromise other VMs. It has been reported that the Google Cloud Platform (GCP) has been abused for launching Denial of Service attacks or intrusion attacks\footnote{https://www.gcppodcast.com/post/episode-47-cloud-abuse-with-swati-and-emeka}. According to a report \cite{c3}, the \enquote{abuse and nefarious} use of {\it Infrastructure-as-a-Service} (IaaS) is one of the most critical security concerns in the cloud. Hackers are attracted IaaS because they can create computing accounts on cloud services with false identities, which enables them operate in obscurity.

This paper reports the findings from interviews conducted with 75 security professionals and ethical hackers who participated in the DEF CON and Black Hat professional hacking conferences.
We observed that these professional hackers often employ common strategies to abuse the cloud platform for its resource-efficient features in order to remain stealth and silent while probing target machines, collecting victim data, discovering vulnerabilities, and launching attacks. This paper makes the following key contributions:

\begin{itemize}
    \item A generalization of cumulative interview data about how cyber attackers utilize cloud platforms for setting up their attack environment while remaining stealth.
    \item Several recommendations for how cloud service providers can impede the misuse of their platforms, drawn from enumerating cyber attackers' generic attack steps.
\end{itemize}
\section{Dynamic Model of Abuse Patterns}
\label{sec:recommend}
As part of a larger project, in which one of the research objectives was to understand and analyze the attackers’ mental models when launching cyber attacks, our research team conducted a series of face-to-face and open-ended interviews with 75 professional hackers and penetration testing experts. We collected the interview responses interactively, and transcribed them into use cases during analysis.
One of the major observations from those interview responses was the abuse of cloud platform to set up the attack environment. We deduced the common patterns of cloud abuse and presented them as a dynamic model of the cloud-based attacks through an activity-flow diagram in Figure \ref{fig:platform}. \\
The Figure \ref{fig:platform} illustrates the common steps an attacker follows to abuse the cloud and performs the exploration, probing, and enumeration for constructing and transmitting malicious payloads to the target in order to launch attacks. The figure depicts the common ground patterns of use cases for creating a VM on IaaS cloud model, setting it up for performing reconnaissance, scanning, and gaining access to the target and launching an exploit, while remaining untraceable throughout various phases of the attack. The participants explained that they could set up a VPS (Virtual Private Server), a multi-hop VPN (Virtual Private Network), or encrypt the communication channel on the cloud VM (virtual machines). They could then install the necessary tool sets on the VM. By running all these tools using the computing resources of the VM on the cloud, attackers could craft and launch a SQL-injection attack, or propagate malware or run malicious scripts on a target machine, or even install software like a \textit{keylogger} on the target machine to obtain credentials.

\begin{figure}
\begin{center}
  \includegraphics[width=0.7\linewidth, height=6.5cm]{AttackModeling.png}
  \caption{A dynamic model and activity-flow diagram for cloud abuse.}
  \label{fig:platform}
  \vspace*{-0.35in}
\end{center}
\end{figure}

\section{Recommendations}
\label{sec:recommend}

The Google Cloud Platform has adequate tracking methods to monitor if any of the VMs are running any suspicious processes to circumvent the resources or network quota\footnote{https://cloud.google.com/compute/quotas}. AWS has GuardDuty \cite{c12} to trace malicious activities on the cloud, alongside Amazon Security Inspector \cite{c8} to perform security assessment. However, from our participants' data, we learned that attackers are erudite enough to evade the prevailing security measures on the cloud. In this section, we enumerate possible countermeasures and mitigation strategies to minimize the likelihood of cyber attackers abusing the IaaS cloud. The recommendations concern three aspects of an automated and effective defense mechanism including: 1) Prevention, 2) Detection, and 3) Recovery.
\vspace{-0.1in}
\subsection{Prevention}
The prevention recommendations are to impede the abuse of the IaaS model in the first place. The following technical and regulatory actions could help prevent attackers from setting up their environments to abuse the cloud.

\noindent \textbf{Account Authentication.} To create a cloud account, an attacker needs only valid credit card information. However, several Websites offer fake credit card numbers\footnote{https://www.getcreditcardinfo.com}, so it is easy for attackers to create cloud accounts anonymously.  Thus, a thorough background check should be employed before activating a cloud account. Multi-factor authorization might also make attackers efforts more difficult.

\noindent \textbf{Tracking Multi-hop VPN.} Cyber attackers frequently reported use  of multi-hop VPNs, which require significant network bandwidth. Thus, the amount of network bandwidth can serve as an indicator to detect suspicious activities. So, tracking the network quota and performing proactive monitoring when certain VMs start to exhaust the network quota can help to detect suspicious accounts.

\noindent \textbf{Setting Up Firewalls and Update Software.} Cloud providers can enforce use of firewalls and encourage updating all software on VMs to latest security patches in order to protect against known vulnerabilities.

\noindent \textbf{Trusted Software Repositories.} Cyber attackers need certain tools for scanning and reconnaissance activities. Cloud providers can enforce downloading of software tools from a trusted repository in order to prevent the use of adversarial tools on their platforms.

  \vspace{-0.1in}
\subsection{Detection}
Detection-based approaches should be implemented as a complimentary step to prevention-based approaches. That way, abusive activities can be identified as they take place.\\
\textbf{Proactive Forensic Analysis.} Public cloud providers can employ automated, periodic, and randomized forensic analysis of the Virtual Hard Drives (VHDs) to identify suspicious accounts. VHDs are the virtualized equivalent of the hard drives of VM instances. VHDs contain information on OS, files and folders, and processes.

\noindent \textbf{Anomaly Detection.} Cloud platforms can benefit from utilizing automated anomaly detection tools and techniques in order to detect any suspicious activities in real time.

   \vspace{-0.07in}
\subsection{Recovery}
Once an IaaS cloud instance is abused to launch an attack, it is important to ascertain  (1) the ways it was abused and (2) reinstate the system states (both attacker and victim)

\noindent \textbf{Blocking Malicious Traffic.} Setting up network rules to block any outgoing traffic from the attack VM.

\noindent \textbf{Isolating VMs.} Perform forensic analysis of the VHDs.

\noindent \textbf{Enforce Blacklisting.} Identify the users of the accounts that perform malicious activity and blacklist them.

\iffalse
Once the IaaS cloud model is set to be abused and identified by the tools and techniques listed at the detection stage, it is necessary to reimpose the VM instance from being used as an attack platform through:

The cloud providers may even identify the suspicious users, and after background checks, can enforce blacklisting.

\fi

\section{Conclusion and Future work}

\label{sec:conclude}
This study highlights the needs for further research on how to prevent, detect, and recover from cyber attacks through cloud platforms. The detection methodologies presented here heavily depend on monitoring VM instances. Cloud service providers such as Amazon implement the Shared Responsibility Model\footnote{https://aws.amazon.com/compliance/shared-responsibility-model}, which makes users of cloud resources responsible for the safety measures \textit{inside} the cloud; whereas, the provider is responsible \textit{for} the cloud. The users of such platforms can programmatically enable alerts to identify various abuses\footnote{https://aws.amazon.com/blogs/mt/automating-processes-for-handling-and-remediating-aws-abuse-alerts/}, essentially making the users accountable for use or abuse of cloud.
The study points to the need for further work on developing a forensics suite and security testing framework \cite{c20} for cloud platforms. In an analogous way to zero-day malware \cite{c30}, it is also important to detect ``{\it zero-day abuse}'' of cloud.
   \vspace{-0.03in}
\section*{Acknowledgment}
This research work is supported by National Science Foundation under Grants No: 1516636, 1723765, 1821560.

   \vspace{-0.03in}


\section{Introduction}
	It is a complicated and laborious task, even for humans, to assemble an IKEA furniture from its parts.
	Without referring to any procedural or external guidance, \textit{e.g. } reading the instruction manual, or watching a step-by-step video demonstration, the task of 3D part assembly involves exploring an extremely large solution spaces and reasoning over the input part geometry for candidate assembly proposals.
	To assemble a physically stable furniture, a rich set of part relations and constraints need to be satisfied for a successful assembly.

	There are some literature in the computer vision and graphics fields that study part-based 3D shape modeling and synthesis.
	For example,
	\cite{chaudhuri2010data,shen2012structure,sung2017complementme} employ a third-party repository of 3D meshes for part retrieval to assemble a complete shape. Benefiting from recent large-scale part-level datasets \cite{mo2019partnet,yi2016scalable} and the advent of deep learning techniques, some recent works \cite{li2019learning,schor2019componet,wu2019pq} leverage deep neural networks to sequentially generate part geometry and posing transform for shape composition.

	Though similar to our task, none of these works addresses the exactly same setting to ours.
	They either allow free-form part generation for the part geometry, or assume certain part priors, such as an available large part pool, a known part count and semantics for the entire category of shapes (\textit{e.g.}, for chairs, four semantic parts: back, seat, leg and arm).
	In our setting, we assume no semantic knowledge upon the input parts and assemble 3D shapes conditioned on a given set of fine-grained part geometry with variable number of parts for different shape instances.

	In this paper, we propose to use a dynamic graph learning framework that predicts a 6-DoF part pose, including a rigid rotation and translation, for each input part point cloud via forming a dynamically varying part graph and iteratively reasoning over the part poses and their relations.
	We employ an iterative graph neural network to gradually refine the part pose estimations in a coarse-to-fine manner.
	At the core of our method, we propose the dynamic part relation reasoning module and the dynamic part aggregation module that jointly learns to dynamically evolve part node and edge features within the part graph.

	Lack of the real-world data for 3D part assembly, we train and evaluate the proposed approach on the synthetic PartNet dataset, which provides a large-scale benchmark with ground-truth part assembly for ShapeNet models at the fine-grained part granularity.
	Although there is no previous work studying the exactly same problem setting as ours,
	we formulate three strong baselines inspired by previously published works on similar task domains and demonstrate that our method outperforms baseline methods by significant margins.

	Diagnostic analysis further indicates that in the iterative part assembly procedure, a set of central parts (\textit{e.g. } chair back, chair seat) learns much faster than the other peripheral parts (\textit{e.g. } chair legs, chair arms), which quickly sketches out the shape backbone in the first several iterations.
	Then, the peripheral parts gradually adjust their poses to match with the central part poses via the graph message-passing mechanism.
	Such dynamic behaviors are automatically emerged without direct supervision and thus demonstrate the effectiveness for our dynamic graph learning framework.

 	\vspace{-2mm}
	\section{Related Work}

	\textbf{Assembly-based 3D modeling.}
	Part assembly is an important task in many fields.

	Recent works in the robotic community \cite{litvak2019learning,zakka2019form2fit,shao2019learning,luo2019reinforcement} emphasize the planning, in-hand manipulation and robot grasping

	using
	a partial RGB-D observation in an active learning manner, while our work shares more similarity with the work in the vision and graphics background, which focuses on the problem of pose or joint estimation for part assembly. On this side, \cite{funkhouser2004modeling} is the pioneering work to construct 3D geometric surface models by assembling parts of interest in a repository of 3D meshes. The follow-ups \cite{chaudhuri2011probabilistic,kalogerakis2012probabilistic,jaiswal2016assembly} learn a probabilistic graphical model that encodes semantic and geometric relationships among shape components to explore the part-based shape modeling. \cite{chaudhuri2010data,shen2012structure,sung2017complementme} model the 3D shape conditioned on
	the single-view scan input,
	rough models created by artists or a partial shape via an assembly manner.

	However, most of these previous works rely on a third-part shape repository to query a part for the assembly. Inspired by
	the recent generative deep learning techniques and benefited from the large-scale annotated object part datasets \cite{mo2019partnet,yi2016scalable}, some recent works \cite{li2019learning,schor2019componet,wu2019pq}
	generate the parts and then predict the per-part transformation to compose the shape. \cite{dubrovina2019composite} introduces a Decomposer-Composer network for a novel factorized shape latent space.
	These existing data-driven approaches mostly focus on creating a novel shape from the accumulated shape prior, and base the estimated transformation parameters on the 6-DoF part pose of translation and scale.
	They assume
	object parts are well rotated to stand in the object canonical space.
	In this work, we focus on a more practical problem setting, similar to assembling parts into a furniture in IKEA, where all the parts are provided and laid out on the ground in the part canonical space. Our goal of part assembly is to estimate the part-wise 6-DoF pose of rotation and translation to compose the parts into a complete shape (furniture). A recent work \cite{li2020learning}
    has
    a
    similar setting
    but requires an input image as guidance.

	\textbf{Structure-aware generative networks.} Deep generative models,
	such as
	generative adversarial networks (GAN) \cite{goodfellow2014generative} and variational autoencoders (VAE) \cite{kingma2013auto}, have been
	explored recently for shape generation tasks. \cite{li2017grass,mo2019structurenet} propose hierarchical generative networks to encode structured models, represented as abstracted bounding box. The follow-up work \cite{mo2019structedit} extends the learned structural variations into conditional shape editing. \cite{gao2019sdm} introduces a two-level VAE to jointly learns the global shape structure and fine part geometries. \cite{wu2019sagnet} proposes a two-branch generative network to exchange information between structure and geometry for 3D shape modeling. \cite{wang2018global} presents a global-to-local adversarial network to construct the overall structure of the shape, followed by a conditional autoencoder for part refinement. Recently, \cite{mo2020pt2pc} employs a conditional GAN to generate a point cloud from an input rough shape structure.
	%

	%
	Most of the aforementioned works couple the shape structure and geometry into the joint learning for diverse and perceptually plausible 3D modeling.
	However, we focus on a more challenging problem

	that aims at generating shapes with only structural variations conditioned on the fixed detailed part geometry.

	\section{Assembly-Oriented Dynamic Graph Learning}
	Given a set of 3D part point clouds $\mathcal{P} = \{p_i\}_{i=1}^N$ as inputs, where $N$ denotes the number of parts which may vary for different shapes, the goal of our task is to predict a 6-DoF part pose $q_i\in SE(3)$ for each input part $p_i$ and form a final part assembly for a complete 3D shape $S=\cup_i q_i(p_i)$, where $q_i(p_i)$ denotes the transformed part point cloud $p_i$ according to $q_i$.

	To tackle this problem, we propose an assembly-oriented dynamic graph learning framework that leverages an iterative graph neural network (GNN) as a backbone, which explicitly conducts sequential part assembly refinements in a coarse-to-fine manner, and exploits a pair of part relation reasoning module and part aggregation module for iteratively adjusting part features and their relations in the part graph.
	Figure~\ref{figure:framework} illustrates our proposed pipeline. Below, we first introduce the iterative GNN backbone and then discuss the dynamic part relation reasoning module and part aggregation module in detail.

	\subsection{Iterative Graph Neural Network Backbone}

	We represent the dynamic part graph at every time step $t$ as a self-looped directed graph $\mathcal{G}^{(t)} = (\mathcal{V}^{(t)}, \mathcal{E}^{(t)})$, where $\mathcal{V}^{(t)} = \{v_i^{(t)}\}_{i=1}^N$ is the set of nodes and $\mathcal{E}^{(t)} = \{e_{ij}^{(t)}\}$ is the set of edges in $\mathcal{G}^{(t)}$.
	We treat each part as a node in the graph and initialize its attribute $v_i^{(0)} \in \mathbb{R}^{256}$ via encoding the part geometry $p_i \in \mathbb{R}^{1000\times3}$ as $v_i^{(0)} = f_{init}(p_i)$, where $f_{init}$ is a parametric function implemented as a vanilla PointNet \cite{qi2017pointnet} that extracts a global permutation-invariant feature summarizing the input part point cloud $p_i$.
	We use a shared PointNet $f_{init}$ to process all the parts.

	We use a fully connected graph, drawing an edge among all pairs of parts, and perform the iterative graph message-passing operations via alternating between updating the edge and node features. To be specific, the edge attribute $e_{ij}^{(t)} \in \mathbb{R}^{256}$ emitting from node $v_j^{(t)}$ to $v_i^{(t)}$ at time step $t$ is calculated as a neural message
	\begin{equation}
	e_{ij}^{(t)} = f_{edge}(v_i^{(t)},v_j^{(t)}),
	\end{equation}
	which is then leveraged to update the node attribute $v_i^{(t+1)}$ at the next time step $t+1$ by aggregating messages from all the other nodes
	\begin{equation}
	v_i^{(t+1)} = f_{node}(v_i^{(t)}, \frac{1}{N} \sum_{j=1}^Ne_{ij}^{(t)} ),
	\end{equation}
	that takes both the previous node attribute and the averaged message among neighbors as inputs.
	The part pose $q_i^{(t+1)} \in \mathbb{R}^{7}$, including a 3-DoF rotation represented as a unit 4-dimensional Quaternion vector and a 3-dimensional translation vector denoting the part center offset, is then regressed by decoding the updated node attribute via
	\begin{equation}
	q_i^{(t+1)} = f_{pose}(v_i^{(0)}, v_i^{(t+1)}, q_i^{(t)}).
	\end{equation}
	Besides the node feature at the current time step $v_i^{(t+1)}$, $f_{pose}$ also takes as input the initial node attribute $v_i^{(0)}$ to capture the raw part geometry information,

	and the estimated pose in the last time step $q_i^{(t)}$ for more coherent pose evolution.
	Note that $q_i^{(0)}$ is not defined and hence not inputted to $f_{pose}$ at the first time step.

	In our implement, $f_{edge}$, $f_{node}$ and $f_{pose}$ are all parameterized as Multi-Layer Perceptrons (MLP) that are shared across all the edges or nodes for each time step.
	Note that we use different network weights for different iterations, since the node and edge features evolve over time and may contain information at different scales.
	Our iterative graph neural network runs for 5 iterations and learns to sequentially refine part assembly in a coarse-to-fine manner.

	\begin{figure*}[t]
		\centering
		\includegraphics[height=0.38\linewidth]{imgs/figure.pdf}
		\caption{The proposed dynamic graph learning framework.
		The iterative graph neural network backbone takes a set of part point clouds as inputs and conducts 5 iterations of graph message-passing for coarse-to-fine part assembly refinements.
		The graph dynamics is encoded into two folds, (a) reasoning the part relation (graph structure) from the part pose estimation, which in turn also evolves from the updated part relations, and (b) alternatively updating the node set by aggregating all the geometrically-equivalent parts (the red and purple nodes), \textit{e.g. } two chair arms, into a single node (the yellow node) to perform graph learning on a sparse node set for even time steps, and unpooling these nodes to the dense node set for odd time steps. Note the semi-transparent nodes and edges are not included in graph learning of certain time steps.}
		\label{figure:framework}
 		\vspace{-1.5mm}
	\end{figure*}

	\subsection{Dynamic Relation Reasoning Module}

	The relationship between entities is known to be important for a better understanding of visual data. There are various relation graphs defined in the literature. \cite{xu2017scene,li2017scene,yang2018graph,chen2019scene} learn the scene graph from the labeled object relationship in a large-scale image dataset namely Visual Genome \cite{krishna2017visual}, in favor of the 2D object detection task. \cite{li2019grains,zhou2019scenegraphnet,wang2019planit,ritchie2019fast} calculate the statistical relationships between objects via some geometrical heuristics for the 3D scene generation problem. In terms of the shape understanding, \cite{mo2019structurenet,gao2019sdm} define the relation as the adjacency or symmetry between every two parts in the full shape.

	In our work, we learn dynamically evolving part relationships for the task of part assembly.

	Different from many previous dynamic graph learning algorithms \cite{wang2019dynamic,zhang2019dynamic} that only evolve the node and edge features implicitly,
	we propose to update the relation graph based on the estimated part assembly at each time step explicitly.
	This is special for our part assembly task and we incorporate the assembly-flavor in our model design.
	At each time step, we predict the part pose for each part and the obtained temporary part assembly enables explicit reasoning about how to refine part poses in the next step considering the current part disconnections and the overall shape geometry.

	To achieve this goal,
	besides the maintained edge attributes, we also learn to reason a directed edge-wise weight scalar $r_{ij}^{(t)}$ to indicate the significance of the relation from node $v_j^{(t)}$ to $v_i^{(t)}$. Then, we update the node attribute at time step $t+1$ via multiplying the weight scalar and edge attribute
	\begin{equation}
	v_i^{(t+1)} = f_{node}\left(v_i^{(t)},\frac{\sum_{j}e_{ij}^{(t)}r_{ij}^{(t)}}{\sum_{j}r_{ij}^{(t)}} \right).
	\end{equation}
	There are various options to implement $r_{ij}^{(t)}$. For example, one can employ the part geometry transformed with the estimated poses to regress the relation, or incorporate the holistic assembled shape feature.

	In our implementation, however, we find that directly exploiting the pose features to learn the relation $r_{ij}^{(t)}$ is already satisfactory. This is mainly caused by the fact that the parts of different semantics may share similar geometries but usually have different poses.
	For instance, the leg and leg stretcher are geometrically similar but placed and oriented very differently.
	Therefore, we adopt the simple solution by reasoning the relation $r_{ij}^{(t)}$ only from the estimated poses
	\begin{equation}
	r_{ij}^{(t)} = f_{relation}\left(f_{feat}\left(q_i^{(t-1)}\right),f_{feat}\left(q_j^{(t-1)}\right)\right),
	\end{equation}
 	where both $f_{feat}$ and $f_{relation}$ are parameterized as MLPs. $f_{feat}$ is used to extract the independent pose feature from each part pose prediction. Note that we set $r_{ij}^{(1)}=1$ in the beginning.

	\subsection{Dynamic Part Aggregation Module}

	The input parts may come in many geometrically-equivalent groups, \textit{e.g.}, four chair legs, two chair arms, where the parts in each group share the same part geometry and hence can be detected via some simple heuristics. We observe that geometrically-equivalent parts are highly correlated and thus very likely to share common knowledge regarding part poses and relationships.

	For example, four long sticks may all serve as legs that stand upright on the ground, or two leg stretchers that are parallel to each other.

	Thus, we propose a dynamic part aggregation module that allows more direct information exchanges among geometrically-equivalent parts.

	To achieve this goal, we explicitly create two sets of nodes at different assembly levels: a dense node set including all the part nodes, and a sparse node set created by aggregating all the geometrically-equivalent nodes into a single node. Then, we perform the graph learning via alternatively updating the relation graph between the dense and sparse node sets.

	In this manner, we allow dynamic communications among geometrically-equivalent parts for synchronizing shared information while learning to diverge to different part poses.

	In our implementation,
	we denote $\mathcal{G}^{(t)}$ as the dense node graph at time step $t$. To create a sparse node graph $\mathcal{G}^{(t+1)}$ at time step $t+1$, we firstly aggregate the node attributes among the geometrically-equivalent parts $\mathcal{V}_g$ via max-pooling into a single node $v_j^{(t)}$ as
	\begin{equation}
	v_j^{(t)} = \pooling_{k \in \mathcal{V}_g}\left(v_k^{(t)}\right)
	\end{equation}
	Then, we aggregate the relation weights from geometrically-equivalent parts to any other node $v_i^{(t)}$
	\begin{equation}
	r_{ij}^{(t)} = f_{relation}\left(f_{feat}\left(q_i^{(t-1)}\right), \pooling_{k \in \mathcal{V}_g}\left(f_{feat}\left(q_k^{(t-1)}\right)\right)\right).
	\end{equation}
	The inverse relation emitted from the node $v_i^{(t)}$ to the aggregated node $v_j^{(t)}$ is computed similarly as
	\begin{equation}
	r_{ji}^{(t)} = f_{relation}\left(\pooling_{k \in \mathcal{V}_g}\left(f_{feat}\left(q_k^{(t-1)}\right)\right), f_{feat}\left(q_i^{(t-1)}\right)\right).
	\end{equation}
	All these equations are conducted once we finish the update of dense node graph $\mathcal{G}^{(t)}$, then we are able to operate on the sparse node graph $\mathcal{G}^{(t+1)}$ following Eq.~(4)~and~(5). To enrich the sparse node set back to dense node set, we simply unpool the node features to a corresponding set of geometrically-equivalent parts.
	We alternatively conduct dynamic graph learning over the dense and sparse node sets at odd and even iterations separately.

	\subsection{Training and Losses. }
	Given an input set of part point clouds, there may be multiple solutions for the shape assembly.

	For example, one can move a leg stretcher up and down as long as it is connected to two parallel legs. The chair back can also be possibly laid down to form a deck chair.

	To address the multi-modal predictions, we employ the Min-of-N (MoN) loss~\cite{fan2017point} to balance between the assembly quality and diversity. Let $f$ denote our whole framework, which takes in the part point cloud set $\mathcal{P}$ and a random noise $z_j$ sampled from unit Gaussian distribution $\mathcal{N}(0,1)$. Let $\mathcal{L}$ be any loss function supervising the network outputs $f(\mathcal{P},z_j)$ and $f^*(\mathcal{P})$ be one provided ground truth sample in the dataset, then the MoN loss is defined as
	\begin{equation}
	\min_{z_j \sim \mathcal{N}(0,1)}\mathcal{L}\left(f(\mathcal{P},z_j),f^*(\mathcal{P})\right).
	\end{equation}
	The MoN loss encourages at least one of the predictions to be close to the ground truth data, which is more tolerant to the dataset of limited diversity and hence more suitable for our problem. In practice, we sample 5 particles of $z_j$ to approximate Eq.~(9).

	The $\mathcal{L}$ is implemented as a weighted combination of both local part and global shape losses, detailed as below. Each part pose $q_i$ can be decomposed into rotation $r_i$ and translation $t_i$. We supervise the translation via an $\mathcal{L}_2$ loss,
	\begin{equation}
	\mathcal{L}_t = \sum_{i=1}^N ||t_i - t_i^*||_2^2
	\end{equation}
    The rotation is supervised via Chamfer distance on the rotated part point cloud
	\begin{equation}
	\mathcal{L}_r = \sum_{i=1}^N \left(\sum_{x \in q_i(p_i)} \min_{y \in q_i^*(p_i)}||x-y||_2^2 + \sum_{x \in q_i^*(p_i)} \min_{y \in q_i(p_i)}||x-y||_2^2\right).
	\end{equation}
	In order to achieve good assembly quality holistically, we also supervise the full shape assembly $S$ using Chamfer distance (CD),
	\begin{equation} \label{equation:chamfer}
	\mathcal{L}_s = \sum_{x \in S} \min_{y \in S^*}||x-y||_2^2 + \sum_{x \in S^*} \min_{y \in S}||x-y||_2^2.
	\end{equation}
	In all equations above, the asterisk symbols denote the corresponding ground-truth values.

	\section{Experiments and Analysis}
	We conduct extensive experiments demonstrating the effectiveness of the proposed method and show quantitative and qualitative comparisons to three baseline methods.
	We also provide diagnostic analysis over the learned part relation dynamics, which clearly illustrates the iterative coarse-to-fine refinement procedure.

	\subsection{Dataset}
	We leverage the recent PartNet \cite{mo2019partnet}, a large-scale shape dataset with fine-grained and hierarchical part segmentations, for both training and evaluation. We use the three largest categories, chairs, tables and lamps, and adopt its default train/validation/test splits in the dataset. In total, there are 6,323 chairs, 8,218 tables and 2,207 lamps. We deal with the most fine-grained level of PartNet segmentation.

	We use Furthest Point Sampling (FPS) to sample 1,000 points for each part point cloud.
	All parts are zero-centered and provided in the canonical part space computed using PCA.

	\subsection{Baseline Approaches}
	Since our task is novel, there is no direct baseline method to compare.
	However, we try to compare to three baseline methods inspired by previous works sharing similar spirits of part-based shape modeling or synthesis. The baseline methods are trained using the same losses and the same termination strategy as our method. We stop training when they achieve the best scores on the validation set.

    \textbf{B-Complement:} ComplementMe~\cite{sung2017complementme} studies the task of synthesizing 3D shapes from a big repository of parts and mostly focus on retrieving part candidates from the part database. We modify the setting to our case by limiting the part repository to the input part set and sequentially predicting a part pose for each part.

	\textbf{B-LSTM:} Instead of leveraging a graph structure to encode and decode part information jointly, we use a bidirectional LSTM module similar to PQ-Net \cite{wu2019pq} to sequentially estimate the part pose. Note that the original PQ-Net studies the task of part-aware shape generative modeling, which is a quite different task from ours.

	\textbf{B-Global:} Without using the iterative GNN, we directly use the per-part feature, augmented with the global shape descriptor, to regress the part pose in one shot.
	Though dealing with different tasks, this baseline method borrows similar network design

	with CompoNet \cite{schor2019componet} and PAGENet \cite{li2019learning}.

	\setlength{\tabcolsep}{6pt}
	\begin{table}[b]
		\small
		\vspace{-0.3cm}
		\begin{center}
			\begin{tabular}{c|ccc|ccc|ccc}
				\toprule
				& \multicolumn{3}{c}{Shape Chamfer Distance $\downarrow$} & \multicolumn{3}{c}{Part Accuracy $\uparrow$} & \multicolumn{3}{c}{Connectivity Accuracy $\uparrow$} \\
				\midrule
				& Chair & Table & Lamp & Chair & Table & Lamp & Chair & Table & Lamp \\
				\midrule
				B-Global & 0.0146 & 0.0112 & 0.0079 & 15.7 & 15.37 & 22.61 & 9.90 & 33.84 & 18.6 \\
				\midrule
				B-LSTM & 0.0131 & 0.0125 & \textbf{0.0077} & 21.77 & 28.64 & 20.78 & 6.80 & 22.56 & 14.05 \\
				\midrule
				B-Complement & 0.0241 & 0.0298 & 0.0150 & 8.78 & 2.32 & 12.67 & 9.19 & 15.57 & 26.56 \\
				\midrule
				\textbf{Ours} & \textbf{0.0091} & \textbf{0.0050} & 0.0093 & \textbf{39.00} & \textbf{49.51} & \textbf{33.33} & \textbf{23.87} & \textbf{39.96} & \textbf{41.70} \\
				\bottomrule
			\end{tabular}
		\end{center}
		\caption{Quantitative Comparison between our approach and the baseline methods.}
		\label{table:performance}
	\end{table}
	\subsection{Evaluation Metrics}

	We use the Minimum Matching Distance (MMD)~\cite{achlioptas2017latent_pc} to evaluate the fidelity of the assembled shape. Conditioned on the same input set of parts, we generate multiple shapes sampled from different Gaussian noises, and measure the minimum distance between the ground truth and the assembled shapes. We adopt three distance metrics, \textit{part accuracy}, \textit{shape chamfer distance} following \cite{li2020learning} and \textit{connectivity accuracy} proposed by us.
	The part accuracy is defined as,
	\begin{equation}
	\frac{1}N \sum_{i=1}^N \mathbbm{1} \left(\left(\sum_{x \in q_i(p_i)} \min_{y \in q_i^*(p_i)}||x-y||_2^2 + \sum_{x \in q_i^*(p_i)} \min_{y \in q_i(p_i)}||x-y||_2^2\right) < \tau_p\right)
	\end{equation}
	where we pick $\tau_p=0.01$.
    Intuitively, it indicates the percentage of parts that match the ground truth parts to a certain CD threshold. Shape chamfer distance is calculated the same as Eq.~\ref{equation:chamfer}.

	\textbf{Connectivity Accuracy.} The part accuracy measures the assembly performance by considering each part separately. In this work, we propose the \textit{connectivity accuracy} to further evaluate how well the parts are connected in the assembled shape. For each connected part pair <$p_i^*, p_j^*$> in the object space, we firstly select one point in part $p_i^*$ that is closest to part $p_j^*$ as $p_i^*$'s contact point $c_{ij}^*$ with respect to $p_j^*$, then select the point in $p_j^*$ that is closest to $c_{ij}^*$ as the corresponding $p_j^*$'s contact point $c_{ji}^*$.
	Given the predefined contact point pair $\{c_{ij}^*, c_{ji}^*\}$ located in the object space, we transform each point into its corresponding canonical part space as $\{c_{ij}, c_{ji}\}$. Then we calculate the connectivity accuracy of an assembled shape as
	\begin{equation}
	\frac{1}{N_{cp}} \sum_{\{c_{ij}, c_{ji}\} \in C} \mathbbm{1} \left(||q_i(c_{ij})-q_j(c_{ji})||_2^2 < \tau_c\right),
	\end{equation}
	where $C$ denotes the set of contact point pairs $\{c_{ij}, c_{ji}\}$ and $\tau_c=0.01$. It evaluates the percentage of correctly connected parts.

	\setlength{\tabcolsep}{2pt}
	\begin{figure}[t]
		\centering
		\includegraphics[scale=0.5, trim={1.3cm, 3.5cm, 1.5cm, 3.2cm}, clip]{imgs/figure-quality-new.pdf}
 		\vspace{-0.6cm}
		\caption{Qualitative Results. Left: visual comparisons between our algorithm and the baseline methods; Right: multiple plausible assembly results generated by our network.}
		\label{figure:quality}
	\end{figure}

	\setlength{\tabcolsep}{2pt}
	\begin{table}[t]
		\small
		\begin{center}
			\begin{tabular}{cccc}
				\toprule
				& Shape Chamfer Distance $\downarrow$ & Part Accuracy $\uparrow$ & Connectivity Accuracy $\uparrow$\\
				\midrule
				Our backbone w/o graph learning & 0.0086 & 26.05 & 28.07 \\
				\midrule
				Our backbone & 0.0055 & 42.09 & 35.87\\
				\midrule
				Our backbone + relation reasoning & 0.0052 & 46.85 & 38.60\\
				\midrule
				Our backbone + part aggregation & 0.0051 & 48.01 & 38.13\\
				\midrule
				Our full algorithm & \textbf{0.0050} & \textbf{49.51} & \textbf{39.96}\\
				\bottomrule
			\end{tabular}
		\end{center}
		\caption{Experiments demonstrate that all the three key components are necessary.}
		\label{table:ablation}
 		\vspace{-0.4cm}
	\end{table}

	\subsection{Results and Comparisons}
	We present the quantitative comparisons with the baselines in Table~\ref{table:performance}.
	Our algorithm outperforms all these approaches by a significant margin for most columns, especially on the part and connectivity accuracy metrics.
	According to the visual results in Figure~\ref{figure:quality} (left), we also observe the best assembly results are achieved by our algorithm, while the baseline methods usually fail in producing well-structured shapes.
	We generate multiple possible assembly outputs with different Gaussian random noise, and measure the minimum distance from the assembly predictions to the ground truth, which allows the models to generate shapes that are different from ground truth while similar to real-world objects, which are demonstrated in Figure~\ref{figure:quality} (right). We see that some bar-shape parts are assembled into different positions to form objects of different structures.

	\setlength{\tabcolsep}{2pt}
	\begin{figure}[t]
		\centering
		\includegraphics[scale=0.4,clip]{imgs/relation-graph.pdf}
		\caption
		Dynamically evolving part relation weights $r_{ij}^{(t)}$ among four common chair part types. The orange cells highlight the four directed edges with the maximal learned relation weight in the matrix, while the yellow cells indicate the minimal ones. The vertical axis denotes the emitting parts, and the horizontal axis denotes the receiving parts.

		}
		\label{figure:relation}
  		\vspace{-0.4cm}
	\end{figure}

	\setlength{\tabcolsep}{2pt}
	\begin{figure}[h]
		\centering
		\begin{tabular}{cccccc}
			\vspace{-1cm}
			\includegraphics[height=2.0cm]{imgs/chair-301-009-00.png}
			&\includegraphics[height=2.0cm]{imgs/chair-301-009-01.png}
			&\includegraphics[height=2.0cm]{imgs/chair-301-009-02.png}
			&\includegraphics[height=2.0cm]{imgs/chair-301-009-03.png}
			&\includegraphics[height=2.0cm]{imgs/chair-301-009-04.png}
			&\includegraphics[height=2.0cm]{imgs/chair-301-009-gt.png}
			\\
			\vspace{-0.4cm}
			\includegraphics[height=2.0cm]{imgs/chair-112-009-00.png}
			&\includegraphics[height=2.0cm]{imgs/chair-112-009-01.png}
			&\includegraphics[height=2.0cm]{imgs/chair-112-009-02.png}
			&\includegraphics[height=2.0cm]{imgs/chair-112-009-03.png}
			&\includegraphics[height=2.0cm]{imgs/chair-112-009-04.png}
			&\includegraphics[height=2.0cm]{imgs/chair-112-009-gt.png}
			\\
			Step 1 & Step 2 & Step 3 & Step 4 & Step 5 & Ground Truth
			\\
		\end{tabular}
		\caption{The time-varying part assembly results.}
		\label{figure:evolution}
	\end{figure}

	We also try to remove the three key components from our method: the iterative GNN, the dynamic part relation reasoning module and the dynamic part aggregation module.
	The results on the PartNet Table category are shown in Table~\ref{table:ablation} and we see that our full model achieves the best performance compared to the ablated versions.
	We firstly justify the effectiveness of our backbone by replacing the graph learning module with a multi-layer perception that estimates the part-wise pose from the concatenated separate and overall part features. We further incorporate the dynamic graph module into our backbone for evaluation. We observe that our proposed backbone and dynamic graph both contribute to the final performance significantly.

	To further investigate the different design choices of our algorithm, we conduct more ablation studies on the Table category regarding the iteration number in GNN (3/5/7, PA: 45.51/49.51/44.6), the number of input points for each part (500/1000, PA: 48.78/49.51), average/max pooling in the part aggregation module (average/max, PA: 48.92/49.51), and different loss designs (w/o $\mathcal{L}_t/\mathcal{L}_r/\mathcal{L}_s$, PA: 24.48/46.94/48.23). PA is short for part accuracy. We find empirically that having too many iterations, makes the network harder to train and difficult to convergence, and hence finally set the iteration number to 5 in our paper. Taking more per-part input points, and using max pooling instead of average pooling both improve the performance. The three loss terms all contribute to the final results.

	\subsection{Dynamic Graph Analysis}

    Figure \ref{figure:relation} summarizes the learned directional relation weights at each time step by averaging $r_{ij}^{(t)}$ over all PartNet chairs. We pick the four common types of parts: back, seat, leg and arm. We see clearly similar statistical patterns for the even iterations and for the odd ones.

	At even iterations, the relation graph is updated from the dense node set. It focuses more on passing messages from the central parts (\textit{i.e. } seat, back) to the peripheral parts (\textit{i.e. } leg, arm) and overlooks the relation between legs.
	While at odd iterations, the relation graph is updated from the sparse node set, where the geometrically-equivalent parts like legs are aggregated to a single node.
	In this case, from the relation graph we can see that all the parts are influenced relatively more by the peripheral parts. Note legs are usually symmetric, so it is very likely that they share the same/similar part orientations, which explains their strong correlations here.
    On the average of all the steps, the central parts have bigger emitting relation weights (1.22) than the peripheral parts (0.94), indicating that the central parts guide the assembly process more.

	We further illustrate the changes of part accuracy and its associated improvement at each time step in Figure~\ref{fig:figure5}. We find that the central parts are consistently predicted more accurately than the peripheral parts.

	Interestingly, the improvement of peripheral parts is relatively higher than central parts at even iterations, which is consistent to the observation of higher directed relations received to minor parts at odd iterations. It demonstrates the fact that central parts guide the pose predictions for the peripheral parts.

    \begin{wrapfigure}{r}{0.45\textwidth}
    \vspace{-7mm}
    \begin{center}
        \includegraphics[scale=0.5, trim = {10cm 3.5cm 11cm 1cm},clip]{imgs/figure3.pdf}
    \end{center}
    \caption{The part accuracy and its relative improvement summarized at each time step.} \label{fig:figure5}
    \end{wrapfigure}

	%

	Figure~\ref{figure:evolution} visualizes the time-varying part assembly results, showing that the poses for the central parts are firstly determined and then the peripheral parts gradually adjust their poses to match the central parts.
	The results finally converge to stable part assembly predictions.

	\section{Conclusion}

	In this work, we propose a novel dynamic graph learning algorithm for the part assembly problem. We develop an iterative graph neural network backbone that learns to dynamically evolve node and edge features within the part graph, augmented with the dynamic relation reasoning module and the dynamic part aggregation module.

	Through thorough experiments and analysis, we demonstrate that the proposed method achieves state-of-the-art performance over the three baseline methods by learning an effective assembly-oriented relation graph.
	Future works may investigate learning better part assembly generative models considering the part joint information and higher-order part relations.

	\clearpage

	\section*{Broader Impact}

Automatic part assembly is an important task in robotics.
 Nowadays, tens of thousands of production line workers are working on product assembly manually.
 Learning how to assemble a set of parts to a complete shape is still a challenging problem.
 This study can enable robots to free human on part assembly task.
 We believe this research benefits both the economy and society.

 However, even with learning algorithms, human trust in machines is still a problem. The algorithms can only be used for assistance in practice and cannot fully replace humans.
 In the assembly task, such machine learning algorithm/system needs to cooperate with human to achieve more effective processes.

	\section*{Acknowledgement}

	This work was supported by the research funds from Key-Area Research and Development Program of Guangdong Province (No.2019B121204008), Peking University (7100602564), the Center on Frontiers of Computing Studies (7100602567), China National Key R\&D Program (2016YFC0700100, 2019YFF0302902), a Vannevar Bush Faculty Fellowship, and gifts from Adobe, Amazon AWS, Google, and Snap. We would also like to thank Imperial Institute of Advanced Technology for GPU supports.

	\section{Appendix}

	This document provides the additional supplemental material that cannot be included into the main paper due to its page limit:

	\begin{itemize}
    \itemsep-0.2em
    \item Additional ablation study.
    \item Analysis of dynamic graph on additional parts and object categories.
    \item Training details.
    \item Failure cases and future work.
    \item Additional results of structural variation.
    \item Additional qualitative results.
    \end{itemize}

    \subsection*{A. Additional ablation study}

In this section, we demonstrate the effectiveness of different components. We test our framework by proposing the following variants, where the results are in Table \ref{table:ablation}.

    \begin{itemize}
    \itemsep-0.2em
    \item \textbf{Our backbone w/o graph learning}: Replacing the graph learning module with a multi-layer perception to estimate the part-wise pose from the concatenated separate and overall part features.
    \item \textbf{Our backbone}: Only the iterative graph learning module.
    \item \textbf{Our backbone + relation reasoning}: Incorporate the dynamic relation reasoning module to our backbone.
    \item \textbf{Our backbone + part aggregation}: Incorporate the dynamic part aggregation module to our backbone.
    \item \textbf{Exchange dense/sparse node set iteration}: Switching the node set order in our algorithm. Specifically, we learn over the dense and sparse node set at even and odd steps respectively.
    \item \textbf{Input GT adjacency relation}: Instead of learning the relation weights from the output poses, we replace the dynamically-evolving relation weights with static ground truth adjacency relation between two parts, which is a binary value. Note the ground truth relation only covers the adjacency, not the symmetry and any other type of relations.
    \item \textbf{Reasoning relation from geometry}: We modify the dynamic relation reasoning module by replacing the input pose information in Equation 5 of the paper with part point cloud transformed by the estimated pose.
    \end{itemize}
    From Table \ref{table:ablation}, we observe that the proposed iterative GNN backbone, dynamic relation reasoning module and dynamic part aggregation module all contribute to the assembly quality significantly. Experimentally, we also exchange the order of sparse and dense node set in the iterative graph learning process, and do not observe much difference compared to our full algorithm. In order to justify our learned relation weights, we employ the ground truth binary adjacency relations to replace learned ones, and observe much worse performance than our learned relations. Finally, instead of learning the relation from estimated poses as in Equation 5 of the main paper, we alternatively replace the pose with transformed part point cloud, and also observe degraded performance as analyzed in the paper.

    \setlength{\tabcolsep}{4pt}
	\begin{table}[h]
		\small
		\begin{center}
			\begin{tabular}{lccc}
				\toprule
				& Shape CD $\downarrow$ & PA $\uparrow$ & CA $\uparrow$\\
				\midrule
				Our backbone w/o graph learning & 0.0086 & 26.05 & 28.07 \\
				\midrule
				Our backbone & 0.0055 & 42.09 & 35.87\\
				\midrule
				Our backbone + relation reasoning & 0.0052 & 46.85 & 38.60\\
				\midrule
				Our backbone + part aggregation & 0.0051 & 48.01 & 38.13\\
				\midrule
				Exchange dense/sparse node set iteration & 0.0052 & 49.19 & 39.62\\
				\midrule
				Input GT adjacency relation & 0.0053 & 45.43 & 35.66\\
				\midrule
				Reasoning relation from geometry & 0.0053 & 45.11 & 39.21\\
				\midrule
				Our full algorithm & \textbf{0.0050} & \textbf{49.51} & \textbf{39.96}\\
				\bottomrule
			\end{tabular}
		\end{center}
		\caption{Ablation study to demonstrate the effectiveness of each component of our algorithm. Shape CD, PA and CA are short for Shape Chamfer Distance, Part Accuracy and Connectivity Accuracy respectively.}
		\label{table:ablation}
		\vspace{-0.5cm}
	\end{table}

    \subsection*{B. Additional analysis of the dynamic graph}
    We demonstrate additional learned relation weights in Figure \ref{figure:relation_sup}. In the chair category, we expend the four parts in the main paper to eight parts, and we observe that the central parts (back, seat, head) have larger emitted relations and the peripheral parts (regular\_leg, arm, footrest, pedestal\_base, star\_leg) have larger received relations. It reveals the same fact as shown in the main paper that the central parts guide the assembly process. Similar phenomenon can also be observed in the lamp and table categories, which demonstrate only four parts due to the limited common parts existing in the dataset.

    \begin{figure*}[t]
		\centering
		\includegraphics[width=1.0\linewidth]{imgs/relation-graph-supp.pdf}
 		\vspace{-0.5cm}
		\caption{Demonstration of the learned relation weights on additional parts and object categories. The number denotes the weight emitted from or received by the specific part averaged over all the other parts. The orange color is the top three or one relation weight in each column.}
		\label{figure:relation_sup}
		\vspace{-3mm}
	\end{figure*}

    \subsection*{C. Training details}
    Our framework is implemented with Pytorch. The network is trained for around 200 epochs to converge. The initial learning rate is set as 0.001, and we employ Adam to optimize the whole framework. We append the supervision on the output poses from all the time steps to accelerate the convergence. The graph neural network loops in five iterations for the final output pose. Experimentally, we find out that the performance tends to approach saturation for five iterations, while we haven't observed obvious improvement with more iterations.

    In order to compute geometrically-equivalent parts, we firstly filter out the parts whose dimension difference of Axis-Aligned-Bounding-Boxes is above a threshold of 0.1, then cope with the remaining parts by excluding all the pairwise parts whose chamfer distance is below an empirical threshold of 0.2.

    \subsection*{D. Failure cases and future work}

    In Figure \ref{figure:fail}, we show a few cases where our algorithm fails to assemble a well-connected shape and hence generates some floating parts. For example, the legs and arms are disconnected/misaligned from the chair base and back. This indicates the fact that our algorithm design lacks the physical constraints to enforce the connection among the parts. Our learned dynamic part graph builds a soft relation between the central and peripheral parts for a progressive part assembly procedure, but is short of the hard connection constraints.
    In the future work, we plan to solve this problem by developing a joint-centric assembly framework to focus more on the relative displacement and rotation between the parts, to facilitate the current part-centric algorithm.

    \setlength{\tabcolsep}{2pt}
	\begin{figure}[h]
		\centering
		\vspace{-3mm}
		\begin{tabular}{cccccc}
			\vspace{-0.3cm}
			\includegraphics[height=2.0cm]{imgs/chair-342-gt.png}
			&\includegraphics[height=2.0cm]{imgs/chair-342-001.png}
			&\includegraphics[height=2.0cm]{imgs/chair-430-gt.png}
			&\includegraphics[height=2.0cm]{imgs/chair-430-001.png}
			&\includegraphics[height=2.0cm]{imgs/chair-602-gt.png}
			&\includegraphics[height=2.0cm]{imgs/chair-602-001.png}
			\\
			Ground truth & Ours & Ground truth & Ours & Ground truth & Ours
			\\
			\vspace{-0.3cm}
		\end{tabular}
		\caption{Failure cases where some parts are floating and disconnected from the other parts.}
		\label{figure:fail}
		\vspace{-0.5cm}
	\end{figure}

    \subsection*{E. Additional results of structural variation}

    Many previous works \cite{gao2019sdm,wu2019sagnet,wang2018global,mo2020pt2pc} learn to create a novel shape from scratch by embedding both the geometric and structural diversity into the generative networks. However, provided the part geometry, our problem only allows structural diversity to be modeled. It poses a bigger challenge to the generative model since some shapes may be assembled differently, while the others can only be assembled uniquely (\emph{i.e.,} only one deterministic result).

    We demonstrate additional diverse assembled shapes with our algorithm in Figure \ref{figure:evolution_sup}. In the top five visual examples, it exhibits various results of structural variation, while in the bottom three examples, our algorithm learns to predict the same assembly results due to the limited input part set.

    \subsection*{F. Additional qualitative results}
    We demonstrate additional visual results predicted by our dynamic graph learning algorithm in Figure~\ref{figure:quality_chair2},~\ref{figure:quality_table2} and~\ref{figure:quality_lamp}.

    \setlength{\tabcolsep}{2pt}
	\begin{figure}[h]
		\centering
		\begin{tabular}{cccc}
 			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-108-000.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-108-002.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-108-031.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-108-gt.png}
			\\
			\vspace{-0.5cm}
            \includegraphics[height=2.0cm]{imgs/diversity/chair/chair-258-007.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-258-021.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-258-022.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-258-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-308-001.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-308-002.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-308-009.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-308-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-030-005.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-030-009.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-030-000.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-030-gt.png}
			\\
			\vspace{-0.3cm}
			\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-755-000.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-755-007.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-755-039.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-755-gt.png}
			\\
			\midrule
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-013-001.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-013-002.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-013-003.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/chair/chair-013-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/diversity/table/table-027-000.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/table/table-027-001.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/table/table-027-002.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/table/table-027-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/diversity/lamp/lamp-028-001.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/lamp/lamp-028-002.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/lamp/lamp-028-003.png}
			&\includegraphics[height=2.0cm]{imgs/diversity/lamp/lamp-028-gt.png}
			\\
			Assembly 1 & Assembly 2 & Assembly 3 & Ground truth
			\\
		\end{tabular}
		\caption{Additional diverse results generated by our network. Top: structural variation demonstrated in part assembly; Bottom: no structural variation for the case with very limited parts.}
		\label{figure:evolution_sup}
		\vspace{-0.5cm}
	\end{figure}

	\setlength{\tabcolsep}{2pt}
	\begin{figure}[h]
		\centering
		\begin{tabular}{ccccc}
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/chair/chair-014-000-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-014-000-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-014-000-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-014-000-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-014-gt.png}
			\\
 			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/chair/chair-301-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-301-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-301-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-301-008-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-301-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/chair/chair-410-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-410-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-410-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-410-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-410-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/chair/chair-385-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-385-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-385-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-385-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-385-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/chair/chair-474-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-474-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-474-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-474-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-474-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/chair/chair-148-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-148-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-148-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-148-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-148-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/chair/chair-113-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-113-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-113-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-113-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-113-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/chair/chair-157-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-157-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-157-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-157-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-157-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/chair/chair-176-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-176-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-176-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-176-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-176-gt.png}
			\\
			\vspace{-0.3cm}
			\includegraphics[height=2.0cm]{imgs/quality/chair/chair-188-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-188-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-188-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-188-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/chair/chair-188-gt.png}
			\\
			B-Complement & B-Global & B-LSTM & Ours & Ground truth
			\\
		\end{tabular}
		\caption{Additional qualitative comparison between our algorithm and the baseline methods on the PartNet Chair dataset}
		\label{figure:quality_chair2}
		\vspace{-0.5cm}
	\end{figure}

		\setlength{\tabcolsep}{2pt}
	\begin{figure}[h]
		\centering
		\begin{tabular}{ccccc}
 			\vspace{-0.5cm}
 			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/table/table-000-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-000-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-000-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-000-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-000-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/table/table-007-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-007-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-007-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-007-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-007-gt.png}
			\\
		    \vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/table/table-514-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-514-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-514-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-514-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-514-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/table/table-520-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-520-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-520-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-520-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-520-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/table/table-542-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-542-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-542-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-542-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-542-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/table/table-387-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-387-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-387-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-387-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-387-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/table/table-136-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-136-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-136-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-136-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-136-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/table/table-834-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-834-008-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-834-008-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-834-008-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-834-gt.png}
			\\
		\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/table/table-836-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-836-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-836-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-836-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-836-gt.png}
			\\
		    \vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/table/table-151-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-151-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-151-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-151-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/table/table-151-gt.png}
			\\
			B-Complement & B-Global & B-LSTM & Ours & Ground truth
			\\
		\end{tabular}
		\caption{Additional qualitative comparison between our algorithm and the baseline methods on the PartNet Table dataset}
		\label{figure:quality_table2}
		\vspace{-0.5cm}
	\end{figure}

	\setlength{\tabcolsep}{2pt}
	\begin{figure}[h]
		\centering
		\begin{tabular}{ccccc}
 			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-013-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-013-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-013-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-013-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-013-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-041-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-041-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-041-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-041-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-041-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-045-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-045-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-045-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-045-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-045-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-061-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-061-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-061-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-061-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-061-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-077-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-077-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-077-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-077-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-077-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-079-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-079-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-079-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-079-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-079-gt.png}
			\\
		\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-095-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-095-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-095-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-095-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-095-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-098-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-098-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-098-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-098-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-098-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-136-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-136-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-136-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-136-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-136-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-146-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-146-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-146-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-146-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-146-gt.png}
			\\
			\vspace{-0.5cm}
			\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-156-009-complementme.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-156-009-componet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-156-009-pqnet.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-156-009-ours.png}
			&\includegraphics[height=2.0cm]{imgs/quality/lamp/lamp-156-gt.png}
			\\
			B-Complement & B-Global & B-LSTM & Ours & Ground truth
			\\
		\end{tabular}
		\caption{Additional qualitative comparison between our algorithm and the baseline methods on the PartNet Lamp dataset}
		\label{figure:quality_lamp}
		\vspace{-0.5cm}
	\end{figure}

	\clearpage