## Multi-path Learning for Object Pose Estimation Across Domains

Martin Sundermeyer 1 2 , , Maximilian Durner 1 2 , , En Yen Puang , Zoltan-Csaba Marton , 1 1 Narunas Vaskevicius , Kai O. Arras , Rudolph Triebel 3 3 1 2 , 1 2 Technical University of Munich (TUM),

German Aerospace Center (DLR), 3 Robert Bosch GmbH

## Abstract

Weintroduce a scalable approach for object pose estimation trained on simulated RGB views of multiple 3D models together. We learn an encoding of object views that does not only describe an implicit orientation of all objects seen during training, but can also relate views of untrained objects. Our single-encoder-multi-decoder network is trained using a technique we denote 'multi-path learning': While the encoder is shared by all objects, each decoder only reconstructs views of a single object. Consequently, views of different instances do not have to be separated in the latent space and can share common features. The resulting encoder generalizes well from synthetic to real data and across various instances, categories, model types and datasets. We systematically investigate the learned encodings, their generalization, and iterative refinement strategies on the ModelNet40 and T-LESS dataset. Despite training jointly on multiple objects, our 6D Object Detection pipeline achieves state-of-the-art results on T-LESS at much lower runtimes than competing approaches. 1

## 1. Introduction

Object pose estimation, i.e. estimating the 3D rotation and translation, is a cornerstone in many perception related applications like augmented reality or robotics. For many years this field was dominated by template- and featurebased approaches. Thereby, a given object model or a set of extracted object features is matched into the underlying sensor data. Depending on the quality of the data and the difficulty of the scene, these approaches are still quite competitive. [22] In recent years however, machine learning, and in particular Deep Learning (DL) techniques have emerged as another key approach []. They are often computationally efficient and can show higher robustness towards sensor noise, clutter and environment changes [45, 35, 41]. On

1 Code can be found here: https://github.com/DLR-RM/

AugmentedAutoencoder/tree/multipath

Figure 1: Training (top) and setup phase (bottom) of the MP-Encoder. During training one encoder is shared among all objects, while each decoder reconstruct views of a single object. This turns the encoder into a viewpoint-sensitive feature extractor, that generates expressive encodings for multiple trained and even untrained objects.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00151v2-figure-567fb069-b045-4a44-8901-bcb54238863e.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00151v2-figure-07512f69-88a5-4fde-a118-0a14256a9725.jpg)

the other hand, the need for pose-annotated data as well as lengthy training phases make them less flexible compared to traditional methods. To tackle the lack of annotations, several methods have recently shifted to train on synthetic data rendered from 3D models [23, 34, 38]. However, it is still common to train individual models for every newly encountered instance which is not scalable. Attempts to train on many objects at once often result in deteriorated performance [45]. Furthermore, due to object discriminative training, most of these approaches are not suited to generalize to untrained objects. Since the real world consists of large amounts of object categories and instances, we propose a more adaptive and scalable approach in this paper.

Inspired by Augmented Autoencoders (AAEs) [34] that

extract pose representative features on an instance level, we propose a single-encoder-multi-decoder network for jointly estimating the 3D object orientation of multiple objects. While the encoder and latent space are shared among all objects, the decoders are trained to reconstruct views of specific instances (top of Fig. 1). This multi-path learning strategy allows similarly shaped objects to share the same encoding space. After training, we can generate instancespecific codebooks containing encodings of synthetic object views from all over SO(3). So each entry contains a shape and viewpoint dependent descriptor mapped to an explicit 3D object orientation that can be retrieved at test time. (bottom of Fig. 1)

As we show experimentally, the learned encodings generalize well to the real domain and are expressive enough to relate views from all trained and even untrained objects in a viewpoint sensitive way. For a large number of objects the performance of the Multi-Path Encoder (MP-Encoder) does not deteriorate compared to separately trained encoders for each object and even slightly improves encoding quality which leads to new state-of-the-art results in 6-DoF object pose estimation on the T-LESS dataset.

Motivated by this, we also introduce an iterative renderinference scheme based on the learned encodings, which enables relative pose estimation on untrained objects. It resembles the online creation of a local codebook which also helps avoiding SO(3) discretization errors. We apply this method to iteratively refine poses of untrained instances from ModelNet40 and outperform DeepIM [25], a state-ofthe-art approach for RGB-based 6D pose refinement.

## 2. Related Work

Object pose estimation has been widely studied in the literature, mainly from the below aspects.

## 2.1. Feature-/Template-based approaches

Traditionally, pose estimation methods rely on matching algorithms with local keypoints and descriptors or templates. E.g ., both Hinterstoisser et al . [17] and Vidal et al . [40] apply so-called Point Pair Features (PPF) to match a 3D model into a given scene outperforming other results in the BOP challenge [21] but having high runtimes per target due to the exhaustive search in the large 6D pose space. While these methods require depth information, which can be noisy and sensible to environment changes, Ulrich et al . [39] use single RGB images and estimate object poses by matching edges of a rendered 3D model into the image. In general, the benefit of such approaches using explicit matching is that no costly training process is required. However, they tend to require longer execution times. Additionally, usually more expert knowledge is required to set up such pipelines, as feature extraction and matching parameters do not generalize well.

## 2.2. Learning-based Approaches

More recently, a number of learning-based approaches have been presented with the advantage of reduced computation time and more complex features that can be learned directly from data. One approach is to classify or regress a representation of the object pose directly [23, 28]. Xiang et al . [44] propose a CNN architecture with multiple stages, where the authors define the pose regression loss as a closest object point measure. Another line of research makes predictions on sparse or dense 2D-3D keypoint locations, e.g. the 3D bounding box corners [32, 38, 37] or 3D object coordinates on the surface [26, 45, 31]. The pose is then computed by solving the Perspective-n-Point (PnP) problem. Apart from occlusions, object and view symmetries cause challenges since they introduce one-tomany mappings from object views to poses. While there are strategies to adapt the loss, network or labels for symmetric objects [42, 15, 23], dealing with view-dependent or occlusion-induced symmetries is often intractable and negatively affects the training process. Therefore, a third approach is to learn descriptors of object views that are either conditioned on the pose [1, 42] or completely implicit [34]. The advantage of the implicit approach is that symmetric views share the same target representation during trainig. Corresponding explicit poses are automatically grouped and can be retrieved at test time. These view-centric representations were also shown to work well for 6D tracking [7].

## 2.3. Synthetic training data

The demand of learning-based methods for large amounts of labeled training data is particularly unfavorable for object pose estimation. Although there are ways to label object poses semi-automatically [30], it remains a cumbersome, error-prone process. Recent research tackles this by exploiting synthetic training data [8] which solves this problem but creates a new one through the domain gap between real and synthetic data.

To bridge this gap, Tremblay et al . [38] use mixed training data consisting of photo-realistic rendering and domain randomized samples. Since photo-realistic renderings are costly, we follow a less expensive Domain Randomization (DR) strategy [23, 37]: 3D models are rendered in poses randomly sampled from SO(3) using OpenGL with random lighting and superimposed onto real images from data sets such as MS COCO [27] or Pascal VOC [10].

## 2.4. Generalization to Novel Objects

Most learning-based methods predict the pose of one [31] or few instances [38, 45, 37, 44] and have to be retrained for every newly encountered object. However, in domains like service and industrial robotics or augmented reality, it would be beneficial to have a general feature extractor that could produce pose-sensitive features for un- trained objects such that testing on a novel object becomes immediately possible.

When trained on few instances, current pose networks like [38, 37] concurrently classify objects which potentially hinders their ability to generalize to untrained objects. Wohlhart et al . [42] and Balntas et al . [1] were the first to report qualitative results of deep pose descriptors applied to untrained objects. However, their descriptors are discriminated by both, orientation and object class. So if an untrained object has similar appearance from any viewpoint to one of the trained objects, the corresponding descriptor will get corrupted. Unlike [42, 1], our multi-path training strategy does not separate different object instances in the encoding space and instead allows them to share the same latent features.

Category-level pose estimation [33, 3, 36] can be used to generalize to novel objects from a given category. It assumes that all instances within a category have similar shape and are aligned in a joint coordinate frame. However, these assumptions often do not hold in practice where semantic and geometric similarities often do not coincide. Assigning aligned coordinate frames can be ambiguous because symmetries of instances can vary within a category. Therefore, in this work, we will not explicitly enforce the alignment within semantic categories and instead leave this decision to the self-supervised, appearance-based training.

CNNs trained on large datasets are frequently used to extract low-level features for downstream tasks, e.g. image retrieval [13] or clustering [9]. A naive baseline to predict the 3D orientation of unknown objects would be to compare feature maps of a network trained on a large dataset like ImageNet or COCO. Unsurprisingly this baseline does not work at all because (1) early features are sensitive to translation while the later layers lost geometric information (2) features strongly differ for synthetic and real object views (3) the dimensionality of feature maps is too high to finely discretize SO(3) while reduction techniques like PCA destroy much information.

The pose refinement methods [29, 25] iteratively predict rotation and translation residuals between an estimated and a target view of an object. The former was shown to generalize to untrained objects of the same category, and the latter even generalizes to objects of new categories. These approaches predict an accurate, relative transformation between two object views in a local neighborhood. In contrast, our method is able to yield both, local relative and global 3D orientation estimates.

## 3. Method

We will first briefly describe the AAE (Sec. 3.1). Building upon these results, we then propose a novel Multi-Path Encoder-Decoder architecture and training strategy (Sec. 3.2). Next, we will investigate the encoder on its ability to extract pose-sensitive features and examine generalization to novel instances (Sec. 3.3). Different application scenarios depending on the test conditions are discussed (Sec. 3.4). Finally, an iterative render-inference optimization for pose refinements is presented (Sec. 3.5).

## 3.1. Implicit Object Pose Representations

Sundermeyer et al . [34] have shown that implicit pose representations can be learned in a self-supervised way using an encoder-decoder architecture. This so-called AAE allows to encode 3D orientations from arbitrary object views, generalizes from synthetic training data to various test sensors, and inherently handles symmetric object views.

The AAE is trained to reconstruct rendered views of a single object. To encode 3D orientation exclusively, the inputs are randomly translated and scaled while the reconstruction target stays untouched. To encode object views from real images, the background of input views are randomized, occlusions at various locations are generated, and various lighting and color augmentations are produced. As a result of this domain randomization , the network learns to represent the objects' orientation of real object views implicitly using the latent code z .

Concretely, an input sample x ∈ R d is randomly augmented by f ( ) . and mapped by an encoder Υ onto a latent code z ∈ R m where m glyph[lessmuch] d . The decoder Λ : R m → R d is trained to map the code back to the target x .

$$\hat { x } = \Lambda ( \Upsilon ( f ( { \mathbf x } ) ) ) = \Lambda ( \Upsilon ( x ^ { \prime } ) ) = \Lambda ( { \mathbf z } ) \quad \ ( 1 )$$

Both Υ and Λ are neural networks, and their weight parameters are trained such that the glyph[lscript] 2 -loss is minimized, i.e .

$$\underset { \text{lexly} } { \text{(2)} } \quad \ell _ { 2 } ( \mathcal { B } ) = \sum _ { i \in \mathcal { B } } \| x _ { i } - \hat { x } _ { i } \| _ { 2 } = \sum _ { i \in \mathcal { B } } \| x _ { i } - \Lambda ( \Upsilon ( f ( x _ { i } ) ) ) \| _ { 2 } \ ( 2 )$$

where B contains the indices of input samples of a given batch. After training, the decoder is discarded and latent encodings of object views from all over SO(3) are saved in a codebook together with their corresponding orientations assigned. At test time, a real object crop is encoded and the nearest code(s) in the codebook according to cosine similarity yield the predicted orientation. The rotation can further be corrected for the translational offset as described in [35].

A downside of this formulation is that a new network must be trained for every new object instance. When naively training the original AAE jointly on several objects, they need to be separated in the latent space so that the decoder is able to reconstruct the correct object. Even when conditioning the decoder on the object by concatenating a one-hot vector to the encoding, it can only reconstruct few instances and it diminishes the ability of the encoder to encode object orientations.

## 3.2. Multi-Path Encoder-Decoder

We propose a simple but effective architecture which, in combination with our multi-path learning strategy, enables the 3D orientation estimation of multiple objects (see Fig. 1). Our architecture consists of a single encoder Υ , an encoding z ∈ R 128 , and n decoders Λ j with j = 1 , ..., n where n is the number of different object instances. The convolutional encoder is fed with the same augmented inputs as an AAE but with heterogeneous batches ˜ B containing multiple objects. The resulting codes are split and each decoder only receives codes that correspond to a single object instance. The multi-path loss function can be written as:

$$\ell _ { m } ( \tilde { \mathcal { B } } ) = & \sum _ { j = 1 } ^ { b } \sum _ { k = 1 } ^ { n } \mathbb { I } ( s _ { j } = k ) \| \mathbf x _ { j } - \Lambda _ { k } ( \Upsilon ( f ( \mathbf x _ { j } ) ) ) \| _ { 2 } \\ = & \sum _ { j = 1 } ^ { b } \sum _ { k = 1 } ^ { n } \mathbb { I } ( s _ { j } = k ) \| \mathbf x _ { j } - \Lambda _ { k } ( \mathbf z _ { j } ) \| _ { 2 } \quad ( 3 ) \\. \quad \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot$$

where I is the indicator function used to select the decoder Λ k that corresponds to the instance s j . Note that in this setting only the encoder Υ receives information from the entire mini-batch, while the decoders Λ j backpropagate a loss glyph[lscript] j from a sub-batch. Since the decoders are only used to learn an efficient encoding, they can be discarded after training, leaving behind a compact encoder model.

In contrast to other approaches [42, 1], where objects are explicitly separated in the descriptor space, our encoder can learn an interleaved encoding where general features can be shared across multiple instances. We consider this ability as the main qualification to obtain encodings that can represent object orientations from novel, untrained instances, even when they belong to untrained categories.

## 3.3. Principal Component Analysis of Encodings

In order to gain insights into the characteristics of the latent space from trained and untrained objects, we perform an experiment on the ModelNet40 [43] dataset. We first train the multi-path encoder-decoder on 80 CAD instances that originate from the car class. A second model is trained on 10 instances from 8 different classes, namely airplane, bed, bench, car, chair, piano, sink, toilet . Training details can be found in the appendix.

After training, we generate 72 viewpoints along a full revolution of azimuth, elevation and in-plane rotation. We record different objects from these viewpoints and feed them into the encoder. From the encoding space z i ∈ R 128 of all objects we compute the first three principal components and project all encodings onto these directions. The interpolated results can be seen in Fig. 2. The top row shows the encodings of a car instance from the training set.

Figure 2: Principal Component Analysis of the learned encodings. Depicted are the encodings of views around elevation (red), azimuth (blue), and in-plane (green). Middle column: Encoder trained only on cars; Right column: Encoder trained on objects from 8 categories

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00151v2-figure-5fd7fdb5-7bbe-4bea-b0e4-9540af3682d9.jpg)

The other rows show instances which are not in the training set of neither model, but different sofa and toilet instances were used to train the second model.

First, it is noticeable that the encodings vary smoothly along each of the rotation axes, even when views of untrained objects are evaluated. It can be seen that the views, while starting at the same point, end up in different submanifolds that capture the shape of the encoded objects and their symmetries. For example, the car instance produces closed trajectories with two revolutions because the car shape looks similar at opposite viewpoints.

Furthermore, it can be seen that the car in the training and test set are encoded similarly by each of the models. This indicates that the encoder as well as the codebook could be reused when predicting orientations of novel cars,

Table 1: Different application scenarios for global and iterative pose estimation and the handling of untrained objects

|                                        | Scenario   |
|----------------------------------------|------------|
| Prerequisites                          | I II       |
| Shape category trained                 | 7 3        |
| 3D Test Model available                | 3 7        |
| 3D Test Model aligned to Train Model   | 7 7        |
| Available Methods                      | ⇓ ⇓        |
| Reuse codebook from a trained instance | 7 3        |
| Create a new full / sparse codebook    | 3 7        |
| Iterative relative pose refinement     | 3 7        |

even without access to their 3D models.

The remaining encodings of untrained objects can still be clearly separated and thus a meaningful pose descriptor could be extracted without retraining the encoder but simply by creating a codebook from their 3D model.

Apart from an offset, both models learn quite similar encodings, even for the sofa category which the first model has not seen during training. This implies that low-level features are extracted that generalize across various shapes and categories. However, to fulfill the reconstruction task the features still need to be robust against lighting and color augmentations as well as translations that are applied to the inputs. In the next section, we will explore different approaches to utilize these properties.

## 3.4. Object Pose Estimation Across Domains

After training, the MP-Encoder can create codebooks for all n training instances for pose estimation. (see Sec. 3.1)

On top of that, it can deal with scenarios inlcuding untrained objects which are depicted in Table 1. Here, the trained encoder model is used as a fixed pose-sensitive feature extractor. Available methods depend on the characteristics and given information on the considered test objects.

If a 3D model of the untrained object is available, it is usually preferable to create a new codebook and estimate the 3D orientation from it (I). If no 3D model is available, but we have codebooks of trained instances from a category with coinciding shapes, we could also reuse these codebooks directly for the test instance (II). However, as we do not explicitly align models of a category in a single coordinate frame, the extracted representation will be more dependent on the shape of an object than on any semantics.

Given a 3D model, we can also use our method for iterative pose refinement at test time. This allows us to refine results from sparser codebooks or to perform local, relative pose estimation directly without any codebook.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00151v2-figure-7418b453-02de-4a61-a133-1e5507b1fc47.jpg)

| Algorithm 1: Iterative 6D Pose                                                                                                                                                                              | Refinement                                                                                                                                                                                                  |
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Input: encoder Υ , init pose q init , t init , target view x ∗                                                                                                                                              | Input: encoder Υ , init pose q init , t init , target view x ∗                                                                                                                                              |
| z ∗ ← Υ( x ∗ )                                                                                                                                                                                              | z ∗ ← Υ( x ∗ )                                                                                                                                                                                              |
| q est ← q init                                                                                                                                                                                              | q est ← q init                                                                                                                                                                                              |
| t est ← t init                                                                                                                                                                                              | t est ← t init                                                                                                                                                                                              |
| for k = 0 . . . 2 do                                                                                                                                                                                        | for k = 0 . . . 2 do                                                                                                                                                                                        |
| for j = 0 . . . 3 do                                                                                                                                                                                        | for j = 0 . . . 3 do                                                                                                                                                                                        |
| for i = 0 . . . 40 - 10 k do α ∼ N (0 , σ 2 j +1 )                                                                                                                                                          | for i = 0 . . . 40 - 10 k do α ∼ N (0 , σ 2 j +1 )                                                                                                                                                          |
| v ∼ N 3 ( 0 , I ) ∆ q ← quat ( v ‖ v ‖ , α ) q i ← ∆ qq est x i ← render ( q i , t est ) z i ← Υ( x i ) end k ← arg max i z i z ∗ ‖ z i ‖‖ z ∗ ‖ q est ← q k est = render ( q est , t est ) ( x ∗ , x est ) | v ∼ N 3 ( 0 , I ) ∆ q ← quat ( v ‖ v ‖ , α ) q i ← ∆ qq est x i ← render ( q i , t est ) z i ← Υ( x i ) end k ← arg max i z i z ∗ ‖ z i ‖‖ z ∗ ‖ q est ← q k est = render ( q est , t est ) ( x ∗ , x est ) |
| q est , t est                                                                                                                                                                                               | q est , t est                                                                                                                                                                                               |
| Result:                                                                                                                                                                                                     | Result:                                                                                                                                                                                                     |
| end                                                                                                                                                                                                         | end                                                                                                                                                                                                         |
| t est = t est +∆ t                                                                                                                                                                                          | t est = t est +∆ t                                                                                                                                                                                          |
| x                                                                                                                                                                                                           | x                                                                                                                                                                                                           |
| end                                                                                                                                                                                                         | end                                                                                                                                                                                                         |
| ∆ t = multiScaleEdgeMatching                                                                                                                                                                                | ∆ t = multiScaleEdgeMatching                                                                                                                                                                                |

## 3.5. Iterative Refinement of Latent Codes

Our method for iterative pose refinement is outlined in Alg. 1. We start with an initial pose estimate and a target view. Next, we render a batch of 3D model views at small rotational perturbations from the initial pose and insert the whole batch into the encoder. The code with the highest cosine similarity to the target view encoding determines the new rotation estimate. We sample again with a smaller perturbation radius. The inner loop of this random optimization scheme, consisting of rendering and inference, can be efficiently parallelized.

Rotation and translation are alternately optimized three times in our experiments. As the MP-Encoder is trained to be invariant against translations, we can first optimize for the more challenging out-of-plane rotation neglecting translation. Afterwards, the rotation is usually roughly aligned to the target and we use a simple edge-based, multi-scale template matching method based on OpenCV [2] to determine the translational offset. The advantage of sampling in SO(3) vs. sampling in the latent space is that (1) the SO(3) space has lower dimensionality and (2) we only search valid orientations.

Apart from relative pose estimation with a novel 3D model without a codebook, the refinement also allows to trade-off between set-up time, inference time, memory resources and accuracy. The full 92232 × 128 codebook creation takes ∼ 5 minutes per object on a modern GPU and 45 MBspace. Inference only takes ∼ 6 ms on a modern GPU while the 6D pose refinement in the presented configura-

Table 2: Pose estimation performance of the MP-Encoder trained on objects 1-18 on the complete T-LESS primesense test set (RGB-only) compared to 30 single object AAEs [34] trained on all objects. We measure recall under the Visible Surface Discrepancy ( err vsd ) metric. The right column shows the performance of a model trained only on 30 instances of the ModelNet40 dataset. We use ground truth bounding boxes (top) and ground truth masks (bottom) in this experiment. It can be observed that a single MP-Encoder can reach similar performance on unknown objects if segmentation masks are given.

| Mean          | 30 separate AAE   | Single Multi-Path Encoder trained on   | Single Multi-Path Encoder trained on   |
|---------------|-------------------|----------------------------------------|----------------------------------------|
| VSD recall    | encoders [34]     | T-Less Objects 1-18                    | 30 Instances ModelNet40                |
| Objects 1-18  | 35.60             | 35.25                                  | 27.64                                  |
| Objects 19-30 | 42.45             | 33.17                                  | 34.57                                  |
| Total         | 38.34             | 34.42                                  | 30.41                                  |
| Objects 1-18  | 38.98             | 43.17                                  | 35.61                                  |
| Objects 19-30 | 45.33             | 43.33                                  | 42.59                                  |
| Total         | 41.52             | 43.24                                  | 38.40                                  |

tion takes approximately 1 second. To further speed this up, the random search could be replaced by more sophisticated black-box optimization tools such as CMA-ES [12]. Depending on the application, i.e. number of objects, available resources and constraints on the set-up time, global initialization with sparse codebooks combined with local pose refinement could be a viable option.

## 4. Evaluation

We focus our analysis on two benchmarks: ModelNet40 [43] and the T-LESS dataset [19]. 2

## 4.1. Metrics

In ModelNet40 we use the absolute angular error

$$e _ { R } = \arccos \, \left ( t r ( \hat { R } ^ { T } \, R - I ) \, / \, 2 \right ) \, \quad \, ( 4 )$$

as well as the ADD metric [16] at an average distance threshold of 0 1 . × object diameter for the model points M

$$A D D = \frac { 1 } { m } \sum _ { x \in \mathcal { M } } | | ( R x + t ) - ( \hat { R } x + \hat { t } ) | | \quad ( 5 )$$

and the 2D projection metric at an average 5px threshold

$$\mathfrak { e } _ { \mathfrak { e } } ^ { \mathfrak { l } } \sum _ { x \in \mathcal { M } } \| K ( R x + t ) - K ( \hat { R } x + \hat { t } ) \| \quad ( 6 )$$

In the T-LESS experiments we use the err vsd [20] since here the above metrics are quite meaningless as a result of various object and view symmetries. The err vsd is an ambiguity-invariant pose error metric that is computed from the distance between the estimated and ground truth visible object surfaces. As in the SIXD challenge 2017 [18] and the BOP benchmark 2018 [22], we report the recall of 6D object poses at err vsd &lt; . 0 3 with tolerance τ = 20 mm and &gt; 10% object visibility.

## 4.2. Generalization Capabilities of the MP-Encoder

We first investigate the joint encoding performance and generalization capabilities of the MP-Encoder in isolation, i.e. with ground truth detections and masks. Therefore, we compare one MP-Encoder against 30 separately trained

2 Further results can be expected at the BOP challenge 2020 [22]

## 3.6. 6DoF Object Detection Pipeline

Our full RGB-based 6DoF object detection pipeline consists of a MaskRCNN with ResNet50 backbone [14], the MP-Encoder for 3D orientation estimation and a projective distance estimate [34] for translation estimation. To compare against other approaches that use depth data, we also report results with a point-to-plane ICP [5, 46] refinement step. Especially in the presence of severe occlusions the RGB-based projective distance estimate does not produce distances accurate enough to fulfill the strict VSD metric. On the other hand, an accurate initialization is crucial for ICP refinement to work well.

For the MaskRCNN we generate training data by pasting object recordings from the T-LESS training set at random translation, scale and in-plane rotation on random background images. Thereby, we produce 80000 images with MS COCO [27] background, 40000 images with black background and 40000 images with random texture backgrounds [6]. We apply several standard online color augmentations like hue, brightness, blur and contrast. Our MaskRCNN reaches an mAP@0.5 performance of 0.68 (bbox detection) and 0.67 (segmentation). Both, MaskRCNN and the MP-Encoder can process multiple objects at once and thus achieve a runtime that stays nearly constant for a large number of objects.

Table 3: Evaluation of our full 6D Object Detection pipeline with MaskRCNN + Multi-Path Encoder + optional ICP. We report the mean VSD recall following the SIXD challenge / BOP benchmark [21] on the T-LESS Primesense test set. See the appendix for object-wise results. A single MP-encoder model outperforms the result of 30 instance-specific AAEs.

|          | Template matching   | PPF based           | PPF based      | PPF based        | Learning-based     | Learning-based   | Learning-based   | Learning-based           | Learning-based           |
|----------|---------------------|---------------------|----------------|------------------|--------------------|------------------|------------------|--------------------------|--------------------------|
|          | Hodan-15 Depth      | Vidal-18 Depth +ICP | Drost-10 Depth | Drost-10-edge    | Brachmann-16 RGB-D | Kehl-16 ICP      | OURS             | Sundermeyer-18 OURS only | Sundermeyer-18 OURS only |
| Average  | 63.18               | 66.51               | 56.81          | Depth + RGB 67.5 | 17.84              | RGB-D + 24.6     | RGB + ICP 69.53  | RGB 19.26                | 20.53                    |
| Time (s) | 13.5                | 4.7                 | 2.3            | 21.5             | 4.4                | 1.8              | 0.8              | 0.1                      | 0.2                      |

AAE models on all scenes of the T-LESS Primesense test set (Table 2). Equivalent encoder and decoder architectures are used. Furthermore, the MP-Encoder is only trained on the first 18 3D object reconstructions of the T-LESS dataset to show the generalization capabilities on untrained objects 19-30. On objects 1-18 which the MP-Encoder has seen during training, the results using ground truth bounding boxes are close to the AAE results. Looking at the next row, the performance on the untrained objects 19-30 is significantly worse compared to the single AAEs. We suppose that the MP-Encoder has difficulties to extract unknown objects from the background. This hypothesis is strongly supported by the results with ground truth masks (bottom) where the MP-Encoder even outperforms the AAEs. Even for the untrained objects 19-30 the performance gap to the AAEs that were trained on these objects is quite small. One possible explanation is that a feature extractor trained on multiple objects learns more versatile and thus more robust features.

The last column of Table 2 depicts the surprisingly good generalization from ModelNet40 to the T-LESS dataset. Here, the MP-Encoder is specifically trained on 30 texturefree CAD models from the 8 categories airplane, bed, bench, car, chair, piano, sink, toilet . Codebooks are created for all T-LESS objects and it is tested on the same real sensor recordings. These results underline that with the multi-path training combined with an input randomization strategy, we can learn to extract orientation variant features that generalize well across greatly differing domains.

## 4.3. 6D Object Detection results

Next, we evaluate our full 6D pose estimation pipeline on the T-LESS dataset which is a particularly challenging 6D object detection benchmark containing texture-less, symmetric objects as well as clutter and severe occlusions. Table 3 presents results using the strict vsd metric (Sec. 3.6). We achieve state-of-the-art results on T-LESS at much lower runtimes than previous methods, both in the RGB domain and also in the depth domain when initializing an ICP with our RGB-based pose estimate. Although the gains are marginal, the results are significant because we only train a single encoder on the whole dataset and still obtain stateof-the-art results. This makes our method scalable, partic-

Figure 3: Left: Qualitative 6D Object Detection results of the RGB-based pipeline on T-LESS Primesense scenes. Only 18 of 30 objects are used to train the Multi-Path encoder; Middle: ICP-refined results; Right: Relative pose refinement of instances from the untrained categories guitar and bathtub. Red is the initial pose, green depicts the refined pose.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00151v2-figure-d8bc981c-86db-4a84-9a45-4ad5d2d3369f.jpg)

Figure 4: Failure case: MaskRCNN predicts the wrong class 20 instead of 19 (the object below). Since the shape is quite similar (except scale) the codebook of object 20 still gives a reasonable pose estimate.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00151v2-figure-15991284-a875-4744-a7b1-2a64e01dedcb.jpg)

ularly considering that no real pose-annotated data is used for training. Fig. 3 (left) shows qualitative examples of the full 6D Object Detection pipeline. Previously, pose estimation of texture-less objects has been dominated by pure depth-based geometric matching approaches that often possess high run-times and do not scale very well with the number of considered objects. Fig. 4 shows a failure case which underlines that strict instance-wise pose estimation may not be suitable for real world applications where objects often

Table 4: Relative Pose Refinement from up to 45 o and ∆ = (10 10 50)[ t , , mm ] perturbations on untrained instances of seen (top) and unseen (bottom) object categories of the ModelNet40 dataset. We measure the recall at the 5 cm, 5 o threshold, the ADD at 0.1d (object diameter) metric and the Proj2D at 5px threshold.

| metric                                                | (5 o , 5 cm )   | (5 o , 5 cm )   | (5 o , 5 cm )   | (5 o , 5 cm )   | 6D Pose (ADD)   | 6D Pose (ADD)   | 6D Pose (ADD)   | 6D Pose (ADD)   | Proj2D (5px)   | Proj2D (5px)   | Proj2D (5px)   | Proj2D (5px)   |
|-------------------------------------------------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|----------------|----------------|----------------|----------------|
|                                                       | DeepIm [25]     | DeepIm [25]     | Ours            | Ours            | DeepIm [25]     | DeepIm [25]     | Ours            | Ours            | DeepIm [25]    | DeepIm [25]    | Ours           | Ours           |
| method                                                | init            | refined         | init            | refined         | init            | refined         | init            | refined         | init           | refined        | init           | refined        |
| airplane car                                          | 0.8             | 68.9            | 0.9             | 96.9            | 25.7            | 94.7            | 33.4            | 97.9            | 0.4            | 87.3           | 0.1            | 97.4           |
|                                                       | 1.0             | 81.5            | 0.4             | 96.4            | 10.8            | 90.7            | 13.4            | 98.5            | 0.2            | 83.9           | 0.1            | 94.0           |
| chair                                                 | 1.0             | 87.6            | 0.3             | 96.4            | 14.6            | 97.4            | 16.3            | 98.3            | 1.5            | 88.6           | 0.0            | 94.6           |
| Mean                                                  | 0.9             | 79.3            | 0.5             | 96.6            | 17.0            | 94.3            | 21.0            | 98.2            | 0.7            | 86.6           | 0.1            | 95.3           |
| bathtub bookshelf guitar range sofa wardrobe tv stand | 0.9             | 71.6            | 0.7             | 85.5            | 11.9            | 88.6            | 15.4            | 91.5            | 0.2            | 73.4           | 0.1            | 80.6           |
|                                                       | 1.2             | 39.2            | 0.7             | 81.9            | 9.2             | 76.4            | 13.7            | 85.1            | 0.1            | 51.3           | 0.0            | 76.3           |
|                                                       | 1.2             | 50.4            | 0.5             | 69.2            | 9.6             | 69.6            | 13.1            | 80.5            | 0.2            | 77.1           | 0.3            | 80.1           |
| hood                                                  | 1.0             | 69.8            | 0.5             | 91.0            | 11.2            | 89.6            | 14.1            | 95.0            | 0.0            | 70.6           | 0.0            | 83.9           |
|                                                       | 1.2             | 82.7            | 0.6             | 91.3            | 9.0             | 89.5            | 12.2            | 95.8            | 0.1            | 94.2           | 0.0            | 86.5           |
|                                                       | 1.4             | 62.7            | 0.7             | 88.7            | 12.5            | 79.4            | 14.8            | 92.1            | 0.2            | 70.0           | 0.0            | 81.1           |
|                                                       | 1.2             | 73.6            | 0.6             | 85.9            | 8.8             | 92.1            | 10.5            | 90.9            | 0.2            | 76.6           | 0.1            | 82.5           |
| Mean                                                  | 1.2             | 64.3            | 0.6             | 84.8            | 10.3            | 83.6            | 13.4            | 90.1            | 0.1            | 73.3           | 0.1            | 81.6           |

differ only in details.

## 4.4. Iterative Refinement on Untrained Objects

the right shows qualitative results of the refinement on unseen categories.

In our final experiment, we evaluate the MP-Encoder on the task of iterative refinement of poses on untrained instances from seen and unseen categories from ModelNet40.

We follow the evaluation protocol of DeepIm [25] where the relative pose between two object views of an untrained instance is sought. The target view is rendered at constant translation t = (0 0 500) , , (mm) and random rotation R ∼ SO (3) . Then we render another object view with that pose perturbed by the angles β x/y/z ∼ N (0 , (15 ) o 2 ) sampled for each rotation axis and a translational offset ∆ x ∼ N (0 , 10 ) 2 , ∆ y ∼ N (0 10 ) , 2 , ∆ z ∼ N (0 , 50 ) 2 (mm). If the total angular perturbation is more than 45 o , it is resampled.

We train category specific MP-Encoders on 80 instances of the airplane, car and chair class and predict the relative pose on novel instances. We also train another MP-Encoder on 80 instances from 8 different categories 3 to obtain a general viewpoint-sensitive feature extractor which we test on instances from novel categories. To retain the relative pose to the target view we use our random optimization scheme described in Alg. 1 with an initial σ = 45 o .

We compare our approach against DeepIm [25] that also minimizes the relative pose between an initial and a target view through an iterative rendering inference cycle. The results in Table 4 demonstrate superior performance of our approach on both, seen and unseen categories. Figure 3 on

3 airplane, bed, bench, car, chair, piano, sink, toilet

## 5. Conclusion

In this paper we presented a novel approach for estimating poses of multiple trained and untrained objects from a single encoder model. In contrast to other methods, training on multiple objects does not degrade performance and instead leads to state-of-the-art results on the textureless, symmetric objects of the T-LESS dataset. The same MP-Encoder architecture is also used for iterative pose refinement on untrained objects outperforming previous approaches on ModelNet40.

The ability of this pose estimator to generalize across objects from different categories, datasets and image domains indicates that low-level viewpoint-sensitive features are shared across various domains. Higher-level features for discriminating semantics require viewpoint-invariance and usually generalize less well. Therefore, our results suggest that these two tasks should be learned separately.

We believe that this is a step towards coping with the large number of instances in industrial settings where 3D models are often available and service robotics where categories often appear repeatedly and interactions with novel objects should be immediately possible.

## References

- [1] Vassileios Balntas, Andreas Doumanoglou, Caner Sahin, Juil Sock, Rigas Kouskouridas, and Tae-Kyun Kim. Pose Guided RGB-D Feature Learning for 3D Object Pose Estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3856-3864, 2017. 2, 3, 4
- [2] G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools , 2000. 5
- [3] Esteves Carlos, Sud Avneesh, Luo Zhengyi, Daniilidis Kostas, and Makadia Ameesh. Cross-domain 3d equivariant image embeddings. arXiv preprint arXiv:1812.02716 , 2018. 3
- [4] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 801-818, 2018. 11
- [5] Yang Chen and G´ erard Medioni. Object modelling by registration of multiple range images. Image and vision computing , 10(3):145-155, 1992. 6, 12
- [6] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2014. 6
- [7] Xinke Deng, Arsalan Mousavian, Yu Xiang, Fei Xia, Timothy Bretl, and Dieter Fox. Poserbpf: A rao-blackwellized particle filter for 6d object pose tracking. Robotics: Science and Systems (RSS) , 2019. 2
- [8] Maximilian Denninger, Martin Sundermeyer, Dominik Winkelbauer, Youssef Zidan, Dmitry Olefir, Mohamad Elbadrawy, Ahsan Lodhi, and Harinandan Katam. Blenderproc. arXiv preprint arXiv:1911.01911 , 2019. 2
- [9] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning , pages 647-655, 2014. 3
- [10] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88(2):303-338, 2010. 2
- [11] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html, 2012. 11
- [12] Nikolaus Hansen, Youhei Akimoto, and Petr Baudis. CMA-ES/pycma on Github. Zenodo, DOI:10.5281/zenodo.2559634, Feb. 2019. 6
- [13] Jiedong Hao, Jing Dong, Wei Wang, and Tieniu Tan. What is the best practice for cnns applied to visual instance retrieval? arXiv preprint arXiv:1611.01640 , 2016. 3
- [14] Kaiming He, Georgia Gkioxari, Piotr Doll´ ar, and Ross Girshick. Mask r-cnn. arXiv preprint arXiv:1703.06870 , 2017. 6
- [15] Stefan Hinterstoisser, Cedric Cagniart, Slobodan Ilic, Peter Sturm, Nassir Navab, Pascal Fua, and Vincent Lepetit. Gra-
- dient response maps for real-time detection of textureless objects. IEEE Transactions on Pattern Analysis and Machine Intelligence , 34(5):876-888, 2012. 2
- [16] Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab. Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes. In Asian conference on computer vision , pages 548-562. Springer, 2012. 6
- [17] Stefan Hinterstoisser, Vincent Lepetit, Naresh Rajkumar, and Kurt Konolige. Going further with Point Pair Features. In European Conference on Computer Vision , pages 834-848. Springer, 2016. 2
- [18] Tomas Hodan. SIXD Challenge 2017, http://cmp. felk.cvut.cz/sixd/challenge\_2017/ , 2017. 6, 13
- [19] Tom´ aˇ s Hodaˇ n, Pavel Haluza, Stˇ ep´n Obdrˇ z´lek, Jiˇ r´ ı Matas, ˇ a a Manolis Lourakis, and Xenophon Zabulis. T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects. IEEE Winter Conference on Applications of Computer Vision (WACV) , 2017. 6
- [20] Tom´ aˇ s Hodaˇ, Jiˇ r´ ı Matas, and Stˇp´n Obdrˇ´lek. n ˇ e a za On evaluation of 6D object pose estimation. In European Conference on Computer Vision , pages 606-619. Springer, 2016. 6
- [21] Tomas Hodan, Frank Michel, Eric Brachmann, Wadim Kehl, Anders GlentBuch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, et al. Bop: benchmark for 6d object pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 19-34, 2018. 2, 7, 14
- [22] Tomas Hodan, Frank Michel, Eric Brachmann, Wadim Kehl, Anders GlentBuch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, Caner Sahin, Fabian Manhardt, Federico Tombari, Tae-Kyun Kim, Jiri Matas, and Carsten Rother. Bop: Benchmark for 6d object pose estimation. In The European Conference on Computer Vision (ECCV) , September 2018. 1, 6
- [23] Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilic, and Nassir Navab. SSD-6D: Making RGB-Based 3D Detection and 6D Pose Estimation Great Again. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1521-1529, 2017. 1, 2
- [24] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. 11
- [25] Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, and Dieter Fox. Deepim: Deep iterative matching for 6d pose estimation. arXiv preprint arXiv:1804.00175 , 2018. 2, 3, 8
- [26] Zhigang Li, Gu Wang, and Xiangyang Ji. Cdpn: Coordinates-based disentangled pose network for real-time rgb-based 6-dof object pose estimation. In The IEEE International Conference on Computer Vision (ICCV) , October 2019. 2
- [27] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740-755. Springer, 2014. 2, 6
- [28] Siddharth Mahendran, Haider Ali, and Ren´ e Vidal. 3d pose regression using convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision , pages 2174-2182, 2017. 2
- [29] Fabian Manhardt, Wadim Kehl, Nassir Navab, and Federico Tombari. Deep model-based 6d pose refinement in rgb. In The European Conference on Computer Vision (ECCV) , September 2018. 3
- [30] Pat Marion, Peter R Florence, Lucas Manuelli, and Russ Tedrake. Label fusion: A pipeline for generating ground truth labels for real rgbd data of cluttered scenes. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 3325-3242. IEEE, 2018. 2
- [31] Kiru Park, Timothy Patten, and Markus Vincze. Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation. In The IEEE International Conference on Computer Vision (ICCV) , October 2019. 2
- [32] Mahdi Rad and Vincent Lepetit. BB8: A scalable, accurate, robust to partial occlusion method for predicting the 3D poses of challenging objects without using depth. In IEEE International Conference on Computer Vision (ICCV) , 2017. 2
- [33] Caner Sahin and Tae-Kyun Kim. Category-level 6d object pose recovery in depth images. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 0-0, 2018. 3
- [34] Martin Sundermeyer, Zoltan-Csaba Marton, Maximilian Durner, Manuel Brucker, and Rudolph Triebel. Implicit 3d orientation learning for 6d object detection from rgb images. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 699-715, 2018. 1, 2, 3, 6, 11
- [35] Martin Sundermeyer, Zoltan-Csaba Marton, Maximilian Durner, and Rudolph Triebel. Augmented autoencoders: Implicit 3d orientation learning for 6d object detection. International Journal of Computer Vision , Oct 2019. 1, 3
- [36] Supasorn Suwajanakorn, Noah Snavely, Jonathan J Tompson, and Mohammad Norouzi. Discovery of latent 3d keypoints via end-to-end geometric reasoning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31 , pages 2059-2070. Curran Associates, Inc., 2018. 3
- [37] Bugra Tekin, Sudipta N. Sinha, and Pascal Fua. Real-Time Seamless Single Shot 6D Object Pose Prediction. In CVPR , 2018. 2, 3
- [38] Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, and Stan Birchfield. Deep object pose estimation for semantic robotic grasping of household objects. In Conference on Robot Learning , pages 306-316, 2018. 1, 2, 3
- [39] Markus Ulrich, Christian Wiedemann, and Carsten Steger. CAD-based recognition of 3D objects in monocular images. In ICRA , volume 9, pages 1191-1198, 2009. 2
- [40] Joel Vidal, Chyi-Yeu Lin, and Robert Mart´ ı. 6D Pose Estimation using an Improved Method based on Point Pair Features. arXiv preprint arXiv:1802.08516 , 2018. 2
- [41] Chen Wang, Danfei Xu, Yuke Zhu, Roberto Mart´ ın-Mart´ ın, Cewu Lu, Li Fei-Fei, and Silvio Savarese. Densefusion:
- 6d object pose estimation by iterative dense fusion. CoRR , abs/1901.04780, 2019. 1
- [42] Paul Wohlhart and Vincent Lepetit. Learning descriptors for object recognition and 3D pose estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3109-3118, 2015. 2, 3, 4
- [43] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1912-1920, 2015. 4, 6
- [44] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.00199 , 2017. 2
- [45] Sergey Zakharov, Ivan Shugurov, and Slobodan Ilic. Dpod: Dense 6d pose object detector in rgb images. arXiv preprint arXiv:1902.11020 , 2019. 1, 2
- [46] Zhengyou Zhang. Iterative point matching for registration of free-form curves and surfaces. International journal of computer vision , 13(2):119-152, 1994. 6, 12

## A. Appendix

In the following we provide implementation details and show qualitative results. The code will be open sourced upon publication.

## A.1. Architecture

We closely follow [34] to design our Multi-Path (MP) encoder and decoders.

In our T-LESS experiments the encoder is symmetric to each of the decoders. The encoder consists of four convolutional layers with kernel size = 5, stride=2, ReLU activation and { 128 256 512 512 , , , } filters respectively. It follows, a shared fully connected layer with W 0 ∈ R 32768 x 128 . The resulting codes are propagated to the corresponding decoders via j ∈ 1 , .., n fully connected layers W j ∈ R 128 x 32768 . After reshaping, each decoder has four consecutive convolutional layers with kernel size = 5, stride=1, ReLU activation and { 512 512 256 128 , , , } filters. Nearest neighbor upsampling is performed after each convolutional layer. The final reconstruction output is preceded by a sigmoid activation.

We initially expected that an encoder with up to hundred corresponding decoders would require a much larger architecture to be able to adequately encode the views of all training objects. Therefore, we first adopted the [4], with a Resnet101 backbone and a spatial pyramid pooling (SPP) layer. Although, the total training loss can be reduced with such a network, the downstream performance on encoding object poses of both, trained and untrained objects, does not increase compared to a more shallow encoder/decoder with 4-5 plain convolutional layers as used by [34]. As usual, multiple explanations are plausible. Shallow features could contain all necessary information for class-agnostic pose estimation. The lost symmetry in the encoder-decoder architecture could make an invertible compression more difficult to learn. A deeper encoder could simply overfit more to synthetic and generalize less to real data. As architecture search and explanation is not our main goal here, we simply stick with the more efficient shallow CNN.

## A.2. Training Details

It is possible to train our architecture with up to 80 decoders on a single GPU. In this case, the encoder receives a joint batch size of 80, while each decoder has a batch size = 1. For single-GPU training on T-LESS with 18 decoders, we use a batch size = 4. We train for 300.000 iterations which takes about 48 hours on a single GPU. Multiple GPUs almost linearly speed up training due to the inherent parallelism in the architecture. Therefore, the batch and the decoders are divided into equal parts on all available GPUs. For the ModelNet experiments where we trained on 80 object instances, we used 4 GPUs with a total encoder batch

Figure 5: Our multi-path training process. Left: Joint augmented input batch with uniformly sampled SO(3) views of n object instances; Right: Individual reconstruction targets x j for each decoder.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00151v2-figure-8ff1e0b7-0cfd-48b5-b1aa-ddd62fb747f7.jpg)

size of 240 and decoder batch size of 4. Higher decoder batch sizes also stabilize the training.

We use Xavier initialization and the Adam [24] optimizer with a learning rate of 10 -4 × b dec , with b dec = decoder batch size. In our experiments we did not have to assign individual learning rates to the encoder and decoder but this could potentially accelerate the training.

## A.3. Synthetic Data Generation

For each instance we render 8000 object views sampled randomly from SO(3) at a constant distance of 700mm. The resulting images are quadratically cropped and resized to 128 × 128 × 3 .

## A.4. Augmentation Parameters

All geometric and color input augmentations besides the random lighting are applied online during training at uniform random strength (see Table 5). As background images we use Pascal VOC2012 [11]. One could argue that random background images during training are redundant where we use MaskRCNN. Training without backgrounds works, but in our experiments it does help with the generalization from synthetic to real data and is also beneficial when the MaskRCNN does not output perfect masks.

Table 5: Augmentation Parameters; Scale and translation is in relation to image shape

| color         | 50% chance (30% per channel)   | light (random position) &geometric   | light (random position) &geometric   |
|---------------|--------------------------------|--------------------------------------|--------------------------------------|
| add           | U ( - 0 . 1 , 0 . 1)           | ambient                              | 0 . 4                                |
| contrast      | U (0 . 4 , 2 . 3)              | diffuse                              | U (0 . 7 , 0 . 9)                    |
| multiply      | U (0 . 6 , 1 . 4)              | specular                             | U (0 . 2 , 0 . 4)                    |
| invert        |                                | scale                                | U (0 . 8 , 1 . 2)                    |
| gaussian blur | σ ∼ U (0 . 0 , 1 . 2)          | translation                          | U ( - 0 . 15 , 0 . 15)               |

Figure 6: Left: MP-Encoder trained on T-LESS objects 118, and tested on a metallic, industrial object; Right: ICPrefined result

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00151v2-figure-c439066c-09fe-426e-804c-5c8f50c9a05e.jpg)

## A.5. ICP Refinement

Optionally, the estimate is refined on depth data using a point-to-plane Iterative Closest Point (ICP) approach with adaptive thresholding of correspondences based on [5, 46]. The refinement is first applied in direction of the vector pointing from camera to the object where most of the RGBbased pose estimation errors stem from and then on the full 6D pose.

## A.6. Results on metallic parts

Webriefly tested the MP-Encoder trained on T-LESS objects on metallic objects from industry (Fig. 6).

## A.7. Full T-LESS results

Table 3 shows the full T-LESS results on each object tested on all scene views of the Primesense test set. Figure 7 show qualitative results with and without refinement on different T-LESS test scenes. Results on objects sometimes differ because the AAE submission uses RetinaNet (0.73mAP@0.5) and the MP-Encoder uses MaskRCNN (0.68mAP@0.5) because it does not learn to distinguish object and background (see Table 2).

## A.8. Runtime

The MaskRCNN with ResNet50 takes ∼ 150 ms for all instances in a scene on a modern GPU. Lighter architectures can be chosen for simpler problems. All resulting image crops can be batched together such that the inference of the MP-Encoder is parallelized. For a single instance the inference takes ∼ 5 ms , nearest neighbor search in the codebook takes 1 -2 ms and the projective distance estimation is negligible. While our RGB-based pipeline runs at high speed and makes our approach applicable for mobile applications, the depth-based ICP refinement takes in average 0 6 . s per target and is thus suitable for robotic manipulation tasks.

Table 6: Evaluation of the full 6D Object Detection pipeline with MaskRCNN + Multi-Path Encoder + optional ICP. We report the mean VSD recall following the SIXD Challenge [18] on the T-LESS Primesense test set.

|          | Template matching   | PPF based           | PPF based      | PPF based           | Learning-based      | Learning-based   | Learning-based     | Learning-based          | Learning-based          |
|----------|---------------------|---------------------|----------------|---------------------|---------------------|------------------|--------------------|-------------------------|-------------------------|
| object   | Hodan-15 Depth      | Vidal-18 Depth +ICP | Drost-10 Depth | Drost-10-edge Depth | Kehl-16 RGB-D + ICP | OURS RGB + ICP   | Brachmann-16 RGB-D | Sundermeyer-18 OURS RGB | Sundermeyer-18 OURS RGB |
| 1        | 66                  | 43                  | 34             | 53                  | 7                   | 73.61            | 8                  | 9.48                    | 5.56                    |
| 2        | 67                  | 46                  | 46             | 44                  | 10                  | 66.40            | 10                 | 13.24                   | 10.22                   |
| 3        | 72                  | 68                  | 63             | 61                  | 18                  | 87.24            | 21                 | 12.78                   | 14.74                   |
| 4        | 72                  | 65                  | 63             | 67                  | 24                  | 82.91            | 4                  | 6.66                    | 6.23                    |
| 5        | 61                  | 69                  | 68             | 71                  | 23                  | 86.16            | 46                 | 36.19                   | 37.53                   |
| 6        | 60                  | 71                  | 64             | 73                  | 10                  | 92.79            | 19                 | 20.64                   | 30.36                   |
| 7        | 52                  | 76                  | 54             | 75                  | 0                   | 80.83            | 52                 | 17.41                   | 14.62                   |
| 8        | 61                  | 76                  | 48             | 89                  | 2                   | 81.32            | 22                 | 21.72                   | 10.73                   |
| 9        | 86                  | 92                  | 59             | 92                  | 11                  | 85.15            | 12                 | 39.98                   | 19.43                   |
| 10       | 72                  | 69                  | 54             | 72                  | 17                  | 82.31            | 7                  | 13.37                   | 32.75                   |
| 11       | 56                  | 68                  | 51             | 64                  | 5                   | 72.60            | 3                  | 7.78                    | 20.34                   |
| 12       | 55                  | 84                  | 69             | 81                  | 1                   | 68.80            | 3                  | 9.54                    | 29.53                   |
| 13       | 54                  | 55                  | 43             | 53                  | 0                   | 53.37            | 0                  | 4.56                    | 12.41                   |
| 14       | 21                  | 47                  | 45             | 46                  | 9                   | 50.54            | 0                  | 5.36                    | 21.30                   |
| 15       | 59                  | 54                  | 53             | 55                  | 12                  | 45.25            | 0                  | 27.11                   | 20.82                   |
| 16       | 81                  | 85                  | 80             | 85                  | 56                  | 82.32            | 5                  | 22.04                   | 33.20                   |
| 17       | 81                  | 82                  | 79             | 88                  | 52                  | 72.27            | 3                  | 66.33                   | 39.88                   |
| 18       | 79                  | 79                  | 68             | 78                  | 22                  | 80.60            | 54                 | 14.91                   | 14.16                   |
| 19       | 59                  | 57                  | 53             | 55                  | 35                  | 48.17            | 38                 | 23.03                   | 9.24                    |
| 20       | 27                  | 43                  | 35             | 47                  | 5                   | 25.74            | 1                  | 5.35                    | 1.72                    |
| 21       | 57                  | 62                  | 60             | 55                  | 26                  | 47.53            | 39                 | 19.82                   | 11.48                   |
| 22       | 50                  | 69                  | 61             | 56                  | 27                  | 50.27            | 19                 | 20.25                   | 8.30                    |
| 23       | 74                  | 85                  | 81             | 84                  | 71                  | 46.99            | 61                 | 19.15                   | 2.39                    |
| 24       | 59                  | 66                  | 57             | 59                  | 36                  | 78.20            | 1                  | 4.54                    | 8.66                    |
| 25       | 47                  | 43                  | 28             | 47                  | 28                  | 50.00            | 16                 | 19.07                   | 22.52                   |
| 26       | 72                  | 58                  | 51             | 69                  | 51                  | 64.61            | 27                 | 12.92                   | 30.12                   |
| 27       | 45                  | 62                  | 32             | 61                  | 34                  | 73.09            | 17                 | 22.37                   | 23.61                   |
| 28       | 73                  | 69                  | 60             | 80                  | 54                  | 86.14            | 13                 | 24.00                   | 27.42                   |
| 29       | 74                  | 69                  | 81             | 84                  | 86                  | 77.66            | 6                  | 27.66                   | 40.68                   |
| 30       | 85                  | 85                  | 71             | 89                  | 69                  | 92.96            | 5                  | 30.53                   | 56.08                   |
| Average  | 63.18               | 66.51               | 56.81          | 67.5                | 24.6                | 69.53            | 17.84              | 19.26                   | 20.53                   |
| Time (s) | 13.5                | 4.7                 | 2.3            | 21.5                | 1.8                 | 0.8              | 4.4                | 0.1                     | 0.2                     |

Figure 7: Qualitative 6D Object Detection results on T-LESS Primesense scenes. Only 18 of 30 objects are used to train the Multi-Path encoder. Left: Predictions of the RGB-based pipeline; Middle-left: Error in depth [mm]; Middle-right: ICP-refined results; Right: Refined error in depth [mm]. Note that only one instance per class is predicted following the BOP [21] rules.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00151v2-figure-bf7a5fd0-8838-45f1-9317-0ce574226f2a.jpg)
Martin Sundermeyer Maximilian Durner En Yen Puang Zoltan-Csaba Marton Narunas Vaskevicius Kai O. Arras Rudolph Triebel
Multi-path Learning for Object Pose Estimation Across Domains
We introduce a scalable approach for object pose estimation trained on
simulated RGB views of multiple 3D models together. We learn an encoding of
object views that does not only describe an implicit orientation of all objects
seen during training, but can also relate views of untrained objects. Our
single-encoder-multi-decoder network is trained using a technique we denote
"multi-path learning": While the encoder is shared by all objects, each decoder
only reconstructs views of a single object. Consequently, views of different
instances do not have to be separated in the latent space and can share common
features. The resulting encoder generalizes well from synthetic to real data
and across various instances, categories, model types and datasets. We
systematically investigate the learned encodings, their generalization, and
iterative refinement strategies on the ModelNet40 and T-LESS dataset. Despite
training jointly on multiple objects, our 6D Object Detection pipeline achieves
state-of-the-art results on T-LESS at much lower runtimes than competing
approaches.