## ShapeCaptioner: Generative Caption Network for 3D Shapes by Learning a Mapping from Parts Detected in Multiple Views to Sentences

Zhizhong Han 1 3 , , Chao Chen 1 2 , , Yu-Shen Liu 1 2 , ∗ , Matthias Zwicker 3 1 School of Software, Tsinghua University, Beijing, China 2 Beijing National Research Center for Information Science and Technology (BNRist) 3 Department of Computer Science, University of Maryland, College Park, USA

h312h@umd.edu, thss15 chenc@163.com, liuyushen@tsinghua.edu.cn, zwicker@cs.umd.edu

## Abstract

3D shape captioning is a challenging application in 3D shape understanding. Captions from recent multi-view based methods reveal that they cannot capture part-level characteristics of 3D shapes. This leads to a lack of detailed part-level description in captions, which human tend to focus on. To resolve this issue, we propose ShapeCaptioner, a generative caption network, to perform 3D shape captioning from semantic parts detected in multiple views. Our novelty lies in learning the knowledge of part detection in multiple views from 3D shape segmentations and transferring this knowledge to facilitate learning the mapping from 3D shapes to sentences. Specifically, ShapeCaptioner aggregates the parts detected in multiple colored views using our novel part class specific aggregation to represent a 3D shape, and then, employs a sequence to sequence model to generate the caption. Our outperforming results show that ShapeCaptioner can learn 3D shape features with more detailed part characteristics to facilitate better 3D shape captioning than previous work.

## 1. Introduction

Jointly understanding 3D shapes and sentences is an important challenge in 3D shape analysis. For example, generated captions are helpful for visually impaired people to understand what a 3D shape looks like, including the category, color, form, and material of the 3D shape, as shown in the examples in Fig. 1. This motivates us to address the issue of automatically generating captions for 3D shapes.

Recently, Text2Shape [1] made an important contribution by proposing a 3D-Text dataset, where voxel-based 3D shapes and their corresponding captions are paired together. To reduce voxel complexity by representing 3D

∗ Corresponding Author. This work was supported by National Key R&amp;D Program of China (2018YFB0505400) and NSF (award 1813583).

Figure 1. Examples of human annotated captions for 3D shapes. Humans tend to describe 3D shapes by focusing on their semantic parts (in red) with their various attributes (in blue).

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-c15e517b-1d75-49ef-a481-17aeb7f63d80.jpg)

shapes as view sequences, Y 2 Seq2Seq [15] was presented to jointly learn a bilateral mapping between view sequences and word sequences. Although Y 2 Seq2Seq can produce plausible captions for 3D shapes, its ability to generate detailed description for parts is limited because the model does not capture part-level characteristics. However, humanprovided captions often focus on part-level details, as the manually annotated ground truth examples in Fig. 1 illustrate. Therefore, how to absorb part characteristics in 3D shape captioning remains challenging.

To resolve this issue, we propose a novel deep neural network, ShapeCaptioner , to generate captions for 3D shapes. By representing a 3D shape as a view sequence, ShapeCaptioner aims to learn a mapping from parts detected in the view sequence to a caption describing the 3D shape. Specifically, ShapeCaptioner first leverages 3D shape segmentation benchmarks to learn the knowledge of detecting parts in terms of their geometry from multiple views, and then, transfers this knowledge to incrementally learn the ability of detecting parts with various attributes from multiple colored views in 3D-Text dataset. ShapeCaptioner further employs a novel part class specific aggregation to aggregate the detected parts over all views for the captioning of a 3D shape. The part class specific aggregation can represent a 3D shape by capturing more part characteristics from different views, which facilitates more detailed 3D shape captioning. In summary, our contributions are as follows:

- i) We propose ShapeCaptioner to enable 3D shape captioning from semantic parts detected in multi-views,

which facilitates more detailed 3D shape captioning.

- ii) We introduce a method to learn the knowledge of part detection in multi-views from 3D shape segmentation benchmarks to facilitate 3D shape captioning.
- iii) With our novel part class specific aggregation, we effectively capture part charicteristics to represent 3D shapes for better shape captioning, inspired by the way humans describe 3D shapes in terms of semantic parts.

## 2. Related work

Image captioning. There is a large volume of work on image captioning. Here we review methods based on object detection, which are most similar to our work. Currently, end-to-end deep learning approaches [22, 21] are most effective for image captioning. These methods try to learn to generate captions from global image features. However, using a global feature limits their interpretability. Other work [4, 35, 37, 36, 34, 19] employs object-level semantics to generate captions. These methods represent images based on occurring semantic concepts or objects. In [36], explicitly detected objects are employed with their category, size and layout to generate captions. In contrast, based on a bag of word model, impressive captions can also be generated by only using the explicitly detected objects [34].

Our method is different from these methods in four aspects. First, the parts we want to detect are more various than the objects. Second, part detection becomes more challenging when considering multiple unaligned and varying viewpoints. Third, how to aggregate the detected parts over different views to enable learning the mapping from 3D shapes to captions represents an additional problem. Fourth, a final obstacle is that there is no labeled dataset available to learn the knowledge of part detection in multiple views of 3D shapes. Yet ShapeCaptioner can resolve these issues to generate captions with detailed part characteristics.

3D shape captioning. Although deep learning models have led to significant progress in feature learning for 3D shapes [9, 8, 11, 10, 14, 16, 6, 17, 12, 7, 13, 24], 3D shape captioning has been less explored due to the lack of training dataset. However, the recently proposed 3D-Text dataset [1] has enabled the research in this area. Y 2 Seq2Seq [15] employs multiple views as a 3D shape representation to address the cubic complexity of voxel representations. It learns 3D shape features by aggregating the global feature of each view. Although Y 2 Seq2Seq can generate plausible captions for 3D shapes, it often fails to generate captions with local part details. To resolve this issue, ShapeCaptioner represents a 3D shape as a set of parts detected in multiple views, which not only avoids the cubic complexity of voxels but also enables the ability of capturing part characteristics. This leads to captions that are more similar to the manually annotated ground truth, which usually includes part details, such as color, form, material, and texture of parts shown in Fig. 1.

## 3. Overview

ShapeCaptioner learns to generate a caption t for a 3D shape s under a 3D-Text dataset D which provides ground truth ( s t , ) pairs, where shape s is represented by a 128 3 dimensional voxel cube and contains various attributes, such as color, material and texture, as demonstrated in Fig. 1. ShapeCaptioner represents shape s as a colored view sequence v = { v , i i ∈ [1 , V ] } , and detects important parts from each view v i to capture part characteristics for caption generation. However, dataset D does not have ground truth parts in multiple views for ShapeCaptioner to learn from.

To resolve this problem, we first leverage a separate 3D shape segmentation benchmark B to learn the knowledge of part geometry detection in multiple views, i.e., detecting parts only in terms of geometry rather than attributes, as demonstrated in Fig. 2 (a) and (b). This is because there is no color, material or texture available in benchmark B . Then, we transfer this knowledge to 3D-Text dataset D to establish the ground truth parts in multiple colored views which enables ShapeCaptioner to incrementally learn the ability of detecting parts in terms of both geometry and attributes, as demonstrated in Fig. 2 (b), (c) and (d). Finally, ShapeCaptioner learns a mapping from parts detected in multiple views of shape s to its description t to facilitate 3D shape captioning, as demonstrated in Fig. 2 (d), (e) and (f).

## 4. ShapeCaptioner

Part geometry detection. As shown in Fig. 2 (a), we propose a method to obtain part geometry ground truth in multiple views from 3D shape segmentation benchmark B . We use the 3D shape segmentation benchmarks involved in [20] as segmentation ground truth.

As illustrated in Fig. 3, starting from each 3D mesh in benchmak B , our method first voxelizes the 3D mesh into voxels, along with labelling each voxel according to the segmentation ground truth on the mesh. In the voxelization, we randomly sample 100 points on each triangle face of the mesh, and label each sampled point by the label of the triangle face. This enables us to perform label voting among points located in the same voxel in the labelling of each voxel. Using randomly sampled points could resist the imbalance effect of triangle face size. Then, we render the voxelized shape from 12 viewpoints and locate different part geometries in each view. Specifically, from each viewpoint, we separately highlight voxels belonging to the same part class in blue, and compute the bounding box of each blue region. We denote p ′ as a one hot probability distribution to indicate which part class the bounding box belongs

Figure 2. The demonstration of ShapeCaptioner. ShapeCaptioner first learns the knowledge of detecting part geometry in multiple views from shape segmentation benchmark B ((a) and (b)). By transferring this knowledge to 3D-Text dataset D , ShapeCaptioner learns the ability of detecting parts containing both geometry and attributes ((b), (c) and (d)). Finally, ShapeCaptioner generates captions by learning the mapping from the detected parts to the description ((d), (e) and (f)).

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-45ac07e8-4022-43d6-b13e-f96e1f4bb08f.jpg)

Figure 3. Illustration of generating part geometry ground truth in multiple views from 3D shape segmentation benchmarks.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-d931b35c-6b78-4c48-b09b-b46909dea131.jpg)

to, and denote l ′ as the location of the bounding box, where ( p , l ′ ′ ) forms the part geometry ground truth.

Take the first viewpoint in Fig. 3 for example, we separately obtain the bounding box of the region formed by voxels in back, arm, seat and leg class of a chair, and finally, we obtain the part geometry ground truth in the first view. By repeating this process, we obtain the part geometry ground truth ( p , l ′ ′ ) in multiple views, as shown by the bounding box on shapes in the dashed box in Fig. 3.

The reason why we obtain views with part geometry ground truth by rendering the voxelized shapes in benchmark B is to avoid the domain gap. Because 3D shapes in 3D-Text dataset D are voxelized, there would be a domain gap if we learn from part geometry ground truth in views rendered from meshes, while detecting part geometry from views rendered from these voxelized shapes.

Subsequently, we train a fasterRCNN [26] as a part geometry detector G under the obtained part geometry ground truth ( p , l ′ ′ ) , as shown in Fig. 2 (b). This enables ShapeCaptioner to detect part geometries ( p , l g g ) from any view v by minimizing the objective function below,

$$O _ { G } ( p _ { g }, p ^ { \prime }, l _ { g }, l ^ { \prime } ) = O _ { p } ( p _ { g }, p ^ { \prime } ) + \lambda O _ { l } ( l _ { g }, l ^ { \prime } ), \quad ( 1 )$$

where O p measures the accuracy in terms of probability by the cross-entropy function of part class labels, while O l measures the accuracy in terms of location by the robust L 1 function as in [5]. The parameter λ balances O p and O l , and a value of 1 works well in all our experiments.

Parts detection. Part geometry detection is not enough for ShapeCaptioner to generate captions with detailed part characteristics, since the detected geometries lack various attributes, such as color or texture. Hence, it also needs to be able to detect parts with attributes.

To resolve this issue, we transfer the learned knowledge of part geometry detection to the 3D-Text dataset D . We introduce a bounding box mapping method to establish part ground truth which contains both geometry and attributes for ShapeCaptioner to learn from.

As demonstrated in Fig. 2 (b), in the test stage of part geometry detector G , we first render multiple views from each 3D shape in the training set of dataset D without color, and then, employ the trained part geometry detector G to detect part geometries ( p , l g g ) in these views. This process is further illustrated by the first column in Fig. 4. Subsequently, we render the 3D shape again but with color, and map the detected part geometries ( p , l g g ) to the corresponding views with color. Finally, we regard the mapped part geometries in views with color as the part ground truth ( p ′′ , l ′′ ) , where l ′′ = l g and p ′′ is a one hot probability distribution by setting the entry of argmax p ( g ) to be 1. Here, we map the detected part geometries with p g &gt; . 0 7 .

Finally, we further push the part geometry detector G to detect parts ( p, l ) from multiple views with color by finetuning G under the part ground truth ( p ′′ , l ′′ ) . We rename G as part detector R , as shown in Fig. 2 (d). Note that these part ground truth ( p ′′ , l ′′ ) are only obtained from shapes in the training set of 3D-Text dataset D . After training, R can detect parts from multiple colored views of shapes in the

Figure 4. Illustration of generating part ground truth in multiple views from 3D-Text dataset.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-993b04be-48e3-4931-920f-b94c50febeb6.jpg)

test set of dataset D . Similar to Eq. (1), we minimize the following objective function with a λ of 1,

$$O _ { R } ( p, p ^ { \prime \prime }, l, l ^ { \prime \prime } ) = O _ { p } ( p, p ^ { \prime \prime } ) + \lambda O _ { l } ( l, l ^ { \prime \prime } ). \quad ( 2 ) \quad \mathbb { S }. 1$$

Part class specific aggregation. Given a 3D shape s , ShapeCaptioner generates its caption t from the parts detected in multiple colored views v i of s , as demonstrated in Fig. 2 (e). We denote the j -th part detected by R from v i as ( p , l j i j i ) . We select the detected parts with p j i &gt; ρ to represent the 3D shape s , where ρ is a probability threshold.

ShapeCaptioner aggregates the selected parts { ( p , l j i j i ) } over all V views in terms of different part classes to represent shape s , as demonstrated in Fig. 2 (e). For each one of C part classes, we group the parts ( p , l j i j i ) belonging to the same c -th part class if argmax p ( j i ) is c . Then, we obtain a part class specific feature F c by aggregating all parts in the c -th part class using pooling procedure as follows,

$$F _ { c } = \text{pool} _ { a r g m a x ( p _ { i } ^ { j } ) = = c } ( f _ { i } ^ { j } ), i \in [ 1, V ], \quad ( 3 ) \quad \text{whi} _ { \text{test} }$$

where f j i is a 4096 dimensional feature extracted from the fc7 layer of part detector R . Finally, we use F to represent shape s as a sequence of part class specific feature F c as follows, where c ∈ [1 , C ] ,

$$F = [ F _ { 1 }, \dots, F _ { c }, \dots, F _ { C } ]. \quad \quad ( 4 ) \quad \underset { \mathfrak { s c } } { \mathfrak { s } }$$

The reason we represent a 3D shape s as a combination of part class specific features F c is to preserve as much part characteristics as possible in the aggregation process. This could reduce the impact of one part class on the others in the multi-view scenario, which enables F c to comprehensively describe what the semantic part looks like over all V views of shape s .

Captioning from parts. ShapeCaptioner leverages the sequence F of part class specific features F c from s to generate a sequence of words t n as a caption t , where t =

[ t 1 , ..., t n , ..., t N ] and n ∈ [1 , N ] . We cast this problem into a sequence to sequence translation model, and implement this seq2seq model by a RNN encoder and a RNN decoder, as demonstrated in Fig. 2 (f). The RNN encoder encodes F by inputting F c at each one of C steps, while the RNN decoder dynamically decodes each word t n in t . Thus, ShapeCaptioner learns the mapping from 3D shape to sentences by minimizing the following objective function,

$$O _ { t } = - \sum _ { t _ { n } \in t } \log p ( t _ { n } | t _ { < n }, F ), \text{ \quad \ \ } ( 5 )$$

where t n is the n -th word in the word sequence t , t &lt;n represents the words in front of t n , p t ( n | t &lt;n , F ) is the probability of correctly predicting the n -th word according to the previous words t &lt;n and the 3D shape feature F . Note that the optimization is conducted under the training set of 3DText dataset D . After training, we can generate captions for shapes in the test set of dataset D .

## 5. Experimental results and analysis

We evaluate ShapeCaptioner in this section. We first explore the effects of some important parameters on the performance of ShapeCaptioner. Then, we conduct some ablation studies to justify the effectiveness of some modules. Finally, we compare it with some state-of-the-art methods. Dataset and metrics. We evaluate ShapeCaptioner under the 3D-Text cross-modal dataset [1], which consists of a primitive subset and a ShapeNet subset. We only employ the ShapeNet subset, because the 3D shapes in the primitive subset are too simple to extract parts. The ShapeNet subset contains 15,038 shapes and 75,344 descriptions in the chair and table classes. We employ the same training/test splitting in each shape class as [1, 15]. Specifically, the chair class is formed by 5954 training shapes and 641 test shapes while the table class contains 7592 training shapes and 851 test shapes. In addition, we employ 487 chairs (out of 537) and 481 tables (out of 520) involved in segmentation benchmarks [20] to train part geometry detector G , respectively, which avoids the 3D shapes that are also in the test set of 3D-Text dataset.

We employ BLUE [25], CIDEr [31], METEOR [3], and ROUGE [23] to evaluate the quality of generated captions according to the ground truth captions, where these metrics are abbreviated as 'B-1', 'B-2', 'B-3', 'B-4', and 'C', 'M', 'R', respectively in the following tables.

Initialization. We extract 3587 unique words from captions of 3D shapes in the training set of ShapeNet subset to form the vocabulary for caption generation. Each word is represented by a 512 dimensional embedding, which is learned along with the other involved parameters in training. Both RNN encoder and decoder are implemented by GRU cells [2]. The learning rate in all the experiments is 0.00001.

Figure 5. Comparison between captions generated by different methods. Compared to ground truth, our method can generate captions with more accurate part characteristics, where ρ &gt; 0 8 . .

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-eb72376c-3ddd-4691-ac8a-c19ff3f356e2.jpg)

Table 1. The comparison on H ρ . = 0 8 . , V = 12 .

32: chair with spherical, revolving steel chair. one vertical central and a round support at bottom. leg;

gray chair with a black back a aaaaaaaaaaaaaaaaaaaaaa.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-647ca80f-9fc3-4949-a942-aa40088f0397.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-0785388b-85c0-405f-ad10-4621ffee4c4b.jpg)

- a black colored chair with a black seat and a black seat and a round support at bottom.

chair a red seat and a red seat. central and round support at bottom. with leg;

|   H |   B-1 |   B-2 |   B-3 |   B-4 |     M |     R |     C |
|-----|-------|-------|-------|-------|-------|-------|-------|
|  16 | 0.122 | 0.072 | 0.041 | 0.021 | 0.121 | 0.168 | 0.002 |
|  32 | 0.937 | 0.917 | 0.894 | 0.878 | 0.55  | 0.847 | 1.789 |
|  64 | 0.794 | 0.639 | 0.537 | 0.475 | 0.284 | 0.551 | 0.644 |
| 128 | 0.761 | 0.604 | 0.493 | 0.421 | 0.273 | 0.529 | 0.56  |
| 256 | 0.772 | 0.615 | 0.503 | 0.431 | 0.276 | 0.534 | 0.576 |
| 512 | 0.736 | 0.559 | 0.442 | 0.371 | 0.265 | 0.503 | 0.506 |

In addition, the number of part classes is C = 4 in the chair class, while C = 3 in the table class, and ShapeCaptioner is trained under each class respectively. We employ ρ = 0 8 . to select parts detected in V = 12 views to represent a 3D shape, and then, use max pooling to aggregate all the selected parts over V views.

Parameters. Here we compare some important parameters under the chair class.

We first explore the effect of dimension H of the RNN encoder and decoder by comparing H ∈ { 16 32 64 128 256 512 , , , , , } . As shown in Table 1, the performance is increased with increasing H until H = 32 , and then, degenerates gradually when H becomes larger. We believe this is caused by overfitting because the training data is not large enough. In addition, we compare the generated captions under different H in Fig. 6, where the detected parts for caption generation are also briefly shown on the views. We also observe the gradually degenerated captions when H becomes larger. For example, the color description becomes inaccurate in the caption with H = 128 . Also, some descriptions are repeating in the captions with H = 256 and H = 512 . In the following, we set H to 32.

Then, we explore the probability threshold ρ of selecting parts to represent a 3D shape. We compare ρ ∈

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-7448a44e-9b57-4c6c-a33f-5f938475732e.jpg)

red round chair. the chair have a round bottom. and a round support at bottom. shape a red color chair with a round base and a circular base leg and a round support at bottom.

a half spherical, revolving steel chair. one vertical central and a round support at bottom.

Figure 6. Comparison between generated captions under different dimensions of the RNN hidden state.

Table 2. The comparison on ρ . H = 32 , V = 12 .

|   ρ |   B-1 |   B-2 |   B-3 |   B-4 |     M |     R |     C |
|-----|-------|-------|-------|-------|-------|-------|-------|
| 0.6 | 0.225 | 0.14  | 0.085 | 0.053 | 0.154 | 0.261 | 0.046 |
| 0.7 | 0.715 | 0.576 | 0.476 | 0.415 | 0.26  | 0.515 | 0.523 |
| 0.8 | 0.937 | 0.917 | 0.894 | 0.878 | 0.55  | 0.847 | 1.789 |
| 0.9 | 0.463 | 0.409 | 0.383 | 0.366 | 0.235 | 0.366 | 0.428 |

{ 0 6 . , 0 7 . , 0 8 . , 0 9 . } . There would be more selected parts when ρ is smaller while the quality of the selected parts is lower, and vice versa, as demonstrated in Fig. 7. We believe both the number and the quality would affect the discriminability of 3D shape features. This is because smaller number of parts would decrease the ability of resisting the effect of viewpoint changing while lower quality would contain inaccurate part characteristics. In Table 2, we found ρ = 0 8 . performs best, hence we use this setting in the following.

Finally, we want to know how the number V of views affects the performance. In this experiment, we use the first

Figure 7. Comparison between detected parts in three views under different probability thresholds ρ .

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-3adbe1aa-0d12-4b13-9505-ea017a7b10c2.jpg)

Table 3. The comparison on V . H = 32 , ρ = 0 8 . .

|   V |   B-1 |   B-2 |   B-3 |   B-4 |     M |     R |     C |
|-----|-------|-------|-------|-------|-------|-------|-------|
|   1 | 0.324 | 0.21  | 0.137 | 0.097 | 0.158 | 0.32  | 0.061 |
|   4 | 0.777 | 0.638 | 0.54  | 0.478 | 0.292 | 0.567 | 0.724 |
|   8 | 0.876 | 0.779 | 0.707 | 0.656 | 0.374 | 0.674 | 1.108 |
|  12 | 0.937 | 0.917 | 0.894 | 0.878 | 0.55  | 0.847 | 1.789 |

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-dbabded0-3b47-48e2-8763-1b81e0e23ddb.jpg)

- a brown chair with a a back and a back and a back. a a

brown chair with a cushion and back rest. physical appearance wooden with 2 hand for rest.

a chair with a back rest. colored, solid physical

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-6ee203a0-25d3-457b-8558-be1d25323708.jpg)

- chair with swing chair, light brown colored, solid physical appearance wooden with 2 hand for rest. GT: rectangular resting chair, light brown colored, solid physical appearance wooden with 2 hand for rest . swing

Figure 8. Comparison between generated captions under different numbers of views.

{ 1 4 8 12 , , , } views to select parts, and then, learn to generate captions. As shown in Table 3, the performance keeps improving along with the increasing number of views. This comparison shows that more views would provide more part characteristics to learn from, which also decreases the effect of inaccurately detected parts. This can also be observed in the comparison of the generated captions under different V in Fig. 8, where the parts employed for caption generation are also briefly shown. For example, we cannot get plausible captions when too few views are available, such as V = 1 . The material and the color gradually appear in the captions when V increases from 4 to 12. We do not explore the results with more views because of the limited computational capacity. In the following, we use V = 12 .

Ablation studies. We first highlight our part class specific aggregation for representing a 3D shape. In the former experiments, we employ max pooling to obtain the part class specific aggregation F c . Here, we try mean pooling to com-

|        |   B-1 |   B-2 |   B-3 |   B-4 |     M |     R |     C |
|--------|-------|-------|-------|-------|-------|-------|-------|
| Mean   | 0.22  | 0.133 | 0.077 | 0.044 | 0.148 | 0.253 | 0.018 |
| Mixed  | 0.843 | 0.744 | 0.668 | 0.617 | 0.36  | 0.653 | 1.019 |
| Max(L) | 0.722 | 0.541 | 0.433 | 0.374 | 0.239 | 0.482 | 0.422 |
| MaxAll | 0.714 | 0.593 | 0.52  | 0.476 | 0.299 | 0.535 | 0.694 |
| Max    | 0.937 | 0.917 | 0.894 | 0.878 | 0.55  | 0.847 | 1.789 |

Table 4. The part aggregation. H = 32 , ρ = 0 8 . , V = 12 .

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-12eae429-6385-4d95-9a38-add46ddbbaaa.jpg)

gray chair with a black cushion and a cushion and back and a cushion and back and a cushion and back.

chair with colored chair; with red colored seating provide; with design. good brown chair with a cushion. colored seating provide; with design good

[MaxAll: a modern and black seat and sit and the design stylish colored chair; with red colored provide, with design. seating good

Figure 9. Comparison between generated captions under different part aggregation methods.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-bf2d8fe6-2a38-4038-9013-417a65114b04.jpg)

pare with the results of max pooling. As shown by the result of 'Mean' in Table. 4, we find that mean pooling is not as good as max pooling ('Max') to aggregate parts in multiple views. To further justify this point, we conduct another experiment to combine max pooling and mean pooling together. Specifically, we use max pooling to aggregate parts in the same part class in the same view while further using mean pooling to obtain F c by aggregating the same part class over different views. As shown by the result of 'Mixed', although it is better than the result of 'Mean', it is still worse than the result of 'Max'. In addition, we also highlight the idea of part class specific features. As shown by the result of 'MaxAll', we max pool all parts over views into a single feature while ignoring the part class, and leverage this feature to generate captions. The degenerated results show that parts in different classes would affect each other in the aggregation. In addition, we compare our employed GRU with LSTM cell in RNN. As shown by 'Max(L)', GRU cell is more suitable in our problem. The captions in Fig. 9 also show the similar comparison results.

Subsequently, we highlight the advantage of our part based features over view based features. Similar to [15], we employ a VGG19 [28] to extract the feature of each view, and employ a RNN encoder to aggregate these view features for caption generation. As shown in Table. 5, without the multi-task scenario and constraints in [15], this aggregation by RNN ('RNN') cannot obtain satisfactory results. In addition, we also try a CNN+RNN architecture similar to the method of image captioning [33]. We employ the first view of each shape to caption a 3D shape. Although the results of 'CNN' are a little bit better than 'RNN', there is still room to improve. We further try to pool the V = 12 view features together into a single feature, and convey this feature to the RNN decoder for caption generation. We find

Table 5. Representations in chair class. H = 32 , ρ = 0 8 . , V = 12 .

|       |   B-1 |   B-2 |   B-3 |   B-4 |     M |     R |     C |
|-------|-------|-------|-------|-------|-------|-------|-------|
| RNN   | 0.231 | 0.148 | 0.09  | 0.055 | 0.172 | 0.252 | 0.009 |
| CNN   | 0.337 | 0.218 | 0.173 | 0.152 | 0.153 | 0.285 | 0.221 |
| VMax  | 0.409 | 0.244 | 0.181 | 0.157 | 0.176 | 0.307 | 0.229 |
| VMean | 0.494 | 0.338 | 0.251 | 0.214 | 0.209 | 0.381 | 0.301 |
| Part  | 0.937 | 0.917 | 0.894 | 0.878 | 0.55  | 0.847 | 1.789 |

Table 6. Representations in table class. H = 32 , ρ = 0 8 . , V = 12 .

|       |   B-1 |   B-2 |   B-3 |   B-4 |     M |     R |     C |
|-------|-------|-------|-------|-------|-------|-------|-------|
| RNN   | 0.31  | 0.194 | 0.11  | 0.054 | 0.161 | 0.317 | 0.073 |
| CNN   | 0.451 | 0.279 | 0.186 | 0.143 | 0.19  | 0.343 | 0.228 |
| VMax  | 0.45  | 0.285 | 0.21  | 0.18  | 0.192 | 0.344 | 0.283 |
| VMean | 0.53  | 0.367 | 0.274 | 0.229 | 0.225 | 0.414 | 0.385 |
| Part  | 0.86  | 0.755 | 0.675 | 0.62  | 0.362 | 0.664 | 1.099 |

both mean pooling ('VMean') and max pooling ('VMax') works well on aggregating views to generate captions, and mean pooling is better than max pooling. However, all these view based shape features cannot capture part characteristics to generate better captions than our part based features ('Part'). Moreover, we also observe similar results in the table class in Table 6.

In addition, we elaborate on the results in Table 5 and Table 6 by cumulative distribution in Fig. 12. For each metric, we use 11 values as probes, and calculate the percentage of samples over the whole test set whose metric scores are bigger than each probe, respectively. These comparisons also demonstrate our significant improvement by higher percentage.

Visualization of detected parts. ShapeCaptioner employs an effective way of detecting parts from multiple colored views of a 3D shape. As demonstrated by the consistent part detection results in Fig. 10, where the bounding boxes of the detected parts are shown on all the V = 12 views, ShapeCaptioner can detect reasonable parts for caption generation by understanding the complex geometry of semantic parts without the impacts by viewpoints and colors.

Comparison with other methods. We evaluate ShapeCaptioner by comparing it with the state-of-the-art methods. As shown in Table 7, we compare to the methods which are able to generate captions from images or image sequences, such as SandT [33] for image captioning, V2T [32] for video captioning, GIF2T [30] for GIF understanding, SLR [27] for video understanding, and Y 2 Seq2Seq ('Y2S') [15] for 3D shape understanding. The results of SandT are produced with mean pooling of all views as shape feature for caption generation, while the results of GIF2T and SLR are produced by the nearest retrieval in the joint feature space of shape and caption.

We can see that ShapeCaptioner ('Ours') significantly outperforms the other view-based methods in all metrics, where 'Our' is the average of results under chair ('Ours(C)') and table ('Ours(T)') classes. We believe our

Table 7. The comparison with others. H = 32 , ρ = 0 8 . , V = 12 .

| Method     |   B-1 |   B-2 |   B-3 |   B-4 |     M |     R |     C |
|------------|-------|-------|-------|-------|-------|-------|-------|
| SandT [33] | 0.494 | 0.338 | 0.251 | 0.214 | 0.209 | 0.381 | 0.301 |
| V2T [32]   | 0.67  | 0.43  | 0.26  | 0.15  | 0.21  | 0.45  | 0.27  |
| GIF2T [30] | 0.61  | 0.35  | 0.21  | 0.12  | 0.16  | 0.36  | 0.14  |
| SLR [27]   | 0.4   | 0.17  | 0.08  | 0.04  | 0.11  | 0.24  | 0.05  |
| Y2S [15]   | 0.8   | 0.65  | 0.54  | 0.46  | 0.3   | 0.56  | 0.72  |
| Ours(C)    | 0.937 | 0.917 | 0.894 | 0.878 | 0.55  | 0.847 | 1.789 |
| Ours(T)    | 0.86  | 0.755 | 0.675 | 0.62  | 0.362 | 0.664 | 1.099 |
| Ours       | 0.899 | 0.836 | 0.785 | 0.749 | 0.456 | 0.756 | 1.444 |

results benefit from the ability of understanding parts of 3D shapes, which captures more part characteristics to generate better captions in a more similar way to humans. We also obtain the captions in the test set generated by Y 2 Seq2Seq from the author, and compare it and SandT ('VMean') to ShapeCaptioner in Fig. 12 in all metrics, where we also observe significant improvement.

Real image test. We further evaluate ShapeCaptioner under a real image set in Fig. 11. What we want to show here is that ShapeCaptioner can also help to caption images by leveraging the detected parts, although it is only trained under 3D data. We select real chair and table images from the Stanford Online Products dataset [29], and we use the trained ShapeCaptioner ('Ours(C)' and 'Ours(T)') in Table 7 to conduct the results. We first compare our method to the state-of-the-art image captioning methods including CaptionBot [18] and NeuralTalk2 [21] in Fig. 11. We find ShapeCaptioner can generate more detailed and accurate descriptions for parts in captions. This benefits from the detected semantic parts with part details, which also justifies that ShapeCaptioner has the ability to overcome the gap between rendered views and real images. Note that the generated captions lack variety due to the absence of training on the real images and captions.

To further evaluate the generated captions, we conduct a user study over randomly selected 30 chairs and tables. We provide all 66 participants each one of the 30 cases accompanied with the generated captions. Then, we ask the participants to select one of three options for each case, i.e., perfectly correct (All descriptions about parts are correct), roughly correct (Some descriptions about parts are correct), and totally wrong (All descriptions about parts are wrong), to evaluate how well the generated caption matches the shape in the real image. Finally, we show the statistical results in Fig. 13 (a). We can see that participants gave almost 90% cases ('Avg') perfectly and roughly correct, where the results on each case are elaborated in Fig. 13 (b). In addition, our results in the table class ('Table') are better than the results in the chair class ('Chair') in terms of percentage of perfectly correct. This is because tables are usually simpler than chairs, and there are more training samples in the table class than the chair class in the 3D-Text dataset.

Figure 10. Demonstration of part detection under 3D-Text dataset, where ρ &gt; 0 8 . . The same color in the first two rows (four chairs) or the last two rows (four tables) indicates the same part class.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-9ca56af4-2375-4f82-bf75-03b3908e71db.jpg)

red color chair with a square shape seat and it be a chair with a red cushion.

back rest. think its a red chair in a room large black chair with a brown colored chair with a round back and seat.

brown colored wooden chair with four and back rest. leg brown chair with a round back and a round back.

black cushion and a back rest. think it's a close up of a chau:

bed with a red blanket and a red chair.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-e8dffc4f-5402-4d58-bd23-9d2858420f23.jpg)

think it's a chair sitting in front of wooden table an old fashioned chair with a laptop on it.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-06ab4d21-1ab8-41d5-9dac-1cc046ad0ea4.jpg)

## wooden table with a drawer and a shelf underneath.

Ithink it's a person sitting on a chair in aroon.

- wooden bench in front of window. sitting
- brown colored wooden table with four leg.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-cdd1b7cc-e3ba-4383-bc40-71ccb53666f3.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-99e01bf3-0bd4-4bef-a411-51349b60d866.jpg)

wooden bench in the near a tree sitting grass a white bed with a white comforter and white blanket.

brown wooden table with a glass top and a shelf underneath.

really describe the picture but I do see indoor table white. a white and black cat sitting on a table

Ithink it's a wooden bench in the grass. sitting think it's a bird chair. sitting

a cat sitting on a chair in a room

Ithink it's a close up of a chair:

of scissors on a table. pair sitting think it's a close up of stool.

- of scissors sitting on a table. sitting on top pair

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-465e2e7d-80fc-4df8-a863-88d1f618094f.jpg)

small bird of a table.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-6f80e284-d30b-4722-baf7-d57195584807.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-487827ef-ebc7-4d57-9c36-24a912ea98cb.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-0e399205-2be9-4b1e-b99b-7d028c838455.jpg)

brown wooden table with a shelf

Ithink it's a close up of furniture.

a brown colored wooden table with a shelf on the top and a shelf on the bottom.  underneath. think it's a book on top of a table. sitting square top.

think it's a chair that table. sitting wooden table with a white and blue cup on it.

Ithink it's a wooden table.

- wooden table with chair and a chair:

on top of a wooden table. sitting a brown colored wooden table with four leg.

[think it's a wooden table a black and white photo of a of scissors\_ pair

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-1d80670c-2be5-49de-9c0b-6a2854cda213.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-13d4f8e4-c9f7-4207-aa8a-c6587884b663.jpg)

a black and white photo of a plane flying in the sky.

Figure 11. Compared to CaptionBot (Black) [18] and NeuralTalk2 (Blue) [21], our results (red) presents more detailed and accurate part characteristics under real images, where ρ &gt; 0 5 . . The same color in the chair class or table class indicates the same part class.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-4f0d80b6-9d15-4277-890c-e7f8cf693724.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-862727c4-2ad8-4727-a309-a84edd55c1c5.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-6c1a83e2-847b-47eb-82c5-51039ff8c479.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-b3cfe45d-9eed-481b-bef9-45652586a9af.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-0c654ecc-b87c-43bb-a864-983b8a21fbe9.jpg)

## 6. Conclusion

## References

- [1] K. Chen, C. B. Choy, M. Savva, A. X. Chang, T. Funkhouser, and S. Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In ACCV , 2018.
- [2] K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio. On the properties of neural machine translation: Encoderdecoder approaches. In SSST@EMNLP , pages 103-111, 2014.
- [3] M. Denkowski and A. Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation , pages 376-380, 2014.
- [4] Z. Gan, C. Gan, X. He, Y. Pu, K. Tran, J. Gao, L. Carin, and L. Deng. Semantic compositional networks for visual captioning. In IEEE Conference on Computer Vision and Pattern Recognition , 2017.

Wepropose ShapeCaptioner to better caption a 3D shape by leveraging more part characteristics. ShapeCaptioner successfully learns the ability of semantic part detection from multiple views of 3D shapes in segmentation benchmarks, and effectively transfers this ability to 3D-Text dataset to caption 3D shapes from the parts detected in multiple colored views. Moreover, our part class specific aggregation can also preserve the part characteristics from different views. Our outperforming results indicate that captioning from parts can produce more accurate descriptions for parts, which is also more similar to human's way of describing 3D shapes.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-42332d01-630f-4c7c-88e1-d9e77276dd3c.jpg)

Figure 12. The comparison with methods in Table 5, Table 6 and state-of-the-art under the test set of chair (1st row) and table (2nd row) class. The comparison is conducted in terms of cumulative distribution of different metrics.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00120v1-figure-dbb14e7f-f590-4dc5-a50c-a496f87d23d5.jpg)

- [13] Z. Han, M. Shang, Y.-S. Liu, and M. Zwicker. View InterPrediction GAN: Unsupervised representation learning for 3D shapes by learning global shape memories to support local view predictions. In AAAI , 2019.

Figure 13. (a) The statistical analysis of the user study over all cases. (b) The statistical analysis of each case.

- [5] R. Girshick. Fast R-CNN. In IEEE International Conference on Computer Vision , pages 1440-1448, 2015.
- [6] Z. Han, X. Liu, Y.-S. Liu, and M. Zwicker. Parts4Feature: Learning 3D global features from generally semantic parts in multiple views. In IJCAI , 2019.
- [7] Z. Han, Z. Liu, J. Han, C. Vong, S. Bu, and C. Chen. Unsupervised learning of 3D local features from raw voxels based on a novel permutation voxelization strategy. IEEE Transactions on Cybernetics , 49(2):481-494, 2019.
- [8] Z. Han, Z. Liu, J. Han, C.-M. Vong, S. Bu, and C. Chen. Mesh convolutional restricted boltzmann machines for unsupervised learning of features with structure preservation on 3D meshes. IEEE Transactions on Neural Network and Learning Systems , 28(10):2268 - 2281, 2017.
- [9] Z. Han, Z. Liu, J. Han, C.-M. Vong, S. Bu, and X. Li. Unsupervised 3D local feature learning by circle convolutional restricted boltzmann machine. IEEE Transactions on Image Processing , 25(11):5331-5344, 2016.
- [10] Z. Han, Z. Liu, C. Vong, Y.-S. Liu, S. Bu, J. Han, and C. Chen. Deep Spatiality: Unsupervised learning of spatially-enhanced global and local 3D features by deep neural network with coupled softmax. IEEE Transactions on Image Processing , 27(6):3049-3063, 2018.
- [11] Z. Han, Z. Liu, C.-M. Vong, Y.-S. Liu, S. Bu, J. Han, and C. Chen. BoSCC: Bag of spatial context correlations for spatially enhanced 3D shape representation. IEEE Transactions on Image Processing , 26(8):3707-3720, 2017.
- [12] Z. Han, H. Lu, Z. Liu, C.-M. Vong, Y.-S. Liu, M. Zwicker, J. Han, and C. P. Chen. 3D2SeqViews: Aggregating sequential views for 3d global feature learning by cnn with hierarchical attention aggregation. IEEE Transactions on Image Processing , 28(8):3986-3999, 2019.
- [14] Z. Han, M. Shang, Z. Liu, C.-M. Vong, Y.-S. Liu, M. Zwicker, J. Han, and C. P. Chen. SeqViews2SeqLabels: Learning 3D global features via aggregating sequential views by rnn with attention. IEEE Transactions on Image Processing , 28(2):1941-0042, 2019.
- [15] Z. Han, M. Shang, X. Wang, Y.-S. Liu, and M. Zwicker. Y2Seq2Seq: Cross-modal representation learning for 3D shape and text by joint reconstruction and prediction of view and word sequences. In AAAI , 2019.
- [16] Z. Han, X. Wang, Y.-S. Liu, and M. Zwicker. Multi-angle point cloud-vae:unsupervised feature learning for 3D point clouds from multiple angles by joint self-reconstruction and half-to-half prediction. In ICCV , 2019.
- [17] Z. Han, X. Wang, C.-M. Vong, Y.-S. Liu, M. Zwicker, and C. P. Chen. 3DViewGraph: Learning global features for 3d shapes from a graph of unordered views with attention. In IJCAI , 2019.
- [18] Q. Huang, P. Zhang, D. Wu, and L. Zhang. Turbo learning for captionbot and drawingbot. In Advances in Neural Information Processing Systems , pages 6455-6465. 2018.
- [19] J. Johnson, A. Karpathy, and L. Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. In IEEE Conference on Computer Vision and Pattern Recognition , 2016.
- [20] E. Kalogerakis, M. Averkiou, S. Maji, and S. Chaudhuri. 3D shape segmentation with projective convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition , pages 6630-6639, 2017.
- [21] A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. IEEE Transactions on Pattern Analysis and Machine Intelligence , 39(4):664676, 2017.
- [22] A. Karpathy and F. Li. Deep visual-semantic alignments for generating image descriptions. In IEEE Conference on Computer Vision and Pattern Recognition , pages 3128-3137, 2015.
- [23] C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. In Proceeding of ACL workshop on Text Summarization Branches Out , 2004.
- [24] X. Liu, Z. Han, Y. Liu, and M. Zwicker. Point2Sequence: Learning the shape representation of 3D point clouds with an attention-based sequence to sequence network. AAAI , 2019.
- [25] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: A method for automatic evaluation of machine translation. In Annual Meeting on Association for Computational Linguistics , pages 311-318, 2002.
- [26] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems , pages 91-99, 2015.
- [27] X. Shen, X. Tian, J. Xing, Y. Rui, and D. Tao. Sequence-tosequence learning via shared latent representation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, February 2-7, 2018 , 2018.
- [28] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR , abs/1409.1556, 2014.
- [29] H. O. Song, Y. Xiang, S. Jegelka, and S. Savarese. Deep metric learning via lifted structured feature embedding. In IEEE Conference on Computer Vision and Pattern Recognition , pages 4004-4012, 2016.
- [30] Y. Song and M. Soleymani. Cross-modal retrieval with implicit concept association. CoRR , abs/1804.04318, 2018.
- [31] R. Vedantam, C. L. Zitnick, and D. Parikh. Cider: Consensus-based image description evaluation. In IEEE Conference on Computer Vision and Pattern Recognition , pages 4566-4575, 2015.
- [32] S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko. Sequence to sequence - video to text. In Proceedings of the IEEE International Conference on Computer Vision , pages 4534-4542, 2015.
- [33] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In IEEE Conference on Computer Vision and Pattern Recognition , 2015.
- [34] J. Wang, P. S. Madhyastha, and L. Specia. Object counts! bringing explicit detections back into image captioning. In NAACL-HLT , pages 2180-2193, 2018.
- [35] Q. Wu, C. Shen, L. Liu, A. Dick, and A. v. d. Hengel. What value do explicit high level concepts have in vision to language problems? In IEEE Conference on Computer Vision and Pattern Recognition , pages 203-212, 2016.
- [36] X. Yin and V. Ordonez. Obj2text: Generating visually descriptive language from object layouts. In EMNLP , pages 177-187, 2017.
- [37] Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo. Image captioning with semantic attention. In IEEE Conference on Computer Vision and Pattern Recognition , pages 46514659, 2016.
Zhizhong Han Chao Chen Yu-Shen Liu Matthias Zwicker
ShapeCaptioner: Generative Caption Network for 3D Shapes by Learning a Mapping from Parts Detected in Multiple Views to Sentences
3D shape captioning is a challenging application in 3D shape understanding.
Captions from recent multi-view based methods reveal that they cannot capture
part-level characteristics of 3D shapes. This leads to a lack of detailed
part-level description in captions, which human tend to focus on. To resolve
this issue, we propose ShapeCaptioner, a generative caption network, to perform
3D shape captioning from semantic parts detected in multiple views. Our novelty
lies in learning the knowledge of part detection in multiple views from 3D
shape segmentations and transferring this knowledge to facilitate learning the
mapping from 3D shapes to sentences. Specifically, ShapeCaptioner aggregates
the parts detected in multiple colored views using our novel part class
specific aggregation to represent a 3D shape, and then, employs a sequence to
sequence model to generate the caption. Our outperforming results show that
ShapeCaptioner can learn 3D shape features with more detailed part
characteristics to facilitate better 3D shape captioning than previous work.