## Learning to Train with Synthetic Humans

David T. Hoffmann , Dimitrios Tzionas , Michael J. Black , and Siyu Tang 1 1 1 1 2 3 , ,

1

Max Planck Institute for Intelligent Systems, Germany 2 University of T¨bingen, Germany u 3 ETH Z¨ urich dhoffmann, dtzionas, black, stang@tuebingen.mpg.de

Abstract. Neural networks need big annotated datasets for training. However, manual annotation can be too expensive or even unfeasible for certain tasks, like multi-person 2D pose estimation with severe occlusions. A remedy for this is synthetic data with perfect ground truth. Here we explore two variations of synthetic data for this challenging problem; a dataset with purely synthetic humans and a real dataset augmented with synthetic humans. We then study which approach better generalizes to real data, as well as the influence of virtual humans in the training loss. Using the augmented dataset, without considering synthetic humans in the loss, leads to the best results. We observe that not all synthetic samples are equally informative for training, while the informative samples are different for each training stage. To exploit this observation, we employ an adversarial student-teacher framework; the teacher improves the student by providing the hardest samples for its current state as a challenge. Experiments show that the student-teacher framework outperforms normal training on the purely synthetic dataset.

## 1 Introduction

The broad success of deep neural networks comes at a price: the ever growing need for huge amounts of labeled training data. For many tasks, the lack of data seems to be one of the major limiting factors of progress. It is particularly problematic for the tasks where manual labeling requires significant human effort, or is even unfeasible. For example, in multi-person 2D pose estimation, a major challenge is that people are often partially visible. Manual annotation of body joints that are severely occluded is error prone and the resulting labels are noisy. Computer graphics can help to resolve these issues. 3D rendering engines offer the opportunity to generate a large amount of data with perfect labels: e.g., the location of occluded body parts and the precise pose of the camera.

Nowadays, large scale synthetic datasets with reasonable realism can be generated relatively easy and the idea of synthesizing training data has been widely explored. In general, there are two common strategies for generating a synthetic dataset: rendering a purely synthetic dataset, and augmenting real training images with synthetic instances. The advantage of the former is the full control over the virtual 3D world and ability to generate high variance datasets [4,5,12,29,35,36,38,44]. The advantage of the second approach is that some of the instances in the dataset are real, resulting in overall higher realism [2,11,40].

Fig. 1: Qualitative comparisons between our models and Cao et al. [8].

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-8b29b173-1b85-4213-b1bd-d22f0a11fce8.jpg)

Cao et al. Ours (using teacher) Ours (best)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-2a1a2a31-6be3-420c-8aed-0c08ff258639.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-bb400b9e-9820-4ca8-94a1-aa4ebde4c414.jpg)

Ours (best)

Cao et al.

Ours (best)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-500e1a1f-69c7-45c8-8412-b837df08a847.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-6e13a8fa-bf4a-45f9-a1d8-bb53de841314.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-1f48b583-069a-4e20-b90e-ae12aa011350.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-2390eaec-b3bf-4843-bb20-446849c82363.jpg)

We generate both types of synthetic datasets. One purely synthetic dataset and a mixed dataset, which is generated by augmenting the MPII pose estimation dataset [3] with synthetic humans. In particular, we design these datasets to improve on frequent failure cases that we observe with state-of-the-art models (see Fig 1), namely uncommon camera angles and strong occlusion. By comparing generalization performance using these two datasets we obtain insights into which way of generating data is preferable. We further investigate how strongly the lack of photorealism of the synthetic humans limits generalization. To make the synthetic data more realistic, we propose a simple synthetic-to-real human style transfer algorithm, based on the work of Dundar et al. [10].

These experiments show that naive training with synthetic data leads only to limited improvements. One explanation is that training on large synthetic datasets leads to overfitting of the model to the features of synthetic data. We observe that some synthetic images convey more information than others. Overfitting to features of synthetic data could be limited by generating only useful, i.e. difficult synthetic data, and thus limiting the training on synthetic data.

As a step in this direction, we propose a method to use synthetic datasets more effectively. Specifically, we introduce an adversarial student-teacher framework. The teacher learns online which training data is still difficult. This information is then used to increase the sampling probability of similar examples. By taking into account feedback from the student, the teacher keeps on updating the sampling probabilities throughout training and adapts them to the specific needs of the student. Training with the teacher on the purely synthetic data outperforms normal training.

Our contributions can be summarized as follows: 1) We propose a large-scale synthetic multi-person dataset, a mixed dataset, and a domain-adapted version of the latter. 2) We explore which way of generating synthetic data is superior for our task. 3) We propose a student-teacher framework to train on the most difficult images and show that this method outperforms random sampling of training data on the synthetic dataset. We provide datasets and code 1 .

1 https://ltsh.is.tue.mpg.de

## 2 Related Work

Synthetic datasets with humans. The need for labelled training data has fueled development of datasets with synthetic humans. Many methods use 3D models of the human body to generate data [4,5,16,35]. Other approaches augment 3D training data by utilizing 2D pose datasets [9,37], while [40,43] augment datasets with cut-outs of objects or animals. Closely related to our approach, [36,44] render the SMPL model [27] on top of random indoor images. These methods generate datasets with a single synthetic human. Multi-person datasets were created by employing video games for pedestrian detection [29] and pose tracking [12]. Similarly, [30] develop a simulation environment in a game engine, including virtual humans. Related to our approach, [38] augment real training images similar to the ones in [44] but with multiple synthetic humans occluding each other.

Domain Adaptation. The quality of synthetic data is often insufficient to generalize well to real data. Several domain adaptation methods have been developed to overcome this problem. Shrivastava et al. [42] train a Generative Adversarial Network (GAN) to refine synthetic images, while keeping the label information intact. Recently, Cycle-GAN has been used to map images from one domain to another [4,31]. However, GAN-based methods are prone to unstable training and require tedious hyper-parameter tuning. As a practical alternative, recently [10] proposed domain stylization to stylize synthetic images to real ones, using the fast photorealistic image stylization method of [25].

Human pose estimation. Multi-person pose estimation has attracted substantial attention over the last years [8,14,15,19,23,28,32,33]. One of the most popular datasets is the MPII multi-person pose estimation dataset [3]. Among the best performing methods on MPII are: [33], which uses a pose partition network, [28], which uses context information, [15], which refines pose predictions, and [32], which predicts 'tag maps' to solve the grouping problem. The most widely used method is OpenPose [8], a bottom-up approach that first predicts keypoints and then estimates Part Affinity Fields (PAFs) to group them.

Learning to train. Bengio et al. [6] introduce curriculum learning for learning systems, exploiting the idea that different data samples are informative at different training stages. As a proof of concept they manually define the samples for each stage with gradually increasing difficulty. Multiple methods have focused on automating curriculum learning [7,13,20,24]. These approaches try to maximize information gain during training by carefully monitoring the learning success of the model. Alternatively, adversarial methods [21,22,41] try to pick the hardest samples at each training stage; [21] favors samples resulting in higher loss, [22] learns weights for the loss of each training sample as a soft curriculum, while [41] uses online hard example mining for object detection. Peng et al. [34] propose an adversarial training scheme to optimize data augmentation online. They train two teacher networks to learn a probability distribution over the hyper-parameters for data augmentation; one predicts the most difficult image rotations, while the other predicts parameters for deep feature occlusion.

Fig. 2: Schematic of the data generation pipeline.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-ecde20a7-a677-4b21-8d5d-4ea271bb4d68.jpg)

## 3 Multi-person Synthetic Data

In the following we describe the generation of the datasets (Sections 3.1, 3.2, 3.3) and the domain adaptation used to increase visual appearance of the mixed dataset (Section 3.4).

## 3.1 Data Generation Pipeline

To generate realistic synthetic training data we build on top of [36,44]. We use multiple different sources of data to build a realistic synthetic scene. Images and ground truth annotations are rendered using Cycles, the rendering engine of Blender . An overview can be seen in Fig. 2. 2

Body Model and MoCap Data. We use the parametric body model SMPL+H [39] to generate realistic synthetic humans. SMPL+H is parameterized by pose θ ∈ R 78 and shape β ∈ R 10 . We collect realistic pose and shape parameters by fitting SMPL+H to standard MoCap data by using MoSh [26].

Details. We draw inspiration from [36,44] who generate small video sequences, each having different but fixed parameters for the position, pose, shape and texture of a synthetic human, the background image, camera position, lighting, etc. In contrast, we generate single images and render multiple synthetic humans, while randomizing the number of them. As a result, we generate a dataset with much higher variance. Images with inter-penetrating meshes of virtual humans are rejected to avoid artifacts in the generated ground truth.

Further details regarding the description of the data generation pipeline, posing of hands and a quantitative comparison of our datasets to other datasets can be found in Supp. Mat.

2 https://www.blender.org

Fig. 3: Example images from the purely synthetic dataset. It contains high occlusion, extreme poses, various camera angles and various challenging backgrounds.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-ddf1fcef-de57-42fa-95f7-50388776a948.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-03f6ce62-c85e-42fd-a100-656bfa52f55c.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-9e119ca6-580f-4c69-915c-f5af5e81ca38.jpg)

## 3.2 Synthetic Dataset

Background Images. To generalize well to in-the-wild pose estimation datasets the background images should come from many different scenes. To this end we use images from SUN397 [45] and reject all images with a resolution smaller than 512 × 512 pixels to ensure high quality backgrounds. Additionally, we reject all images containing humans, as we do not have ground truth annotations for them. We use mask-RCNN [1,17] as our human detector.

Generative Factors. We sample the number of synthetic humans per image from a Poisson distribution with λ = 9, to encourage many humans per image, while avoiding too extreme values. The datasets of [36,44] have only very small variance in camera position. However, preliminary experiments show that the camera position significantly influences the difficulty of multi-person pose estimation. Therefore we increase the range of possible camera positions, by sampling the camera pitch uniformly from [0 45] . The resolution of the final , ◦ rendered images is set to 640 × 640 pixels. We refer to this dataset as D S .

## 3.3 Mixed Dataset

We build upon the finding of [40] that realistic occluding objects lead to larger improvements than abstract objects. We choose our occluders to be from the same class as our target objects; i.e. humans, to simulate crowded scenes with multiple humans. To generate the dataset we use the pipeline described above with a few differences. Instead of SUN397 [45] we use the training images of the MPII human pose dataset [3] as background images. To keep the MPII ground truth intact, we render the images with the same resolution as the background MPII image, and keep the camera pose fixed. We then augment the MPII human pose dataset by superimposing synthetic humans. Their number is drawn from a Poisson distribution with λ = 4 to introduce interesting and intense occlusions as shown in Fig. 4 (A), without extreme occlusions by too many synthetic humans. We render each of the 15 956 images in our training set 5 times with different , parameters for increased variance. We refer to this dataset as D M .

Fig. 4: (A) Example images from D M . (B) Corresponding images of D Style . For the last image, the segmentation network included non-human parts in the segmentation masks. Resulting artifacts can be seen for rightmost synthetic human.

Mixed

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-6f27bb1b-d4b9-4dcd-a5e1-5447cadd47fa.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-22bdbeb2-0738-46bc-884c-a569969515d1.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-18178b87-cab5-4714-a0a9-a7bca659cb06.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-b1753e25-851b-4cec-8b75-9eda7aec0e73.jpg)

Mixed

Mixed

Stylized

Stylized

Stylized

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-98a76985-ff81-4f4a-b2ec-722db65d10ba.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-1491bba5-bbfc-4edd-b10b-a02a21fc582e.jpg)

## 3.4 Domain Stylization

The appearance of real and synthetic humans differs strongly. Factors contributing to these differences are the low quality of textures and differences in lighting conditions for synthetic humans and background images. Additionally, the small number of human textures limits the variability. These differences in appearance might limit the generalization. We draw inspiration from Dundar et al. [10] to reduce these differences by using the fast photorealistic style transfer method of [25]. Style transfer methods require a pair of images as input, a content image I C and a style image I S . While the style of these images can be largely different, their content should have similarities. Finding such pairs is a non-trivial problem. However, for the D M , we have a canonical choice of image pairs: the image from D M and its background as I C and I S , respectively.

Naive application of style transfer methods on the whole image, leads to severe artifacts. Therefore, only the style of semantically similar classes should be transferred. To obtain a good semantic segmentation network Dundar et al. [10] iterate between stylizing a dataset and training a network for semantic segmentation on the stylized dataset. Here, we are only interested in transferring the style of real to synthetic humans. Fine-grained human detection is important to avoid parts of the background bleeding into the foreground after style transfer (see Fig. 4 (B) right panel). We therefore employ Mask-RCNN [1,17] to predict pixel-wise masks for humans. Ground truth masks for the synthetic humans are generated during data generation. Since the style-transfer algorithm can not handle images of arbitrary size, we rescale the larger images to 600 pixels before applying the style transfer. We refer to the resulting dataset as D Style . Examples can be seen in Fig. 4 (B).

## 4 Learning to Train with Multi-person Synthetic Data

We now turn to the task of training a pose estimation network with synthetic data. We first provide general information about our model. This is followed by a brief description of the training procedure. Finally, we detail the student-teacher framework and explain how the teacher is trained.

Pose Estimation Network. For all experiments we use our Tensorflow implementation of the network proposed in [8], which we will refer to as OpenPose network. Our training differs only slightly from [8]; details are provided in Supp. Mat. Because of these differences, we use a self-trained model on real data as the baseline throughout the paper.

Training with Synthetic Data. Following the advice of [18], we freeze the weights of our feature extractor whenever training on synthetic data. In particular, we freeze the first 4 layers of the OpenPose network. Additionally, we make sure that each batch is composed of 50% real and 50% synthetic images. More details on our training procedure with synthetic data are provided in Supp. Mat.

Grouping. Sampling the most difficult samples with higher probability comes with the risk of oversampling a small amount of data. To avoid such behavior, and ease the task for the teacher, we group the synthetic data into meaningful groups. We found empirically that the position of the camera and the distance of people in an image contribute to the difficulty of multi-person pose estimation. Thus, we use these two image characteristics to group our data. We assume 10 groups to yield a good trade-off between precision and difficulty for the teacher. For minimal distance grouping , denoted as 3 mD , we decide for linearly spaced values between [0 640) px. For the camera pitch grouping, denoted as , C we space the group boundaries linearly in the interval [min( X )+Var( X , ) max( X ) -Var( X )), where X contains all values for the camera pitch in the dataset.

## 4.1 Adversarial Teacher

The teacher network is trained simultaneously with the student network to adapt the sampling strategy dynamically to the current training state of the student. A schematic of the forward pass can be seen in Fig. 5. The input to the teacher should represent the training state of the student. We choose the real image resulting in highest mean loss per joint within the previous N training steps. This provides some information about the type of images that are still difficult for the student. The output of the teacher network is a probability distribution ˜ P over a set of groups { g , . . . , g 1 i } . This probability distribution is used to sample one of them. For the next N training steps, synthetic training samples are drawn from this group only.

Training the Teacher. The objective of the adversarial teacher should be to maximize the loss, glyph[lscript] D S t . Unfortunately, sampling and data augmentation

3 We sample persons not images. The image is cropped around this person. The minimal distance is defined as smallest distance of this person to any other person.

Fig. 5: Diagram of the forward pass. The total loss is denoted as glyph[lscript] t , the loss on real images as glyph[lscript] D R t and on synthetic images as glyph[lscript] D S t . The reward signal and backward pass are explained in the main text. For clarity only 6 instead of 10 groups are displayed.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-45b5e2ec-430e-457b-988c-1d56c52e7d72.jpg)

are non-differentiable operations, prohibiting end-to-end training of the teacher network. An alternative method to provide a supervision signal is needed. We draw inspiration from [34] and employ a reward/penalty training scheme. To determine whether the teacher is rewarded or penalized we monitor glyph[lscript] D S t . If glyph[lscript] D S t ≥ glyph[lscript] D S t -1 the teacher succeeded in finding more difficult examples than before and is rewarded. Unfortunately, glyph[lscript] D S t has a high variance. To reduce the variance of the teaching signal we reward if

$$\ell _ { t } ^ { \mathcal { D } _ { S } } & \geq \frac { 1 } { H } \sum _ { h = 0 } ^ { H } \ell _ { t - 1 - h } ^ { \mathcal { D } _ { S } },$$

glyph[negationslash]

$$P _ { i } = \tilde { P } _ { i } + \delta \alpha \tilde { P } _ { i }, \text{ and } P _ { j } = \tilde { P } _ { j } - \delta \frac { \alpha \tilde { P } _ { i } } { | g | - 1 }. \text{ \quad \ } ( 2 ) \\ \text{ denotes the prediction of the teacher } P _ { i } \text{ is the undated second round}$$

where H denotes the number of past loss values to be considered. To avoid favoring images with many people, we use the mean loss per joint on the image. Eq. 1 provides the direction of the gradient descent step but no ground truth is given to compute the gradients. To efficiently get a reward signal we follow [34] and increase the probability of a group being chosen, if the teacher gets rewarded. Probabilities for other groups are decreased accordingly. Formally, we update P i and P j , where i denotes the selected group and j = i denotes all other groups, by

Here ˜ P i denotes the prediction of the teacher, P i is the updated pseudo ground truth probability, 0 ≤ α ≤ 1 controls the size of the update, | g | denotes the number of groups and δ is a sign indicator δ = +1 { , if Eq. 1 holds; -1 , otherwise } . Finally, we obtain gradients to update the teacher network by computing the KL-divergence loss between P and ˜ P . Information on optimization related hyperparameters and the architecture are provided in Supp. Mat.

During training we face a exploration/exploitation trade-off. The group with highest probability, might not be optimal. To overcome this problem, we sample

Table 1: Results on the held out validation set. M D R is trained only with real data. M D S was trained solely on synthetic data. M D R + D S is trained on real and synthetic data. M D R + D M was trained on real and the mixed dataset.

| Model       |   Head |   Shoulder |   Elbow |   Wrist |   Hip |   Knee |   Ankle |   mAP |
|-------------|--------|------------|---------|---------|-------|--------|---------|-------|
| M D R       |   91.3 |       89.1 |    79.2 |    70.4 |  75.9 |   71.5 |    66.7 |  77.7 |
| M D S       |   37.9 |       23.5 |    12.7 |     7.3 |   5.6 |    3.4 |     3.2 |  13.4 |
| M D R + D S |   91.1 |       89.2 |    80.5 |    71   |  75.2 |   73.6 |    68.1 |  78.4 |
| R + M       |   91.3 |       89.5 |    80.7 |    71.7 |  75.4 |   72.5 |    67.7 |  78.4 |

M

D

D

a group from a uniform distribution instead of the predicted probabilities with probability glyph[epsilon1] = 0 1. .

## 5 Experiments

We test our models on the MPII multi-person pose dataset [3]. For that purpose we use the toolkit provided by [3]. Following the standard validation procedure, the test metric is only computed for people within close proximity. We split the real training data, denoted as D R , into a training and a validation set. Our validation set consist of 343 randomly selected groups of people in close proximity. The respective images are not used for training. We report mAP (mean Average Precision), the main metric of the benchmark [3] for each model.

## 5.1 Which Dataset Generalizes Best?

We hypothesize that multi-person pose estimation methods are limited by a lack of training data. To test this hypothesis we train our model on D S and D R + D S . Interestingly, for M D S , the model resulting from training only on D S , mAP is very low, suggesting that the model overfits to the synthetic data. However, when training on D R + D S the mAP improves over the mAP of M D R (Tab. 1).

While synthetic data can improve the accuracy of multi-person pose estimation, the improvements are relatively small given the extensive amount of additional training data. Multiple factors might limit the generalization. It could well be that the dataset bias between the synthetic and real dataset is just too strong. Generating the D M is a straightforward way of generating a dataset with similar dataset bias as the original dataset. By training on D M we can quantify the influence of it on the generalization. When training with D M we consider only real humans as samples. The rationale behind that decision is that we primarily want to increase the frequency of occlusion of real humans. However, the network is also trained on all synthetic humans that are within the cropped training image. As can be seen in Tab. 1, training on D R + D M results in similar accuracy as M D R + D S . Therefore, the two methods of generating data are equivalently good. The dataset bias seems not to be the main limiting factor.

Table 2: Results for the models M D R + D M , M D R + D Style , M D R + D M +masks and M D R + D Style +masks datasets, where '+ masks' denotes masking out loss generated by synthetic humans. All reported results are on the held out validation set.

| Model                |   Head |   Shoulder |   Elbow |   Wrist |   Hip |   Knee |   Ankle |   mAP |
|----------------------|--------|------------|---------|---------|-------|--------|---------|-------|
| M D R + D M          |   91.3 |       89.5 |    80.7 |    71.7 |  75.4 |   72.5 |    67.7 |  78.4 |
| M D R + D M +masks   |   92.3 |       90.9 |    80.5 |    72.2 |  76   |   71.7 |    68.3 |  78.9 |
| M D R + D Style      |   91.8 |       89.8 |    80.4 |    70.9 |  75.5 |   71.6 |    67.9 |  78.3 |
| D R + D Style +masks |   91.6 |       90.6 |    80.8 |    71.8 |  77.7 |   72.2 |    68.8 |  79.1 |

M

## 5.2 Does Stylization Improve Generalization?

The generalization might be limited by the appearance of synthetic humans. To overcome the limited generalization and measure how the difference in appearance influences performance we train on D R + D Style . This improves accuracy over M D R , but leads to a decrease of mAP compared to M D R + D M (see Tab. 2). This result is surprising, as many factors believed to limit generalization are improved and the data is visually more realistic. A possible explanation is that the network learns to detect artifacts of the style transfer. Alternatively, the failure cases of the style transfer method might lead to 'confusion' of the network.

In an ablation study, we test whether training on synthetic humans actually improves the mAP or if the improvement when training with D M or D Style is mostly caused by additional occlusion. For that purpose, we mask out all the loss that is generated by synthetic humans. Comparing M D R + D M +masks with M D R + D M , it can be seen that masking out the loss generated by synthetic humans increases the accuracy. Therefore, the domain gap between synthetic and real humans limits the generalization, and the improvement of M D R + D M over M D R is due to more occlusion. Stylization in combination with masks leads to the best model (see Fig. 1 and Supp. Mat. for qualitative results), suggesting that a smaller gap between synthetic occluder and real parts of the image improves generalization.

## 5.3 Does Informed Sampling Improve Results?

Finally, we test whether the teacher network can help to use the synthetic data more effectively. For that purpose we use the adversarial teacher with D R + D S . The results can be seen in Tab. 3. Grouping according to the camera pitch leads to an additional improvement of 0 5 mAP in comparison to . M D R + D S . See Fig. 1 and Supp. Mat. for qualitative results. For usage of the teacher in combination with the mixed and stylized datasets we do not observe further improvements. Since we only consider real humans as samples some groups are very small. We assume that oversampling of these groups inhibits improvements.

Table 3: Comparison of the M D R and M D R + D S baselines (copied from Tab. 1) to the student-teacher model using different groupings. Results on the validation set.

| Model                 | Grouping Head Shoulder Elbow Wrist Hip Knee Ankle mAP   |   Grouping Head Shoulder Elbow Wrist Hip Knee Ankle mAP |   Grouping Head Shoulder Elbow Wrist Hip Knee Ankle mAP |   Grouping Head Shoulder Elbow Wrist Hip Knee Ankle mAP |   Grouping Head Shoulder Elbow Wrist Hip Knee Ankle mAP |   Grouping Head Shoulder Elbow Wrist Hip Knee Ankle mAP |   Grouping Head Shoulder Elbow Wrist Hip Knee Ankle mAP |   Grouping Head Shoulder Elbow Wrist Hip Knee Ankle mAP |   Grouping Head Shoulder Elbow Wrist Hip Knee Ankle mAP |
|-----------------------|---------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------|
| M D R                 |                                                         |                                                    91.3 |                                                    89.1 |                                                    79.2 |                                                    70.4 |                                                    75.9 |                                                    71.5 |                                                    66.7 |                                                    77.7 |
| M D R + D S           |                                                         |                                                    91.1 |                                                    89.2 |                                                    80.5 |                                                    71   |                                                    75.2 |                                                    73.6 |                                                    68.1 |                                                    78.4 |
| 'adversarial Teacher' | C                                                       |                                                    91.7 |                                                    90   |                                                    80.9 |                                                    71.2 |                                                    77.1 |                                                    73.6 |                                                    67.7 |                                                    78.9 |
| 'adversarial Teacher' | mD                                                      |                                                    91.5 |                                                    90.4 |                                                    80.5 |                                                    72.2 |                                                    75.8 |                                                    73.1 |                                                    67.6 |                                                    78.7 |

Fig. 6: Detection performance for varying ratio of visible joints (mAP). The validation data is grouped into 5 linearly spaced groups in range [0 , 1]. The groups contain 14, 111, 289, 286, 180 persons respectively. The teacher methods are hard to distinguish because of similar values.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-86e91ecc-87ca-4a5f-9517-ecfcfb9f9b37.jpg)

As can be seen in Fig. 6, improvements for highly occluded people are strongest for models trained with the teacher. Clear differences to M D R + D M +masks and M D R + D Style +masks can only be seen for the highest occlusion level. For all but the lowest occlusion levels M D R is outperformed by all other models. Thus, our methods of training improve accuracy for difficult high occlusion cases.

Sampling Probabilities. The teacher often assigns high probability to few groups early in training. In most cases the teacher converges to a uniform sampling strategy over the groups, as training progresses. This is not equivalent to random sampling, as samples are not uniformly distributed across groups. As a result the training data follows a uniform distribution for the respective image characteristic. A uniform distribution over an image characteristic like camera pitch is more extreme than the distribution in the real training data. Training on more extreme samples seems to improve the generalization. We find that im-

provements are largest for the camera pitch grouping. This might be due to the bias to small camera pitch in the real training data.

## 6 Conclusion

In summary, we created multiple synthetic datasets and analyze their effectiveness. We show that training with synthetic data improves multi-person pose estimation methods. We find that both our methods of generating synthetic datasets perform on par. Surprisingly, our approach for improving visual appearance of synthetic humans decreased the accuracy. More elaborate domain adaptation methods might provide better results. For example, GAN-based approaches ensure that no obvious artifacts can be used to distinguish real from synthetic. We find that improvements of the M D R + D M and M D R + D Style can be explained by more occlusion, as mAP increases when masking out the loss of synthetic humans. Here the stylization leads to further improvements, suggesting that visual quality of occluding objects is important. Finally, we find that training on the most difficult synthetic samples at each point of training improves the results. This suggest that, for large synthetic datasets, random sampling is not optimal and better strategies exist. More research in this direction is necessary to draw final conclusions. M D R + D Style +masks outperforms the 'adversarial Teacher' C model. We assume that the potential of the teacher is limited by the small amount and the quality of human textures. Given better textures we expect this approach to outperform M D R + D Style +masks .

Limitations. The teacher network is limited in multiple ways. First, the current implementation requires grouping of data. The size and spacing of groups might have a large influence on the training success and applicability of teacher networks. Furthermore, the grouping is based on one feature only. This is suboptimal, since difficulty of an image is determined by multiple factors. We plan to extend the teachers to handle multiple characteristics at once. A more elaborate formulation that does not require grouping might be superior. Last, the teacher can be applied to other tasks and networks, here we evaluate it only for multi-person pose estimation with the OpenPose network.

The style transfer occasionally produces artifacts in the stylized image. Especially the skin color of the synthetic humans might be unnatural. In rare cases, large parts of the background are included in the human mask. These failures in segmentation can lead to ghost-like synthetic humans (see Supp. Mat. ).

Acknowledgement. S. Tang acknowledges funding by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) Projektnummer 276693517 SFB 1233.

Disclosure. MJB has received research gift funds from Intel, Nvidia, Adobe, Facebook, and Amazon. While MJB is a part-time employee of Amazon, his research was performed solely at, and funded solely by, MPI. MJB has financial interests in Amazon and Meshcapade GmbH.

## References

- 1. Abdulla, W.: Mask r-cnn for object detection and instance segmentation on keras and tensorflow. https://github.com/matterport/Mask\_RCNN (2017)
- 2. Alhaija, H.A., Mustikovela, S.K., Mescheder, L., Geiger, A., Rother, C.: Augmented reality meets computer vision: Efficient data generation for urban driving scenes. IJCV 126 (9), 961-972 (2018)
- 3. Andriluka, M., Pishchulin, L., Gehler, P., Schiele, B.: 2d human pose estimation: New benchmark and state of the art analysis. In: CVPR (June 2014)
- 4. Bak, S., Carr, P., Lalonde, J.F.: Domain adaptation through synthesis for unsupervised person re-identification. In: ECCV (2018)
- 5. Barbosa, I.B., Cristani, M., Caputo, B., Rognhaugen, A., Theoharis, T.: Looking beyond appearances: Synthetic training data for deep cnns in re-identification. CVIU 167 , 50-62 (2018)
- 6. Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: ICML (2009)
- 7. B¨ uchler, U., Brattoli, B., Ommer, B.: Improving spatiotemporal self-supervision by deep reinforcement learning. In: ECCV (2018)
- 8. Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2d pose estimation using part affinity fields. In: CVPR (2017)
- 9. Chen, W., Wang, H., Li, Y., Su, H., Wang, Z., Tu, C., Lischinski, D., Cohen-Or, D., Chen, B.: Synthesizing training images for boosting human 3d pose estimation. In: 3DV. IEEE (2016)
- 10. Dundar, A., Liu, M.Y., Wang, T.C., Zedlewski, J., Kautz, J.: Domain stylization: A strong, simple baseline for synthetic to real image domain adaptation. arXiv preprint arXiv:1807.09384 (2018)
- 11. Dvornik, N., Mairal, J., Schmid, C.: On the importance of visual context for data augmentation in scene understanding. arXiv preprint arXiv:1809.02492 (2018)
- 12. Fabbri, M., Lanzi, F., Calderara, S., Palazzi, A., Vezzani, R., Cucchiara, R.: Learning to detect and track visible and occluded body joints in a virtual world. In: ECCV (2018)
- 13. Fan, Y., Tian, F., Qin, T., Bian, J., Liu, T.Y.: Learning what data to learn. arXiv preprint arXiv:1702.08635 (2017)
- 14. Fang, H., Xie, S., Tai, Y.W., Lu, C.: Rmpe: Regional multi-person pose estimation. In: ICCV (2017)
- 15. Fieraru, M., Khoreva, A., Pishchulin, L., Schiele, B.: Learning to refine human pose estimation. In: CVPR Workshops (2018)
- 16. Ghezelghieh, M.F., Kasturi, R., Sarkar, S.: Learning camera viewpoint using cnn to improve 3d body pose estimation. In: 3DV (2016)
- 17. He, K., Gkioxari, G., Doll´r, P., Girshick, R.: Mask r-cnn. In: ICCV (2017) a
- 18. Hinterstoisser, S., Lepetit, V., Wohlhart, P., Konolige, K.: On pre-trained image features and synthetic images for deep learning. In: ECCV (2018)
- 19. Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., Schiele, B.: Deepercut: A deeper, stronger, and faster multi-person pose estimation model. In: European Conference on Computer Vision. pp. 34-50. Springer (2016)
- 20. Jiang, L., Zhou, Z., Leung, T., Li, L.J., Fei-Fei, L.: Mentornet: Learning datadriven curriculum for very deep neural networks on corrupted labels. arXiv preprint arXiv:1712.05055 (2017)
- 21. Katharopoulos, A., Fleuret, F.: Biased importance sampling for deep neural network training. arXiv preprint arXiv:1706.00043 (2017)

- 22. Kim, T.H., Choi, J.: Screenernet: Learning self-paced curriculum for deep neural networks. arXiv preprint arXiv:1801.00904 (2018)
- 23. Kocabas, M., Karagoz, S., Akbas, E.: Multiposenet: Fast multi-person pose estimation using pose residual network. In: ECCV (2018)
- 24. Kumar, M.P., Packer, B., Koller, D.: Self-paced learning for latent variable models. In: NIPS (2010)
- 25. Li, Y., Liu, M.Y., Li, X., Yang, M.H., Kautz, J.: A closed-form solution to photorealistic image stylization. In: ECCV (2018)
- 26. Loper, M., Mahmood, N., Black, M.J.: Mosh: Motion and shape capture from sparse markers. TOG 33 (6), 220 (2014)
- 27. Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: Smpl: A skinned multi-person linear model. TOG 34 (6), 248 (2015)
- 28. Luo, Y., Xu, Z., Liu, P., Du, Y., Guo, J.M.: Multi-person pose estimation via multi-layer fractal network and joints kinship pattern. TIP 28 (1), 142-155 (2019)
- 29. Marin, J., V´zquez, D., Ger´nimo, D., L´pez, A.M.: Learning appearance in virtual a o o scenarios for pedestrian detection. In: CVPR (2010)
- 30. M¨ uller, M., Casser, V., Lahoud, J., Smith, N., Ghanem, B.: Sim4cv: A photorealistic simulator for computer vision applications. IJCV pp. 1-18 (2018)
- 31. Murez, Z., Kolouri, S., Kriegman, D., Ramamoorthi, R., Kim, K.: Image to image translation for domain adaptation. In: CVPR (2018)
- 32. Newell, A., Huang, Z., Deng, J.: Associative embedding: End-to-end learning for joint detection and grouping. In: NIPS (2017)
- 33. Nie, X., Feng, J., Xing, J., Yan, S.: Pose partition networks for multi-person pose estimation. In: ECCV (2018)
- 34. Peng, X., Tang, Z., Yang, F., Feris, R.S., Metaxas, D.: Jointly optimize data augmentation and network training: Adversarial data augmentation in human pose estimation. In: CVPR (2018)
- 35. Pishchulin, L., Jain, A., Wojek, C., Andriluka, M., Thorm¨hlen, T., Schiele, B.: a Learning people detection models from few training samples. In: CVPR (2011)
- 36. Ranjan, A., Romero, J., Black, M.J.: Learning human optical flow. In: BMVC (2018)
- 37. Rogez, G., Schmid, C.: Image-based synthesis for deep 3d human pose estimation. IJCV 126 (9), 993-1008 (2018)
- 38. Rogez, G., Weinzaepfel, P., Schmid, C.: Lcr-net++: Multi-person 2d and 3d pose detection in natural images. TPAMI (2019)
- 39. Romero, J., Tzionas, D., Black, M.J.: Embodied hands: Modeling and capturing hands and bodies together. ACM TOG, (Proc. SIGGRAPH Asia) 36 (6) (Nov 2017)
- 40. S´ ar´ndi, a I., Linder, T., Arras, K.O., Leibe, B.: How robust is 3d human pose estimation to occlusion? arXiv preprint arXiv:1808.09316 (2018)
- 41. Shrivastava, A., Gupta, A., Girshick, R.: Training region-based object detectors with online hard example mining. In: CVPR (2016)
- 42. Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., Webb, R.: Learning from simulated and unsupervised images through adversarial training. In: CVPR (2017)
- 43. Tripathi, S., Chandra, S., Agrawal, A., Tyagi, A., Rehg, J.M., Chari, V.: Learning to generate synthetic data via compositing. arXiv preprint arXiv:1904.05475 (2019)
- 44. Varol, G., Romero, J., Martin, X., Mahmood, N., Black, M.J., Laptev, I., Schmid, C.: Learning from synthetic humans. In: CVPR (2017)
- 45. Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: Large-scale scene recognition from abbey to zoo. In: CVPR (2010)

## Learning to Train with Synthetic Humans **Supplementary Material**

David T. Hoffmann , Dimitrios Tzionas , Michael J. Black , and Siyu Tang 1 1 1 1 2 3 , ,

1 Max Planck Institute for Intelligent Systems, Germany 2 University of T¨bingen, Germany u 3 ETH Z¨ urich

dhoffmann, dtzionas, black, stang@tuebingen.mpg.de

## 1 Data Generation

MoCap Data. To pose the body we use 1 515 MoCap sequences from the CMU , dataset [1], 62 from HumanEva [14], 41 from the PosePrior dataset [3] and 157 unpublished sequences recorded with our own motion capture system. To reduce the similarity of poses we subsample every 10th frame, resulting in 12 fps for most of the datasets. We end up with 253 762 individual poses. ,

Hand Poses. Without variations of hand poses, a keypoint detector trained on the synthetic data might not generalize to other hand poses. To avoid this, we use SMPL+H [13] and pose the hands and fingers. However, conventional MoCap systems do not record the pose of fingers [1, 3, 14]. To obtain realistic poses for hands and fingers we use the 'embodied hands' dataset [13].

Shape. Besides body pose, humans differ in their body shape. To express these differences in our synthetic datasets we extract shape parameters β from standard MoCap datasets using MoSh [9].

Textures. Textures have a large influence on the perceived realism of synthetic data. For the synthetic humans we use the textures published with the SURREAL dataset [18]. The dataset provides 772 textures of people in casual clothing and 157 in minimal clothing. The former were collected by the authors of [18]. The latter were acquired from the CAESAR dataset [11]. We use 80% of these textures and keep the remaining 20% for validation and test data, even though we do not use any synthetic validation or test data for this particular work. The reason for this decision is to keep the data splits for training, test and validation consistent across projects.

Noise and Lighting. Real images are subject to multiple sources of noise, for example motion blur or defocus. Furthermore, they are taken under varying lighting conditions. We model noise with Gaussian blur. We blur x and y direction independently with a probability of 0 5, each. The size of the Gaussian . kernel in pixels is given by the absolute value sampled from a standard normal distribution. We vary lighting conditions similarly as [18, 10] using spherical harmonics [6] with randomly drawn coefficients.

Sampling Motions. Sampling MoCap sequences randomly does not guarantee that every MoCap sequence is used. To ensure that, we select the sequence

for the first synthetic human deterministically. MoCap sequences for all other synthetic humans are sampled randomly. The probability of sampling a MoCap sequence S j is given by p j = | S j | ∑ | S | i =1 | S i | . Here | S i | denotes the number of frames of sequence i and | S | the number of sequences. Since the chosen sequences might have different length, we further randomly sample for each one the frame ID for the starting frame, to encourage the use of the whole sequences.

Changing Rendering Parameters. To increase variance in the dataset multiple parameters are changed. For each MoCap sequence we change the camera position, number of humans and the global rotation of each synthetic human. In contrast, background image and lighting as well as position, pose, shape and texture of synthetic humans are changed for each frame.

Scene Generation. After sampling all these parameters, each virtual human is placed on an invisible ground plane in the field of view of the camera. Maximal distance to the camera are 12 m.

Collision Detection. The meshes of synthetic humans in the 3D scene might intersect, resulting in physically impossible configurations. A quick and easy way of detecting collisions are axis aligned bounding boxes. However, their usage results in a large number of false positives and limits the distances between virtual humans. For multi-person pose estimation, however, small distances between humans are frequent and should be represented in the training data. We draw inspiration from [17] and use bounding volume hierarchies (BVH) [16] instead. This is an efficient method to check for collisions on triangle level. Thus, it allows for smallest possible distances between the meshes.

Whenever two synthetic humans collide, we search for a new, valid position for one of them before rendering the image. Frames with mesh self-collisions are not rejected, as they are very frequent for highly articulated poses, which might be important for a pose-estimation dataset.

## 2 Datasets

Tab. A.1 shows a comparison of our datasets to related other datasets. JTA and SURREAL are much larger, however SURREAL only considers a single person in indoor environments and JTA only urban scenes in surveillance scenarios. Our datasets have a much higher variety in terms of scenes.

Example images for D S can be seen in Fig. A.1

## 2.1 D M and D Style -Qualitative Results

Fig. A.2 and A.3 show example images of D M and the corresponding images from D Style . In particular Fig. A.2 (A) shows that the style transfer method generates realistic variations of textures and changes their color. In Fig. A.2 (B, C) and Fig. A.3 (A, B, C) it can be seen that the method adapts the lighting of textures to fit better into the scene. It greatly improves blending-in of synthetic humans. Fig A.2 (D) and Fig A.3 (D) show failure cases. Here, larger parts of the background were included in the human mask of mask-RCNN [7, 2]. As a

Fig. A.1. Example images from D S . The dataset contains images with interesting poses, challenging backgrounds, variance in camera position and heavy occlusion.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-23e902a1-bdb5-43b9-9fbd-4e7b3a58ce9b.jpg)

Table A.1. Number of frames and poses for our datasets and for the most related synthetic datasets. All datasets are multi-person datasets, except for SURREAL. For D M and D Style '#Synthetic Humans' corresponds to the additional number of synthetic humans. For D S , SURREAL, JTA and SURREAL-style multi-person '#Synthetic Humans' corresponds to the number of annotated poses.

| Dataset                    |      | #frames       | #Synthetic Humans   |
|----------------------------|------|---------------|---------------------|
| D S                        | ours | 70 , 379      | 580 , 693           |
| D M                        | ours | 74 , 628      | 279 , 605           |
| D Style                    | ours | 74 , 352      | 279 , 278           |
| SURREAL                    | [18] | 6 , 536 , 752 | 6 , 536 , 752       |
| JTA                        | [5]  | 460 , 800     | 10 , 000 , 000      |
| SURREAL-style multi-person | [12] | 40 , 000      | 186 , 000           |

result, synthetic humans are partially stylized in the style of the background. This leads to the observable ghost-effect.

## 3 Training

Pose Estimation Network. Unfortunately not all hyper-parameters used for training on the MPII pose estimation dataset were provided for the OpenPose network. Our hyper-parameter search results in a model that approaches the original performance. Differences in performance may be due to a different choice of optimizer. We use the more common Adam algorithm [8], whereas Cao et al. [4] rely on SGD. Our hyperparameter search lead to a learning rate of lr = 0 0001, . β 1 = 0 8 and . β 2 = 0 999. Furthermore, we found that a learning rate decay . improves results. We decay the learning rate every 20 000 steps with a decay . rate of 0 66. We use a batch size of 32. .

Most of our models are initialized with weights pretrained on real data. The only 2 models that are not pretrained on real pose estimation data are M D R and M D S . For these models we follow the same procedure as proposed by Cao et al. [4] and initialize them with the first 10 layers of VGG-19. The remaining weights are randomly initialized.

Training with Synthetic Data. All models trained with synthetic data are initialized with pretrained weights. These weights are obtained by training OpenPose for 70 000 steps on real data. Whenever we start from these pretrained . weights, we use an initial learning rate of lr = 0 00005. .

## 3.1 Data Augmentation

To increase the number of training samples we apply standard data augmentation techniques. We implement the same data augmentation pipeline as Cao et al. [4], however the hyperparameters might be different. We scale the image in the range [0 4 . , 1 6], rotate it by a uniformly sampled value in the interval [ . -45 ◦ , 45 ]. ◦

Fig. A.2. Example images from D M and the respective image from D Style . Last row shows a failure case with ghost-like appearance of synthetic humans.

D M

D Style

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-842347bb-2b23-41e4-894f-aca83beb0fcf.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-e86b8f3c-a7a4-4581-a63a-53ad3c7a60e6.jpg)

B

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-f5194505-370c-4b34-86ba-5c810ff3fe9e.jpg)

C

D M

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-3217d812-c208-49d6-8293-9427fdd37372.jpg)

D M

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-8f7cae30-54d6-4046-8ab3-f129c38d9a2f.jpg)

D M

D Style

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-ae7a96b1-d569-4880-8aec-3aa181fab1a9.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-1ee8e78b-3c60-4c27-a96f-4a4699ebd886.jpg)

D Style

D Style

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-68adee32-37dd-492d-a7e6-0a259f6b5406.jpg)

Fig. A.3. Example images from D M and the respective image from D Style . Last row shows a failure case with ghost-like appearance of synthetic humans.

D

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-7c2c913c-b230-471c-82f7-df5a1d232bd6.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-aa103aac-98f3-4e31-9000-ed1012e99858.jpg)

M

D

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-104c38be-85ba-4440-a688-71f2ce8db95b.jpg)

M

D M

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-0f0e3077-bdf7-4ad5-b2f5-f26ef68d8ae5.jpg)

D M

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-41a36a2c-7e4a-4e22-b47d-b45e88b6b42f.jpg)

D

D Style

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-822b02ca-df79-467d-9103-ef916f7841b1.jpg)

D Style

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-d03eff44-67a8-4773-8478-eed954bbd96c.jpg)

D Style

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-76607ded-d697-4df2-af4d-0f14598b1f76.jpg)

Furthermore, each image is cropped around a target person. Size of the crop is 368 × 368 px. We add noise to the center of the crop. It is uniformly sampled from [ -50 50] px. Finally, with probability of 0 5 the image is flipped horizontally. , .

## 3.2 The Teacher Network

To optimize the teacher we use a constant learning rate of 0 00005. Similar to . optimization of the OpenPose network we use the Adam algorithm with β 1 = 0 8 . and β 2 = 0 999. .

Architecture. The teacher network consists of two parts. The first part are the first 10 layers of VGG-19 [15] and is identical to the OpenPose feature extractor. Similar to the OpenPose feature extractor the weights of VGG-19 are used to initialize it. The second part of the teacher network differs from the OpenPose network. It consists of 3 × × 3 256 and 3 × × 3 128 convolutional layers, followed by two 3 × 3 × 64 layers. After every two layers we add a max pooling layer. The last two layers are fully-connected with 512 units each.

## 4 Multi-Person Pose Estimation - Qualitative Results

Qualitative results for M D R , 'adversarial Teacher' C and M D R + D Style +masks can be seen in Fig. A.4, Fig. A.5 and Fig. A.6. While a clear improvement can be seen by 'adversarial Teacher' C and M D R + D Style +masks over M D R , differences between 'adversarial Teacher' C and M D R + D Style +masks are more subtle. We found that their predictions are qualitatively on par for most images.

As can be seen in Fig. A.4 (A, B) and Fig. A.6 (A) grouping of joints fails frequently for M D R . In contrast, erroneous grouping can be less frequently observed for our models. Thus, the models trained on our datasets improve in their ability to group joints. For very crowded scenes as shown in Fig. A.4 (A) and Fig. A.6 (A) the 'adversarial Teacher' C seems to outperform M D R + D Style +masks . Thus, adversarial training on the more crowded purely synthetic dataset is beneficial for real crowded scenes.

Besides grouping, training on our datasets seems to improve the detection of occluded people and occluded keypoints. Examples for this can be seen in Fig. A.4 (A, B, C), Fig. A.5 (B, C) and Fig. A.6 (A). Of particular interest is Fig. A.5 (A). Here it can be seen that the training on purely synthetic data with adversarial teacher leads to better detection of people under challenging imaging conditions and improves the detection and grouping of joints in front of highly cluttered backgrounds.

Last, Fig. A.6 (A, B) suggest that 'adversarial Teacher' C improves the prediction for uncommon camera positions. Thus, training on the most challenging camera positions improves the predictions for such images in comparison to M D R and M D R + D Style +masks .

Fig. A.4. Example images with detected poses for M D R , the model trained with adversarial teacher using the camera pitch grouping ('adversarial Teacher' C ) and our best model M D R + D Style +masks .

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-9f56c137-070a-4bca-9a29-7850f4303882.jpg)

B

M D R

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-f59018f0-314f-4cf4-a5ee-b1610e136c72.jpg)

M D R

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-3a783ec1-6fc2-436b-a29e-5770a8a2b7e1.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-bf29d31d-3dc8-415a-8e08-fe14af810246.jpg)

adversarial teacher C

M D R + D Style +masks

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-4bc28f7d-a9ca-427e-96f9-77539fc602b5.jpg)

M D R

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-0e62ca2c-de55-466e-b940-9f7d98a7cac9.jpg)

M D R + D Style +masks

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-edfa67a0-f932-4007-af10-e11cd76cd0fd.jpg)

adversarial teacher C

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-e8748064-ff60-4172-aaf8-a3631ffa3aba.jpg)

adversarial teacher C

M D R + D Style +masks

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-857cc7a0-7948-4da1-ad50-37055e597a20.jpg)

M D R

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-e94ff306-fd6d-4fde-8fb1-57026ff2dcf9.jpg)

Fig. A.5. Example images with detected poses for M D R , the model trained with adversarial teacher using the camera pitch grouping ('adversarial Teacher' C ) and our best model M D R + D Style +masks .

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-59c1a961-d2de-4fd2-b3a7-d01c21362341.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-90db936f-5e68-4f22-adde-c6237b3cdc7a.jpg)

adversarial teacher C

M D R + D Style +masks

M D R

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-fd2c1a32-3b24-4fc2-9d23-2520002df068.jpg)

C

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-995f5fb2-ff91-4c35-bf98-91c0c06a2ec9.jpg)

M D R

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-72450740-bf05-4233-9995-420a07f5fd8b.jpg)

adversarial teacher C

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-9300e1d9-5882-480d-bd97-d0812ef04604.jpg)

adversarial teacher C

M D R + D Style +masks

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-a34a9824-7b74-4c45-932e-afc846ff31a2.jpg)

M D R + D Style +masks

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-be56f25e-b0f6-47a8-bf11-8962506874f5.jpg)

Fig. A.6. Example images with detected poses for M D R , the model trained with adversarial teacher using the camera pitch grouping ('adversarial Teacher' C ) and our best model M D R + D Style +masks .

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-23579bc2-9269-4ebe-b315-97e0326a5fa3.jpg)

M D R

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-d9bf720b-e936-48fe-b3ff-c04b51711d81.jpg)

M D R

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-5295514b-b071-488f-9da1-e7fb8be0771f.jpg)

adversarial teacher C

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-80431e9b-3906-465e-8385-38e75aa3ca9e.jpg)

M D R + D Style +masks

adversarial teacher C

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-55f1e112-364f-4188-a91e-cc1b64e68094.jpg)

M D R + D Style +masks

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00967v1-figure-cd51bc52-6bbe-4989-ad5d-f1ee853f077b.jpg)

## References

- 1. Carnegie-mellon mocap database. http://mocap.cs.cmu.edu
- 2. Abdulla, W.: Mask r-cnn for object detection and instance segmentation on keras and tensorflow. https://github.com/matterport/Mask RCNN (2017)
- 3. Akhter, I., Black, M.J.: Pose-conditioned joint angle limits for 3d human pose reconstruction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1446-1455 (2015)
- 4. Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2d pose estimation using part affinity fields. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1302-1310. IEEE (2017)
- 5. Fabbri, M., Lanzi, F., Calderara, S., Palazzi, A., Vezzani, R., Cucchiara, R.: Learning to detect and track visible and occluded body joints in a virtual world. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 430-446 (2018)
- 6. Green, R.: Spherical Harmonic Lighting: The Gritty Details. Archives of the Game Developers Conference (Mar 2003), http://www.research.scea.com/gdc2003/spherical-harmonic-lighting.pdf
- 7. He, K., Gkioxari, G., Doll´r, P., Girshick, R.: Mask r-cnn. In: Computer Vision a (ICCV), 2017 IEEE International Conference on. pp. 2980-2988. IEEE (2017)
- 8. Kinga, D., Adam, J.B.: A method for stochastic optimization. In: International Conference on Learning Representations (ICLR). vol. 5 (2015)
- 9. Loper, M., Mahmood, N., Black, M.J.: Mosh: Motion and shape capture from sparse markers. ACM Transactions on Graphics (TOG) 33 (6), 220 (2014)
- 10. Ranjan, A., Romero, J., Black, M.J.: Learning human optical flow. In: 29th British Machine Vision Conference (Sep 2018)
- 11. Robinette, K.M., Blackwell, S., Daanen, H., Boehmer, M., Fleming, S.: Civilian american and european surface anthropometry resource (caesar), final report. volume 1. summary. Tech. rep., DTIC Document (2002)
- 12. Rogez, G., Weinzaepfel, P., Schmid, C.: Lcr-net++: Multi-person 2d and 3d pose detection in natural images. IEEE transactions on pattern analysis and machine intelligence (2019)
- 13. Romero, J., Tzionas, D., Black, M.J.: Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia) 36 (6) (Nov 2017)
- 14. Sigal, L., Balan, A.O., Black, M.J.: Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. International journal of computer vision 87 (1-2), 4 (2010)
- 15. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
- 16. Teschner, M., Kimmerle, S., Heidelberger, B., Zachmann, G., Raghupathi, L., Fuhrmann, A., Cani, M.P., Faure, F., Magnenat-Thalmann, N., Strasser, W., Volino, P.: Collision detection for deformable objects. In: Eurographics. pp. 119139 (2004)
- 17. Tzionas, D., Ballan, L., Srikantha, A., Aponte, P., Pollefeys, M., Gall, J.: Capturing hands in action using discriminative salient points and physics simulation. International Journal of Computer Vision (IJCV) 118 (2), 172-193 (Jun 2016), https://doi.org/10.1007/s11263-016-0895-4
- 18. Varol, G., Romero, J., Martin, X., Mahmood, N., Black, M.J., Laptev, I., Schmid, C.: Learning from synthetic humans. In: CVPR (2017)
David T. Hoffmann Dimitrios Tzionas Micheal J. Black Siyu Tang
Learning to Train with Synthetic Humans
Neural networks need big annotated datasets for training. However, manual
annotation can be too expensive or even unfeasible for certain tasks, like
multi-person 2D pose estimation with severe occlusions. A remedy for this is
synthetic data with perfect ground truth. Here we explore two variations of
synthetic data for this challenging problem; a dataset with purely synthetic
humans and a real dataset augmented with synthetic humans. We then study which
approach better generalizes to real data, as well as the influence of virtual
humans in the training loss. Using the augmented dataset, without considering
synthetic humans in the loss, leads to the best results. We observe that not
all synthetic samples are equally informative for training, while the
informative samples are different for each training stage. To exploit this
observation, we employ an adversarial student-teacher framework; the teacher
improves the student by providing the hardest samples for its current state as
a challenge. Experiments show that the student-teacher framework outperforms
normal training on the purely synthetic dataset.