summaries: This paper introduces a framework to utilize the synthetic data as augmentations in the scene graph generation task, which is able to narrow the domain gap by decomposing it into several discrepancies between the two domains. They are the first to propose the synthetic-to-real transfer learning for SGG. The experimental results show the Sim2SG can improve the baseline models in three different scenarios: CLEVR, Dining-Sim, and Drive-Sim. The paper addresses the problem of learning scene graphs from synthetic data and unlabeled real data while performing well on real data by narrowing the content and appearance gap between the two domains when training on synthetic data. Scene graphs are extracted in a two-step process, mapping input to an intermediate latent space and generating the final prediction from the latent space. The authors decompose the content gap into two components: (a) label discrepancy, i.e. how much do the label distributions between the two domains differ, and (b) prediction discrepancy, i.e. the difference in distributions of outputs predicted from the latent space for the two domains. They further model the appearance gap by aligning the latent representation for both domains after accounting for the content gap (to avoid spurious influence of differing content distributions as the latent space is expected to comprise content and appearance). Most of these components are intractable and the paper provides approximations. Empirical investigation on two entirely synthetic and one real/synthetic data set provide evidence for the benefit of the method in closing the posed domain gaps as well as the quality of chosen approximations, the influence of the individual content and appearance gap terms, and the effectiveness of the optimization procedure. The paper tackles the problem of sim2real transfer for scene graph inference. It proposes an approach for closing the gap between simulated training data and real test data, to allow models trained purely on simulated data to be deployed on real images. The approach is tested on multiple environments, including transfer from a driving scene simulator to real KITTI scenes.
abstractText: Scene graph (SG) generation has been gaining a lot of traction recently. Current SG generation techniques, however, rely on the availability of expensive and limited number of labeled datasets. Synthetic data offers a viable alternative as labels are essentially free. However, neural network models trained on synthetic data, do not perform well on real data because of the domain gap. To overcome this challenge, we propose Sim2SG, a scalable technique for sim-to-real transfer for scene graph generation. Sim2SG addresses the domain gap by decomposing it into appearance, label and prediction discrepancies between the two domains. We handle these discrepancies by introducing pseudo statistic based self-learning and adversarial techniques. Sim2SG does not require costly supervision from the real-world dataset. Our experiments demonstrate significant improvements over baselines in reducing the domain gap both qualitatively and quantitatively. We validate our approach on toy simulators, as well as realistic simulators evaluated on real-world data.
references.authors: Kamyar Azizzadenesheli Anqi Liu Fanny Yang Animashree Anandkumar
references.title: Regularized learning for domain adaptation under label shifts
references.venue: In International Conference on Learning Representations,
references.authors: J. Borrego Atabak Dehban Rui Figueiredo Plinio Moreno Alexandre Bernardino J. Santos-Victor
references.title: Applying domain randomization to synthetic data for object category detection
references.authors: Daniel J. Butler Jonas Wulff Garrett B. Stanley Michael J. Black
references.title: A naturalistic open source movie for optical flow evaluation
references.venue: In A. Fitzgibbon et al. (Eds.) (ed.), ECCV, Part IV,
references.authors: Angel X. Chang Thomas Funkhouser Leonidas Guibas Pat Hanrahan Qixing Huang Zimo Li Silvio Savarese Manolis Savva Shuran Song Hao Su Jianxiong Xiao Li Yi Fisher Yu
references.title: Shapenet: An information-rich 3d model repository
references.authors: Wei-Lun Chang Hui-Po Wang Wen-Hsiao Peng Wei-Chen Chiu
references.title: All about structure: Adapting structural information across domains for boosting semantic segmentation
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2019. doi: 10.1109/cvpr.2019.00200. URL http://dx.doi. org/10.1109/CVPR.2019.00200
references.authors: Yuhua Chen Wen Li Christos Sakaridis Dengxin Dai Luc Van Gool
references.title: Domain adaptive faster r-cnn for object detection in the wild
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Yun-Chun Chen Yen-Yu Lin Ming-Hsuan Yang Jia-Bin Huang
references.title: Crdoco: Pixel-level domain transfer with cross-domain consistency
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2019. doi: 10.1109/cvpr.2019.00189. URL http://dx.doi.org/10.1109/CVPR
references.authors: Bo Dai Yuqi Zhang Dahua Lin
references.title: Detecting visual relationships with deep relational networks
references.venue: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017. doi: 10.1109/cvpr.2017.352. URL http://dx.doi.org/10.1109/CVPR.2017.352
references.authors: Matt Deitke Winson Han Alvaro Herrasti Aniruddha Kembhavi Eric Kolve Roozbeh Mottaghi Jordi Salvador Dustin Schwenk Eli VanderBilt Matthew Wallingford Luca Weihs Mark Yatskar Ali Farhadi
references.title: RoboTHOR: An Open Simulation-to-Real Embodied AI Platform
references.authors: Apoorva Dornadula Austin Narcomey Ranjay Krishna Michael Bernstein Li Fei-Fei
references.title: Learning predicates as functions to enable few-shot scene graph prediction, 2019
references.authors: A. Dosovitskiy P. Fischer E. Ilg P. Häusser C. Hazırbaş V. Golkov P. Smagt D. Cremers T. Brox
references.title: FlowNet: Learning optical flow with convolutional networks
references.venue: In IEEE International Conference on Computer Vision (ICCV),
references.authors: Alexey Dosovitskiy German Ros Felipe Codevilla Antonio Lopez Vladlen Koltun
references.title: CARLA: An open urban driving simulator
references.venue: In CORL,
references.authors: Geoffrey French Michal Mackiewicz Mark Fisher
references.title: Self-ensembling for visual domain adaptation
references.venue: In ICLR,
references.authors: Yaroslav Ganin Evgeniya Ustinova Hana Ajakan Pascal Germain Hugo Larochelle François Laviolette Mario Marchand Victor Lempitsky
references.title: Domain-adversarial training of neural networks
references.venue: Advances in Computer Vision and Pattern Recognition,
references.authors: Andreas Geiger Philip Lenz Raquel Urtasun
references.title: Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite
references.venue: In CVPR,
references.authors: Robert Geirhos Patricia Rubisch Claudio Michaelis Matthias Bethge Felix A. Wichmann Wieland Brendel
references.title: Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
references.venue: In International Conference on Learning Representations,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2016. doi: 10.1109/cvpr.2016.90. URL http://dx.doi.org/10.1109/cvpr.2016.90
references.authors: Roei Herzig Moshiko Raboh Gal Chechik Jonathan Berant Amir Globerson
references.title: Mapping images to scene graphs with permutation-invariant structured prediction
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Roei Herzig Amir Bar Huijuan Xu Gal Chechik Trevor Darrell Amir Globerson
references.title: Learning canonical representations for scene graph to image generation
references.venue: In European Conference on Computer Vision,
references.authors: Judy Hoffman Eric Tzeng Taesung Park Jun-Yan Zhu Phillip Isol Kate Saenko Alexei A. Efros Trevor Darrell
references.title: Cycada: Cycle-consistent adversarial domain adaptation
references.authors: Xun Huang Ming-Yu Liu Serge Belongie Jan Kautz
references.title: Multimodal unsupervised image-to-image translation
references.venue: In ECCV,
references.authors: Justin Johnson Bharath Hariharan Laurens van der Maaten Li Fei-Fei C Lawrence Zitnick Ross Girshick
references.title: Clevr: A diagnostic dataset for compositional language and elementary visual reasoning
references.authors: Amlan Kar Aayush Prakash Ming-Yu Liu Eric Cameracci Justin Yuan Matt Rusiniak David Acuna Antonio Torralba Sanja Fidler
references.title: Meta-sim: Learning to generate synthetic datasets
references.venue: IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2019. doi: 10.1109/iccv.2019.00465. URL http://dx.doi. org/10.1109/ICCV.2019.00465
references.authors: Seunghyeon Kim Jaehoon Choi Taekyung Kim Changick Kim
references.title: Self-training and adversarial background regularization for unsupervised domain adaptive one-stage object detection
references.venue: IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2019. doi: 10.1109/iccv.2019.00619. URL http://dx.doi. org/10.1109/ICCV.2019.00619
references.authors: Eric Kolve Roozbeh Mottaghi Winson Han Eli VanderBilt Luca Weihs Alvaro Herrasti Daniel Gordon Yuke Zhu Abhinav Gupta Ali Farhadi
references.title: AI2-THOR: An interactive 3D environment for visual AI
references.venue: arXiv preprint arXiv:1712.05474,
references.authors: Ranjay Krishna Yuke Zhu Oliver Groth Justin Johnson Kenji Hata Joshua Kravitz Stephanie Chen Yannis Kalantidis Li-Jia Li David A. Shamma
references.title: Visual genome: Connecting language and vision using crowdsourced dense image annotations
references.venue: International Journal of Computer Vision,
references.authors: Ranjay Krishna Vincent Chen Paroma Varma Michael Bernstein Christopher Re Li Fei-Fei
references.title: Scene graph prediction with limited labels
references.venue: IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2019. doi: 10.1109/iccv.2019.00267. URL http://dx.doi.org/10.1109/ICCV.2019.00267
references.authors: Congcong Li Dawei Du Libo Zhang Longyin Wen Tiejian Luo Yanjun Wu Pengfei Zhu
references.title: Spatial attention pyramid network for unsupervised domain adaptation, 2020
references.authors: Peilun Li Xiaodan Liang Daoyuan Jia Eric P. Xing
references.title: Semantic-aware grad-gan for virtual-to-real urban scene adaption
references.venue: In BMVC,
references.authors: Yikang Li Wanli Ouyang Bolei Zhou Kun Wang Xiaogang Wang
references.title: Scene graph generation from objects, phrases and region captions
references.venue: IEEE International Conference on Computer Vision (ICCV), Oct 2017. doi: 10.1109/iccv.2017.142. URL http://dx.doi.org/10.1109/ICCV.2017.142
references.authors: Yunsheng Li Lu Yuan Nuno Vasconcelos
references.title: Bidirectional learning for domain adaptation of semantic segmentation
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2019. doi: 10.1109/cvpr.2019.00710. URL http://dx.doi.org/10.1109/CVPR.2019.00710
references.authors: Zachary C. Lipton Yu-Xiang Wang Alex Smola
references.title: Detecting and correcting for label shift with black box predictors, 2018
references.authors: Cewu Lu Ranjay Krishna Michael Bernstein Li Fei-Fei
references.title: Visual relationship detection with language priors
references.venue: In European Conference on Computer Vision,
references.authors: Yawei Luo Liang Zheng Tao Guan Junqing Yu Yi Yang
references.title: Taking a closer look at domain shift: Categorylevel adversaries for semantics consistent domain adaptation
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition
references.authors: Rui Ma Akshay Gadi Patil Matthew Fisher Manyi Li Sören Pirk Binh-Son Hua Sai-Kit Yeung Xin Tong Leonidas Guibas Hao Zhang
references.title: Language-driven synthesis of 3d scenes from scene databases
references.venue: In SIGGRAPH Asia
references.authors: Nikolaus Mayer Eddy Ilg Philip Hausser Philipp Fischer Daniel Cremers Alexey Dosovitskiy Thomas Brox
references.title: A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation
references.authors: John McCormac Ankur Handa Stefan Leutenegger Andrew J Davison
references.title: Scenenet rgb-d: 5m photorealistic images of synthetic indoor trajectories with ground truth
references.venue: arXiv preprint arXiv:1612.05079,
references.authors: Lars Mescheder Michael Oechsle Michael Niemeyer Sebastian Nowozin Andreas Geiger
references.title: Occupancy networks: Learning 3d reconstruction in function space
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Matthias Mueller Vincent Casser Jean Lahoud Neil Smith Bernard Ghanem
references.title: Sim4CV: A photo-realistic simulator for computer vision applications
references.authors: Alejandro Newell Jia Deng
references.title: Pixels to graphs by associative embedding
references.venue: In Advances in neural information processing systems,
references.authors: Aayush Prakash Shaad Boochoon Mark Brophy David Acuna Eric Cameracci Gavriel State Omer Shapira Stan Birchfield
references.title: Structured domain randomization: Bridging the reality gap by context-aware synthetic data
references.venue: In 2019 International Conference on Robotics and Automation (ICRA),
references.authors: Weichao Qiu Alan Yuille
references.title: Unrealcv: Connecting computer vision to unreal engine
references.venue: In European Conference on Computer Vision,
references.authors: Shaoqing Ren Kaiming He Ross B. Girshick Jian Sun
references.title: Faster r-cnn: Towards real-time object detection with region proposal networks
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Stephan R Richter Vibhav Vineet Stefan Roth Vladlen Koltun
references.title: Playing for data: Ground truth from computer games
references.venue: In ECCV,
references.authors: German Ros Laura Sellart Joanna Materzynska David Vazquez Antonio Lopez
references.title: The SYNTHIA Dataset: A large collection of synthetic images for semantic segmentation of urban scenes
references.authors: Kuniaki Saito Yoshitaka Ushiku Tatsuya Harada Kate Saenko
references.title: Strong-weak distribution alignment for adaptive object detection
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2019. doi: 10.1109/cvpr.2019.00712. URL http://dx.doi.org/10.1109/CVPR
references.authors: Yu Sun Eric Tzeng Trevor Darrell Alexei A. Efros
references.title: Unsupervised domain adaptation through selfsupervision, 2019
references.authors: Supasorn Suwajanakorn Noah Snavely Jonathan Tompson Mohammad Norouzi
references.title: Discovery of latent 3D keypoints via end-to-end geometric reasoning
references.authors: Shuhan Tan Xingchao Peng Kate Saenko
references.title: Generalized domain adaptation with covariate and label shift co-alignment, 2019
references.authors: Thang To Jonathan Tremblay Duncan McKay Yukie Yamaguchi Kirby Leung Adrian Balanon Jia Cheng Stan Birchfield
references.title: NDDS: NVIDIA deep learning dataset synthesizer, 2018. https://github.com/ NVIDIA/Dataset_Synthesizer
references.authors: Josh Tobin Rachel Fong Alex Ray Jonas Schneider Wojciech Zaremba Pieter Abbeel
references.title: Domain randomization for transferring deep neural networks from simulation to the real world
references.authors: A. Torralba A.A. Efros
references.title: Unbiased look at dataset bias
references.authors: Jonathan Tremblay Thang To Balakumar Sundaralingam Yu Xiang Dieter Fox Stan Birchfield
references.title: Deep object pose estimation for semantic robotic grasping of household objects, 2018
references.authors: Apostolia Tsirikoglou Joel Kronander Magnus Wrenninge Jonas Unger
references.title: Procedural modeling and physically based rendering for synthetic data generation in automotive applications
references.authors: Nanyang Wang Yinda Zhang Zhuwen Li Yanwei Fu Wei Liu Yu-Gang Jiang
references.title: Pixel2mesh: Generating 3d mesh models from single rgb images, 2018
references.authors: Magnus Wrenninge Jonas Unger
references.title: Synscapes: A photorealistic synthetic dataset for street scene parsing
references.venue: In arXiv:1810.08705,
references.authors: Yifan Wu Ezra Winston Divyansh Kaushik Zachary Lipton
references.title: Domain adaptation with asymmetricallyrelaxed distribution alignment, 2019
references.authors: Fanbo Xiang Yuzhe Qin Kaichun Mo Yikuan Xia Hao Zhu Fangchen Liu Minghua Liu Hanxiao Jiang Yifu Yuan He Wang
references.title: Sapien: A simulated part-based interactive environment
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Danfei Xu Yuke Zhu Christopher B. Choy Li Fei-Fei
references.title: Scene graph generation by iterative message passing
references.venue: IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Minghao Xu Hang Wang Bingbing Ni Qi Tian Wenjun Zhang
references.title: Cross-domain detection via graph-induced prototype alignment
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2020. doi: 10.1109/cvpr42600.2020.01237. URL http://dx.doi.org/10.1109/CVPR42600
references.authors: Qiangeng Xu Weiyue Wang Duygu Ceylan Radomir Mech Ulrich Neumann
references.title: Disn: Deep implicit surface network for high-quality single-view 3d reconstruction, 2019
references.authors: Jianwei Yang Jiasen Lu Stefan Lee Dhruv Batra Devi Parikh
references.title: Graph r-cnn for scene graph generation
references.venue: Lecture Notes in Computer Science,
references.authors: Rowan Zellers Mark Yatskar Sam Thomson Yejin Choi
references.title: Neural motifs: Scene graph parsing with global context
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Y. Zhang W. Qiu Q. Chen X. Hu A. Yuille
references.title: UnrealStereo: A synthetic dataset for analyzing stereo vision
references.authors: Han Zhao Remi Tachet des Combes Kun Zhang Geoffrey J. Gordon
references.title: On learning invariant representation for domain adaptation, 2019
references.authors: Zhedong Zheng Yi Yang
references.title: Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation, 2020
references.authors: Jun-Yan Zhu Taesung Park Phillip Isola Alexei A Efros
references.title: Unpaired image-to-image translation using cycle-consistent adversarial networkss
references.venue: In ICCV,
references.authors: Yang Zou Zhiding Yu B.V.K. Vijaya Kumar Jinsong Wang
references.title: Domain adaptation for semantic segmentation via class-balanced self-training
references.authors: Xu
references.title: Baselines: We adapt domain adaptation baselines (Chen et
references.venue: We do not adapt SAPNet (Li et al.,
references.authors: SG GraphRCNN
references.title: predictor) for 60k iterations using the same optimizer and learning rate as Sim2SG. For self-learning based on pseduo labels (Zou et al., 2018), we obtain the pseudo labels on KITTI images using the most confident predictions by synthetic pretrained GraphRCNN network (as per the authors). We then train these labeled KITTI images on Sim2SG framework for 60k iterations using the same optimizer and learning rate as Sim2SG
sections.heading: 1 INTRODUCTION
sections.text: Scene Graphs (SGs) in both computer vision and computer graphics are an interpretable and structural representation of scenes. A scene graph summarizes entities in the scene and plausible relationships among them. SGs (Dai et al., 2017; Herzig et al., 2018; Li et al., 2017; Newell & Deng, 2017; Xu et al., 2017; Yang et al., 2018; Zellers et al., 2018) are a manifestation of vision as inverse graphics. They have found a variety of applications such as image captioning, visual question answering, high level reasoning tasks, image retrieval, image generation, etc. However, most prior work on SG generation relies on the availability of expensive and limited number of labeled datasets such as Visual Genome (Krishna et al., 2017) and Visual Relationship Dataset (VRD) (Lu et al., 2016).
  One of the main limitations in machine learning applications is the general lack of sufficient labeled data for supervised learning tasks. Synthetic data is a viable alternative to this problem since annotations are essentially free. Synthetic data has been used for a variety of tasks such as image classification, object detection, semantic segmentation, optical flow modeling, 3D keypoint extraction, object pose estimation, 3D reconstruction, etc. (Borrego et al., 2018; Butler et al., 2012; Dosovitskiy et al., 2015; McCormac et al., 2016; Mueller et al., 2017; Richter et al., 2016; Ros et al., 2016; Suwajanakorn et al., 2018; Tremblay et al., 2018; Tsirikoglou et al., 2017). It has also been shown to be effective in initializing task networks (Prakash et al., 2019) and for data augmentation. However, the use of synthetic data for SG generation and visual relationships is yet to be explored.
  One crucial issue with training on a labeled source domain (synthetic data) and evaluating on an unlabeled target domain (real data) is the performance gap known as domain gap (Torralba & Efros, 2011). This gap is due to the difference of data distribution between the source and target domains. Kar et al. (2019) argue that domain gap can be divided into appearance and content gap. The appearance gap can be addressed by making scenes photo-realistic (McCormac et al., 2016; Wrenninge & Unger, 2018), by using image translations (Hoffman et al., 2018; Huang et al., 2018; Zhu et al., 2017), by feature alignment (Chang et al., 2019; Chen et al., 2018; Li et al., 2019; Luo et al., 2019; Saito et al., 2019; Sun et al., 2019), or by learning robust representations based on domain randomization (Prakash et al., 2019; Tobin et al., 2017). There are also studies that address the content gap for image classification (Azizzadenesheli et al., 2019; Lipton et al., 2018; Tan et al., 2019). We present a thorough investigation of the domain gap between source and target domains. We assume a gap in both appearance and content, expand those gaps into different sub-components and provide a way to address them. We primarily apply our method to reduce the domain gap for
  SG generation. Nonetheless, our techniques can also be applied to other vision tasks such as image classification, image segmentation and object detection among others.
  We propose Sim2SG (Simulation to Scene Graph); a model that learns sim-to-real scene graph generation leveraging labeled synthetic data and unlabeled real data. Extending the formulation in (Wu et al., 2019), Sim2SG addresses the domain gap by bounding the task error (where the task is scene graph generation) on real data through appearance, prediction, label (ground truth) discrepancies between the two domains and task error on synthetic data. Our work differs from (Wu et al., 2019) as they do not provide a way to address the content gap, and their risk discrepancy is intractable. To the best of our knowledge, Sim2SG is the first work to introduce a tractable error bound on the content component of the domain gap.
  We minimize the appearance and prediction discrepancies by aligning the corresponding latent and output distributions via Gradient Reversal Layers (Ganin et al., 2017). We address discrepancy in the label using principles of self-learning (Zou et al., 2018). However, self-learning based on pseudo labels often suffer from the inaccurately generated labels (e.g. predicted bounding boxes are ill-categorized or imprecise, hence, the model will regress on the wrong objects) (Zheng & Yang, 2020; Kim et al., 2019). Therefore, we instead propose to collect a higher level statistic (e.g. list of objects and their type, position and relationships for placement), that we call pseudo-statistics, from target data and leverage the synthetic data generator to produce valid objects with their precise labels (e.g. bounding boxes). We experimentally demonstrate our method in three distinct environments– all synthetic CLEVR (Johnson et al., 2017), more realistic Dining-Sim and Drive-Sim with a driving simulator evaluated on KITTI (Geiger et al., 2012). We almost close the domain gap in the Clevr environment and we show significant improvements over respective baselines in Dining-Sim and Drive-Sim. Through ablations, we validate our assumptions about appearance and content gaps. Sim2SG differs from other unsupervised domain adaptation methods (Chen et al., 2018; Xu et al., 2020; Li et al., 2020) as it can modify the source distribution (via self-learning based on pseudostatistics to align with the target distribution) with access to a synthetic data generator. We also outperform these domain adaptation baselines (Chen et al., 2018; Xu et al., 2020; Li et al., 2020) as shown in Section 4.3.
  Contributions: Our contributions are three-fold: In terms of methodology, to the best of our knowledge, (1) We are the first to propose sim-to-real transfer learning for scene graph generation. We do not require costly supervision from the target real-world dataset. (2) We study domain gap from synthetic to real data in detail, provide a tractable error bound on the content component of the gap and propose a novel pipeline including pseudo statistics to fully handle the gap. Experimentally, (3) we show that Sim2SG can learn SG generation and obtains significant improvements over baselines in all three scenarios - Clevr, Dining-Sim and Drive-Sim. We also present ablations to illustrate the effectiveness of our technique.
sections.heading: 2 PROPOSED METHOD: SIM2SG
sections.text: Our proposed Sim2SG pipeline is illustrated in Figure 1. We first describe how we generate scene graphs in Section 2.1. When we naïvely train on a source distribution (synthetic data) and evaluate on a target distribution (real data), we have a domain gap (Torralba & Efros, 2011). We study it in more detail in Section 2.2 and propose methods to address it.
sections.heading: 2.1 SCENE GRAPHS
sections.text: This section describes scene graphs (SGs) and how we train the SG predictor module using labels from the source domain.
  Notation: We represent a scene graph of a given image I as a graph G with nodes o and edges r. Each node is a tuple oi = 〈bi, ci〉 of bounding box bi = {xmini, ymini, wi, hi} and category ci. Relationships r are a triplet of 〈oi, p, oj〉 where p is a predicate. SG prediction has two key components: feature extractor φ and graph predictor h. φ maps input space x to a latent space z and h maps from latent space z to output space y. The predicted SG is G = h (φ(x)).
  We use Resnet 101 (He et al., 2016) to implement φ and GraphRCNN (Yang et al., 2018) architecture to implement h. We train the networks φ and h using the following task loss (Yang et al., 2018): cross entropy loss for object classification & relationship classification and `1 loss for bounding boxes.
  Our framework Sim2SG is illustrated in Figure 1. It is worth noting that we predict the relationships among objects in the scene, but not their attributes like (Yang et al., 2018).
sections.heading: 2.2 DOMAIN GAP
sections.text: We now study the domain gap between source and target domains and formulate the SG generation task error on real domain as a function of the task error on synthetic domain and appearance & content gap between the domains. We then propose methods to address each subcomponent of the gap.
  Notation and assumptions: We have two domains: synthetic 〈xs, ys〉 ∼ p(x, y) and real domain 〈xr, yr〉 ∼ q(x, y). Note that xs or xr is the input (image) and ys or yr is the output (SG) as introduced in Section 2.1 It is also worth noting that yr is not known. We assume that synthetic and real domains contain the same categories of objects. Both domains also share similar scenarios (e.g. both have driving scenes). However, joint distributions of scenes (images) are different in both domains (i.e. p(x) 6= q(x)). The label (ground truth) distributions are also different in the two domains (i.e. p(y) 6= q(y)). Using the formulation in (Wu et al., 2019), the task (SG generation) error (i.e. risk) on synthetic domain as a function of the latent space z is given by:
  s(φ, h) = ∫ p(z)esdz (1)
  where es is the risk defined as es = |p(y|z) − ys|. p(z) is the distribution of features, p(y) is the distribution of labels and p(y|z) is the output distribution. The bound on task error in real domain r(φ, h) from Wu et al. (2019) is a function of three terms which are: (1) task error on the synthetic domain s(φ, h), (2) risk discrepancy between the domains c(φ, h) and (3) feature discrepancy between the two domains a(φ, h). Our goal is to minimize the task error on the real domain:
  r(φ, h) = ∫ q(z)erdz = ∫ q(z)erdz + ∫ p(z)esdz − ∫ p(z)esdz + ∫ q(z)esdz − ∫ q(z)esdz
  = ∫ p(z)esdz + ∫ q(z)(er − es)dz ∫ (q(z)− p(z))esdz
  = s(φ, h) + c(φ, h) + a(φ, h)
  (2)
  We drop the terms φ, h from now on. We would need to minimize s, a and c to reduce the task error on the real domain r ≥ 0. If the error r reduces to zero on the target domain, we have closed the domain gap. Since we have access to the label ys, we can minimize the task error on synthetic domain s as described in Section 2.1. However, we do not have access to the label distribution q(y) for the real domain. This makes the risk discrepancy c intractable. We split this discrepancy into tractable components and address them in Section 2.2.1. We call the discrepancy content gap based on our
  Algorithm 1 Pseudocode for Sim2SG training 1: Given: φθ, hθ . Encoder, Scene Graph predictor 2: Given: Xs, Ys, R,Xr . Synthetic Images, Synthetic Labels, Data Generator, Real Images 3: Hyperparameters: E, I . Epochs, Iterations 4: for e ∈ E do . Training 5: loss = 0; 6: for i ∈ Im do 7: loss += a(φ(xs), φ(xr)) . Appearance gap 8: loss += c,pred(h(φ(xs)), h(φ(xr))) . Content prediction gap 9: loss += task(xs, ys) . Scene Graph generation task loss 10: φθ , hθ = optimize(φθ ,hθ , loss); . Network parameters θ ; SGD step 11: end for 12: ps = {h (φ(xr)) : xr ∈ Xr} . extracting pseudo statistics from predictions 13: Xs, Ys = R(ps) . generating synthetic data with labels aligned to real data 14: end for
  ablations in Section 4.1. Recent work shows that features frequently account for appearance (Geirhos et al., 2019), and therefore we call the feature discrepancy a as appearance gap and empirically show through ablations in Section 4.1 to be the case. We address this discrepancy in Section 2.2.2. The pseudo-code of our proposed method is illustrated in Algorithm 1.
  2.2.1 CONTENT GAP: c
  Content gap refers to discrepancies between the two domains including the difference in distribution of the number of objects and their class, placement, pose and scale. This affects the position, dimension and the type of the labels leading to the problem of label shift (i.e. discrepancy in the ground truth distribution). However, minimizing c in the current form is not tractable. Since the task error (risk) r is non-negative and both r and er are lower bounded by zero, we also assume a lower bound of zero for c. Hence, we approximate the (er - es) as ((q(y|z)− p(y|z)) + (ys − yr)). We show this in equation 3. We assume ys ≡ yr and q(y|z) ≡ p(y|z) as a sufficient condition for ((q(y|z) − p(y|z)) + (ys − yr)) to be zero and to avoid any degenerate solutions. Based on these assumptions, we also empirically demonstrate in Section 4 that the label discrepancy c,label and the prediction discrepancy c,pred help reduce the domain gap.
  argmin θ c = argmin θ
  ∫ q(z)(er − es)dz
  ' argmin θ
  ∫ q(z)(ys − yr)dz + ∫ q(z)(q(y|z)− p(y|z))dz
  = argmin θ
  c,label + c,pred
  (3)
  Label discrepancy and Pseudo-Statistics: Minimizing c,label is challenging because we do not have access to the label (yr) of the target (real) domain. We then propose to get an estimate of yr through principles of self-learning (Zou et al., 2018) based on minimum reliable statistic (pseudostatistic) of target data and generate aligned synthetic data (ys) to narrow the gap. We generate SGs for all input images (xr) of the target domain and derive the pseudo statistic ps from each SG by retaining the minimum information needed for 3D representation (e.g. we discard the entire bounding box and keep the centroid position). The statistic ps is a list of objects with each object’s type, centroid location and relationship with others. Using either known or assumed camera intrinsic (e.g. car dash cam), we map ps to a full 3D scene. Some unknown parameters (e.g. texture or pose) and context (ground, sky, light) are randomized as done in (Prakash et al., 2019). We use a synthetic data generator to render those scenes as shown in Figure 11. More details on scene generation for different environments are in Sections A.2.1, A.2.2 and A.2.3 of Appendix. This is also analogous to an Expectation Maximization algorithm where we compute pseudo-statistics and generate aligned synthetic data (E-step) and then use it to train the Sim2SG model (M-step).
  Contrarily, self-learning based on pseudo labels (Zou et al., 2018) generated from and trained on real data often suffer from poorly generated labels (Zheng & Yang, 2020; Kim et al., 2019). For instance, training SG generation model on ill-categorized or imprecise bounding boxes can lead to poor detection results. Pseudo-Statistics (described previously) integrated with a synthetic data generator (renderer) will still produce a valid scene for those ill-detected objects (false positives) with
  precise labels (bounding boxes). We show through quantitative experiments that our method performs better than pseudo label based self-learning in Section 4.3. Please note that our method is orthogonal to pseudo labels based self-learning and the latter can potentially be applied in conjunction.
  Prediction discrepancy: The output of the scene graph generation model should be the same for same categories in different domains. To address the prediction discrepancy c,pred, we propose to align output distribution for latent code belonging to the same visual category. To this end, we align the output distributions p(y|z) and q(y|z) using a GRL based technique (Ganin et al., 2017) that we will discuss in Section 2.2.2.
  2.2.2 APPEARANCE GAP: a Appearance gap is the discrepancy in the appearance of the two domains. This includes differences in texture, color, light, reflectance, etc. of objects in the scene. To address the appearance gap, we want to avoid photo-realism in synthetic data as it requires high quality assets and a tremendous amount of effort from artists (McCormac et al., 2016; Wrenninge & Unger, 2018). Therefore, we propose to learn an appearance invariant representation z so we can avoid the appearance bias in the model. However, the features z : φ(x) have both content and appearance components and aligning them may be detrimental (Saito et al., 2019; Wu et al., 2019) as shown in the ablation in Section 4.3. To overcome this challenge, we align the appearance gap a only after we align the content gap c of the two domains using the method described in Section 2.2.1. We minimize this appearance gap a by aligning the feature distributions p(z) and q(z) as follows:
  θ∗ = argmin θ
  ∫ (q(z)− p(z))esdz (4)
  where we exploit the fact that p(z) ≡ q(z) is a sufficient condition for a to be zero (Wu et al., 2019). We use Gradient Reversal Layer (GRL) (Ganin et al., 2017) to align the distributions p(z) and q(z) along with a domain classifier D to classify them. We minimize the D’s loss w.r.t. its own parameters while maximizing w.r.t. the network parameters of φ (θ). We do so through GRL that acts as an identity function during forward propagation and flips the sign of the gradients during back propagation from D to φ. We provide details of GRL layers in Section A.1
sections.heading: 3 RELATED WORK
sections.text: Scene Graphs: There has been a plethora of work on Visual Relationships and Scene Graph (SG) generation (Dai et al., 2017; Herzig et al., 2018; Li et al., 2017; Newell & Deng, 2017; Xu et al., 2017; Yang et al., 2018; Zellers et al., 2018). While all this work is fully supervised and some use priors, they still depend on availability of expensive, large scale data (Krishna et al., 2017; Lu et al., 2016). Krishna et al. (2019) propose a way to generate SGs using limited labels and Dornadula et al. (2019) learn to generalize SG prediction to rare categories in a few shot setting. However, they still use either existing object detection models trained on large data or ground truth object labels. We, on the other hand, require no labels whatsoever in the target domain. Alternatively, there are also works on scene generation from SG (Ma et al., 2018; Herzig et al., 2020)
  Synthetic Data has been used for many tasks including, but not limited to, object detection (Kar et al., 2019; Prakash et al., 2019), semantic segmentation (Richter et al., 2016; Ros et al., 2016; Tsirikoglou et al., 2017), optical flow modeling (Butler et al., 2012; Dosovitskiy et al., 2015), scene flow (Mayer et al., 2016), classification (Borrego et al., 2018), stereo (Qiu & Yuille, 2016; Zhang et al., 2016), 3D keypoint extraction (Suwajanakorn et al., 2018), object pose estimation (Mueller et al., 2017; Tremblay et al., 2018) and 3D reconstruction (McCormac et al., 2016). However, to the best of our knowledge, synthetic data has not been applied to scene graph generation.
  Domain Gap is the performance gap when the network is trained on a synthetic domain and evaluated on real data. Kar et al. (2019) argue that domain gap has two components which are appearance and content. We work with the same assumptions. Most prior work addresses the appearance gap by image translations (Chen et al., 2019; French et al., 2018; Hoffman et al., 2018; Huang et al., 2018; Li et al., 2018; Zhu et al., 2017), clever feature alignment (Chen et al., 2018; Li et al., 2019; Luo et al., 2019; Saito et al., 2019; Xu et al., 2020; Li et al., 2020) and domain randomization (Prakash et al., 2019; Tobin et al., 2017). There are few works which handle the content gap (Azizzadenesheli et al., 2019; Lipton et al., 2018; Tan et al., 2019; Zhao et al., 2019). More specifically, they address the label shift between the two domains. However, they do not exploit the unlabeled images from
  the target domain. We, on the other hand, leverage the images from the target domain to reduce the domain gap further. We are also interested in a scene graph generation task which is more complex than classification. The idea of self-training with pseudo labels (Zou et al., 2018) is used in (Li et al., 2019; Tan et al., 2019) to learn models from target distribution. However, the labels predicted by the model on the target are often inaccurate because of domain gap (Zheng & Yang, 2020). We instead rely on pseudo statistics and use a synthetic data generator to produce accurate labels. Similar to us, Chang et al. (2019); Sun et al. (2019) also train their task model on top of domain invariant features for image classification and image segmentation.
sections.heading: 4 EXPERIMENTS
sections.text: We evaluate Sim2SG in three different environments with increasing complexity. In each environment we have a fully labeled source domain and unlabeled target domain with labeled test data. We present a simple environment using CLEVR (Johnson et al., 2017) as described in Section 4.1 and a more realistic synthetic environment in Dining-Sim using ShapeNet (Chang et al., 2015) in Section 4.2. We then use an Unreal Engine 4 based driving simulator similar to (Prakash et al., 2019) as the source domain, and real images from KITTI (Geiger et al., 2012) as the target domain in Section 4.3. Using these experiments, we show that Sim2SG learns sim-to-real scene graph (SG) generation, it reduces the domain gap, and our label alignment (Eq. (3)), prediction alignment (Eq. (3)) and appearance alignment (Eq. (4)), work as intended. We compare Sim2SG to several baselines in DriveSim environment (Section 4.3). Our quantitative evaluation metric includes detection mAP (mean average precision) @ 0.5 IoU (Intersection over Union) and relationship triplet recall @20 or @50 (Krishna et al., 2017). Note that relationship triplet recall implicitly includes object detection recall as well (see Appendix). All the mean and standard deviations are based on five runs.
sections.heading: 4.1 TOY EXPERIMENTS ON THE CLEVR DATASET
sections.text: The goal of the experiments on the CLEVR environment (Johnson et al., 2017) is to show that Sim2SG can learn sim-to-real scene graph generation on an unlabeled domain while addressing the domain gap between source and target domains. Additionally, we show that appearance alignment a and label alignment c,label work as intended through ablations. For our experiments, we have 3 classes of objects: cube, sphere and cylinder and 4 kinds of relationships: front, behind, left and right. Our source domain mimics the synthetic environment by having simple texture, different number of objects (4) and closer objects. Nonetheless, objects are placed at random locations in the scene as shown in the top row of Figure 4. The target domain is representative of real data with a different set of colors, number of objects (2 or 3) and objects placed with larger margin. Regardless, the objects can be placed anywhere and have more complex texture by applying a style transfer network to the generated scenes (second row of Figure 4). We use 1000 labeled images of source, 1000 unlabeled images of target for training and 200 labeled images of source and target for evaluation. Details of the environment, training and hyper-parameters are in Section A.2.1 of Appendix.
  Results: Quantitative evaluation of Sim2SG is reported in Table 1. When we train on the source domain and evaluate on a test set from the source domain itself, we achieve 1.0 mAP @0.5 IoU for detection 0.986 recall@20 for relationship triplets. Hence, the first row shows that there is a domain gap from source to target as seen from the lower mAP and recall values. Second and third rows show how label alignment c,label and appearance alignment a, drastically reduce the domain gap compared to baseline (source only). We see that c,label improves detection performance as it is possible that the detection recall is more sensitive to content gap. Nonetheless, a is more effective in improving the relationship triplet recall as we have found it to reduce false positive detections effectively (Figure 6). The domain gap reduces further by combining both c,label and a terms (fourth row). Please note that we report saturation performance (details in Appendix). Qualitative improvements of scene graph recall over baseline are shown in first row of Figure 2.
  Ablations: We conduct two sets of experiments on the CLEVR dataset and quantitative results are shown in Table 2. The first experiment studies appearance gap: source and target have the same number of objects and range of margin, but they use different color and texture. Additionally, we transform the target by using the style transfer network. We observe that style alignment a reduces the domain gap with significant improvement over baseline. However, the label alignment c,label fails to have significant improvement on relationship triplet recall. Similarly, the second experiment studies content gap only (i.e. source and target use same color and texture but different number of objects and margin). We observe that label alignment c,label closes the domain gap completely. However a leads to performance degradation. These experiments show our label alignment c,label reduces content gap and appearance alignment a addresses appearance gap.
sections.heading: 4.2 EXPERIMENTS ON DINING-SIM
sections.text: The goal of our experiments on Dining-Sim created from ShapeNet objects (Chang et al., 2015) is to show that Sim2SG works as intended on a more complex dataset where target domain is representative of real data. This dataset has 3 classes of objects – chair, table and laptop. There are 5 kinds of relationships – front, behind, left, right, and on. The source and target domains are illustrated in the top two rows of Figure 7 and details are in Appendix. The results agree with the findings of Section 4.1. We also see domain gap in this environment as evaluation on source domain (1.000
  mAP @0.5 IoU & 0.995 recall@50) is higher than on target domain (0.584 mAP @0.5 IoU & 0.331 recall@50). The oracle performance on target domain stands at 0.904 mAP@0.5 IoU & 0.846 recall@50. Label alignment c,label drastically improves performance on target domain (0.713 mAP @0.5 IoU & 0.501 recall@50). a reduces false positives (Figure 10). We achieve the best scene graph recall @50 using a combination of label alignment c,label, appearance alignment a and prediction alignment c,pred (0.729 mAP @0.5 IoU & 0.547 recall@50). Complete results that validate our approach are in Table 4 in Appendix. Qualitative improvements of scene graph generation over baseline are illustrated in the second row of Figure 2.
sections.heading: 4.3 REAL-WORLD EXPERIMENTS ON DRIVE-SIM
sections.text: In this section, we validate our approach on a real-world dataset. For synthetic data, we use a simulator similar to (Prakash et al., 2019) with minor simplifications (fixing camera and road spline parameters, exclusion of some objects, see details in Appendix) to make our generation easier. The number of lanes, sidewalk, cars, vegetation, houses, pedestrians; their positions, pose, color, texture; light settings are randomly picked from a set of realistic values (details in Appendix.) akin to (Prakash et al., 2019) as shown in Figure 13. We use four classes: car, pedestrian, vegetation, house and four types of relationships: front, left, right, behind. All our relationships have the car as the subject. For example, ’car behind car’, ’vegetation left car’, etc. Although we show the ’on’ relationship to work in Dining-Sim environment, we found these relationships were always trivial to predict because they are always true: e.g. cars are always ’on’ road, pedestrian ’on’ sidewalk, etc. Therefore, we did not include them in the experiments. We use KITTI (Geiger et al., 2012) as the target domain. We need a small amount of labels on KITTI for evaluation only. Hence, while keeping the existing annotations for cars and pedestrians, we add annotations for vegetation and houses along with relationships among them. We will release these annotations to the community. We use 6000 labeled synthetic images, 6000 unlabeled KITTI images for training, and 1000 labeled synthetic and 550 labeled KITTI images for evaluation. See Appendix for details on KITTI annotation schema, training and hyper-parameters.
  Baselines: We compare Sim2SG to the randomization based method (Prakash et al., 2019), the method addressing content gap (Kar et al., 2019), self-learning based on pseudo labels (Zou et al., 2018) and domain adaptation methods for object detection (Chen et al., 2018; Xu et al., 2020; Li et al., 2020).
  Prakash et al. (2019) use a context based randomization of pose, position and texture of objects of interest(car). Kar et al. (2019) learn the parameters of a renderer to match the target distribution to address the content gap. Unsupervised domain adaptation methods (Chen et al., 2018; Xu et al., 2020; Li et al., 2020) align the features from source and target domain. We discuss self-learning based on pseudo labels (Zou et al., 2018) in Section 2.2.1. We train our baselines on 6000 images, using the hyper parameters provided by the authors. We adapt the baselines to our framework with Resnet101 as backbone. The details can be found in Section A.2.3.
  Results: We evaluate scene graph generation on three KITTI evaluation modes : easy, moderate and hard based on object size, occlusion and truncation (Appendix). In all three settings, we improve over our baselines. We report the results for KITTI hard in Table 3, the detailed results for KITTI easy and moderate can be found in Table 6. We perform better than Meta-Sim as it cannot align the structure of the scenes (e.g. number of object in a scene) to match the target distribution. Our predicted scene graphs also differ from the intermediate scene graphs in Meta-Sim which lacks the notion of relationships. We outperform other domain adaptation methods (adapted to our framework) for object detection (Chen et al., 2018; Xu et al., 2020; Li et al., 2020) because we believe that feature alignment without good label/content alignment may not be effective. We discussed this
  briefly in Section 2.2.2 and further show an ablation (next paragraph) that Sim2SG also suffers from the same issue. We believe that our label alignment c,label can be used in conjunction with other domain adaptation methods. The reason we improve over self-learning based on pseudo labels is also discussed in Section 2.2.1. The last three rows of Table 3 show that most improvements come from label alignment c,label and appearance alignment a and the combination of c,label, a and c,pred achieves the best relationship triplet recall. We notice that the AP of the pedestrian category does not improve with a and c,pred. The reason might be that pedestrians are under-represented, small and hard to detect class in KITTI. Sim2SG can align the label distribution but cannot address the class imbalance in the target domain. The qualitative results are shown in last three rows of Figure 2. We see that Sim2SG significantly improves on both false positives and recall of objects. As a result, it generates more accurate scene graphs. This is because label alignment c,label generates nicely aligned data (Fig. 11) and appearance alignment a reduces false positives (see Fig. 2). Figure 3 qualitatively shows how synthetic data is adjusted over the duration of training towards more label alignment w.r.t KITTI using c,label.
  Ablations: As briefly discussed in Section 2.2.2, we run the label alignment c,label before appearance alignment a and prediction alignment c,pred to address the fact that feature alignment can be detrimental if the content of both domains are not aligned. We indeed found that our performance drops significantly when we train Sim2SG without c,pred and evaluate in the same setting as Table 3. Sim2SG with a+ c,pred gives a 0.246 mAP @0.5 IoU for detection & 0.076 recall@50 for relationship triplets while simply adding c,pred to it, we get 0.316 mAP @0.5 IoU for detection & 0.139 recall@50 for relationship triplets (KITTI Hard). This shows the effectiveness of c,label and importance of the entire Sim2SG framework.
sections.heading: 5 CONCLUSION
sections.text: In this work, we propose Sim2SG, a model that achieves sim-to-real transfer learning for scene graph generation on unlabeled real-world datasets. We decompose the domain gap into label, prediction and appearance discrepancies between synthetic and real domains. We propose methods to address these discrepancies and achieve significant improvements over baselines in all three environments - Clevr, Dining-Sim and Drive-Sim. We do require access to simulator and 3D assets. However, this limitation is mitigated with the availability of open source simulators (Dosovitskiy et al., 2017; Deitke et al., 2020; Kolve et al., 2017; To et al., 2018; Crespi et al., 2020; Denninger et al., 2019; Xiang et al., 2020) and exciting work around object mesh generation (Mescheder et al., 2019; Xu et al., 2019; Gkioxari et al., 2020; Wang et al., 2018). For future work, we plan to learn more complex relationships and explore generative modeling for learning our representation.
sections.heading: A APPENDIX
sections.text: A.1 ARCHITECTURE DETAILS
  Encoder φ and SG Predictor h We use Resnet 101 (He et al., 2016) with imagenet pretraining as the backbone or encoder neural network. We use the Faster-RCNN (Ren et al., 2015) and Graph Convolution Network based architecture from GraphRCNN (Yang et al., 2018) to implement the SG Predictor h.
  GRL For appearance alignment a, we use a 2 layer 2D convolution neural network based discriminator with Relu activation. For prediction alignment c,pred we use 2 fully connected neural network based discriminator. We also scale the gradients to the encoder network φ from the discriminator by a factor of 4 in above cases.
  A.2 EXPERIMENTS
  A.2.1 CLEVR
  Setup The source and target domains of the CLEVR (Johnson et al., 2017) environment leverage Blender (Community, 2018) to render 320x240 images and corresponding ground truth scene graphs. Details of the two domains are available in Section 4.1. We use colors (blue, green, magenta, yellow) and material (metal) for source domain and different colors (pink, brown, white) and material (rubber) for target domain. Additionally, we transform the target by using a style transfer network 1. For both domains, we sample each class and their size(small, medium & large) with equal probability. The environment has three lights and a fixed camera. We add a small random jitter to their initial positions during the rendering process. Some samples of source and target domain are shown in Figure 4.
  Details of Generation using Pseudo-Statistics The Label Discrepancy in Section 2.2.1 describes how we generate scenes from pseudo statistics. We assume access to camera parameters.
  Training Details We run our experiments in two stages. In the first stage, we train with appearance alignment a for 70k iterations. In the second stage, we continue training the model using pseudo statistic based self-learning (label alignment c,label) for 3 epochs each with 10k iterations.
  We optimize the model using a SDG optimizer with learning rate of 0.0001 and momentum of 0.9. We train our model using a batch size 4 on NVIDIA DGX workstations. We report saturation peak performance in all our tables. We give equal regularization weights to source task loss s, appearance alignment a and label alignment c,label.
  Results More qualitative results of Sim2SG evaluated on the target domain for CLEVR are shown in Figure 5. We see better recall and fewer false positive object detections leading to more accurate scene graphs. Label alignment c,label improves object recall, but occasionally introduces some false positive detections. Our appearance alignment a helps in reducing such false positives as shown in Figure 6.
  A.2.2 DINING-SIM
  Setup The Dining-Sim environment is written using Pixar’s USD API and rendered with a proprietary renderer. The source domain is rendered with 2 spp (samples per pixels) followed by denoiser. We select 1 chair (cantilever chair), 1 table (workshop table) and 1 laptop (PC). We randomly place chair and table on the floor and laptop on the floor as well as on the table with a random orientation. The asset for each subcategory is randomly chosen from a list of subcategory specific ShapeNet (Chang et al., 2015) assets. We also ensure that objects do not overlap by applying collision avoidance with simple box collision volumes. A subset of 4 to 5 simple materials that vary only in diffuse colour is created for each of the walls, floor, chair and table. Laptops use the original asset texture.
  1 https://github.com/pytorch/examples/tree/master/fast_neural_style
  The target domain is rendered using path tracing with 20 spp (samples per pixels) followed by denoiser. We use 4 chairs (Windsor chair), 1 table (kitchen table) and 2 laptops (MacBook). We first place the table with a random orientation and position on the floor. We then place the four chairs at each side of the table, oriented towards the table centre. Two laptops are then placed randomly on the table surface with a random rotation. The asset for each subcategory is randomly chosen from a list of subcategory specific ShapeNet (Chang et al., 2015) assets. For materials, we use a subset of 4 to 6 physically based, highly detailed materials for each of the walls, floor, chair and table. Laptops use the original asset texture.
  Both domains share room parameters: a fixed camera (60 degree field of view, positioned at far side of the room) and 3 fixed spherical lights. Samples from the source and the target domains are shown in Figure 7. There are five kinds of relationships - front, behind, left, right, and on with table as subject. We use 5000 labeled images from source, 5000 unlabeled images from target for training and 1000 labeled images from both source & target domains for evaluation. We use 1024 x 768 image resolution for training and evaluation.
  Details of Generation using Pseudo-Statistics Section 2.2.1 describes how we generate synthetic data using pseudo-statistics. We assume access to camera parameters. Pseudo statistic derived from a SG contains a list of objects, their type, 2D centroid position and relationship with others. We filter the objects and relationships among them using an adaptive threshold (details in the next paragraph) for the generation. Using camera parameters, we place each object in the 3D scene by picking a random 3D asset according its type(class) and assigning random pose in the range 0◦–360◦. We assume context like ground, wall as described in the previous paragraph. We refine the 3D scene further according to the predicted relationships among objects. For example, we use "on" relationship to refine object placements by adjusting the object (laptop or chair) elevation to match the table top.
  Training Details We optimize the model using a SGD optimizer with learning rate of 0.0001 and momentum of 0.9. We train our model using a batch size 2 on NVIDIA DGX workstations. We report saturation peak performance in all our tables. We give equal weights to source task loss s, appearance alignment a, prediction alignment c,pred and label alignment c,label.
  We conduct training in three stages. In the first stage, we train for 40,000 iterations on the source domain. Second stage is training the model using pseudo statistic based self-learning (label alignment c,label) for 6 epochs each with 10k iterations and score threshold of 0.5. We use the aligned synthetic data from this stage (generated with a score threshold of 0.9) to train the next stage. In the third stage, we add appearance alignment a and prediction alignment c,pred and train for an additional 20,000 iterations. It takes 24 hours for full training including rendering time.
  Results The purpose of the Dining-Sim environment is to show that the Sim2SG works as intended in a more complex setting that is similar to a real-world application. We present the full quantitative results in the table 4 and qualitative results in the Figure 8. We observe that the combination of all alignment terms c,label, a and c,pred gives the best relationship triplet recall of 0.547@50. In order to keep our approach as general as possible, we do not enforce strict rules on object placements and prefer to randomize parameters that are not predicted such as orientation as evident in the qualitative results of label alignment c,label in Figure 9. When target domain assets are too dissimilar from the assets in the source domain, it often results in incorrect reconstructions as shown in Figure 9 (last column). We also observe that after label alignment c,label, the model occasionally has false positive detections, particularly in areas of the floor that have intricate patterns. We qualitatively show that these false positives disappear with the addition of appearance alignment a term (Figure 10).
  Ablations We conduct two sets of experiments on the Dining-Sim environment. The first experiment studies appearance gap: source domain has different appearance but similar content from the target domain. The source domain is generated using the target generation scheme but using source dataset materials as shown in Figure 7. We observe that the appearance alignment a helps reduce the appearance gap, increasing relationship triplet recall from 0.625@50 to 0.821@50. Similarly, the second experiment studies content gap where the source and target use the same materials but have different assets, object positions and number of objects. We accomplish this by modifying the source generation scheme to select materials from the target dataset. Samples of source and target are shown in rows 3, 4 of Figure 7. We observe that the label alignment c,label term aids in reducing the
  content gap, increasing relationship triplet recall from 0.468@50 to 0.539@50. The relatively modest improvement makes sense as the two domains still differ in content (source and target domain assets differ).
  A.2.3 DRIVE-SIM
  Setup As mentioned in Section 4.3, we use an Unreal Engine 42 based driving simulator akin to (Prakash et al., 2019) to generate synthetic data. We have cars(1-2 per lane), trees(1-3), houses/buildings(1-3), pedestrians(0-2), sidewalk(2), roads(2-6). We do not have poles, street signs or any other objects. We have straight roads. We use realistic random placements, e.g. cars can only be placed on a lane, pedestrians on sidewalk, houses on ground and trees on both sidewalk and ground. We randomize the time of the day, cloud density and use directional light. We assume real world scale. We place our camera at a car height on a random right lane with fixed camera parameters (0 yaw, 0 pitch, 90 fov). We add realistic texture and color to each object similar to (Prakash et al., 2019). We use 1242 x 375 image resolution for training and evaluation.
  Details on Synthetic Data generation using Pseudo-Statistics Section 2.2.1 describes how we generate synthetic data using pseudo-statistics. We do not have access to KITTI camera parameters and we use the camera parameters described in the previous paragraph. Pseudo statistic derived from a SG contains a list of objects, their type, 2D centroid position and relationship with others. We filter the objects and relationships among them using a confidence threshold of 0.2. Using camera parameters, we place each object in the 3D scene by picking a random 3D asset according its type(class) and assigning random pose in the range 0◦–360◦ (except cars that are aligned to the lane). We assume context like road, ground, sky, sidewalk as described in the previous paragraph. We refine the 3D scene further according to the predicted relationships among objects. We also assume a consistent lane width, and number of roads are determined by positions of the detected vehicles in the scene. We place multiple Trees if the projected 3D volume permits.
  2 https://www.unrealengine.com/
  Training Details We optimize the model using a SGD optimizer with learning rate of 0.0001 and momentum of 0.9. We train our model using a batch size 2 on NVIDIA DGX workstations. We report saturation peak performance in all our tables. We give equal regularization weights to source task loss s, appearance alignment a, prediction alignment c,pred and label alignment c,label.
  We run our experiments in two stages. First stage is training the model using pseudo statistic based self-learning (label alignment c,label) for 3 epochs each with 10k iterations. We use the aligned synthetic data from this stage to train the next stage. During second stage we train the model with appearance alignment a and prediction alignment c,pred for an additional 60,000 iterations. This makes sense as a works better when content/labels are aligned between the two domains. The total training takes 24 hours including the rendering time.
  Baselines: We adapt domain adaptation baselines (Chen et al., 2018; Xu et al., 2020) to our framework by using the same backbone (Resnet 101) and SG Predictor (GraphRCNN (Yang et al., 2018)) network as Sim2SG, but their loss function. We do not adapt SAPNet (Li et al., 2020). We train these baselines on 6000 images from the source domain (Prakash et al., 2019) using the same optimizer and learning rate as Sim2SG for 60k iterations. We found GPA (Xu et al., 2020) and SAPNet (Li et al., 2020) detection performance to be lower than that reported in their work especially for pedestrian, vegetation and house classes. It is worth noting that their reported class-wise performance numbers only overlap with some of the classes in our work.
  We train (Kar et al., 2019) for 40 epochs with a batch size of 16 and learning rate 0.001 as per the authors. We then obtain 6000 images and train it on Sim2SG framework (Resnet 101 backbone and GraphRCNN SG predictor) for 60k iterations using the same optimizer and learning rate as Sim2SG. For self-learning based on pseduo labels (Zou et al., 2018), we obtain the pseudo labels on KITTI images using the most confident predictions by synthetic pretrained GraphRCNN network (as per the authors). We then train these labeled KITTI images on Sim2SG framework for 60k iterations using the same optimizer and learning rate as Sim2SG.
  KITTI Annotation We use the existing bounding box annotations of Vehicle and Pedestrians. We annotate Trees and Houses/Buildings of all sizes, occlusion and truncation in KITTI. We use the available camera parameters to project the 2D bounding box into 3D space to help us annotate spatial relationships–front, behind, left and right.
  Results Full quantitative evaluations results are in Table 6 on all KITTI (Geiger et al., 2012) evaluation criteria– easy, moderate and hard. In all three criteria, Sim2SG is able to achieve significantly better results (higher detection mAP @0.5 IoU and relationship triplet recall @ 50) than source only baseline (Prakash et al., 2019). More qualitative results of label alignment c,label is in Figure 11. We show qualitative improvements (better object recall and fewer false positive detections) over source only (Prakash et al., 2019) baseline in Figure 12 and the corresponding accurate and full scene graphs in Figure 14.
title: SIM2SG: SIM-TO-REAL SCENE GRAPH GENERATION