summaries: This paper introduces VRDP, a method that learns to answer predictive, counterfactual, explanatory and descriptive yes or no questions about videos depicting the evolution of physical scenes. VRDP is composed of three modules, an object detector, a neuro-symbolic concept learner and differentiable physics model.  The paper shows that VRDP surpasses or closely matches the SOTA on two datasets, a synthetic dataset CLEVERER and a dataset of real world videos. This paper proposes to do visual question answering by learning a differentiable physics model from video. The pipeline is divided to several modular and interpretable parts. Firstly, a visual perception module parse the input video to several objects’ features. Secondly, a concept learner parse the input questions to a set of programs and grounded concepts. The output of the above two modules are used to estimate the objects’ physical parameters and fed to a differentiable physical engine, which is used to do future trajectory predictions. Finally, a symbolic execution part is used to output answer given the future predictions. The paper tackles the problem of dynamic visual reasoning by using a modular unit that is composed of three components: visual perception, concept learner, and physics simulator. This system is capable of performing system identification from visual and language modalities with assistance from a physics model. Having access to the physics model, it can produce imagined future trajectories for reasoning about possible future outcomes. This gave them an advantage of outperforming competitive methods, especially in counterfactual tasks.  - The paper presents a composite model for visual question answering in scenarios which require dynamic visual reasoning. - The model pipeline consists of a visual perception module which extracts object-centric trajectories from an input video, a concept learner which parses the natural language QA pair into a set of neurosymbolic concept vectors, a differentiable physics model which can simulate the evolution of rigid body trajectories and a neurosymbolic program executor which computes or selects the answer for a given input question using the intermediate concept and trajectory features. - As a contribution over prior art in neurosymbolic visual reasoning, the authors propose the utilisation of a differentiable physics module for accurate trajectory prediction based on physics parameters which are inferred from the video input sample. Since explicit physical properties like mass, friction and restitution are inferred in the model, the computed output is very amenable to human interpretation. - The proposed approach is comprehensively evaluated on the synthetic CLEVRER and realistic Real-billiard benchmarks and compared against related neurosymbolic and end-to-end VQA models. It demonstrates remarkable boosts in data efficiency on CLEVRER, sets a new state-of-the-art performance on counterfactual questions and significantly outperforms prior art in a few-shot setup where generalisation to modified physical attributes like 'heavier' or 'lighter' is required.
abstractText: In this work, we propose a unified framework, called Visual Reasoning with Differentiable Physics (VRDP) 1, that can jointly learn visual concepts and infer physics models of objects and their interactions from videos and language. This is achieved by seamlessly integrating three components: a visual perception module, a concept learner, and a differentiable physics engine. The visual perception module parses each video frame into object-centric trajectories and represents them as latent scene representations. The concept learner grounds visual concepts (e.g., color, shape, and material) from these object-centric representations based on the language, thus providing prior knowledge for the physics engine. The differentiable physics model, implemented as an impulse-based differentiable rigid-body simulator, performs differentiable physical simulation based on the grounded concepts to infer physical properties, such as mass, restitution, and velocity, by fitting the simulated trajectories into the video observations. Consequently, these learned concepts and physical models can explain what we have seen and imagine what is about to happen in future and counterfactual scenarios. Integrating differentiable physics into the dynamic reasoning framework offers several appealing benefits. More accurate dynamics prediction in learned physics models enables state-of-the-art performance on both synthetic and real-world benchmarks while still maintaining high transparency and interpretability; most notably, VRDP improves the accuracy of predictive and counterfactual questions by 4.5% and 11.5% compared to its best counterpart. VRDP is also highly data-efficient: physical parameters can be optimized from very few videos, and even a single video can be sufficient. Finally, with all physical parameters inferred, VRDP can quickly learn new concepts from few examples.
authors.name: Mingyu Ding
authors.name: Zhenfang Chen
references.authors: P. Agrawal A.V. Nair P. Abbeel J. Malik S. Levine
references.title: Learning to poke by poking: Experiential learning of intuitive physics
references.venue: In NeurIPS,
references.authors: A. Ajay M. Bauza J. Wu N. Fazeli J.B. Tenenbaum A. Rodriguez L.P. Kaelbling
references.title: Combining physical simulators and object-based networks for control
references.venue: In ICRA,
references.authors: S. Amizadeh H. Palangi O. Polozov Y. Huang K. Koishida
references.title: Neuro-symbolic visual reasoning: Disentangling" visual" from" reasoning
references.venue: In ICML,
references.authors: J. Andreas M. Rohrbach T. Darrell D. Klein
references.title: Neural module networks
references.venue: In CVPR,
references.authors: S. Antol A. Agrawal J. Lu M. Mitchell D. Batra C. Lawrence Zitnick D. Parikh
references.title: Vqa: Visual question answering
references.venue: In ICCV,
references.authors: D. Bahdanau K. Cho Y. Bengio
references.title: Neural machine translation by jointly learning to align and translate
references.venue: In ICLR,
references.authors: A. Bakhtin L. van der Maaten J. Johnson L. Gustafson R. Girshick
references.title: Phyre: A new benchmark for physical reasoning
references.venue: In NeurIPS,
references.authors: F. Baradel N. Neverova J. Mille G. Mori C. Wolf
references.title: Cophy: Counterfactual learning of physical dynamics
references.venue: In ICLR,
references.authors: P.W. Battaglia J.B. Hamrick J.B. Tenenbaum
references.title: Simulation as an engine of physical scene understanding
references.venue: Proceedings of the National Academy of Sciences,
references.authors: P.W. Battaglia R. Pascanu M. Lai D. Rezende K. Kavukcuoglu
references.title: Interaction networks for learning about objects, relations and physics
references.venue: arXiv preprint arXiv:1612.00222,
references.authors: D.M. Bear C. Fan D. Mrowca Y. Li S. Alter A. Nayebi J. Schwartz L. Fei-Fei J. Wu J.B. Tenenbaum
references.title: Learning physical graph representations from visual scenes
references.venue: NeurIPS,
references.authors: Y. Bengio J. Louradour R. Collobert J. Weston
references.title: Curriculum learning
references.venue: In ICML, pages
references.authors: J.C. Butcher
references.title: A stability property of implicit runge-kutta methods
references.venue: BIT Numerical Mathematics, pages 358–361,
references.authors: E. Catto
references.title: Modeling and solving constraints
references.venue: In Game Developers Conference,
references.authors: M.B. Chang T. Ullman A. Torralba J.B. Tenenbaum
references.title: A compositional object-based approach to learning physical dynamics
references.venue: In ICLR,
references.authors: Z. Chen J. Mao J. Wu K.-Y.K. Wong J.B. Tenenbaum C. Gan
references.title: Grounding physical concepts of objects and events through dynamic visual reasoning
references.venue: In ICLR,
references.authors: F. de Avila Belbute-Peres K. Smith K. Allen J. Tenenbaum J.Z. Kolter
references.title: End-to-end differentiable physics for learning and control
references.venue: In NeurIPS,
references.authors: J. Degrave M. Hermans J. Dambre
references.title: A differentiable physics engine for deep learning in robotics
references.venue: Frontiers in neurorobotics,
references.authors: D. Ding F. Hill A. Santoro M. Botvinick
references.title: Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures
references.venue: arXiv preprint arXiv:2012.08508,
references.authors: M. Ding Y. Huo H. Yi Z. Wang J. Shi Z. Lu P. Luo
references.title: Learning depth-guided convolutions for monocular 3d object detection
references.venue: In CVPR,
references.authors: M. Ding X. Lian L. Yang P. Wang X. Jin Z. Lu P. Luo
references.title: Hr-nas: Searching efficient high-resolution neural architectures with lightweight transformers
references.venue: In CVPR,
references.authors: C. Fan X. Zhang S. Zhang W. Wang C. Zhang H. Huang
references.title: Heterogeneous memory enhanced multimodal attention model for video question answering
references.venue: In CVPR,
references.authors: C. Finn I. Goodfellow S. Levine
references.title: Unsupervised learning for physical interaction through video prediction
references.venue: In NeurIPS,
references.authors: C. Gan Y. Li H. Li C. Sun B. Gong
references.title: VQS: Linking segmentations to questions and answers for supervised attention in vqa and question-focused semantic segmentation
references.venue: In ICCV,
references.authors: C. Gan J. Schwartz S. Alter M. Schrimpf J. Traer J. De Freitas J. Kubilius A. Bhandwaldar N. Haber M. Sano
references.title: Threedworld: A platform for interactive multi-modal physical simulation
references.venue: NeurIPS,
references.authors: C. Gan S. Zhou J. Schwartz S. Alter A. Bhandwaldar D. Gutfreund D.L. Yamins J.J. DiCarlo J. McDermott A. Torralba
references.title: The threedworld transport challenge: A visually guided task-and-motion planning benchmark for physically realistic embodied ai
references.venue: arXiv preprint arXiv:2103.14025,
references.authors: R. Girdhar D. Ramanan
references.title: Cater: A diagnostic dataset for compositional actions and temporal reasoning
references.venue: In ICLR,
references.authors: G. Gkioxari J. Malik
references.title: Finding action tubes
references.venue: In CVPR,
references.authors: C. Han J. Mao C. Gan J. Tenenbaum J. Wu
references.title: Visual concept-metaconcept learning
references.venue: In NeurIPS,
references.authors: X. Han S. Wang C. Su W. Zhang Q. Huang Q. Tian
references.title: Interpretable visual reasoning via probabilistic formulation under natural supervision
references.venue: In ECCV,
references.authors: K. He G. Gkioxari P. Dollár R. Girshick
references.title: Mask r-cnn
references.venue: In ICCV,
references.authors: K. He X. Zhang S. Ren J. Sun
references.title: Deep residual learning for image recognition
references.venue: In CVPR,
references.authors: E. Heiden D. Millard E. Coumans Y. Sheng G.S. Sukhatme
references.title: Neuralsim: Augmenting differentiable simulators with neural networks
references.venue: arXiv preprint arXiv:2011.04217,
references.authors: E. Heiden D. Millard H. Zhang G.S. Sukhatme
references.title: Interactive differentiable simulation
references.venue: arXiv preprint arXiv:1905.10706,
references.authors: S. Hochreiter J. Schmidhuber
references.title: Long short-term memory
references.venue: Neural Computation,
references.authors: R. Hu J. Andreas T. Darrell K. Saenko
references.title: Explainable neural computation via stack neural module networks
references.venue: In ECCV,
references.authors: Y. Hu L. Anderson T.-M. Li Q. Sun N. Carr J. Ragan-Kelley F. Durand
references.title: Difftaichi: Differentiable programming for physical simulation
references.venue: In ICLR,
references.authors: D. Huang P. Chen R. Zeng Q. Du M. Tan C. Gan
references.title: Location-aware graph convolutional networks for video question answering
references.venue: In AAAI,
references.authors: Z. Huang Y. Hu T. Du S. Zhou H. Su J.B. Tenenbaum C. Gan
references.title: Plasticinelab: A soft-body manipulation benchmark with differentiable physics
references.venue: In ICLR,
references.authors: D.A. Hudson C.D. Manning
references.title: Compositional attention networks for machine reasoning
references.venue: In ICLR,
references.authors: D.A. Hudson C.D. Manning
references.title: Gqa: A new dataset for real-world visual reasoning and compositional question answering
references.venue: In CVPR,
references.authors: Y. Jang Y. Song Y. Yu Y. Kim G. Kim
references.title: Tgif-qa: Toward spatio-temporal reasoning in visual question answering
references.venue: In CVPR,
references.authors: M. Janner S. Levine W.T. Freeman J.B. Tenenbaum C. Finn J. Wu
references.title: Reasoning about physical interactions with object-oriented prediction and planning
references.venue: In ICLR,
references.authors: K.M. Jatavallabhula M. Macklin F. Golemo V. Voleti L. Petrini M. Weiss B. Considine J. Parent- Levesque K. Xie K. Erleben
references.title: gradsim: Differentiable simulation for system identification and visuomotor control
references.venue: In ICLR,
references.authors: J. Johnson B. Hariharan L. Van Der Maaten J. Hoffman L. Fei-Fei C. Lawrence Zitnick R. Girshick
references.title: Inferring and executing programs for visual reasoning
references.venue: In ICCV,
references.authors: T. Kipf E. van der Pol M. Welling
references.title: Contrastive learning of structured world models
references.venue: In ICLR,
references.authors: T.N. Kipf M. Welling
references.title: Semi-supervised classification with graph convolutional networks
references.venue: In ICLR,
references.authors: T.M. Le V. Le S. Venkatesh T. Tran
references.title: Hierarchical conditional relation networks for video question answering
references.venue: In CVPR,
references.authors: J. Lei L. Yu M. Bansal T.L. Berg
references.title: Tvqa: Localized, compositional video question answering
references.venue: In EMNLP, 2018
references.authors: A. Lerer S. Gross R. Fergus
references.title: Learning physical intuition of block towers by example
references.venue: In ICML,
references.authors: X. Li J. Song L. Gao X. Liu W. Huang X. He C. Gan
references.title: Beyond rnns: Positional self-attention with co-attention for video question answering
references.venue: In AAAI,
references.authors: Y. Li T. Lin K. Yi D. Bear D. Yamins J. Wu J. Tenenbaum A. Torralba
references.title: Visual grounding of learned physical models
references.venue: In ICML,
references.authors: Y. Li J. Wu R. Tedrake J.B. Tenenbaum A. Torralba
references.title: Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids
references.venue: In ICLR,
references.authors: Y. Li J. Wu J.-Y. Zhu J.B. Tenenbaum A. Torralba R. Tedrake
references.title: Propagation networks for modelbased control under partial observation
references.venue: In ICRA,
references.authors: J. Liang M.C. Lin
references.title: Differentiable physics simulation
references.venue: In ICLR Workshop,
references.authors: D.C. Liu J. Nocedal
references.title: On the limited memory bfgs method for large scale optimization
references.venue: Mathematical programming,
references.authors: F. Locatello D. Weissenborn T. Unterthiner A. Mahendran G. Heigold J. Uszkoreit A. Dosovitskiy T. Kipf
references.title: Object-centric learning with slot attention
references.venue: arXiv preprint arXiv:2006.15055,
references.authors: J. Mao C. Gan P. Kohli J.B. Tenenbaum J. Wu
references.title: The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision
references.venue: In ICLR,
references.authors: D. Mascharka P. Tran R. Soklaski A. Majumdar
references.title: Transparency by design: Closing the gap between performance and interpretability in visual reasoning
references.venue: In CVPR,
references.authors: I. Misra R. Girshick R. Fergus M. Hebert A. Gupta L. Van Der Maaten
references.title: Learning by asking questions
references.venue: In CVPR,
references.authors: R. Mottaghi M. Rastegari A. Gupta A. Farhadi
references.title: what happens if...” learning to predict the effect of forces in images
references.venue: In ECCV. Springer,
references.authors: M. Müller J. Stam D. James N. Thürey
references.title: Real time physics: class notes
references.venue: In ACM SIGGRAPH 2008 classes,
references.authors: H. Qi X. Wang D. Pathak Y. Ma J. Malik
references.title: Learning long-term visual dynamics with region proposal interaction networks
references.venue: In ICLR,
references.authors: R. Riochet M.Y. Castro M. Bernard A. Lerer R. Fergus V. Izard E. Dupoux
references.title: Intphys: A framework and benchmark for visual intuitive physics reasoning
references.venue: arXiv preprint arXiv:1803.07616,
references.authors: T. Shao A. Monszpart Y. Zheng B. Koo W. Xu K. Zhou N.J. Mitra
references.title: Imagining the unseen: Stability-based cuboid arrangements for scene understanding
references.venue: ACM TOG,
references.authors: K. Smith L. Mei S. Yao J. Wu E. Spelke J. Tenenbaum T. Ullman
references.title: Modeling expectation violation in intuitive physics with coarse probabilistic object representations
references.venue: NeurIPS, 32:8985–8995,
references.authors: M. Tapaswi Y. Zhu R. Stiefelhagen A. Torralba R. Urtasun S. Fidler
references.title: MovieQA: Understanding Stories in Movies through Question-Answering
references.venue: In CVPR,
references.authors: M.A. Toussaint K.R. Allen K.A. Smith J.B. Tenenbaum
references.title: Differentiable physics and stable modes for tool-use and manipulation planning
references.venue: In IJCAI,
references.authors: A. Vaswani N. Shazeer N. Parmar J. Uszkoreit L. Jones A.N. Gomez Ł. Kaiser I. Polosukhin
references.title: Attention is all you need
references.venue: In NeurIPS,
references.authors: R. Veerapaneni J.D. Co-Reyes M. Chang M. Janner C. Finn J. Wu J. Tenenbaum S. Levine
references.title: Entity abstraction in visual model-based reinforcement learning
references.venue: In CoRL,
references.authors: N. Watters D. Zoran T. Weber P. Battaglia R. Pascanu A. Tacchetti
references.title: Visual interaction networks: Learning a physics simulator from video
references.venue: NeurIPS, 30:4539–4547,
references.authors: J. Wu E. Lu P. Kohli B. Freeman J. Tenenbaum
references.title: Learning to see physics via visual de-animation
references.venue: In NeurIPS,
references.authors: J. Wu I. Yildirim J.J. Lim W.T. Freeman J.B. Tenenbaum
references.title: Galileo: Perceiving physical object properties by integrating a physics engine with deep learning
references.venue: NeurIPS,
references.authors: Q. Wu C. Shen L. Liu A. Dick A. van den Hengel
references.title: What value do explicit high level concepts have in vision to language problems
references.venue: In CVPR,
references.authors: D. Xu Z. Zhao J. Xiao F. Wu H. Zhang X. He Y. Zhuang
references.title: Video question answering via gradually refined attention over appearance and motion
references.venue: In ACM MM,
references.authors: T. Ye X. Wang J. Davidson A. Gupta
references.title: Interpretable intuitive physics model
references.venue: In ECCV,
references.authors: Y. Ye M. Singh A. Gupta S. Tulsiani
references.title: Compositional video prediction
references.venue: In ICCV,
references.authors: Y. Ye Z. Zhao Y. Li L. Chen J. Xiao Y. Zhuang
references.title: Video question answering via attribute-augmented attention network learning
references.venue: In ICLR,
references.authors: K. Yi C. Gan Y. Li P. Kohli J. Wu A. Torralba J.B. Tenenbaum
references.title: Clevrer: Collision events for video representation and reasoning
references.venue: In ICLR,
references.authors: K. Yi J. Wu C. Gan A. Torralba P. Kohli J. Tenenbaum
references.title: Neural-symbolic vqa: Disentangling reasoning from vision and language understanding
references.venue: In NeurIPS,
references.authors: A. Zadeh M. Chan P.P. Liang E. Tong L.-P. Morency
references.title: Social-iq: A question answering benchmark for artificial social intelligence
references.venue: In CVPR,
references.authors: H. Zhou A. Kadav F. Lai A. Niculescu-Mizil M.R. Min M. Kapadia H.P. Graf
references.title: Hopper: Multi-hop transformer for spatiotemporal reasoning
references.venue: In ICLR,
references.authors: Y. Zhu O. Groth M. Bernstein L. Fei-Fei
references.title: Visual7w: Grounded question answering in images
references.venue: In CVPR, 2016
sections.heading: 1 Introduction
sections.text: Dynamic visual reasoning about objects, relations, and physics is essential for human intelligence. Given a raw video, humans can easily use their common sense of intuitive physics to explain what has happened, predict what will happen next, and infer what would happen in counterfactual situations. Such human-like physical scene understanding capabilities are also of great importance in practical applications such as industrial robot control [2, 53].
  1Project page: http://vrdp.csail.mit.edu/
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  Previous works have made great efforts to build artificial intelligence (AI) models with such physical reasoning capabilities. One popular strategy is to develop pure neural-network-based models [63, 19, 40]. These methods typically leverage end-to-end neural networks [32, 35] with powerful attention modules such as Transformer [69, 21] to extract attended features from both video frames and question words, based on which they answer questions directly. Despite their high question-answering accuracy on CLEVRER [79], a challenging dynamic visual question-answering benchmark, these black-box models neither learn concepts nor model objects’ dynamics. Therefore, they lack transparency, interpretability, and generalizability to new concepts and scenarios. Another common approach to dynamic visual reasoning is to build graph neural networks (GNNs) [47] to capture the dynamics of the scenes. These GNN models [54, 79, 16] treat objects in the video as nodes and perform objectand relation-centric updates to predict objects’ dynamics in future or counterfactual scenes. Such systems achieve decent performance with good interpretability on CLEVRER by combining the GNN-based dynamics models with neural-symbolic execution [58, 80]. However, these dynamic models do not explicitly consider laws of physics or use concepts encoded in the question-answer pairs associated with the videos. As a result, they show limitations in counterfactual situations that require long-term dynamics prediction.
  Although (graph-)neural-network-based approaches have achieved competitive performance on CLEVRER, dynamic visual reasoning is still far from being solved perfectly. In particular, due to the lack of explicit physics models, existing models [79, 19, 16] typically struggle to reason about future and counterfactual events, especially when training data is limited. For this reason, one appealing alternative is to develop explicit physics-based methods to model and reason about dynamics, as highlighted in the recent development of differentiable physics engines [9, 17, 68, 18, 66] and their applications in robotics [9, 17, 68]. However, these physics engines typically take as input a full description of the scene (e.g., the number of objects and their shapes) which usually requires certain human priors, limiting their availability to applications with well-defined inputs only.
  In this work, we take an approach fundamentally different from either network-based methods or physics-based methods. Noting that deep learning based methods excel at parsing objects and learning concepts from videos and language, and physics laws are good at capturing object dynamics, we propose Visual Reasoning with Differentiable Physics (VRDP), a unified framework that combines a visual perception module, a concept learner, and a differentiable physics engine. VRDP jointly learns object trajectories, language concepts, and objects’ physics models to make accurate dynamic predictions. It starts with a perception module running an object detector [31] on individual frames to generate object proposals and connect them into trajectories based on a motion heuristic. Then, a concept learner learns object- and event-based concepts, such as ‘shape’, ‘moving’, and ‘collision’ as in DCL [16, 58]. Based on the obtained object trajectories and attributes, the differentiable physics engine estimates all dynamic and physical properties (e.g., velocity, angular velocity, restitution, mass, and the coefficient of resistance) by comparing the simulated trajectories with the video observations. With these explicit physical parameters, the physics engine reruns the simulation to reason about future motion and causal events, which a program executor then executes to get the answer. The three components of VRDP cooperate seamlessly: the concept learner grounds physical concepts needed by the physics engine like ‘shape’ onto the objects detected by the perception module; the differentiable physics engine estimates all physical parameters and simulates accurate object trajectories, which in turn help the concept learning process in the concept learner.
  Compared with existing methods, VRDP has several advantages thanks to its carefully modularized design. First, it achieves the state-of-the-art performance on both synthetic videos (CLEVRER [79]) and real-world videos (Real-Billiard [63]) without sacrificing transparency or interpretability, especially in situations that require long-term dynamics prediction. Second, it has high data efficiency thanks to the differentiable physics engine and symbolic representation. Third, it shows strong generalization capabilities and can capture new concepts with only a few examples.
sections.heading: 2 Related Work
sections.text: Visual Reasoning Our model is related to reasoning on vision and natural language. Existing works can be generally categorized into two streams as end-to-end approaches [40, 83, 74, 41, 5] and neurosymbolic approaches [80, 29, 30, 24, 4, 59, 45, 58, 36, 3]. The end-to-end methods [40, 83, 74, 60] typically tackle the visual question answering (VQA) problem by designing monolithic multi-modal deep networks [32, 35]. They directly output answers without explicit and interpretable mechanisms.
  Beyond question answering, neuro-symbolic methods [80, 29, 59, 45, 58] propose a set of visualreasoning primitives, which manifest as an attention mechanism capable of performing complex reasoning tasks in an explicitly interpretable manner.
  Dynamic visual reasoning in videos has attracted much research attention. Many video question answering datasets [27, 49, 42, 67, 81] and the methods [82, 51, 38, 75, 78, 22] built on them mainly focus on understanding diverse visual scenes, such as human actions (MovieQA [67]) or 3D object movements without physical and language cues (CATER [27]). Differently, CLEVRER [79] targets the physical and causal relations grounded in dynamic videos of rigid-body collisions and asks a range of questions that requires the modeling of long-term dynamic predictions. For this reason, we evaluate our method and compare it with other state-of-the-arts on CLEVRER.
  Both end-to-end [63, 19] and neuro-symbolic methods [79, 16] have been explored on CLEVRER. However, they either lack transparency or struggle for long-term dynamic prediction. In this paper, we perform high-performance and interpretable reasoning by recovering the physics model of objects and their interactions (e.g., collisions) from visual perception and language concepts.
  Physical Models Physical models are widely used in video prediction [50, 23, 76, 77], neural simulation and rendering [52, 54], and dynamic reasoning [10, 71]. For example, PhysNet and its variants [50, 23, 76, 77] leverage global or object-centric deep features to predict the physical motion in video frames. Some other related works [61, 1, 65, 46] extend physical models to predict the effect of forces and infer the geometric attributes and topological relations of cuboids.
  In this work, we focus on dynamic visual reasoning about object interactions, dynamics, and physics with question answering, which is central to human intelligence and a key goal of artificial intelligence. Solving such tasks requires a good representation and understanding of physics models. A common choice is to train a deep neural network for physical property estimation (e.g., location and velocity) based on learned visual and dynamic priors [15, 10, 71, 70, 43, 54, 55, 34, 33]. However, since these neural networks do not model physics laws, generalizing them to unseen events or counterfactual scenarios could result in unexpected results. Our work is different and more physics-based: Inspired by the recent advances in differentiable physics [9, 17, 68, 18, 66, 37], we implement an impulsebased differentiable rigid-body simulator and leverage the power of its gradients to infer dynamics information about the scene.
  Physical Scene Understanding Our work is also relevant to studies on physical scene understanding [7, 73, 66, 64, 8, 25, 26, 20, 50, 39], most of which propose pure neural-network solutions without explicitly incorporating physics models. Benchmarks like PHYRE [7] study physical understanding and reasoning based on pure videos without concept learning and language inference. Based on such benchmarks, some works [72, 11, 50, 54] learn compositional scene structure and estimate states through physical motions and visual de-animation. Recently, two papers propose pure physics-based methods [44, 39] that make heavy use of differentiable physics simulators, but they typically assume concepts in the scene are given as input. Our work is unique in that we learn video and language concepts from raw videos and infer dynamics information from a differentiable simulator, combining the benefits of both learning and physics.
sections.heading: 3 Method
sections.text: By integrating differentiable physics into the dynamic reasoning framework, VRDP jointly learns visual perception, language concepts, and physical properties of objects. The first two provide prior knowledge for optimizing the third one to reason about the physical world, and the optimized physical properties in turn help to learn better concepts. In the following, we first give an overview of our framework and then describe each of its components in detail.
sections.heading: 3.1 Framework Overview
sections.text: An overview of VRDP is illustrated in Fig. 1. It contains three components: a visual perception module, a concept learner, and a physics model. The input to the framework is a video and reasoning questions, where the former is processed by the visual perception module to get object trajectories and corresponding visual features, and the latter is parsed into executable symbolic programs with language concepts by the concept learner. Similar to DCL [58, 16], the concept learner first grounds object properties (e.g., color and shape) and event concepts (e.g., collision and moving) by aligning the
  visual features and the corresponding concept embeddings in the (explanatory or descriptive) program that does not require dynamic predictions, e.g., “what is the shape of ...”. With those perceptually grounded object trajectories and properties, the physical model then performs differentiable simulation to learn all physical parameters of the scene and objects by comparing the simulated trajectories with the video observations. After that, the physics engine simulates unseen trajectories for predictive and counterfactual scenarios and generates their features, in turn enabling the concept learner to finetune event concepts from the program that requires dynamic predictions, e.g., “what will happen ...” and “what if ...”. Finally, a symbolic executor executes the parsed programs with the dynamic predictions to get the answer.
sections.heading: 3.2 Model Details
sections.text: Visual Perception Module Given a video with the number of frames T , the visual perception module parses the video frame-by-frame and associate the parsed objects in each frame into object trajectories L = {ln}Nn=1, where ln denotes the object trajectory of the nth object and N is the number of the objects in the video. Specifically, we leverage a pretrained Faster R-CNN [31] as the object detector to get the Region of Interest (ROI) feature ft ∈ RN×D and the object location of objects bt = [x2Dt , y 2D t , x BEV t , y BEV t ] ∈ RN×4 at frame t, where D is the feature dimension, (x2Dt , y 2D t ) denotes the normalized object bounding box center in the image coordinate frame, and (xBEVt , y BEV t ) denotes the projected bird’s-eye view (BEV) location in the BEV coordinate frame using the calibrated camera matrix. Following works [16, 28], we associate object proposals in adjacent frames by thresholding their intersection over union (IoU) and obtain the object trajectory ln = {bnt }Tt=1 for the nth object. The visual perception module then constructs object and interactive representations for concept learning. The object representation Fobj ∈ RN×(D+4T ) contains both appearance-based Fa = avg({ft}Tt=1) and trajectory-based feature Fl = {bt}Tt=1 for modeling static properties and dynamic concepts, respectively, where avg(·) here represents the average ROI feature over time. The interactive feature Fpair ∈ RT×N×N×12S , where S denotes a fixed temporal window size, is built on every
  pair of objects. It contains object trajectories {bit} t0+S/2 t0−S/2, {b j t} t0+S/2 t0−S/2 of the objects i and j and their distance {abs(bit − b j t )} t0+S/2 t0−S/2 to model the collision event of the objects at a specific moment t0.
  Concept Learner The concept learner grounds the physical and event concepts (e.g., shape and color) as prior knowledge for the physics model from the video representation and language. It first leverages a question parser to translate the input questions and choices into executable neuro-symbolic programs where each language concept in the program is represented by a concept embedding. Similar to [58, 16], this work adopts a seq2seq model [6] with an attention mechanism to translate language into a set of symbolic programs, e.g., retrieving objects with certain colors, getting future or counterfactual events, finding the causes of an event, thus decomposing complex questions into step-by-step dynamic visual reasoning processes. The concept learner assigns each concept in the program (e.g., color, shape, and collision) a randomly initialized embedding e ∈ RC so that the symbolic program can be formulated as differentiable vector operations.
  After that, it projects the visual representation into concept embedding spaces and performs Nearest Neighbor (NN) search to quantize concepts for the object attributes and events. We implement the projection through a linear layer P(·) and calculate the cosine similarity between two vectors in the embedding space for NN search. For example, the confidence score of whether the nth object is a cube can be represented by [cos(P(Fna ), ecube)− µ]/σ, where ecube is the embedding of concept ‘cube’, µ and σ are the shifting and scaling scalars, and P(·) maps a D-dimensional visual feature into a C-dimensional vector in this case.
  Physics Model The differentiable physics model captures objects’ intrinsic physical properties and makes accurate dynamic predictions for reasoning. With the perceptually grounded object shapes and trajectories from the above two components of VRDP, it performs differentiable simulation to optimize the physical parameters of the scene and objects by comparing the simulation with the video observations L. Our physics model is implemented as an impulse-based differentiable rigid-body simulator [37, 62, 14]. It iteratively simulates a small time step of ∆t based on the objects’ state in the BEV coordinate through inferring collision events, forces (including resistance and collision force), and impulses acting on the object, and updating the state of each object. When an object moves on the ground with velocity −→v and angular velocity ω, we consider three kinds of forces that affect the movement of the object: sliding friction, rolling resistance, and air resistance. We use λ1, λ2, λ3 to denote their coefficients and have:
  −→a =
  { − −→v |−→v | (λ1g + λ3|
  −→v |2) if the shape is not sphere − −→v |−→v | (λ2g + λ3| −→v |2) if the shape is sphere (1)
  where g = 9.81m/s2 is the standard gravity and −→a denotes the acceleration of the object, whose direction is opposite to the velocity. The velocity −→v and the location −→ l′ = (x′, y′) are then updated accordingly by the second order Runge-Kutta (RK2) algorithm [13]. Similarly, the angular velocity ω also decreases at each time step due to the angular drag, and the angle α of the object is updated by the RK2 algorithm.
  The physics engine checks whether the boundaries of two objects with radius R are overlapped in the BEV coordinate frame to detect collision events. Based on the fact that the total momentum of an isolated system should be constant in the absence of net external forces, we compute the impulse of collided objects and ignore the friction caused by the collision. Let (m1,m2), (r1, r2), (α1, α2), (−→v1 ,−→v2), ( −→ l′1 , −→ l′2 ) denote the mass, restitution, angle, velocity and BEV location of two collided objects at the moment of the collision, respectively; −→ d1, −→ d2 represent their collision unit directions that the force is acting on, where −→ d1 + −→ d2 = −→ 0 . The change of velocity ∆−→v1 ,∆−→v2 at the moment of collision can be obtained by calculating the impulse on the collision direction: −−→ ∆v1 = −(1 + r1r2)(m2/(m1 +m2))( −→ d1 • ( −→v1 −−→v2)) −→ d1
  −−→ ∆v2 = −(1 + r1r2)(m1/(m1 +m2))( −→ d2 • ( −→v2 −−→v1)) −→ d2,
  (2)
  the velocity −→v is then updated by −→v ← −→v + ∆−→v . Similarly, the angular velocity ω can be updated by ω ← ω + ∆ω, where ∆ω is computed based on conservation of angular momentum. Given an initial state of the scene and objects, our physics engine simulates force, impulse, and collision events and iteratively updates the state of each element. All physical parameters including
  R, λ,m, r, α,−→v , −→ l are initialized and then optimized with L-BFGS algorithm [56] by fitting the simulated trajectories L′ = {(x′t, y′t)}Tt=1 into the perceptual trajectories LBEV = {(xBEVt , yBEVt )}Tt=1. To alleviate the difficulty of the optimization, we mark the time frame of each object’s first collision by calculating the BEV distance between every pair and decompose the differentiable physical optimization and simulation into the following steps: 1) Since radius R and resistance coefficients λ are consistent in all videos, we use K videos to jointly learn those physical parameters and fix them for the optimization of other sample-dependent parameters. 2) For each video, we then use the frames before the collision to optimize the collision-independent physical parameters, such as initial velocity −→v0 , initial location −→ l0 , and initial angle α0. 3) With the above parameters learned and fixed, we optimize the remaining collision-dependent parameters, including mass m and resistance r of each object. This process follows the curriculum learning paradigm [12] by optimizing from fewer to more frames, e.g., multi-step optimization on [0, 40], [0, 80], and [0, 128] frames, where the parameters in each step are initialized from the optimization of the previous step. 4) With all parameters of the physical model learned, the engine runs simulations and re-calculates the trajectorybased representations Fl for answering counterfactual, descriptive, and explanatory questions. 5) For the predictive case, we leverage the learned physical model as initialization and re-optimize all sample-dependent parameters with only the last 20 frames to reduce the cumulative error over time.
  Symbolic Execution As in [58, 16], we perform reasoning with a program executor, which is a collection of deterministic functional modules designed to realize all logic operations specified in symbolic programs. Its input consists of the parsed programs, learned concept embeddings, and visual representations, including the appearance-based feature Fa from the visual perception module and the updated trajectory feature Fl from the physics engine. Given a set of parsed programs, the program executor runs them step-by-step and derives the answer based on these representations. For example, the ‘counting’ program outputs the number of objects which meet specific conditions (e.g., red sphere). In this process, the executor leverages the concept learner to filter out eligible objects.
  Our reasoning process is designed fully differentiable w.r.t. the visual representations and the concept embeddings by representing all object states, events, and results of all operators in a probabilistic manner during training, supporting gradient-based optimization. Moreover, it works seamlessly with our explicit physics engine, which simulates dynamic predictions through real physical parameters, forming a symbolic and deterministic physical reasoning process. The whole reasoning process is fully transparent and step-by-step interpretable.
sections.heading: 3.3 Training Objectives
sections.text: Similar to [16, 79], we train the program parser with program labels using cross-entropy loss,
  Lprogram = − J∑
  j=1
  1{yp = j} log(pj), (3)
  where J is the size of the pre-defined program set, pj is the probability for the j-th program and yp is the ground-truth program label.
  We optimize the physical parameters in the physical model by comparing the simulation trajectories with the video observations. All physical parameters includingR, λ,m, r, α,−→v , −→ l are initialized and then optimized with L-BFGS algorithm [56] by fitting the simulated trajectories L′ = {(x′t, y′t)}Tt=1 into the perceptual trajectories LBEV = {(xBEVt , yBEVt )}Tt=1. We have:
  LPhysics = ‖L′ − LBEV‖22, (4)
  We optimize the feature extractor and the concept embeddings in the concept learner by question answering. We treat each option of a multiple-choice question as an independent boolean question during training. Specifically, we use cross-entropy loss to supervise open-ended questions and use mean square error loss to supervise counting questions. Formally, for counting questions, we have
  LQA,count = (ya − z)2, (5) where z is the predicted number and ya is the ground-truth number label. For other open-ended questions and multiple-choice questions, we have
  LQA,others = − A∑
  a=1
  1{ya = a} log(pa), (6)
  where A is the size of the pre-defined answer set, pa is the probability for the a-th answer and ya is the ground-truth answer label.
sections.heading: 4 Experiments
sections.text: By recovering physics models of objects and their interactions from video and language, VRDP enjoys the following benefits: 1) high accuracy and full transparency, 2) superior data efficiency, and 3) high generalizability. In this section, we first evaluate the accuracy and data efficiency of VRDP on the widely used dynamic visual reasoning benchmark CLEVRER [79] and its subsets; we then validate the model’s generalizability on adapting to new concepts with few-shot data; lastly, we experiment on the real-world dataset Real-Billiard [63] to show that VRDP works well in real-world dynamic prediction and reasoning.
  Datasets and Evaluation Settings To validate the effectiveness of our method for reasoning about the physical world, we conduct main experiments on the CLEVRER [79] dataset, as it contains both language and physics cues such as rigid body collisions and dynamics, compared to other benchmarks that focus on either action understanding without physical inferring [49, 42] or temporal reasoning without language concepts [27, 8]. CLEVRER includes four types of question answering (QA): descriptive, explanatory, predictive, and counterfactual, where the first two types concern more on video understanding, while the latter two types involve physical dynamics and predictions in reasoning. Therefore, we mainly focus on the predictive and counterfactual questions in this work and use QA accuracy as the evaluation metric. Note the multi-choice question (explanatory, predictive, and counterfactual) contains multiple options. Only if all the options (per opt.) are answered correctly can it be regarded as a correct question (per ques.).
  We then collect a few-shot physical reasoning dataset with novel language and physical concepts (e.g., “heavier” and “lighter”), termed generalized CLEVRER, containing 100 videos (split into 25/25/50 for train/validation/test) with 375 options in 158 counterfactual questions. This dataset is supplementary to CLEVRER [79] for generalizing to new concepts with very few samples. For real-world scenarios, we conduct experiments on the Real-Billiard [63] dataset, which contains three-cushion billiards videos captured in real games for dynamics prediction. We generate 6 reasoning questions (e.g., “will one billiard collide with ...?”) for each video and evaluate both the prediction error and QA accuracy.
  Implementation Details We follow the experimental setting in [79, 16] using a pre-trained Faster RCNN model [31] to generate object proposals for each frame and training the language program parser with 1,000 programs for all question types. We implement three versions of VRDP models, where our unsupervised VRDP leverage a Slot-Attention model [57] to parse the objects unsupervisedly, while VRDP † use Faster-RCNN [31] as the object detector. In addition to our standard model that grounds object properties from question-answer pairs, we also train a variant (VRDP †‡) on CLEVRER with an explicit rule-based program executor [79] and object attribute supervision. The camera matrix is optimized from 20 training videos. We set ∆t = 0.004s,K = 10, S = 10, and T = 128 for CLEVRER [79] and T = 20 for Real-Billiard [63]. More details of the dataset and settings can be found in Supplemental Materials.
sections.heading: 4.1 Comparative Results on CLEVRER
sections.text: We conduct experiments on CLEVRER against several counterparts: TVQA+ [49], Memory [22], IEP (V) [45], TbD-net (V) [59], HCRN [48], MAC [40], NS-DR [79], DCL [16], and Object-based Attention [19]. Among them, NS-DR [79] and DCL [16] are high-performance interpretable symbolic models, while Object-based Attention [19] is the state-of-the-art end-to-end method.
  From Tab. 1 we observe that: 1) Counterfactual and predictive questions are more difficult than descriptive and explanatory ones as they require accurate physical dynamics and prediction hence our main focus. By reconstructing the physical world explicitly, our method outperforms all existing works on these two types by large margins. For example, VRDP † improves the per question accuracy of counterfactual questions by 11.5% and 79.7% compared to the best end-to-end [19] and neural-symbolic [16] counterparts.
  2) The end-to-end model [19] improves the accuracy at the cost of losing model transparency and interpretability. However, by leveraging object attribute supervision and explicit program executors [79], our VRDP †‡ achieves new state-of-the-art overall performance on CLEVRER. It
  closes the performance gap between interpretable models and state-of-the-art end-to-end methods. Moreover, it shows the flexibility of our physics model that can be combined with various physical concepts and program executors while achieving impressive performance.
  3) We conducted ablative experiments to study the impact of pre-trained object detection modules of our framework by replacing the supervised visual model [31] in VRDP † with an unsupervised one [57] in VRDP. We observe that although the use of unsupervised detectors decreases the performance slightly, our framework still enjoys higher performance than previous methods in counterfactual and predictive questions.
  4) Neither the neuro-symbolic nor end-to-end works employ explicit dynamic models with physical meanings. In contrast, our model is fully transparent with step-by-step interpretable programs and meaningful physical parameters powered by a differentiable engine.
sections.heading: 4.2 Detailed Analysis
sections.text: Evaluation of Data Efficiency We evaluated the data efficiency of VRDP with two representative models: MAC (V) [40] and Object-based Attention [19]. From Fig. 2 we see that: VRDP is highly data-efficient. When the amount of data is reduced, the accuracy of our model drops slightly, while the performance of MAC (V) [40] and Object-based Attention [19] drops drastically due to insufficient data. For example, we improve the counterfactual accuracy of Object-based Attention [19] by 426%
  Question: If the blue sphere were much heavier, which of the events that happened would not have happened? Choice: The blue cylinder collides with the cube.
  under the setting of 20% data. Notably, our model uses 20% of the dataset to achieve comparable performance to other works that use 80% of data. This is because the components of VRDP, e.g., perception module and question parser, can be trained with a small amount of data. More importantly, our physics model is built based on an explicit physics engine, which can be optimized from the trajectory of a single video.
  Evaluation of Generalizability This part studies the generalization capabilities of VRDP against MAC (V) [40] and Object-based Attention [19] on the generalized CLEVRER dataset. Tab. 2 shows our model outperforms other works by a large margin (75.6 vs. 26.7) on per question accuracy, demonstrating our model can quickly learn new concepts from few examples by reconstructing the physics world. An example of generalization with few-shot data is shown in Fig. 3. Our model learns a novel concept “heavier” from only 25 videos and the corresponding question-answer pairs. The simulation is then run with 5 times the mass to answer the question correctly.
  Ablation Study on the Learning of Physics Models In this work, sample-independent physical parameters (R, λ) are learned from multiple training videos. In contrast, the sample-dependent parameters, such as m, r, α, v, l, can only be learned with a single video, leading to difficulties in optimization, especially when there are many collisions. This part studies the optimization of these sample-dependent parameters by making comparisons among the following four simplified learning processes on CLEVRER [79]: 1) Baseline – optimize all target parameters directly from all frames simultaneously. 2) Collision-independent First – first use the frames before the collision to optimize collision-independent parameters for each object, including initial velocity −→v0 , initial location −→ l0 , and initial angle α0; then optimize mass m and restitution r from all video frames. 3) Curriculum Optimization – optimizem and r by performing multiple steps on [0, 40], [0, 80], and [0, 128] frames, where each step is initialized from the optimization of the previous step. 4) Re-optimization for Prediction (Full model) – leverage the learned physical parameters as initialization and re-optimize all sample-dependent parameters with the last 20 frames to reduce the cumulative error over time.
  Tab. 3 shows that the performance continuously increases when more optimization steps are used, demonstrating the contribution of each part. The “Collision-independent First” rule offers the greatest improvement, especially for counterfactual questions, as counterfactual simulations only rely on the initial state. “Curriculum Optimization” improves all types of questions, and “Re-optimization for Prediction” re-calculates the dynamics of the last 20 frames, thus mainly affect predictive questions.
  Failure Analysis VRDP learns the physics model from object trajectories in videos and language concepts in question-answer pairs. It is data-efficient and robust enough to work well when there exists inaccurate perception or incorrect concept learning in some video frames. However, we noticed
  that the model might fail in the following cases: 1) If the object collides immediately after entering the image plane, there are insufficient frames before the collision to learn the initial velocity v0. 2) If no collision occurs on an object, its restitution r and mass m cannot be optimized (unknown). We set default values for them. 3) The optimization becomes difficult if there are many cubes and collisions between them in the scene, because cube collisions (considering the sides and corners) are more complicated than sphere and cylinders’. These issues are challenging and will be our future work.
sections.heading: 4.3 Comparative Results on Real-World Billiards
sections.text: We also conduct experiments on the real-world dataset Real-billiard [63] with our supplemented question-answer pairs. Note that the billiard table is a chaotic system, and highly accurate long-term prediction is intractable. Fig. 4 shows an example of the ground truth video and our simulated prediction based on the perceptual grounded physics model. It can be seen that the predicted collision events and trajectories are of good quality. Tab. 4 evaluates the prediction errors under two different rollout timesteps and QA accuracy with 5 competitors: VIN [71], OM [43], CVP [77], IN [10], and CIN [63]. For the prediction task, the rollout timesteps are chosen to be the same (S1= [0, T ]) and twice (S2= [T, 2T ]) as the training time, where the training time T = 20. We refer interested readers to CIN [63] for more details. We find that VRDP is superior to these methods on both prediction and question answering tasks. Moreover, VRDP works well in long-term prediction. It reduces the S2 error on CIN [63] by 62.4%.
sections.heading: 5 Conclusion
sections.text: This work introduces VRDP, a unified framework that integrates powerful differentiable physics models into dynamic visual reasoning. It contains three mutually beneficial components: a visual perception module, a concept learner, and a differentiable physics engine. The visual perception module parses the input video into object trajectories and visual representations; the concept learner grounds language concepts and object attributes from question-answer pairs and the visual representations; with object trajectories and attributes as prior knowledge, the physics model optimizes all physical parameters of the scene and objects by differentiable simulation. With these explicit physical parameters, the physics model reruns the simulation to reason about future motion and causal events, which are then executed by a symbolic program executor to get the answer. Equipped with the powerful physics model, VRDP is of highly data-efficient and generalizable that adapts to novel concepts quickly with few-shot data. Moreover, both the explicit physics engine and the symbolic executor are step-by-step interpretable, making VRDP fully transparent. Extensive experiments on CLEVRER and Real-Billiards show that VRDP outperforms state-of-the-art dynamic reasoning methods by large margins.
  Acknowledgments and Disclosure of Funding
  This work was supported by MIT-IBM Watson AI Lab and its member company Nexplore, ONR MURI, DARPA Machine Common Sense program, ONR (N00014-18-1-2847), and Mitsubishi Electric. Ping Luo was supported by the General Research Fund of Hong Kong No.27208720.
title: Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language