summaries: This paper presented a Dual-stream Network for image classification. It combines the representation of local and global pattern features by using self-attention and convolution together. It also proposed an Intrascale Propagation module to process two different resolutions in each block.  In addition, it also introduced such block to FPN to demonstrate the benefit.

summaries: This paper proposes Dual-stream Network (DS-Net) for visual recognition. The two streams correspond to high-resolution convolutions for local features and low-resolution vision transformers for global patterns. The paper proposes an intra-scale propagation module to process the two streams and an inter-scale alignment module to fuse them into one. For object detection, it designs a Dual-stream FPN (DS-FPN) that adds dual-stream blocks to every feature pyramid. The proposed DS-Net achieves good results for image classification, object detection, and instance segmentation.

summaries: This paper proposes a hybrid convolution+transformer model for 2d images. To summarize the model, it is broken up into 4 stages that operate at successively higher resolutions much like a typical Resnet.  At each stage, there are two parallel paths, one that apples a depthwise convolution at high resolution (capturing local effects), and one that applies a standard a multi-headed attention at coarse (capturing global effects) resolution.  These paths are fused by a final attention module which allows for local to attend to global and vice versa.  Another contribution is the DS-FPN (which simply slots one of the above “dual stream” modules into the lateral paths of a typical FPN). Experiments show strong performance on both image classification for ImageNet and detection/segmentation for COCO (relative to competing Transformer-only models like Deit).

summaries: The authors propose a new architecture for visual recognition. Like standard convolutional architectures, the representation is structured in a multi-scale hierarchy, with deeper layers being coarser and more abstract. At every level of the hierarchy, a coarse, global representation is appended and simultaneously processed with the more fine-grained, local one. Local features are processed with depth-wise convolutions, and global ones with multi-head self-attention. In addition to this separate processing of local and global features, they are co-processed by cross-attending between local and global features.   In addition to this recognition backbone, the authors adapt Feature Pyramid Networks in a similar spirit. Whereas FPN's use dense convolutions to fuse an intermediate representation with a coarser one, the authors instead use the proposed "Dual-Stream" block, combining local depth-wise convolutions, global self-attention, and local/global cross-attention.   The authors evaluate the recognition backbone for classification on ImageNet and detection and instance segmentation on COCO, reporting large gains over standard convolutional architectures such as ResNet, and reasonable gains over more recent transformer-based ones.

abstractText: Transformers with remarkable global representation capacities achieve competitive results for visual tasks, but fail to consider high-level local pattern information in input images. In this paper, we present a generic Dual-stream Network (DS-Net) to fully explore the representation capacity of local and global pattern features for image classification. Our DS-Net can simultaneously calculate fine-grained and integrated features and efficiently fuse them. Specifically, we propose an Intra-scale Propagation module to process two different resolutions in each block and an Inter-Scale Alignment module to perform information interaction across features at dual scales. Besides, we also design a Dual-stream FPN (DS-FPN) to further enhance contextual information for downstream dense predictions. Without bells and whistles, the proposed DS-Net outperforms DeiT-Small by 2.4% in terms of top-1 accuracy on ImageNet-1k and achieves state-of-the-art performance over other Vision Transformers and ResNets. For object detection and instance segmentation, DS-Net-Small respectively outperforms ResNet-50 by 6.4% and 5.5 % in terms of mAP on MSCOCO 2017, and surpasses the previous state-of-the-art scheme, which significantly demonstrates its potential to be a general backbone in vision tasks. The code will be released soon.

authors:
T.B. Brown
B. Mann
N. Ryder
M. Subbiah
J. Kaplan
P. Dhariwal
A. Neelakantan
P. Shyam
G. Sastry
A. Askell

title: et al

venue: Language models are few-shot learners. arXiv preprint arXiv:2005.14165

authors:
Z. Cai
N. Vasconcelos

title: Cascade r-cnn: Delving into high quality object detection

venue: IEEE CVPR, pages 6154–6162

authors:
N. Carion
F. Massa
G. Synnaeve
N. Usunier
A. Kirillov
S. Zagoruyko

title: End-to-end object detection with transformers

authors:
C.-F. Chen
Q. Fan
R. Panda

title: Crossvit: Cross-attention multi-scale vision transformer for image classification

title: Xception: Deep learning with depthwise separable convolutions

authors:
X. Chu
Z. Tian
Y. Wang
B. Zhang
H. Ren

title: and X

venue: W. et al. Twins: Revisiting the design of spatial attention in vision transformers. arXiv:2103.00112

authors:
X. Chu
Z. Tian
B. Zhang
X. Wang
X. Wei
H. Xia

title: Conditional positional encodings for vision transformers

venue: IEEE CVPR

authors:
J. Deng
W. Dong
R. Socher
L.-J. Li
K. Li
L. Fei-Fei

title: Imagenet: A large-scale hierarchical image database

venue: IEEE CVPR, pages 248–255

authors:
J. Devlin
M.-W. Chang
K. Lee
K. Toutanova

title: Bert: Pre-training of deep bidirectional transformers for language understanding

venue: arXiv preprint arXiv:1810.04805

authors:
A. Dosovitskiy
L. Beyer
A. Kolesnikov
D. Weissenborn
X. Zhai

title: and T

venue: U. et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR

authors:
K. Duan
S. Bai
L. Xie
H. Qi
Q. Huang
Q. Tian

title: Centernet: Object detection with keypoint triplets

authors:
P. Gao
S. Geng
R. Zhang
T. Ma
R. Fang
Y. Zhang
H. Li
Y. Qiao

title: Clip-adapter: Better vision-language models with feature adapters

venue: arXiv preprint arXiv:2110.04544

authors:
P. Gao
Z. Jiang
H. You
P. Lu
S.C.H. Hoi
X. Wang
H. Li

title: Dynamic fusion with intra- and inter-modality attention flow for visual question answering

venue: IEEE CVPR, pages 6632–6641

authors:
P. Gao
J. Lu
H. Li
R. Mottaghi
A. Kembhavi

title: Container: Context aggregation network

venue: arXiv preprint arXiv:2106.01401

authors:
P. Gao
M. Zheng
X. Wang
J. Dai
H. Li

title: Fast convergence of detr with spatially modulated co-attention

venue: arXiv preprint arXiv:2101.07448

authors:
K. Han
A. Xiao
E. Wu
J. Guo
C. Xu
Y. Wang

title: Transformer in transformer

authors:
K. He
G. Gkioxari
P. Dollár
R.B. Girshick

title: Mask R-CNN

venue: IEEE ICCV, pages 2980–2988

authors:
K. He
X. Zhang
S. Ren
J. Sun

title: Deep residual learning for image recognition

venue: IEEE CVPR, pages 770–778

authors:
A.G. Howard
M. Zhu
B. Chen
D. Kalenichenko
W. Wang
T. Weyand
M. Andreetto
H. Adam

title: Mobilenets: Efficient convolutional neural networks for mobile vision applications

authors:
J. Hu
L. Shen
G. Sun

title: Squeeze-and-excitation networks

venue: IEEE CVPR)

authors:
H. Law
J. Deng

title: Cornernet: Detecting objects as paired keypoints

venue: ECCV, pages 765–781

authors:
Y. Lecun
L. Bottou
Y. Bengio
P. Haffner

title: Gradient-based learning applied to document recognition

venue: Proceedings of the IEEE, 86(11):2278–2324

authors:
T. Lin
P. Dollár
R.B. Girshick
K. He
B. Hariharan
S.J. Belongie

title: Feature pyramid networks for object detection

venue: IEEE CVPR, pages 936–944

authors:
T. Lin
P. Goyal
R.B. Girshick
K. He
P. Dollár

title: Focal loss for dense object detection

venue: IEEE ICCV, pages 2999–3007

authors:
T.-Y. Lin
M. Maire
S.J. Belongie
L.D. Bourdev
R.B. Girshick
J. Hays
P. Perona
D. Ramanan
P. Dollár
C.L. Zitnick

title: Microsoft coco: Common objects in context

venue: ECCV, pages 740–755

authors:
W. Liu
D. Anguelov
D. Erhan
C. Szegedy
S.E. Reed
C. Fu
A.C. Berg

title: SSD: single shot multibox detector

venue: ECCV, pages 21–37

authors:
Z. Liu
Y. Lin
Y. Cao
H. Hu
Y. Wei

title: and Z

venue: Z. et al. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv:2103.14030

authors:
J. Long
E. Shelhamer
T. Darrell

title: Image segmentation fully convolutional networks for semantic segmentation

authors:
J. Lu
D. Batra
D. Parikh
S. Lee

title: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

venue: arXiv preprint arXiv:1908.02265

authors:
M. Mao
Y. Tian
B. Zhang
Q. Ye
W. Liu
G. Guo
D. Doermann

title: iffDetector: Inferenceaware feature filtering for object detection

venue: arXiv preprint arXiv:2006.12708

authors:
J. Pang
K. Chen
J. Shi
H. Feng
W. Ouyang
D. Lin

title: Libra r-cnn: Towards balanced learning for object detection

authors:
G. Peng
H. You
Z. Zhang
X. Wang
H. Li

title: Multi-modality latent interaction network for visual question answering

venue: IEEE ICCV, pages 5824–5834

authors:
Z. Peng
W. Huang
S. Gu
L. Xie
Y. Wang
J. Jiao
Q. Ye

title: Conformer: Local features coupling global representations for visual recognition

authors:
A. Radford
K. Narasimhan
T. Salimans
I. Sutskever

title: Improving language understanding by generative pre-training

authors:
I. Radosavovic
R.P. Kosaraju
R. Girshick
K. He
P. Dollár

title: Designing network design spaces

authors:
J. Redmon
S.K. Divvala
R.B. Girshick
A. Farhadi

title: You only look once: Unified

venue: real-time object detection. In IEEE CVPR, pages 779–788

authors:
S. Ren
K. He
R.B. Girshick
J. Sun

title: Faster R-CNN: towards real-time object detection with region proposal networks

venue: NeurIPS, pages 91–99

authors:
K. Simonyan
A. Zisserman

title: Very deep convolutional networks for large-scale image recognition

authors:
C. Szegedy
W. Liu
Y. Jia
P. Sermanet
S. Reed
D. Anguelov
D. Erhan
V. Vanhoucke
A. Rabinovich

title: Going deeper with convolutions

venue: IEEE CVPR, pages 1–9

authors:
H. Tan
M. Bansal

title: Lxmert: Learning cross-modality encoder representations from transformers

venue: arXiv preprint arXiv:1908.07490

authors:
M. Tan
Q. Le

title: Efficientnet: Rethinking model scaling for convolutional neural networks

venue: ICML

authors:
H. Touvron
M. Cord
M. Douze
F. Massa
A. Sablayrolles
H. Jégou

title: Training data-efficient image transformers and distillation through attention

authors:
A. Vaswani
N. Shazeer
N. Parmar
J. Uszkoreit
L. Jones

title: and A

venue: N. G. et al. Attention is all you need. In NeurIPS

authors:
H. Wang
Y. Zhu
H. Adam
A. Yuille
L.-C. Chen

title: MaX-DeepLab: End-to-end panoptic segmentation with mask transformers

authors:
W. Wang
E. Xie
X. Li
D.-P. Fan
K. Song

title: and D

venue: L. et al. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv:2102.12122

authors:
H. Wu
B. Xiao
N. Codella
M. Liu
X. Dai

title: and L

venue: Y. et al. Cvt: Introducing convolutions to vision transformers. arXiv:2103.15808

authors:
H. Yan
Z. Li
W. Li
C. Wang
M. Wu

title: and C

venue: Zhang. Contnet: Why not use convolution and transformer at the same time? arXiv:2104.13497

authors:
L. Yuan
Y. Chen
T. Wang
W. Yu
Y. Shi

title: and Z

venue: J. et al. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv:2101.11986

authors:
D. Zhang
H. Zhang
J. Tang
M. Wang
X. Hua
Q. Sun

title: Feature pyramid transformer

venue: ECCV

authors:
Q. Zhang
Y. Yang

title: Rest: An efficient transformer for visual recognition

venue: arXiv preprint arXiv:2105.13677

authors:
X. Zhang
F. Wan
C. Liu
R. Ji
Q. Ye

title: FreeAnchor: Learning to match anchors for visual object detection

venue: NeurIPS

authors:
X. Zhang
X. Zhou
M. Lin
J. Sun

title: Shufflenet: An extremely efficient convolutional neural network for mobile devices

venue: IEEE CVPR, pages 6848–6856

authors:
Z. Zhao
K. Samel
B. Chen
L. Song

title: Proto: Program-guided transformer for programguided tasks

venue: arXiv preprint arXiv:2110.00804

authors:
S. Zheng
J. Lu
H. Zhao
X. Zhu
Z. Luo

title: and Y

venue: W. et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In IEEE CVPR

authors:
D. Zhou
B. Kang
X. Jin
L. Yang
X. Lian

title: and Z

venue: J. et al. Deepvit: Towards deeper vision transformer. In IEEE CVPR

authors:
Z. Zhu
M. Xu
S. Bai
T. Huang
X. Bai

title: Asymmetric non-local neural networks for semantic segmentation

venue: IEEE ICCV, pages 593–602

heading: 1 Introduction

text:
In recent years, convolutional neural networks (CNNs) have dominated various vision tasks including but not limited to image recognition [22, 18, 38, 39, 5, 19, 41, 20, 52], object detection [37, 2, 26, 36, 24, 51, 30, 21, 11] and segmentation [17, 28], thanks to their unprecedented representation capacity. However, limited receptive fields of convolutions inevitably neglect the global patterns in images, which might be crucial during inference. For example, it is more likely a chair than a elephant nearby a table. Such object-level information cannot be fully explored by local convolutions, that hampers further improvement of CNNs. Motivated by the success of Transformer architecture [43] in Natural Language Processing (NLP) [9, 34, 1] and Multi-modality Fusion [29, 40, 12], researchers are trying to apply Transformer to vision tasks and have obtained promising results. In [10, 55, 48, 4, 7, 45, 46, 27, 3, 54, 14, 15, 44, 53], many novel architectures and optimizing strategies are proposed and achieve comparable or even better performance than CNNs, emphasizing the significance of extracting global features in vision tasks.
Inspired by the verified efficacy of CNN and Transformer, many concurrent works, such as ContNet [47] a CvT [46], attempt to introduce convolutions to vision transformers in different manners, hoping to combine both advantages. In CvT [46], linear projection of self-attention block in the Transformer module are replaced with convolutional projection. In contrast, ContNet[47] performs
35th Conference on Neural Information Processing Systems (NeurIPS 2021)
convolution on different token maps to build mutual connections, which is similar to Swin Transformer [27].
Attention maps
(a) (b)Input image
Feature maps
(c) (d)
Deit DS-Net Deit DS-Net
However, some drawbacks of existing models still remain to be solved. Firstly, works mentioned above either perform convolution and attention mechanisms sequentially, or just replace linear projection with convolutional projection in attention mechanisms, which may not be the most ideal design. Additionally, the conflicting properties of such two operations, convolution for local patterns but attention for global patterns, might cause ambiguity during training, which prevents their merits from merging to the maximum extent. Furthermore, self-attention can capture long-range information via the built-in all-pair interaction in theory, but it is quite possible that attentions might be confused and disturbed by neighboring details in high resolution feature maps, fail to build up object-level global patterns(see Fig. 1(a)). Finally, the computation cost of self-attention is unaffordable due to the quadratic computation complexity of sequence length. Although PVT [45] and APNB [56] downsample the key-query features to improve the efficiency of self-attention operator, they both abandon fine-graind local details of the image, which greatly impairs their performance.
In this paper, we address the these issues by introducing a Dual-stream Network (DS-Net). Instead of single stream architecture as previous works, our DS-Net adopts Dual-stream Blocks (DS-Blocks), which generates two feature maps with different resolutions, and retains both local and global information of the image via two parallel branches. We propose a Intra-scale Propagation module here to process two feature maps. Specifically, high-resolution features are used to extract fine-grained local patterns with depth-wise convolution, while low-resolution features are expected to summarize long-range global patterns. Considering low-resolution features themselves contain more integrated information, it would be much more easier for self-attention mechanism to capture object-level patterns rather than overwhelmed by trivial details (see Fig. 1(b)). Such dual-stream architecture disentangles local and global representations, which helps maximize both their merits, and thus generates better representations compared to DeiT baseline (see (c) and (e) in Fig. 1). Besides, low-resolution feature maps for self-attention dramatically reduce the memory cost and computation complexity. After parallelly processing dual streams, we present Inter-scale Alignment module based on co-attention mechanism at the end of DS-Blocks. This is because local details and global patterns capture different perspectives of the image, which are misaligned not only in pixel positions but also in semantics. Hence, this module is designed for modeling complex cross-scale relations and adaptively fuse local and global patterns.
Besides, we apply our DS-Blocks to Feature Pyramid Networks for further feature refinement, named DS-FPN. In this way, multi-level features are capable of extracting contextual information from both local and global views, improving performance of downstream tasks. This demonstrates that our Dual Stream design could be utilized as a plug-in building block not only for image recognition, but for many other vision tasks.
The contributions of this work are concluded as follows:
1. We present a novel Dual-Stream Network, named DS-Net, which retains both local and global features in DS-Block. The independent propagation maximize the advantages of convolution and self-attention, thus eliminating the conflicts during training.
2. We propose Intra-scale Propagation and Inter-Scale Alignment mechanism to achieve effective information flows within and between features of different resolutions, thus generating better representations of the image.
3. We introduce Dual-stream Feature Pyramid Network (DS-FPN) to enhance contextual information for downstream dense tasks and achieves better performance with little extra costs.
4. Without bells and whistles, the proposed DS-Net outperforms DeiT baseline by significant margins in terms of top-1 accuracy on ImageNet-1k and achieves state-of-the-art performance over other Vision Transformers and CNN-based networks on image classification and downstream tasks, including object detection and instance segmentation.

heading: 2 Related work

text:
Vision Transformers. Motivated by the great success of Transformer in natural language processing, researchers are trying to apply Transformer architecture to Computer Vision tasks. Unlike mainstream CNN-based models, Transformer is capable of capturing long-distance visual relations by its selfattention module and provides the paradigm without image-specific inductive bias. ViT [10] views 16 × 16 image patches as token sequence and predicts classification via a unique class token, which shows promising results. Subsequently, many works, such as DeiT [42] and PVT [45] achieve further improvement on ViT, making it more efficient and applicable in downstream tasks. ResT [50] proposed a efficient transformer model. Besides, models based on Transformer also give leading performance in many other vision tasks such as object tracking and video analysis.
Local and Global Features in Transformers. Despite the Transformer’s superiority upon extracting global representations, image-level self-attention is unable to capture fine-grained details. To tackle this issue, previous works assert to obtain the local information at the same time and then fuse features of the two scales. TNT [16] proposes intra-patch self-attention block to model local structures and aggregates them with global patch embeddings. Swin Transformer [27] extracts local features within each partitioned window and fuses them by self-attention in successive shifted windows. Similarly, shifted windows are replaced by convolutions in ContNet [47]. Following such design of local windows, Twins [6] uses inter-window attention to aggregate global features. Swin Transformer and Twins both give advanced performance, but still rely on local-window self-attention to capture fine-level features, which works intuitively worse than CNN. Besides, Swin’s shifted windows are complex for device optimization and Twins [6] recurrently local and global attentions would constrain each other’s representation learning.
Equip Transformers with Convolution. Compared to self-attention mechanism, CNN has its unique advantages of local modeling and translation invariance. Therefore, some existing transformers explore the hybrid architecture to incorporate both merits for better visual representation. T2T [48] progressively aggregate neighboring tokens to one token to capture local structure information, similar to convolution operation. Followingly, CvT [46] designs Convolutional Token Embedding and Convolutional Transformer Block for capturing more precise local spatial context. CPVT [7] utilizes one convolutional layer to dynamically generate positional encodings, adaptive for varying image scales. Conformer [33] combines transformer with an independent CNN network for specializing local features and flows information via lateral connections. Nonetheless, none of the aforementioned models maximizes the potential of the hybrid architecture. They either use CNN for marginal operations to boost Transformer, e.g., tokens generation in CvT [46] and positional encodings in CPVT [7], or view CNN as a relatively separate branch, like Conformer.
Different from previous works, in proposed DS-Blocks, we treat self-attention and convolution as dual resolution processing paths via Intra-scale Propagation module, where the former aims to extract local fine-grained details, and the latter focus on exploring features from a global view. On top of that, their information is reasonably fused by Inter-scale Alignment module based on co-attention, which addresses the issue of feature misalignment. By this design, the local and global patterns could be simultaneously extracted and the capability of CNN is fully exerted in Transformer architecture.

heading: 3 Method

heading: 3.1 Dual-scale Representations

text:
The overall pipeline of our proposed DS-Net is shown in Fig. 2. Motivated by stage-wise design in previous networks like ResNet [18], we set 4 stages in the architecture, whose down-sampling factors
are 4, 8, 16, 32, respectively. Within each stage, Dual-stream Blocks (DS-Blocks) are adopted to generate and combine dual-scale representations.
Our key idea is to keep local features in a relatively high resolution to reserve local details, while represent global features with a lower resolution (1/32 of the image size) to retain global patterns. Specifically, in each DS-block, we split the input feature map into two parts at the channel dimension. One is for extracting local features, denoted as fl, the other is for summarizing global features, denoted as fg. Notably, we keep the size of fg unchanged in all stages across the network by downsampling it with a proper factor, W32 × H 32 × Ci 2 , where W and H represents the width and height of the input image, and Ci represents the channel number of input features in current stage. Aided by the high resolution of fl, the local patterns could be much reserved for subsequent extraction, but thanks to the low resolution of fg , the exploration of non-local and object-level information is greatly benefited.

heading: 3.2 Intra-scale Propagation

text:
Local and global features are representations of one image from two totally different views. The former focuses on fine-grained details, essential for tiny-object detection and pixel-level localization, while the latter aims at modeling object-level relations between long-range parts. Therefore, given dual-scale features fl and fg , we perform Intra-scale Propagation module to parallelly process them.
Local representation. For high-resolution fl with the size of Wi ×Hi × Cl, where Cl equals Ci2 , we perform 3×3 depth-wise convolution to extract local features as follows and obtain fL:
fL(i, j) = M,N∑ m,n W (m,n)⊙ fl(i+m, j + n), (1)
where W (m,n), (m,n) ∈ (−1, 0, 1) represents the convolution filters, W (m,n) and fl(i, j) are both Ci 2 dimensional vectors, ⊙ denotes element-wise product. By the power of depth-wise convolution, fL is able to contain fine-grained local details of the input image.
Global representation. For low-resolution representation fg with the fixed size of W32 × H 32 × Ci 2 as illustrated in Section 3.1, we first flatten fg to a sequence with the length of lg , the product of W32 and
H 32 , in which each element is a Ci 2 dimensional vector. By doing this, each vector in the sequence is treated as a visual token without spatial information. Here, the dependencies between different token pairs are unrelated with their spatial positions in the feature map, which is totally different with the convolution. Then, we summarize the global information and model the object-level coupling relation via self-attention mechanism:
fQ = fgWQ, fK = fgWK , fV = fgWV , (2)
where WQ, WK , WV denote matrixes to generate queries, keys and values respectively. By calculating the similarity between fQ and fV , we obtain attention weights for aggregating information from different locations of fg. Finally, we calculate the weighted sum of attention weights and fV , thus obtaining integrated features:
fG = softmax( fQf T K√ d )fV , (3)
where d equals Ci 2
N , N denotes the number of attention head that we set to 1, 2, 5, 8 for 4 stages respectively in DS-Net.
Our dual-stream architecture disentangles fine-grained and integrated features in two pathways, which significantly eliminates the ambiguity during training. Additionally, the Intra-scale Propagation module processes feature maps with dual resolutions by two domain-specifically effective mechanisms separately, extracting local and global features to the maximum extent.

heading: 3.3 Inter-scale Alignment

text:
A delicate fusion of dual-scale representations is vital for the success of DS-Net, since they capture two different perspectives of one image. To address this, a naive idea is to upsample the low-resolution representation using bilinear interpolation, and then fuse the dual-scale representations via 1 by 1 convolution after simply concatenating them position-wisely. Such naive fusion is not convincing enough. Furthermore, by visualizing the feature maps of dual-scale features, we observe that global features in low resolutions and local features in high-resolutions are actually misaligned (see (b) and (d) in Fig. 1. Therefore, considering the relation of two representations are not explicitly explored, it is unpersuasive to pre-define a fixed strategy to fuse them, such as concatenation, element-wise addition or production. Enlightened by [13, 32], we propose a novel co-attention-based Interscale Alignment module, whose scheme of Inter-scale Alignment module is shown in Fig. 2. This module aims to capture the mutual correlations between each local-global token pair, and propagate information bidirectionally in a learnable and dynamic manner. Such mechanism prompts local features to adaptively explore their relations with global information, enabling themselves to be more representative and informative, and vice versa.
Given extracted local features fL with the size of Wi ×Hi × Ci2 and global features fG with the size of lg × Ci2 , we first flatten fL to a sequence with the length of ll, the product of Wi and Hi, in which each element is a Ci2 dimensional vector. fL and fG are now in the same format but with different length. Then we perform co-attention on two sequences as follows:
QL = fLW l Q, KL = fLW l K , VL = fLW l V , QG = fGW g Q, KG = fGW g K , VG = fGW g V ,
(4)
where the sizes of W are all Ci2 × dim, and the dim is a hyper-parameter. Thus we have the transformed features of local and global representations. Then we calculate the similarities between every pair of fL and fA to obtain the corresponding attention weights:
WG→L = softmax( QLK T G√
d ), WL→G = softmax(
QGK T L√
d ). (5)
The size of WG→L and WL→G are ll × lg and lg × ll respectively. The non-linear function softmax is performed at the last dimension. WG→L reflects the importance of different tokens in global features to the local tokens. Likewise, global features can also extract useful information from local features via WL→G. Instead of fixed fusion strategy that might bring constraints, what and how the
information transfers are automatically determined by features themselves here. We can then obtain hybrid features as:
hL = WG→LVG, hG = WL→GVL, (6)
where the size of hL and hG are ll×dim and lg×dim respectively. Then we add a 1 × 1 convolution layer after hybrid features to further fuse the channels and reshape them to Wi × Hi × Ci2 and W 32 × H 32 × Ci 2 .
Such a bidirectional information flow is able to identify cross-scale relations between local and global tokens, by which dual-scale features are highly aligned and coupled with each other. After this, we could safely upsample low-resolution representation hG, concatenate it with high-resolution hL and perform 1 by 1 convolution for channel-wise dual-scale information fusion. At the end of the last block in stage 4, we add a fully-connected layer as the classifier to conduct classification.

heading: 3.4 Dual-stream Feature Pyramid Networks

text:
Introducing contextual information into Feature Pyramid Networks(FPN) [23] has been explored by [49, 31]. However, previous methods often cause large extra memory and computation costs, due to their complicated architectures and utilized high resolution feature maps. Besides, Non-local contexts would miss local details, which is disastrous for tiny object detection and segmentation. Here, we apply our Dual-stream design into FPN, named Dual-stream Feature Pyramid Networks(DS-FPN), by simply adding DS-Blocks to every feature pyramid scale. In this way, DS-FPN is able to better attain non-local patterns and local details with marginal increased costs at all scales, which further enhance the performance of subsequent object detection and segmentation heads. This shows our DS-Net can not only serve as a backbone but also a general plug-in building block in many other vision architectures.
Similar to FPN [23], we take image features from various scales from the backbone as input, and output corresponding refined feature maps of fixed channels number by a top-down aggregation methods. Our structure is composed of bottom-up pathways, Dual-stream lateral connections, and top-down pathways. The bottom-up and top-down pathways follow the design of FPN, but lateral connection here adopts DS-Block to process features in dual scales via Intra-scale Propagation and Inter-scale Alignment. See Fig. 3.

heading: 4 Experiments

text: In this section, we first provide three ablation studies to explore the optimal structure of DS-Net and interpret the necessity of dual-stream design. Then we give the experimental results of image classification and downstream tasks including object detection and instance segmentation. Specifically, We use ImageNet-1K [8] for classification and MSCOCO 2017 [25] for object detection and instance segmentation. All experiments are conducted on 8 V100 GPUs and the throughput is tested on 1 V100 GPU. As the number of blocks we set in different stages in the architecture (see Fig. 2) is flexible, we conduct experiments on 3 models with different parameter size, denoted as DS-Net-T (Tiny), DS-Net-S (Small) and DS-Net-B (Big), which have comparable parameters with ResNet-18, ResNet-50 and ResNet-101 respectively. The detailed setup is shown in Table 1.

heading: 4.1 Ablation Study

heading: 4.1.1 Ratio of Local to Global Features

text:
Settings. As described in Section 3.1, we split the features into two parts in channel dimension as local and global representations, which are later processed by convolution and self-attention, respectively. We conduct an experiment to explore a appropriate partition ratio. We use α to denote the proportion of channel number of fg (global features) to the total channel number. When α equals 0, only depth-wise convolution is performed, and when α equals 1, only self-attention is performed. Noting that in this experiment, we choose DS-Net-T as the testing model, which simply concatenates fL and fA at the end of each block, without Inter-scale Alignment module.
Results. The results in Table 2 indicates that when α equals 0.5, DS-Net-T achieves the highest accuracy on ImageNet-1k validation set, which, to some extent, implies that local and global information play equally important roles for visual representations. Thus, in the following experiments, we equally split features into two parts at the channel dimension to obtain dual-scale representations.

heading: 4.1.2 None is Dispensable in DS-Block

text:
Settings. Here, we illustrate that every component within DS-Block plays a vital role in the prediction, including either pathway of Intra-scale Propagation and bidirectional co-attention of Inter-scale Alignment. In Table 3, we respectively remove one of the four modules in every block of DS-Net-T∗ (DS-Net∗ represents the corresponding DS-Net version with Inter-scale Alignment module), while remain the others the same. Therein, w/o fL denotes that fl split at the beginning is directly fed to Inter-scale Alignment Module without exploring local features, and w/o G → L denotes that co-attention from global features to local features is not implemented, analogous to w/o fG and w/o L → G. The version w/o L ↔ G removes the entire co-attention, which equals DS-Net-T.
Results. As shown in Table 3, any absence of the components in DS-Block deteriorates the performance. Compared to extracting local and global features simultaneously, the model’s representation capacity is constrained with either fL or fG. Surprisingly, the unilateral co-attention of w/o G → L and w/o G → L performs worse than removing any fusion strategy, because implementing G → L would confuse the original local features fL, and only by L → G could the missing local features be complemented. This further demonstrates the importance of the designed Inter-scale Alignment module.

heading: 4.2 Image Classification

text:
Settings. Image classification experiments are performed on ImageNet-1K [8], comprising 1.28M training images and 50K validation images of 1000 classes. For fair comparison with other works, we follow the training settings in DeiT. We train our model for 300 epochs by AdamW optimizer. The initial learning rate is set to 1e-3 and scheduled by the cosine strategy.
Results. We report the results in Table 4, where DS-Net and DS-Net∗ denote our proposed networks with and without Inter-scale Alignment module respectively. For DS-Net, only concatenation operation is used to fuse the local and global features. DS-Net∗ is the version with Inter-scale Alignment module. With comparable parameters and complexity, our proposed DS-Net consistently outperforms the DeiT [42] baseline by significant margins, which is, specifically, 6.8% improvement for tiny model, 2.4% improvement for small model and 1.3% for big model with only half parameters. The efficacy of Inter-scale Alignment module is also clearly indicated in Table 4, which gains further 0.9% , 0.4% and 0.3% improvement compared to the simple version of DS-Net-T, DS-Net-S and DS-Net-B. Notably, without bells and whistles, DS-Net-T∗ DS-Net-S∗ also outperform other state-of-the-art methods, including CNN, vision transformer and some hybrid architectures.

heading: 4.3 Object Detection and instance segmentation

text: Settings. Experiments in this part are conducted on the challenging MSCOCO 2017 [25], containing 118K training images and 5K validation images. Specifically, the backbone is initialized by pretraind weights from ImageNet-1K, and other layers adopt Xavier. We evaluate our DS-Net on typical detectors: RetinaNet and Mask R-CNN [17]. As the standard 1× schedule(12 epochs), we adopt AdamW optimizer with initial learning rate of 1e-4, decayed by 0.1 at epoch 8 and 11. We set stochastic drop path regularization of 0.1 and weight decay of 0.05. Following previous methods, training and testing images are resized to 800×1333. Results. The results are shown in Table 5. For object detection, DS-Net-S∗ achieves significant improvement compared to ResNet-50 backbone [24] by 6.4% with RetinaNet and 6.1% with Mask R-CNN in terms of AP bbox, and outperforms the previous state-of-the-art Swin-T [27] by 1.2% and 0.6%. For instance segmentation, DS-Net-S∗ achieves 40.2% in terms of AP segm, which surpasses the ResNet-50 and Swin-T by 5.5% and 0.4%.

heading: 4.4 Dual-stream Feature Pyramid Networks

text:
Settings. Experiments of DS-FPN are implemented on DS-Net-T and Swin Transformer [27] for object detection and instance segmentation on MSCOCO 2017 [25], by replacing the original FPN with DS-FPN. The training setting and scheme are the same as those mentioned above, which is 1× schedule(12 epochs) and AdamW optimizer with initial learning rate of 1e-4. Likewise, the backbone is initialized with pretrained weights, and DS-FPN is trained from scratch with newly added heads.
Ablation Study. In Table 6, we report results of several different designs combining DS-Blocks with FPN on RetinaNet, with DS-Net-T backbone. Four inserted positions of DS-Blocks have been
experimented, where "Last" means replacing the last 3×3 convolution layer, and "Lateral" equips them on lateral connections. More specifically, "Lateralrev" moves the inner channel-alignment 1 ×1 convolution layer from the last place to the first, and "Lateralextra" adds DS-Blocks also on extra feature outputs. The results show that "Lateralextra" has the best performance. Therefore, we set "Lateralextra" as our default, which is 1.4% higher than FPN with marginal memory costs.
Comparison with FPN.As reported in Table 7. For object detection, DS-Net-T and Swin Transformer with DS-FPN gain 1.4% and 0.8% mAP improvement, respectively. For instance segmentation, DSFPN helps to gain 0.7% and 1.4% AP segm improvement for DS-Net-T and Swin Transformer, respectively. The results significantly prove the generality and effectiveness our Dual-stream design.

heading: 5 Conclusion

text:
In this paper, we propose a generic Dual-stream Network, called DS-Net, which disentangles local and global features by generating two representations with different resolutions. We present Intrascale Propagation module and Inter-scale Alignment module to combine the merits of convolution and self-attention mechanisms, and identify cross-scale relations between local and global tokens. Besides, for downstream tasks, we design a Dual-stream Feature Pyramid Network (DS-FPN) to introduce contextual information to feature pyramid for further refinement with marginal costs. With the outstanding performance on image classification and dense downstream tasks including object detection and instance segmentation, our proposed DS-Net has shown its promising potential in vision tasks.
Disclosure of Funding
This work was supported by the National Natural Science Foundation of China under Grant 62076016 and the Shanghai Committee of Science and Technology, China (Grant No. 21DZ1100100).

title: Dual-stream Network for Visual Recognition


summaries: This paper introduces VRDP, a method that learns to answer predictive, counterfactual, explanatory and descriptive yes or no questions about videos depicting the evolution of physical scenes. VRDP is composed of three modules, an object detector, a neuro-symbolic concept learner and differentiable physics model.  The paper shows that VRDP surpasses or closely matches the SOTA on two datasets, a synthetic dataset CLEVERER and a dataset of real world videos.

summaries: This paper proposes to do visual question answering by learning a differentiable physics model from video. The pipeline is divided to several modular and interpretable parts. Firstly, a visual perception module parse the input video to several objects’ features. Secondly, a concept learner parse the input questions to a set of programs and grounded concepts. The output of the above two modules are used to estimate the objects’ physical parameters and fed to a differentiable physical engine, which is used to do future trajectory predictions. Finally, a symbolic execution part is used to output answer given the future predictions.

summaries: The paper tackles the problem of dynamic visual reasoning by using a modular unit that is composed of three components: visual perception, concept learner, and physics simulator. This system is capable of performing system identification from visual and language modalities with assistance from a physics model. Having access to the physics model, it can produce imagined future trajectories for reasoning about possible future outcomes. This gave them an advantage of outperforming competitive methods, especially in counterfactual tasks.

summaries: - The paper presents a composite model for visual question answering in scenarios which require dynamic visual reasoning. - The model pipeline consists of a visual perception module which extracts object-centric trajectories from an input video, a concept learner which parses the natural language QA pair into a set of neurosymbolic concept vectors, a differentiable physics model which can simulate the evolution of rigid body trajectories and a neurosymbolic program executor which computes or selects the answer for a given input question using the intermediate concept and trajectory features. - As a contribution over prior art in neurosymbolic visual reasoning, the authors propose the utilisation of a differentiable physics module for accurate trajectory prediction based on physics parameters which are inferred from the video input sample. Since explicit physical properties like mass, friction and restitution are inferred in the model, the computed output is very amenable to human interpretation. - The proposed approach is comprehensively evaluated on the synthetic CLEVRER and realistic Real-billiard benchmarks and compared against related neurosymbolic and end-to-end VQA models. It demonstrates remarkable boosts in data efficiency on CLEVRER, sets a new state-of-the-art performance on counterfactual questions and significantly outperforms prior art in a few-shot setup where generalisation to modified physical attributes like 'heavier' or 'lighter' is required.

abstractText: In this work, we propose a unified framework, called Visual Reasoning with Differentiable Physics (VRDP) 1, that can jointly learn visual concepts and infer physics models of objects and their interactions from videos and language. This is achieved by seamlessly integrating three components: a visual perception module, a concept learner, and a differentiable physics engine. The visual perception module parses each video frame into object-centric trajectories and represents them as latent scene representations. The concept learner grounds visual concepts (e.g., color, shape, and material) from these object-centric representations based on the language, thus providing prior knowledge for the physics engine. The differentiable physics model, implemented as an impulse-based differentiable rigid-body simulator, performs differentiable physical simulation based on the grounded concepts to infer physical properties, such as mass, restitution, and velocity, by fitting the simulated trajectories into the video observations. Consequently, these learned concepts and physical models can explain what we have seen and imagine what is about to happen in future and counterfactual scenarios. Integrating differentiable physics into the dynamic reasoning framework offers several appealing benefits. More accurate dynamics prediction in learned physics models enables state-of-the-art performance on both synthetic and real-world benchmarks while still maintaining high transparency and interpretability; most notably, VRDP improves the accuracy of predictive and counterfactual questions by 4.5% and 11.5% compared to its best counterpart. VRDP is also highly data-efficient: physical parameters can be optimized from very few videos, and even a single video can be sufficient. Finally, with all physical parameters inferred, VRDP can quickly learn new concepts from few examples.

authors:
P. Agrawal
A.V. Nair
P. Abbeel
J. Malik
S. Levine

title: Learning to poke by poking: Experiential learning of intuitive physics

venue: In NeurIPS,

authors:
A. Ajay
M. Bauza
J. Wu
N. Fazeli
J.B. Tenenbaum
A. Rodriguez
L.P. Kaelbling

title: Combining physical simulators and object-based networks for control

venue: In ICRA,

authors:
S. Amizadeh
H. Palangi
O. Polozov
Y. Huang
K. Koishida

title: Neuro-symbolic visual reasoning: Disentangling" visual" from" reasoning

venue: In ICML,

authors:
J. Andreas
M. Rohrbach
T. Darrell
D. Klein

title: Neural module networks

venue: In CVPR,

authors:
S. Antol
A. Agrawal
J. Lu
M. Mitchell
D. Batra
C. Lawrence Zitnick
D. Parikh

title: Vqa: Visual question answering

venue: In ICCV,

authors:
D. Bahdanau
K. Cho
Y. Bengio

title: Neural machine translation by jointly learning to align and translate

venue: In ICLR,

authors:
A. Bakhtin
L. van der Maaten
J. Johnson
L. Gustafson
R. Girshick

title: Phyre: A new benchmark for physical reasoning

authors:
F. Baradel
N. Neverova
J. Mille
G. Mori
C. Wolf

title: Cophy: Counterfactual learning of physical dynamics

authors:
P.W. Battaglia
J.B. Hamrick
J.B. Tenenbaum

title: Simulation as an engine of physical scene understanding

venue: Proceedings of the National Academy of Sciences,

authors:
P.W. Battaglia
R. Pascanu
M. Lai
D. Rezende
K. Kavukcuoglu

title: Interaction networks for learning about objects, relations and physics

venue: arXiv preprint arXiv:1612.00222,

authors:
D.M. Bear
C. Fan
D. Mrowca
Y. Li
S. Alter
A. Nayebi
J. Schwartz
L. Fei-Fei
J. Wu
J.B. Tenenbaum

title: Learning physical graph representations from visual scenes

venue: NeurIPS,

authors:
Y. Bengio
J. Louradour
R. Collobert
J. Weston

title: Curriculum learning

venue: In ICML, pages

title: A stability property of implicit runge-kutta methods

venue: BIT Numerical Mathematics, pages 358–361,

title: Modeling and solving constraints

venue: In Game Developers Conference,

authors:
M.B. Chang
T. Ullman
A. Torralba
J.B. Tenenbaum

title: A compositional object-based approach to learning physical dynamics

authors:
Z. Chen
J. Mao
J. Wu
K.-Y.K. Wong
J.B. Tenenbaum
C. Gan

title: Grounding physical concepts of objects and events through dynamic visual reasoning

authors:
F. de Avila Belbute-Peres
K. Smith
K. Allen
J. Tenenbaum
J.Z. Kolter

title: End-to-end differentiable physics for learning and control

authors:
J. Degrave
M. Hermans
J. Dambre

title: A differentiable physics engine for deep learning in robotics

venue: Frontiers in neurorobotics,

authors:
D. Ding
F. Hill
A. Santoro
M. Botvinick

title: Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures

venue: arXiv preprint arXiv:2012.08508,

authors:
M. Ding
Y. Huo
H. Yi
Z. Wang
J. Shi
Z. Lu
P. Luo

title: Learning depth-guided convolutions for monocular 3d object detection

authors:
M. Ding
X. Lian
L. Yang
P. Wang
X. Jin
Z. Lu
P. Luo

title: Hr-nas: Searching efficient high-resolution neural architectures with lightweight transformers

authors:
C. Fan
X. Zhang
S. Zhang
W. Wang
C. Zhang
H. Huang

title: Heterogeneous memory enhanced multimodal attention model for video question answering

authors:
C. Finn
I. Goodfellow
S. Levine

title: Unsupervised learning for physical interaction through video prediction

authors:
C. Gan
Y. Li
H. Li
C. Sun
B. Gong

title: VQS: Linking segmentations to questions and answers for supervised attention in vqa and question-focused semantic segmentation

authors:
C. Gan
J. Schwartz
S. Alter
M. Schrimpf
J. Traer
J. De Freitas
J. Kubilius
A. Bhandwaldar
N. Haber
M. Sano

title: Threedworld: A platform for interactive multi-modal physical simulation

authors:
C. Gan
S. Zhou
J. Schwartz
S. Alter
A. Bhandwaldar
D. Gutfreund
D.L. Yamins
J.J. DiCarlo
J. McDermott
A. Torralba

title: The threedworld transport challenge: A visually guided task-and-motion planning benchmark for physically realistic embodied ai

venue: arXiv preprint arXiv:2103.14025,

authors:
R. Girdhar
D. Ramanan

title: Cater: A diagnostic dataset for compositional actions and temporal reasoning

authors:
G. Gkioxari
J. Malik

title: Finding action tubes

authors:
C. Han
J. Mao
C. Gan
J. Tenenbaum
J. Wu

title: Visual concept-metaconcept learning

authors:
X. Han
S. Wang
C. Su
W. Zhang
Q. Huang
Q. Tian

title: Interpretable visual reasoning via probabilistic formulation under natural supervision

venue: In ECCV,

authors:
K. He
G. Gkioxari
P. Dollár
R. Girshick

title: Mask r-cnn

authors:
K. He
X. Zhang
S. Ren
J. Sun

title: Deep residual learning for image recognition

authors:
E. Heiden
D. Millard
E. Coumans
Y. Sheng
G.S. Sukhatme

title: Neuralsim: Augmenting differentiable simulators with neural networks

venue: arXiv preprint arXiv:2011.04217,

authors:
E. Heiden
D. Millard
H. Zhang
G.S. Sukhatme

title: Interactive differentiable simulation

venue: arXiv preprint arXiv:1905.10706,

authors:
S. Hochreiter
J. Schmidhuber

title: Long short-term memory

venue: Neural Computation,

authors:
R. Hu
J. Andreas
T. Darrell
K. Saenko

title: Explainable neural computation via stack neural module networks

authors:
Y. Hu
L. Anderson
T.-M. Li
Q. Sun
N. Carr
J. Ragan-Kelley
F. Durand

title: Difftaichi: Differentiable programming for physical simulation

authors:
D. Huang
P. Chen
R. Zeng
Q. Du
M. Tan
C. Gan

title: Location-aware graph convolutional networks for video question answering

venue: In AAAI,

authors:
Z. Huang
Y. Hu
T. Du
S. Zhou
H. Su
J.B. Tenenbaum
C. Gan

title: Plasticinelab: A soft-body manipulation benchmark with differentiable physics

authors:
D.A. Hudson
C.D. Manning

title: Compositional attention networks for machine reasoning

title: Gqa: A new dataset for real-world visual reasoning and compositional question answering

authors:
Y. Jang
Y. Song
Y. Yu
Y. Kim
G. Kim

title: Tgif-qa: Toward spatio-temporal reasoning in visual question answering

authors:
M. Janner
S. Levine
W.T. Freeman
J.B. Tenenbaum
C. Finn
J. Wu

title: Reasoning about physical interactions with object-oriented prediction and planning

authors:
K.M. Jatavallabhula
M. Macklin
F. Golemo
V. Voleti
L. Petrini
M. Weiss
B. Considine
J. Parent- Levesque
K. Xie
K. Erleben

title: gradsim: Differentiable simulation for system identification and visuomotor control

authors:
J. Johnson
B. Hariharan
L. Van Der Maaten
J. Hoffman
L. Fei-Fei
C. Lawrence Zitnick
R. Girshick

title: Inferring and executing programs for visual reasoning

authors:
T. Kipf
E. van der Pol
M. Welling

title: Contrastive learning of structured world models

authors:
T.N. Kipf
M. Welling

title: Semi-supervised classification with graph convolutional networks

authors:
T.M. Le
V. Le
S. Venkatesh
T. Tran

title: Hierarchical conditional relation networks for video question answering

authors:
J. Lei
L. Yu
M. Bansal
T.L. Berg

title: Tvqa: Localized, compositional video question answering

venue: In EMNLP, 2018

authors:
A. Lerer
S. Gross
R. Fergus

title: Learning physical intuition of block towers by example

authors:
X. Li
J. Song
L. Gao
X. Liu
W. Huang
X. He
C. Gan

title: Beyond rnns: Positional self-attention with co-attention for video question answering

authors:
Y. Li
T. Lin
K. Yi
D. Bear
D. Yamins
J. Wu
J. Tenenbaum
A. Torralba

title: Visual grounding of learned physical models

authors:
Y. Li
J. Wu
R. Tedrake
J.B. Tenenbaum
A. Torralba

title: Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids

authors:
Y. Li
J. Wu
J.-Y. Zhu
J.B. Tenenbaum
A. Torralba
R. Tedrake

title: Propagation networks for modelbased control under partial observation

authors:
J. Liang
M.C. Lin

title: Differentiable physics simulation

venue: In ICLR Workshop,

authors:
D.C. Liu
J. Nocedal

title: On the limited memory bfgs method for large scale optimization

venue: Mathematical programming,

authors:
F. Locatello
D. Weissenborn
T. Unterthiner
A. Mahendran
G. Heigold
J. Uszkoreit
A. Dosovitskiy
T. Kipf

title: Object-centric learning with slot attention

venue: arXiv preprint arXiv:2006.15055,

authors:
J. Mao
C. Gan
P. Kohli
J.B. Tenenbaum
J. Wu

title: The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision

authors:
D. Mascharka
P. Tran
R. Soklaski
A. Majumdar

title: Transparency by design: Closing the gap between performance and interpretability in visual reasoning

authors:
I. Misra
R. Girshick
R. Fergus
M. Hebert
A. Gupta
L. Van Der Maaten

title: Learning by asking questions

authors:
R. Mottaghi
M. Rastegari
A. Gupta
A. Farhadi

title: what happens if...” learning to predict the effect of forces in images

venue: In ECCV. Springer,

authors:
M. Müller
J. Stam
D. James
N. Thürey

title: Real time physics: class notes

venue: In ACM SIGGRAPH 2008 classes,

authors:
H. Qi
X. Wang
D. Pathak
Y. Ma
J. Malik

title: Learning long-term visual dynamics with region proposal interaction networks

authors:
R. Riochet
M.Y. Castro
M. Bernard
A. Lerer
R. Fergus
V. Izard
E. Dupoux

title: Intphys: A framework and benchmark for visual intuitive physics reasoning

venue: arXiv preprint arXiv:1803.07616,

authors:
T. Shao
A. Monszpart
Y. Zheng
B. Koo
W. Xu
K. Zhou
N.J. Mitra

title: Imagining the unseen: Stability-based cuboid arrangements for scene understanding

venue: ACM TOG,

authors:
K. Smith
L. Mei
S. Yao
J. Wu
E. Spelke
J. Tenenbaum
T. Ullman

title: Modeling expectation violation in intuitive physics with coarse probabilistic object representations

authors:
M. Tapaswi
Y. Zhu
R. Stiefelhagen
A. Torralba
R. Urtasun
S. Fidler

title: MovieQA: Understanding Stories in Movies through Question-Answering

authors:
M.A. Toussaint
K.R. Allen
K.A. Smith
J.B. Tenenbaum

title: Differentiable physics and stable modes for tool-use and manipulation planning

venue: In IJCAI,

authors:
A. Vaswani
N. Shazeer
N. Parmar
J. Uszkoreit
L. Jones
A.N. Gomez
Ł. Kaiser
I. Polosukhin

title: Attention is all you need

authors:
R. Veerapaneni
J.D. Co-Reyes
M. Chang
M. Janner
C. Finn
J. Wu
J. Tenenbaum
S. Levine

title: Entity abstraction in visual model-based reinforcement learning

venue: In CoRL,

authors:
N. Watters
D. Zoran
T. Weber
P. Battaglia
R. Pascanu
A. Tacchetti

title: Visual interaction networks: Learning a physics simulator from video

authors:
J. Wu
E. Lu
P. Kohli
B. Freeman
J. Tenenbaum

title: Learning to see physics via visual de-animation

authors:
J. Wu
I. Yildirim
J.J. Lim
W.T. Freeman
J.B. Tenenbaum

title: Galileo: Perceiving physical object properties by integrating a physics engine with deep learning

authors:
Q. Wu
C. Shen
L. Liu
A. Dick
A. van den Hengel

title: What value do explicit high level concepts have in vision to language problems

authors:
D. Xu
Z. Zhao
J. Xiao
F. Wu
H. Zhang
X. He
Y. Zhuang

title: Video question answering via gradually refined attention over appearance and motion

venue: In ACM MM,

authors:
T. Ye
X. Wang
J. Davidson
A. Gupta

title: Interpretable intuitive physics model

authors:
Y. Ye
M. Singh
A. Gupta
S. Tulsiani

title: Compositional video prediction

authors:
Y. Ye
Z. Zhao
Y. Li
L. Chen
J. Xiao
Y. Zhuang

title: Video question answering via attribute-augmented attention network learning

authors:
K. Yi
C. Gan
Y. Li
P. Kohli
J. Wu
A. Torralba
J.B. Tenenbaum

title: Clevrer: Collision events for video representation and reasoning

authors:
K. Yi
J. Wu
C. Gan
A. Torralba
P. Kohli
J. Tenenbaum

title: Neural-symbolic vqa: Disentangling reasoning from vision and language understanding

authors:
A. Zadeh
M. Chan
P.P. Liang
E. Tong
L.-P. Morency

title: Social-iq: A question answering benchmark for artificial social intelligence

authors:
H. Zhou
A. Kadav
F. Lai
A. Niculescu-Mizil
M.R. Min
M. Kapadia
H.P. Graf

title: Hopper: Multi-hop transformer for spatiotemporal reasoning

title: Visual7w: Grounded question answering in images

heading: 1 Introduction

text:
Dynamic visual reasoning about objects, relations, and physics is essential for human intelligence. Given a raw video, humans can easily use their common sense of intuitive physics to explain what has happened, predict what will happen next, and infer what would happen in counterfactual situations. Such human-like physical scene understanding capabilities are also of great importance in practical applications such as industrial robot control [2, 53].
1Project page: http://vrdp.csail.mit.edu/
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Previous works have made great efforts to build artificial intelligence (AI) models with such physical reasoning capabilities. One popular strategy is to develop pure neural-network-based models [63, 19, 40]. These methods typically leverage end-to-end neural networks [32, 35] with powerful attention modules such as Transformer [69, 21] to extract attended features from both video frames and question words, based on which they answer questions directly. Despite their high question-answering accuracy on CLEVRER [79], a challenging dynamic visual question-answering benchmark, these black-box models neither learn concepts nor model objects’ dynamics. Therefore, they lack transparency, interpretability, and generalizability to new concepts and scenarios. Another common approach to dynamic visual reasoning is to build graph neural networks (GNNs) [47] to capture the dynamics of the scenes. These GNN models [54, 79, 16] treat objects in the video as nodes and perform objectand relation-centric updates to predict objects’ dynamics in future or counterfactual scenes. Such systems achieve decent performance with good interpretability on CLEVRER by combining the GNN-based dynamics models with neural-symbolic execution [58, 80]. However, these dynamic models do not explicitly consider laws of physics or use concepts encoded in the question-answer pairs associated with the videos. As a result, they show limitations in counterfactual situations that require long-term dynamics prediction.
Although (graph-)neural-network-based approaches have achieved competitive performance on CLEVRER, dynamic visual reasoning is still far from being solved perfectly. In particular, due to the lack of explicit physics models, existing models [79, 19, 16] typically struggle to reason about future and counterfactual events, especially when training data is limited. For this reason, one appealing alternative is to develop explicit physics-based methods to model and reason about dynamics, as highlighted in the recent development of differentiable physics engines [9, 17, 68, 18, 66] and their applications in robotics [9, 17, 68]. However, these physics engines typically take as input a full description of the scene (e.g., the number of objects and their shapes) which usually requires certain human priors, limiting their availability to applications with well-defined inputs only.
In this work, we take an approach fundamentally different from either network-based methods or physics-based methods. Noting that deep learning based methods excel at parsing objects and learning concepts from videos and language, and physics laws are good at capturing object dynamics, we propose Visual Reasoning with Differentiable Physics (VRDP), a unified framework that combines a visual perception module, a concept learner, and a differentiable physics engine. VRDP jointly learns object trajectories, language concepts, and objects’ physics models to make accurate dynamic predictions. It starts with a perception module running an object detector [31] on individual frames to generate object proposals and connect them into trajectories based on a motion heuristic. Then, a concept learner learns object- and event-based concepts, such as ‘shape’, ‘moving’, and ‘collision’ as in DCL [16, 58]. Based on the obtained object trajectories and attributes, the differentiable physics engine estimates all dynamic and physical properties (e.g., velocity, angular velocity, restitution, mass, and the coefficient of resistance) by comparing the simulated trajectories with the video observations. With these explicit physical parameters, the physics engine reruns the simulation to reason about future motion and causal events, which a program executor then executes to get the answer. The three components of VRDP cooperate seamlessly: the concept learner grounds physical concepts needed by the physics engine like ‘shape’ onto the objects detected by the perception module; the differentiable physics engine estimates all physical parameters and simulates accurate object trajectories, which in turn help the concept learning process in the concept learner.
Compared with existing methods, VRDP has several advantages thanks to its carefully modularized design. First, it achieves the state-of-the-art performance on both synthetic videos (CLEVRER [79]) and real-world videos (Real-Billiard [63]) without sacrificing transparency or interpretability, especially in situations that require long-term dynamics prediction. Second, it has high data efficiency thanks to the differentiable physics engine and symbolic representation. Third, it shows strong generalization capabilities and can capture new concepts with only a few examples.

heading: 2 Related Work

text:
Visual Reasoning Our model is related to reasoning on vision and natural language. Existing works can be generally categorized into two streams as end-to-end approaches [40, 83, 74, 41, 5] and neurosymbolic approaches [80, 29, 30, 24, 4, 59, 45, 58, 36, 3]. The end-to-end methods [40, 83, 74, 60] typically tackle the visual question answering (VQA) problem by designing monolithic multi-modal deep networks [32, 35]. They directly output answers without explicit and interpretable mechanisms.
Beyond question answering, neuro-symbolic methods [80, 29, 59, 45, 58] propose a set of visualreasoning primitives, which manifest as an attention mechanism capable of performing complex reasoning tasks in an explicitly interpretable manner.
Dynamic visual reasoning in videos has attracted much research attention. Many video question answering datasets [27, 49, 42, 67, 81] and the methods [82, 51, 38, 75, 78, 22] built on them mainly focus on understanding diverse visual scenes, such as human actions (MovieQA [67]) or 3D object movements without physical and language cues (CATER [27]). Differently, CLEVRER [79] targets the physical and causal relations grounded in dynamic videos of rigid-body collisions and asks a range of questions that requires the modeling of long-term dynamic predictions. For this reason, we evaluate our method and compare it with other state-of-the-arts on CLEVRER.
Both end-to-end [63, 19] and neuro-symbolic methods [79, 16] have been explored on CLEVRER. However, they either lack transparency or struggle for long-term dynamic prediction. In this paper, we perform high-performance and interpretable reasoning by recovering the physics model of objects and their interactions (e.g., collisions) from visual perception and language concepts.
Physical Models Physical models are widely used in video prediction [50, 23, 76, 77], neural simulation and rendering [52, 54], and dynamic reasoning [10, 71]. For example, PhysNet and its variants [50, 23, 76, 77] leverage global or object-centric deep features to predict the physical motion in video frames. Some other related works [61, 1, 65, 46] extend physical models to predict the effect of forces and infer the geometric attributes and topological relations of cuboids.
In this work, we focus on dynamic visual reasoning about object interactions, dynamics, and physics with question answering, which is central to human intelligence and a key goal of artificial intelligence. Solving such tasks requires a good representation and understanding of physics models. A common choice is to train a deep neural network for physical property estimation (e.g., location and velocity) based on learned visual and dynamic priors [15, 10, 71, 70, 43, 54, 55, 34, 33]. However, since these neural networks do not model physics laws, generalizing them to unseen events or counterfactual scenarios could result in unexpected results. Our work is different and more physics-based: Inspired by the recent advances in differentiable physics [9, 17, 68, 18, 66, 37], we implement an impulsebased differentiable rigid-body simulator and leverage the power of its gradients to infer dynamics information about the scene.
Physical Scene Understanding Our work is also relevant to studies on physical scene understanding [7, 73, 66, 64, 8, 25, 26, 20, 50, 39], most of which propose pure neural-network solutions without explicitly incorporating physics models. Benchmarks like PHYRE [7] study physical understanding and reasoning based on pure videos without concept learning and language inference. Based on such benchmarks, some works [72, 11, 50, 54] learn compositional scene structure and estimate states through physical motions and visual de-animation. Recently, two papers propose pure physics-based methods [44, 39] that make heavy use of differentiable physics simulators, but they typically assume concepts in the scene are given as input. Our work is unique in that we learn video and language concepts from raw videos and infer dynamics information from a differentiable simulator, combining the benefits of both learning and physics.

heading: 3 Method

text: By integrating differentiable physics into the dynamic reasoning framework, VRDP jointly learns visual perception, language concepts, and physical properties of objects. The first two provide prior knowledge for optimizing the third one to reason about the physical world, and the optimized physical properties in turn help to learn better concepts. In the following, we first give an overview of our framework and then describe each of its components in detail.

heading: 3.1 Framework Overview

text:
An overview of VRDP is illustrated in Fig. 1. It contains three components: a visual perception module, a concept learner, and a physics model. The input to the framework is a video and reasoning questions, where the former is processed by the visual perception module to get object trajectories and corresponding visual features, and the latter is parsed into executable symbolic programs with language concepts by the concept learner. Similar to DCL [58, 16], the concept learner first grounds object properties (e.g., color and shape) and event concepts (e.g., collision and moving) by aligning the
visual features and the corresponding concept embeddings in the (explanatory or descriptive) program that does not require dynamic predictions, e.g., “what is the shape of ...”. With those perceptually grounded object trajectories and properties, the physical model then performs differentiable simulation to learn all physical parameters of the scene and objects by comparing the simulated trajectories with the video observations. After that, the physics engine simulates unseen trajectories for predictive and counterfactual scenarios and generates their features, in turn enabling the concept learner to finetune event concepts from the program that requires dynamic predictions, e.g., “what will happen ...” and “what if ...”. Finally, a symbolic executor executes the parsed programs with the dynamic predictions to get the answer.

heading: 3.2 Model Details

text:
Visual Perception Module Given a video with the number of frames T , the visual perception module parses the video frame-by-frame and associate the parsed objects in each frame into object trajectories L = {ln}Nn=1, where ln denotes the object trajectory of the nth object and N is the number of the objects in the video. Specifically, we leverage a pretrained Faster R-CNN [31] as the object detector to get the Region of Interest (ROI) feature ft ∈ RN×D and the object location of objects bt = [x2Dt , y 2D t , x BEV t , y BEV t ] ∈ RN×4 at frame t, where D is the feature dimension, (x2Dt , y 2D t ) denotes the normalized object bounding box center in the image coordinate frame, and (xBEVt , y BEV t ) denotes the projected bird’s-eye view (BEV) location in the BEV coordinate frame using the calibrated camera matrix. Following works [16, 28], we associate object proposals in adjacent frames by thresholding their intersection over union (IoU) and obtain the object trajectory ln = {bnt }Tt=1 for the nth object. The visual perception module then constructs object and interactive representations for concept learning. The object representation Fobj ∈ RN×(D+4T ) contains both appearance-based Fa = avg({ft}Tt=1) and trajectory-based feature Fl = {bt}Tt=1 for modeling static properties and dynamic concepts, respectively, where avg(·) here represents the average ROI feature over time. The interactive feature Fpair ∈ RT×N×N×12S , where S denotes a fixed temporal window size, is built on every
pair of objects. It contains object trajectories {bit} t0+S/2 t0−S/2, {b j t} t0+S/2 t0−S/2 of the objects i and j and their distance {abs(bit − b j t )} t0+S/2 t0−S/2 to model the collision event of the objects at a specific moment t0.
Concept Learner The concept learner grounds the physical and event concepts (e.g., shape and color) as prior knowledge for the physics model from the video representation and language. It first leverages a question parser to translate the input questions and choices into executable neuro-symbolic programs where each language concept in the program is represented by a concept embedding. Similar to [58, 16], this work adopts a seq2seq model [6] with an attention mechanism to translate language into a set of symbolic programs, e.g., retrieving objects with certain colors, getting future or counterfactual events, finding the causes of an event, thus decomposing complex questions into step-by-step dynamic visual reasoning processes. The concept learner assigns each concept in the program (e.g., color, shape, and collision) a randomly initialized embedding e ∈ RC so that the symbolic program can be formulated as differentiable vector operations.
After that, it projects the visual representation into concept embedding spaces and performs Nearest Neighbor (NN) search to quantize concepts for the object attributes and events. We implement the projection through a linear layer P(·) and calculate the cosine similarity between two vectors in the embedding space for NN search. For example, the confidence score of whether the nth object is a cube can be represented by [cos(P(Fna ), ecube)− µ]/σ, where ecube is the embedding of concept ‘cube’, µ and σ are the shifting and scaling scalars, and P(·) maps a D-dimensional visual feature into a C-dimensional vector in this case.
Physics Model The differentiable physics model captures objects’ intrinsic physical properties and makes accurate dynamic predictions for reasoning. With the perceptually grounded object shapes and trajectories from the above two components of VRDP, it performs differentiable simulation to optimize the physical parameters of the scene and objects by comparing the simulation with the video observations L. Our physics model is implemented as an impulse-based differentiable rigid-body simulator [37, 62, 14]. It iteratively simulates a small time step of ∆t based on the objects’ state in the BEV coordinate through inferring collision events, forces (including resistance and collision force), and impulses acting on the object, and updating the state of each object. When an object moves on the ground with velocity −→v and angular velocity ω, we consider three kinds of forces that affect the movement of the object: sliding friction, rolling resistance, and air resistance. We use λ1, λ2, λ3 to denote their coefficients and have:
−→a =
{ − −→v |−→v | (λ1g + λ3|
−→v |2) if the shape is not sphere − −→v |−→v | (λ2g + λ3| −→v |2) if the shape is sphere (1)
where g = 9.81m/s2 is the standard gravity and −→a denotes the acceleration of the object, whose direction is opposite to the velocity. The velocity −→v and the location −→ l′ = (x′, y′) are then updated accordingly by the second order Runge-Kutta (RK2) algorithm [13]. Similarly, the angular velocity ω also decreases at each time step due to the angular drag, and the angle α of the object is updated by the RK2 algorithm.
The physics engine checks whether the boundaries of two objects with radius R are overlapped in the BEV coordinate frame to detect collision events. Based on the fact that the total momentum of an isolated system should be constant in the absence of net external forces, we compute the impulse of collided objects and ignore the friction caused by the collision. Let (m1,m2), (r1, r2), (α1, α2), (−→v1 ,−→v2), ( −→ l′1 , −→ l′2 ) denote the mass, restitution, angle, velocity and BEV location of two collided objects at the moment of the collision, respectively; −→ d1, −→ d2 represent their collision unit directions that the force is acting on, where −→ d1 + −→ d2 = −→ 0 . The change of velocity ∆−→v1 ,∆−→v2 at the moment of collision can be obtained by calculating the impulse on the collision direction: −−→ ∆v1 = −(1 + r1r2)(m2/(m1 +m2))( −→ d1 • ( −→v1 −−→v2)) −→ d1
−−→ ∆v2 = −(1 + r1r2)(m1/(m1 +m2))( −→ d2 • ( −→v2 −−→v1)) −→ d2,
(2)
the velocity −→v is then updated by −→v ← −→v + ∆−→v . Similarly, the angular velocity ω can be updated by ω ← ω + ∆ω, where ∆ω is computed based on conservation of angular momentum. Given an initial state of the scene and objects, our physics engine simulates force, impulse, and collision events and iteratively updates the state of each element. All physical parameters including
R, λ,m, r, α,−→v , −→ l are initialized and then optimized with L-BFGS algorithm [56] by fitting the simulated trajectories L′ = {(x′t, y′t)}Tt=1 into the perceptual trajectories LBEV = {(xBEVt , yBEVt )}Tt=1. To alleviate the difficulty of the optimization, we mark the time frame of each object’s first collision by calculating the BEV distance between every pair and decompose the differentiable physical optimization and simulation into the following steps: 1) Since radius R and resistance coefficients λ are consistent in all videos, we use K videos to jointly learn those physical parameters and fix them for the optimization of other sample-dependent parameters. 2) For each video, we then use the frames before the collision to optimize the collision-independent physical parameters, such as initial velocity −→v0 , initial location −→ l0 , and initial angle α0. 3) With the above parameters learned and fixed, we optimize the remaining collision-dependent parameters, including mass m and resistance r of each object. This process follows the curriculum learning paradigm [12] by optimizing from fewer to more frames, e.g., multi-step optimization on [0, 40], [0, 80], and [0, 128] frames, where the parameters in each step are initialized from the optimization of the previous step. 4) With all parameters of the physical model learned, the engine runs simulations and re-calculates the trajectorybased representations Fl for answering counterfactual, descriptive, and explanatory questions. 5) For the predictive case, we leverage the learned physical model as initialization and re-optimize all sample-dependent parameters with only the last 20 frames to reduce the cumulative error over time.
Symbolic Execution As in [58, 16], we perform reasoning with a program executor, which is a collection of deterministic functional modules designed to realize all logic operations specified in symbolic programs. Its input consists of the parsed programs, learned concept embeddings, and visual representations, including the appearance-based feature Fa from the visual perception module and the updated trajectory feature Fl from the physics engine. Given a set of parsed programs, the program executor runs them step-by-step and derives the answer based on these representations. For example, the ‘counting’ program outputs the number of objects which meet specific conditions (e.g., red sphere). In this process, the executor leverages the concept learner to filter out eligible objects.
Our reasoning process is designed fully differentiable w.r.t. the visual representations and the concept embeddings by representing all object states, events, and results of all operators in a probabilistic manner during training, supporting gradient-based optimization. Moreover, it works seamlessly with our explicit physics engine, which simulates dynamic predictions through real physical parameters, forming a symbolic and deterministic physical reasoning process. The whole reasoning process is fully transparent and step-by-step interpretable.

heading: 3.3 Training Objectives

text:
Similar to [16, 79], we train the program parser with program labels using cross-entropy loss,
Lprogram = − J∑
j=1
1{yp = j} log(pj), (3)
where J is the size of the pre-defined program set, pj is the probability for the j-th program and yp is the ground-truth program label.
We optimize the physical parameters in the physical model by comparing the simulation trajectories with the video observations. All physical parameters includingR, λ,m, r, α,−→v , −→ l are initialized and then optimized with L-BFGS algorithm [56] by fitting the simulated trajectories L′ = {(x′t, y′t)}Tt=1 into the perceptual trajectories LBEV = {(xBEVt , yBEVt )}Tt=1. We have:
LPhysics = ‖L′ − LBEV‖22, (4)
We optimize the feature extractor and the concept embeddings in the concept learner by question answering. We treat each option of a multiple-choice question as an independent boolean question during training. Specifically, we use cross-entropy loss to supervise open-ended questions and use mean square error loss to supervise counting questions. Formally, for counting questions, we have
LQA,count = (ya − z)2, (5) where z is the predicted number and ya is the ground-truth number label. For other open-ended questions and multiple-choice questions, we have
LQA,others = − A∑
a=1
1{ya = a} log(pa), (6)
where A is the size of the pre-defined answer set, pa is the probability for the a-th answer and ya is the ground-truth answer label.

heading: 4 Experiments

text:
By recovering physics models of objects and their interactions from video and language, VRDP enjoys the following benefits: 1) high accuracy and full transparency, 2) superior data efficiency, and 3) high generalizability. In this section, we first evaluate the accuracy and data efficiency of VRDP on the widely used dynamic visual reasoning benchmark CLEVRER [79] and its subsets; we then validate the model’s generalizability on adapting to new concepts with few-shot data; lastly, we experiment on the real-world dataset Real-Billiard [63] to show that VRDP works well in real-world dynamic prediction and reasoning.
Datasets and Evaluation Settings To validate the effectiveness of our method for reasoning about the physical world, we conduct main experiments on the CLEVRER [79] dataset, as it contains both language and physics cues such as rigid body collisions and dynamics, compared to other benchmarks that focus on either action understanding without physical inferring [49, 42] or temporal reasoning without language concepts [27, 8]. CLEVRER includes four types of question answering (QA): descriptive, explanatory, predictive, and counterfactual, where the first two types concern more on video understanding, while the latter two types involve physical dynamics and predictions in reasoning. Therefore, we mainly focus on the predictive and counterfactual questions in this work and use QA accuracy as the evaluation metric. Note the multi-choice question (explanatory, predictive, and counterfactual) contains multiple options. Only if all the options (per opt.) are answered correctly can it be regarded as a correct question (per ques.).
We then collect a few-shot physical reasoning dataset with novel language and physical concepts (e.g., “heavier” and “lighter”), termed generalized CLEVRER, containing 100 videos (split into 25/25/50 for train/validation/test) with 375 options in 158 counterfactual questions. This dataset is supplementary to CLEVRER [79] for generalizing to new concepts with very few samples. For real-world scenarios, we conduct experiments on the Real-Billiard [63] dataset, which contains three-cushion billiards videos captured in real games for dynamics prediction. We generate 6 reasoning questions (e.g., “will one billiard collide with ...?”) for each video and evaluate both the prediction error and QA accuracy.
Implementation Details We follow the experimental setting in [79, 16] using a pre-trained Faster RCNN model [31] to generate object proposals for each frame and training the language program parser with 1,000 programs for all question types. We implement three versions of VRDP models, where our unsupervised VRDP leverage a Slot-Attention model [57] to parse the objects unsupervisedly, while VRDP † use Faster-RCNN [31] as the object detector. In addition to our standard model that grounds object properties from question-answer pairs, we also train a variant (VRDP †‡) on CLEVRER with an explicit rule-based program executor [79] and object attribute supervision. The camera matrix is optimized from 20 training videos. We set ∆t = 0.004s,K = 10, S = 10, and T = 128 for CLEVRER [79] and T = 20 for Real-Billiard [63]. More details of the dataset and settings can be found in Supplemental Materials.

heading: 4.1 Comparative Results on CLEVRER

text:
We conduct experiments on CLEVRER against several counterparts: TVQA+ [49], Memory [22], IEP (V) [45], TbD-net (V) [59], HCRN [48], MAC [40], NS-DR [79], DCL [16], and Object-based Attention [19]. Among them, NS-DR [79] and DCL [16] are high-performance interpretable symbolic models, while Object-based Attention [19] is the state-of-the-art end-to-end method.
From Tab. 1 we observe that: 1) Counterfactual and predictive questions are more difficult than descriptive and explanatory ones as they require accurate physical dynamics and prediction hence our main focus. By reconstructing the physical world explicitly, our method outperforms all existing works on these two types by large margins. For example, VRDP † improves the per question accuracy of counterfactual questions by 11.5% and 79.7% compared to the best end-to-end [19] and neural-symbolic [16] counterparts.
2) The end-to-end model [19] improves the accuracy at the cost of losing model transparency and interpretability. However, by leveraging object attribute supervision and explicit program executors [79], our VRDP †‡ achieves new state-of-the-art overall performance on CLEVRER. It
closes the performance gap between interpretable models and state-of-the-art end-to-end methods. Moreover, it shows the flexibility of our physics model that can be combined with various physical concepts and program executors while achieving impressive performance.
3) We conducted ablative experiments to study the impact of pre-trained object detection modules of our framework by replacing the supervised visual model [31] in VRDP † with an unsupervised one [57] in VRDP. We observe that although the use of unsupervised detectors decreases the performance slightly, our framework still enjoys higher performance than previous methods in counterfactual and predictive questions.
4) Neither the neuro-symbolic nor end-to-end works employ explicit dynamic models with physical meanings. In contrast, our model is fully transparent with step-by-step interpretable programs and meaningful physical parameters powered by a differentiable engine.

heading: 4.2 Detailed Analysis

text:
Evaluation of Data Efficiency We evaluated the data efficiency of VRDP with two representative models: MAC (V) [40] and Object-based Attention [19]. From Fig. 2 we see that: VRDP is highly data-efficient. When the amount of data is reduced, the accuracy of our model drops slightly, while the performance of MAC (V) [40] and Object-based Attention [19] drops drastically due to insufficient data. For example, we improve the counterfactual accuracy of Object-based Attention [19] by 426%
Question: If the blue sphere were much heavier, which of the events that happened would not have happened? Choice: The blue cylinder collides with the cube.
under the setting of 20% data. Notably, our model uses 20% of the dataset to achieve comparable performance to other works that use 80% of data. This is because the components of VRDP, e.g., perception module and question parser, can be trained with a small amount of data. More importantly, our physics model is built based on an explicit physics engine, which can be optimized from the trajectory of a single video.
Evaluation of Generalizability This part studies the generalization capabilities of VRDP against MAC (V) [40] and Object-based Attention [19] on the generalized CLEVRER dataset. Tab. 2 shows our model outperforms other works by a large margin (75.6 vs. 26.7) on per question accuracy, demonstrating our model can quickly learn new concepts from few examples by reconstructing the physics world. An example of generalization with few-shot data is shown in Fig. 3. Our model learns a novel concept “heavier” from only 25 videos and the corresponding question-answer pairs. The simulation is then run with 5 times the mass to answer the question correctly.
Ablation Study on the Learning of Physics Models In this work, sample-independent physical parameters (R, λ) are learned from multiple training videos. In contrast, the sample-dependent parameters, such as m, r, α, v, l, can only be learned with a single video, leading to difficulties in optimization, especially when there are many collisions. This part studies the optimization of these sample-dependent parameters by making comparisons among the following four simplified learning processes on CLEVRER [79]: 1) Baseline – optimize all target parameters directly from all frames simultaneously. 2) Collision-independent First – first use the frames before the collision to optimize collision-independent parameters for each object, including initial velocity −→v0 , initial location −→ l0 , and initial angle α0; then optimize mass m and restitution r from all video frames. 3) Curriculum Optimization – optimizem and r by performing multiple steps on [0, 40], [0, 80], and [0, 128] frames, where each step is initialized from the optimization of the previous step. 4) Re-optimization for Prediction (Full model) – leverage the learned physical parameters as initialization and re-optimize all sample-dependent parameters with the last 20 frames to reduce the cumulative error over time.
Tab. 3 shows that the performance continuously increases when more optimization steps are used, demonstrating the contribution of each part. The “Collision-independent First” rule offers the greatest improvement, especially for counterfactual questions, as counterfactual simulations only rely on the initial state. “Curriculum Optimization” improves all types of questions, and “Re-optimization for Prediction” re-calculates the dynamics of the last 20 frames, thus mainly affect predictive questions.
Failure Analysis VRDP learns the physics model from object trajectories in videos and language concepts in question-answer pairs. It is data-efficient and robust enough to work well when there exists inaccurate perception or incorrect concept learning in some video frames. However, we noticed
that the model might fail in the following cases: 1) If the object collides immediately after entering the image plane, there are insufficient frames before the collision to learn the initial velocity v0. 2) If no collision occurs on an object, its restitution r and mass m cannot be optimized (unknown). We set default values for them. 3) The optimization becomes difficult if there are many cubes and collisions between them in the scene, because cube collisions (considering the sides and corners) are more complicated than sphere and cylinders’. These issues are challenging and will be our future work.

heading: 4.3 Comparative Results on Real-World Billiards

text: We also conduct experiments on the real-world dataset Real-billiard [63] with our supplemented question-answer pairs. Note that the billiard table is a chaotic system, and highly accurate long-term prediction is intractable. Fig. 4 shows an example of the ground truth video and our simulated prediction based on the perceptual grounded physics model. It can be seen that the predicted collision events and trajectories are of good quality. Tab. 4 evaluates the prediction errors under two different rollout timesteps and QA accuracy with 5 competitors: VIN [71], OM [43], CVP [77], IN [10], and CIN [63]. For the prediction task, the rollout timesteps are chosen to be the same (S1= [0, T ]) and twice (S2= [T, 2T ]) as the training time, where the training time T = 20. We refer interested readers to CIN [63] for more details. We find that VRDP is superior to these methods on both prediction and question answering tasks. Moreover, VRDP works well in long-term prediction. It reduces the S2 error on CIN [63] by 62.4%.

heading: 5 Conclusion

text:
This work introduces VRDP, a unified framework that integrates powerful differentiable physics models into dynamic visual reasoning. It contains three mutually beneficial components: a visual perception module, a concept learner, and a differentiable physics engine. The visual perception module parses the input video into object trajectories and visual representations; the concept learner grounds language concepts and object attributes from question-answer pairs and the visual representations; with object trajectories and attributes as prior knowledge, the physics model optimizes all physical parameters of the scene and objects by differentiable simulation. With these explicit physical parameters, the physics model reruns the simulation to reason about future motion and causal events, which are then executed by a symbolic program executor to get the answer. Equipped with the powerful physics model, VRDP is of highly data-efficient and generalizable that adapts to novel concepts quickly with few-shot data. Moreover, both the explicit physics engine and the symbolic executor are step-by-step interpretable, making VRDP fully transparent. Extensive experiments on CLEVRER and Real-Billiards show that VRDP outperforms state-of-the-art dynamic reasoning methods by large margins.
Acknowledgments and Disclosure of Funding
This work was supported by MIT-IBM Watson AI Lab and its member company Nexplore, ONR MURI, DARPA Machine Common Sense program, ONR (N00014-18-1-2847), and Mitsubishi Electric. Ping Luo was supported by the General Research Fund of Hong Kong No.27208720.

title: Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language