summaries: This paper proposes a new type of recurrent neural network architecture called schema / object-file factorization (SCOFF). This model contains multiple weight-sharing GRU cells. The input information is fed into each GRU cells through an attention layer. The output information is fetched from these GRU cells and mixed with another attention layer. The model is tested on several intuitive physics benchmarks and basic reinforcement learning environment. This model demonstrates superior performance than other modular RNN architectures such as RIM on specific tasks.

summaries: The authors propose SCOFF, a novel architectural motif, one with memory, which, as they describe, can serve as a drop-in for an LSTM or GRU within any architecture. It is inspired by the notion that when modeling a structured, dynamic environment (such as one with objects moving around), one must keep track of both declarative knowledge and procedural knowledge. They propose that these two types of knowledge be factored, creating an architecture consisting of "object files" (OF) whose evolution is governed by input, all objects, and  "schemata" which can be selectively applied to each OF.

summaries: The motivation and the proposal for splitting the schema from the procedural (representational) block makes sense. This is a good idea. A the authors build on top of RIMs, which have shown reasonable ways to model dynamical systems. However the paper itself needs to be improved and we need to evaluate the model more before publication.

abstractText: Modeling a structured, dynamic environment like a video game requires keeping track of the objects and their states (declarative knowledge) as well as predicting how objects behave (procedural knowledge). Black-box models with a monolithic hidden state often fail to apply procedural knowledge consistently and uniformly, i.e., they lack systematicity. For example, in a video game, correct prediction of one enemy’s trajectory does not ensure correct prediction of another’s. We address this issue via an architecture that factorizes declarative and procedural knowledge and that imposes modularity within each form of knowledge. The architecture consists of active modules called object files that maintain the state of a single object and invoke passive external knowledge sources called schemata that prescribe state updates. To use a video game as an illustration, two enemies of the same type will share schemata but will have separate object files to encode their distinct state (e.g., health, position). We propose to use attention to determine which object files to update, the selection of schemata, and the propagation of information between object files. The resulting architecture is a drop-in replacement conforming to the same input-output interface as normal recurrent networks (e.g., LSTM, GRU) yet achieves substantially better generalization on environments that have multiple object tokens of the same type, including a challenging intuitive physics benchmark.

authors:
Jacob Andreas
Marcus Rohrbach
Trevor Darrell
Dan Klein

title: Neural module networks

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Léon Bottou
Patrick Gallinari

title: A framework for the cooperation of learning algorithms

venue: In Advances in neural information processing systems,

authors:
Christopher P Burgess
Loic Matthey
Nicholas Watters
Rishabh Kabra
Irina Higgins
Matt Botvinick
Alexander Lerchner. Monet

title: Unsupervised scene decomposition and representation

authors:
Maxime Chevalier-Boisvert
Dzmitry Bahdanau
Salem Lahlou
Lucas Willems
Chitwan Saharia
Thien Huu Nguyen
Yoshua Bengio

title: Babyai: First steps towards grounded language learning with a human in the loop

venue: arXiv preprint arXiv:1810.08272,

authors:
Junyoung Chung
Caglar Gulcehre
KyungHyun Cho
Yoshua Bengio

title: Empirical evaluation of gated recurrent neural networks on sequence modeling

venue: arXiv preprint arXiv:1412.3555,

authors:
Chrisantha Fernando
Dylan Banarse
Charles Blundell
Yori Zwols
David Ha
Andrei A Rusu
Alexander Pritzel
Daan Wierstra

title: Pathnet: Evolution channels gradient descent in super neural networks

venue: arXiv preprint arXiv:1701.08734,

authors:
Anirudh Goyal
Alex Lamb
Jordan Hoffmann
Shagun Sodhani
Sergey Levine
Yoshua Bengio
Bernhard Schölkopf

title: Recurrent independent mechanisms

authors:
Alex Graves
Greg Wayne
Malcolm Reynolds
Tim Harley
Ivo Danihelka
Agnieszka GrabskaBarwińska
Sergio Gómez Colmenarejo
Edward Grefenstette
Tiago Ramalho
John Agapiou

title: Hybrid computing using a neural network with dynamic external memory

authors:
Klaus Greff
Raphaël Lopez Kaufman
Rishabh Kabra
Nick Watters
Christopher Burgess
Daniel Zoran
Loic Matthey
Matthew Botvinick
Alexander Lerchner

title: Multi-object representation learning with iterative variational inference

venue: In International Conference on Machine Learning,

authors:
Mikael Henaff
Jason Weston
Arthur Szlam
Antoine Bordes
Yann LeCun

title: Tracking the world state with recurrent entity networks

venue: arXiv preprint arXiv:1612.03969,

authors:
Sepp Hochreiter
Jürgen Schmidhuber

title: Long short-term memory

venue: Neural computation,

authors:
Robert A Jacobs
Michael I Jordan
Steven J Nowlan
Geoffrey E Hinton

title: Adaptive mixtures of local experts

authors:
Eric Jang
Shixiang Gu
Ben Poole

title: Categorical reparameterization with gumbel-softmax

venue: arXiv preprint arXiv:1611.01144,

authors:
D. Kahneman
A. Treisman
B.J. Gibbs

title: The reviewing of object files: object-specific integration of information

venue: Cognitive psychology,

authors:
Diederik Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Louis Kirsch
Julius Kunze
David Barber

title: Modular networks: Learning to decompose neural computation

authors:
Michael C Mozer
Sreerupa Das

title: A connectionist symbol manipulator that discovers the structure of context-free languages

venue: Advances in Neural Information Processing Systems

authors:
Ronan Riochet
Mario Ynocente Castro
Mathieu Bernard
Adam Lerer
Rob Fergus
Véronique Izard
Emmanuel Dupoux

title: Intphys: A benchmark for visual intuitive physics

authors:
Eric Ronco
Henrik Gollee
Peter J Gawthrop

title: Modular neural networks and self-decomposition

venue: Technical Report CSC-96012,

authors:
Clemens Rosenbaum
Tim Klinger
Matthew Riemer

title: Routing networks: Adaptive selection of non-linear functions for multi-task learning

venue: arXiv preprint arXiv:1711.01239,

authors:
Clemens Rosenbaum
Ignacio Cases
Matthew Riemer
Tim Klinger

title: Routing networks and the challenges of modular and compositional computation

authors:
Noam Shazeer
Azalia Mirhoseini
Krzysztof Maziarz
Andy Davis
Quoc Le
Geoffrey Hinton
Jeff Dean

title: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

venue: arXiv preprint arXiv:1701.06538,

authors:
Sainbayar Sukhbaatar
arthur szlam
Jason Weston
Rob Fergus

title: End-to-end memory networks

authors:
Sjoerd Van Steenkiste
Michael Chang
Klaus Greff
Jürgen Schmidhuber

title: Relational neural expectation maximization: Unsupervised discovery of objects and their interactions

venue: arXiv preprint arXiv:1802.10353,

authors:
Nicholas Watters
Loic Matthey
Christopher P Burgess
Alexander Lerchner

title: Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes

heading: 1 INTRODUCTION

text:
An intelligent agent that interacts with its world must not only perceive objects but must also remember its past experience with these objects. The wicker chair in one’s living room is not just a chair, it is the chair which has an unsteady leg and easily tips. Your keys may not be visible, but you recall placing them on the ledge by the door. The annoying fly buzzing in your left ear is the same fly you saw earlier which landed on the table.
Visual cognition requires a short-term memory that keeps track of an object’s location, properties, and history. In the cognitive science literature, this particular form of state memory is often referred to as an object file (Kahneman et al., 1992), which we’ll abbreviate as OF. An OF serves as a temporally persistent reference to an external object, permitting object constancy and permanence as the object and the viewer move in the world.
Complementary to information in the OF is abstract knowledge about the dynamics and behavior of an object. We refer to this latter type of knowledge as a schema (plural schemata), another term borrowed from the cognitive-science literature. The combination of OFs and schemata is sufficient to predict future states of object-structured environments, critical for planning and goal-seeking behavior.
To model a complex, structured visual environment, multiple OFs must be maintained in parallel. Consider scenes like a PacMan video-game screen in which the ghosts chase the PacMan, a public
01 Mila, University of Montreal, 2 IIT BHU, Varanasi, 3 Waverly, 4 UC Berkeley, 5 Deepmind, 6 Google Research, Brain Team, Corresponding author: anirudhgoyal9119@gmail.com
Object Schema 1 Schema 2 Schema 3
files PacMan Normal Scared
(OFs) Ghost Ghost
Left Frame
A X B X C X D X E X
Right Frame
A X B X C X D X E X
Figure 1: Two successive frames of PacMan, illustrating the factorization of knowledge. Each ghost is represented by a persistent OF (maintaining its location and velocity), but all ghosts operate according to one of two schemata, depending on whether the ghost is in a normal or scared state.
square or sports field in which people interact with one another, or a pool table with rolling and colliding balls. In each of these environments, multiple instances of the same object class are present; all operate according to fundamentally similar dynamics. To ensure systematic modeling of the environment, the same dynamics must be applied to multiple object instances. Toward this goal, we propose a method of separately representing the state of an individual object—via an OF—and how its state evolves over time—via a schema.
Object-oriented programming (OOP) provides a metaphor for thinking about the relationship between OFs and schemata. In OOP, each object is an instantiation of an object class and it has a self-contained collection of variables whose values are specific to that object and methods that operate on all instances of the same class. The relation between objects and methods mirrors the relationship between our OFs and schemata. In both OOP and our view of visual cognition, a key principle is the encapsulation of knowledge: internal details of objects (OFs) are hidden from other objects (OFs), and methods (schemata) are accessible to all and only objects (OFs) to which they are applicable.
The modularity of knowledge in OOP supports human programmers in writing code that is readily debugged, extended, and reused. We conjecture that the corresponding modularity of OFs and schemata will lead to neural-network models with more efficient learning and more robust generalization, thanks to appropriate disentangling and separation of concerns.
Modularity is the guiding principle of the model we propose, which we call SCOFF, an acronym for schema / object-file factorization. Like other neural net models with external memory (e.g., Mozer and Das, 1993; Graves et al., 2016; Sukhbaatar et al., 2015), SCOFF includes a set of slots which are each designed to contain an OF (Figure 2). In contrast to most previous external memory models, the slots are not passive contents waiting to be read or written by an active process, but are dynamic, modular elements that seek information in the environment that is relevant to the object they represent, and when critical information is observed, they update their states, possibly via information provided by other OFs. Event-based OOP is a good metaphor for this active process, where external events can trigger the action of objects.
As Figure 2 suggests, there is a factorization of declarative knowledge—the location, properties, and history of an object, as contained in the OFs—and procedural knowledge—the rules of object behavior, as contained in the schemata. Whereas declarative knowledge can change rapidly, procedural knowledge is more stable over time. This factorization allows any schema to be applied to any OF as
deemed appropriate. The model design ensures systematicity in the operation of a schema, regardless of the slot to which an OF is assigned. Similarly, an OF can access any applicable schema regardless of which slot it sits in. Furthermore, a schema can be applied to multiple OFs at once, and multiple schemata could be applied to an OF (e.g., Figure 1). In OOP, systematicity is similarly achieved by virtue of the fact that the same method can be applied to any object instantiation and that multiple methods exist which can be applied to an object of the appropriate type.
Our key contribution is to demonstrate the feasibility and benefit of factorizing declarative knowledge (the location, properties, and history of an object) and procedural knowledge (the way objects behave). This factorization enforces not only an important form of systematicity, but also of exchangeability: the model behaves exactly the same regardless of the assignment of schemata to schemata-slots and the assignment of objects to OF-slots, i.e., the neural network operates on a set of objects and a set of schemata. With this factorization, we find improved accuracy of next-state prediction models and improved interpretability of learned parameters.

heading: 2 THE SCHEMATA / OBJECT-FILE FACTORIZATION (SCOFF) MODEL

text:
SCOFF, shown in Figure 2, is an architectural backbone that supports the separation of procedural and declarative knowledge about dynamical entities (objects) in an input sequence. The input sequence {x1, . . . ,xt, . . . ,xT }, indexed by time step t is processed by a neural encoder (e.g., a fully convolutional net for images) to obtain a deep embedding, {z1, . . . ,zt, . . . ,zT }, which then drives a network with nf OFs and ns schemata.
OFs are active processing components that maintain and update their internal state. Essentially, an OF is a layer of GRU (Chung et al., 2014) or LSTM (Hochreiter and Schmidhuber, 1997) units with three additional bits of machinery, which we now describe.
1. Our earlier metaphor identifying OFs in SCOFF with objects in OOP is apropos in the sense that OFs are event driven. OFs operate in a temporal loop, continuously awaiting relevant input signals. Relevance is determined by an attention-based soft competition among OFs: the current input serves a key that is matched to a query generated from the state of each OF; based on the goodness of match, each OF is provided with a value extracted from the input.
2. Each OF performs a one-step update of its state layer of GRU or LSTM units, conditioned on the input signal received. The weight parameters needed to perform this update, which we will denote generically as θ, are not—as in a standard GRU or LSTM—internal to the layer but rather are provided externally. Each schema j is nothing more than a set of parameters θj which can be plugged into this layer. SCOFF uses a key-value attention mechanism to perform Gumbel-based hard selection of the appropriate schema (parameters).
3. OFs may pass information to other OFs (analogous to arguments being passed to a method in OOP), again using a soft attention mechanism by which each OF queries all other OFs. Keys provided by the other OFs are matched to the query, and a soft selection of the best matching OFs determines the weighting of values transmitted by the other OFs.
This operation cycle ensures that OFs can update their state in response to both the external input and the internal state comprised of all the OFs’ contents. This updating is an extra wrapper around the ordinary update that takes place in a GRU or LSTM layer. It provides additional flexibility in that (1) The external input is routed to OFs contingent on their internal state, (2) OFs can switch their dynamics from one time step to the next conditioned on their internal state, (3) OFs are modular in that they do not communicate with one another except via state-dependent selective message passing.
OFs are placeholders in that a particular OF has no weight parameter specific to that OF. Instead, the parameters are provided from two sources: either the schemata or a pool of generic parameters shared by the nf OFs. This generic parameter pool is used to implement key-value attention over the input, the schemata, and communication among OFs. The sharing of schemata ensures systematicity; the sharing of the generic parameter pool ensures exchangeability—model behavior is unaffected by the assignment of object instances to OF slots.
2.1 SCOFF SPECIFICS
Algorithm 1 SCOFF model Input: Current sequence element, xt and previous OF state, {ht−1,k| k ∈ {1, . . . , nf}}
Step 1: Process image by position p with fully convolutional net • cp = [CNN(xt)]p • ap = [cp ep] (concatenate encoding of position to CNN output)
Step 2: Soft competition among OFs to select regions of the input to process • qk =W qht−1,k
• sk,p = softmaxk ( qTk κp√
de
) ,where κp = (apW e)T
• zk = ∑ p sk,pvp where vp = apW v ∀ k ∈ {1, . . . , nf}
Step 3: OFs pick the most relevant schema and update • h̃t,k,j = GRUθj (zk,ht−1,k) ∀k ∈ {1, . . . , nf}, j ∈ {1, . . . , ns} • q̃k = ht−1,kW̃ q
• ik = argmaxj ( q̃Tk κ̃k,j + γ ) , where κ̃k,j = (h̃t,k,jW̃ e)T and γ ∼ Gumbel(0, 1)
• ht,k = h̃t,k,ik
Step 4: Soft competition among OFs to transmit relevant information to each OF • q̂k = ht−1,kŴ
q ∀k{1, . . . , nf} • sk,k′ = softmaxk′ ( q̂Tk κ̂k′√
de
) where κ̂k′ = (ht,k′Ŵ e)T ∀ k, k′ ∈ {1, . . . , nf}
• ht,k ← ht,k + ∑ k′ sk,k′ v̂k′ where v̂k′ = ht,k′Ŵ v ∀k ∈ {1, . . . , nf}
Algorithm 1 provides a precise specification of SCOFF broken into four steps. Step 1 is external to SCOFF and involves processing an image input to obtain a deep embedding. The processing is performed by a fully Convolutional Neural Net (CNN) that preserves positional information (typically 64× 64 in our simulations), and for each position p encodes the processed input cp and concatenates a learned encoding of position, ep to produce a position-specific hidden state. (SCOFF can also handle non-visual inputs, such as vectors or discrete tokens; in this case the CNN is replaced by an MLP.) The subsequent core steps are as follows.
Step 2: Soft competition among OFs to select regions of the input to process. The state of each OF k, ht−1,k, is used to form a query, qk, which determines the input positions that it will attend to. The query is matched to a set of position-specific input keys, κk,p for position p, producing a position-specific match score, sk,p. Soft position-specific competition among the OFs results in the OFs selecting complementary image regions to attend to. The contents of the attended positions are combined yielding an OF-specific input encoding, zk.
Step 3: OFs pick the most relevant schema and update. OF k picks one schema via attention as follows. OF k binds to each schema j, and then performs a hypothetical update, yielding h̃t,k,j . In experiments below, the OF state is maintained by a GRU layer, and schema j is a parameterization of the GRU, denoted GRUθj , which determines the update. The previous state of the OF, ht−1,k serves as a query in key-value attention against a key derived from the hypothetical updated state, h̃t,k,j . The schema ik corresponding to the best query-key match for OF k is used to update OF k’s state. Selection is based on the straight-through Gumbel-softmax method (Jang et al., 2016), which makes a hard choice during forward propagation and during backward propagation, it considers a softened version of the output to permit gradient propagation to non-selected schemata.
Step 4: Soft competition among OFs to transmit relevant information to each OF. This step allows for interactions among OFs. Each OF k queries other OFs for information relevant to its update as follows. The OF’s previous state, ht−1,k, is used to form a query, q̂k, in an attention mechanism against a key derived from the new state of each other OF k′, κ̂k′ , and softmax selection (sk,k′) is used to obtain weighted information from other OFs, v̂′k, into OF k’s state.
Number of Parameters. SCOFF can be used as a drop-in replacement for a LSTM/GRU layer. There is a subtlety that must be considered for successful integration. If the total size of the hidden state is kept the same, integrating SCOFF dramatically reduces the total number of recurrent parameters in
the model because of its block-sparse structure. The majority of SCOFF parameters are the schemata, {θj |j ∈ {1, . . . , ns}}. The remaining parameters are those of query (W q, W̃ q, Ŵ q), key (W e, W̃ e, Ŵ e), and value W v, Ŵ v) functions. Note that these linear functions could be replaced by nonlinear functions. Its also interesting to note that SCOFF actually has far fewer parameters than other modular architectures (e.g., RIMs, Recurrent Entity Networks, Neural Module Networks) when ns < nf (which holds true for most of our experiments), because of the potential one-to-many mapping between schemata and OFs. Optionally, at Step 2, during training instead of activating all the OFs, we can use a sparse attention, to only activate a subset of OFs that are relevant at that time step t, and only update the state of the activated OFs. We note that this is not specific to the proposed method, but the performance of the method can be improved by only selectively activating the relevant OFs. We also note that the different schemata in the proposed method are only used to update the state of OFsand hence is agnostic as to how one obtains a set of OFs. Any other method for entity extraction (Burgess et al., 2019; Greff et al., 2019) can be used for obtaining a set of OFs. Similarly, other methods for decoding which preserves equivariance (Watters et al., 2019) among different OFscan be used for decoding the contents of different OFs.

heading: 3 RELATED WORK

text:
CNNs. SCOFF applies the same knowledge (schemata) to multiple OFs, yielding systematicity. Similarly, a CNN is a highly restrictive instantiation of this same notion, where knowledge—in the form of a convolutional filter—is applied uniformly to every location in an image, yielding equivariance. SCOFF is a more flexible architecture than a CNN in that OFs are defined by abstract notion of objects not a physical patch of an image, and schemata are applied dynamically and flexibly over time, not in a fixed, rigid manner as the filters in a CNN.
Memory Networks. A variety of existing models leverage an external slot-based, content-addressible memory (e.g., Graves et al., 2016; Sukhbaatar et al., 2015). The memory slots are passive elements that are processed by a differentiable neural controller such as an LSTM. Whereas traditional memory networks have many dumb memory cells and one smart controller, SCOFF has many smart memory cells—the OFs—which also act as local controllers. However, SCOFF shares the notion with memory networks that the same knowledge is applied systematically to every cell.
Relational RNN (RMC). The RMC (Santoro et al., 2018) has a multi-head attention mechanism which allows it to share information between multiple memory locations. RMC is like Memory Networks in that dynamics are driven by a central controller: memory is used to condition the dynamics of an RNN.
Recurrent Entity Networks. Henaff et al. (2016) describe a collection of recurrent modules that update independently and in parallel in response to each input in a sequence. The module outputs are integrated to form a response. It shares a modular architecture with SCOFF, but the modules have a fixed function and do not directly communicate with one another. This earlier work focused on language not images.
Recurrent Independent Mechanisms (RIMs). RIMs (Goyal et al., 2019) are a key inspiration for our work. RIMs are a modular neural architecture consisting of an ensemble of dynamical components which have sparse interactions through the bottleneck of attention. Each RIM module has at its core an LSTM layer (Hochreiter and Schmidhuber, 1997). A RIM module is much like our OF. Both are meant to be dynamical entities with a time-evolving state. However, in contrast to OFs in SCOFF, RIM modules operate according to fixed dynamics and each RIM module is specialized for a particular computation. RIM modules are thus not interchangeable.
Neural Module Networks. A modular module network (Jacobs et al., 1991; Bottou and Gallinari, 1991; Ronco et al., 1997; Reed and De Freitas, 2015; Andreas et al., 2016; Rosenbaum et al., 2017; Fernando et al., 2017; Shazeer et al., 2017; Kirsch et al., 2018; Rosenbaum et al., 2019) has an architecture which is composed dynamically from several neural modules, where each module is meant to perform a distinct function. Each module has a fixed function, unlike our OFs, and modules are applied one at a time, in contrast to our OFs, which can update and be used for prediction in parallel.

heading: 4 METHODOLOGY

text:
SCOFF is a drop-in replacement for a standard LSTM or GRU layer, conforming to the same inputoutput interface. Because of their interchangeability, we compare SCOFF to LSTM and GRUs. We also compare SCOFF to two alternative modular architectures: RMC, a memory based relational recurrent model with attention between memory elements and hidden states (Santoro et al., 2018), and Recurrent Independent Mechanisms (RIMs), a modular memory based on a single layered recurrent model with attention modulated input and communication between modules (Goyal et al., 2019).
In all simulations we present, the input is a video sequence, each frame of which is preprocessed by a CNN backbone. We consider two task types: video prediction and reinforcement learning (RL). For video prediction, the simulation output is a prediction of the next frame in the sequence. For RL, the output is an action choice. In both cases, SCOFF’s internal state is mapped to an output in the following manner. The state of each OF is remapped by a siamese network that transforms the state. (By siamese network, we mean that every OF is remapped by the same function.) The transformed state is concatenated and used by an attention-based mechanism operating over the OF slots to ensure exchangeability of the OFs. In the case of video prediction, a deconvolutional backbone yields an image as output; in the case of reinforcement learning, the output is a distribution over actions.
The heart of the model is a single SCOFF layer, consisting of nf OFs and ns schemata. Most simulation details are contained in section A of the Appendix, but we summarize some key points here. Unless otherwise indicated, we always train in an end-to-end fashing using the Adam (Kingma and Ba, 2014) optimizer with a learning rate of 0.0001 and momentum of 0.9. As a default, we use nf = 6 and ns = 4, except where we are specifically exploring the effects of manipulating these hyperparameters. We include more experimental results in the Appendix, and we will release the code.

heading: 5 EXPERIMENTS

text:
Our experiments address the following questions. (1) Does SCOFF successfully factorize knowledge into OFs and schemata? (2) Do the learned schemata have semantically meaningful interpretations? (3) Is the factorization of knowledge into object files and schemata helpful in downstream tasks? (4) Does SCOFF outperform state-of-the-art approaches, both modular and non-modular, which lack SCOFF’s strong inductive bias toward systematicity and knowledge factorization?
We begin with a series of experiments involving greyscale synthetic video sequences consisting of a single ball-like object moving over time according to switchable dynamics. We model this scenario with SCOFF having a single (nf = 1) OF and the number of schemata matching the number of different types of dynamics.
Single object with fixed dynamics. To demonstrate that the proposed model is able to factorize knowledge, we consider video scenes in which a single object, starting in a random location, has dynamics that cause it to either (a) accelerate in a particular direction, (b) move at a constant velocity in a particular direction, or (c) take a random walk with a constant velocity. Following training, SCOFF can predict trajectories after being shown the first few frames. It does so by activating a schema that has a one-to-one correspondence with the three types of dynamics (Figure 3, left panel), leading
Ball
Two Visual Markers
Horizontal dynamics (correlates with right visual marker)
Vertical dynamics (correlates with left visual marker)
Tw o
vi su
al m
ar ke
rs
Ba ll Figure 4: Switching dynamics task.
Sequence of steps showing horizontal (top row) and vertical (bottom row) dynamics of a ball. Frames contain visual markers that indicate the current dynamics.
to interpretable semantics and a clean factorization of knowledge. For additional details, refer to Appendix C.
Single object with switching dynamics. In this experiment, video scenes contain a single object, starting in a random location, operating according to one of two dynamics: vertical and horizontal motion. In contrast to the previous experiment, the object can switch dynamics in the course of a sequence. The current dynamics are indicated by markers in the image (see Figure 4) which SCOFF must learn to detect in order to select the corresponding schema. In this experiment, we find that SCOFF is able to learn a one-to-one correspondence with two types of dynamics, yielding high-precision prediction of the next frame, even on the first frame after a switch.
Single object with switching dynamics in an RL paradigm. In this experiment, we use the partiallyobservable GotoObjMaze environment from Chevalier-Boisvert et al. (2018), in which an agent must learn to navigate to navigate a 2D multi-room grid world to locate distinct objects such as a key. The world consists of 6 × 6 square rooms that are randomly connected through doors to form a 3× 3 maze of rooms. The agent’s view of the world is an egocentric 5× 5 patch. Our experiment involves ns = 4 schemata whose activation pattern over time steps is shown in the right panel of Figure 3. The schema activation pattern is interpretable; for example, schema 4 is triggered when the ‘key’ is in the agent’s field of view (see Appendix, Figure 13 and Section D). To obtain quantitative evaluations, we test transfer by increasing the room size to 10× 10 during testing. SCOFF is able to successfully reach its goal on 82% of trials, whereas a GRU baseline succeeds only 56% of trials. Even an overparameterized GRU, which matches SCOFF in number of free parameters, still succeeds less often—on 74% of trials. All models are trained for an equal number of time steps. To summarize, the factorization of knowledge in SCOFF leads not only to better next-step prediction but also improve performance on a downstream control task.
Multiple objects with multiple dynamics. We now turn from single-object sequences to sequences involving multiple objects which operate according complex dynamics. We consider a bouncing-balls environment in which multiple balls move with billiard-ball dynamics (Van Steenkiste et al., 2018). The dataset consists of 50,000 training examples and 10,000 test examples showing ∼50 frames of either 4 solid same-color balls bouncing in a confined square geometry with different masses corresponding to their radii (4Balls), 6-8 same-color balls bouncing in a confined geometry (678Balls), 3 same-color balls bouncing in a confined geometry with a central occluder (Curtain), or balls of four different colors (Colored 678Balls). Although each ball has a distinct state (position, velocity, and possibly color), they share the same underlying dynamics of motion and collision. We expected that SCOFF would learn to dissect these dynamics, e.g., by treating collisions, straight-line motion, interactions with the walls, and moving behind an occluder as distinct schemata. We use the encoder and decoder architecture of (Van Steenkiste et al., 2018) for all models. SCOFF has nf = 4 for 4Balls and Curtain, and nf = 8 for 678Balls and Coloured678Balls, and ns = 4 for all simulations. All models are trained for 100 epochs. As shown in Figure 5, SCOFF achieves dramatic improvements in successfully predicting 10- and 30-step iterated predictions relative to GRUs, and RIMs. (For further details, see Appendix Section C.) We omitted RMC (Santoro et al., 2018) from Figures 5a-d because RIMs performs strictly better than RMC in these environments (Goyal et al., 2019), but have included RMC in the Appendix.
Increasing the number of schemata. Our previous experiment used ns = 4 for all simulations. In order to study what happens if SCOFF has access to a large number of schemata, we perform an experiment with ns = 10. For the curtain task, we found that only three schemata were being used, and performance was about the same as when training with ns = 3 or ns = 4. For 678Balls, we found that by the end of training only four schemas were being used, and performance is about the same as when training with ns = 4. Thus, providing SCOFF with excess resources leads to some not being used which is a waste of computation and memory but does not lead to suboptimal performance.
For example, one might have been concerned about overfitting with too many resources. However, with ns = 3 schemata, we never observe unused schemata, suggesting that the model does not have difficulty using the resources we provide to it. That is, there are no ‘dead’ schemata that fail to be trained due to local optima.
Modeling physical laws in a multi-object environment. Modeling a physical system, such as objects in a world obeying laws of gravity and momentum, requires factorization of state (object position, velocity) and dynamics. All objects must obey the same laws while each object must maintain its distinct state. We used the Intuitive Physics Benchmark (Riochet et al., 2019) in which balls roll behind a brick wall such that they are briefly occluded. The training data is constructed such that the examples are all physically realistic. The test set consists of both sequences that behave according to the laws used to synthesize the training data and sequences that follow unrealistic physical laws. We test three forms of unrealistic physics: balls disappearing behind the wall (O1 task), balls having their shape change for no reason (O2 task), and balls teleporting (O3 task). The Benchmark has three subsets of experiments, and we chose the challenging subset with significant occlusions. We trained models to perform next state prediction on the training set and we use model likelihood (Riochet et al., 2019) to discriminate between realistic and unnatural test sequences. Further details are in Appendix E. As Table 1 indicates, SCOFF significantly outperforms two competitors on all three tasks.

heading: 6 CONCLUSIONS

text: Understanding the visual world requires interpreting images in terms of distinct independent physical entities. These entities have persistent intrinsic properties, such as a color or velocity, and they have dynamics that transform the properties. We explored a mechanism that is able to factorize declarative knowledge (the properties) and procedural knowledge (the dynamics). Using attention, our SCOFF model learns this factorization into representations of entities—OFs—and representations of how they transform over time—schemata. By applying the same schemata to multiple OFs, SCOFF achieves systematicity of prediction, resulting in significantly improved generalization performance over state-of-the-art methods. It also addresses a fundamental issue in AI and cognitive science: the distinction between types and tokens. SCOFF is also interpretable, in that we can identify the binding between schemata and entity behavior. The factorization of declarative and procedural knowledge has broad applicability to a wide variety of challenging deep learning prediction tasks.

heading: B Adding Task 11

heading: C Bouncing Ball 12

heading: D BabyAI: Reinforcement Learning 12

text: E Intuitive Physics 13

heading: F Switching Dynamics 13

text: F.1 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 F.2 The factorisation of dynamics into schemata . . . . . . . . . . . . . . . . . . . 15

heading: G Pseudocode for SCOFF Algorithm 24

text:
A IMPLEMENTATION DETAILS AND HYPERPARAMETERS
The model setup consists of three main components: an encoder, the process of interaction between object files and schemata followed by a decoder. The images are first processed by an encoder, which is parameterized as a CNN. For experiments containing multiple entities, we use a spatial attention (i.e., output of the CNN preserves the spatial information) such that different OFs can attend to different spatial regions.
Resources Used. It takes about 2 days to train the proposed model on bouncing ball task for 100 epochs on V100 (32G). We did not do any hyper-parameter search specific to a particular dataset (i.e 4Balls or 678Balls or Curtain Task). We ran the proposed model for different number of schemata (i.e 2/4/6). Similarly, it takes about 3 days to run for 20M steps for the Reinforcement learning task.

heading: B ADDING TASK

text:
We analyzed the proposed method on the adding task. This is a standard task for investigating recurrent models (Hochreiter and Schmidhuber, 1997). The input consists of two co-occuring sequences: 1) N numbers (a0 · · · aN−1) sampled independently from U [0, 1], 2) an index i0 in the first half of the sequence, and an index i1 in the second half of the sequence together encoder as a one hot sequences. The target output is ai0 + ai1 . As shown in figure 6 (a), we can clearly observe the factorisation of procedural knowledge into two schemata effectively, one schema is triggered when an operand is encountered and the other when non-operand is encountered.
Generalization Result: For demonstrating the generalization capability of SCOFF, we consider a scenario where the models are trained to add a mixture of two and four numbers from sequences of length 50. They are evaluated on adding variable number (2-10) of numbers on 200 length sequences. As shown in Table 1, we note better generalization when using SCOFF. The dataset consists of 50,000 training sequences and 20,000 testing sequences for each different number of numbers to add.
All the models are trained for 100 epochs with a learning rate of 0.001 using the Adam optimizer. We use 300 as the hidden dimension for both the LSTM baseline and the LSTM’s in RIMS, SCOFF. Table 2 lists the different hyperparameters used for training SCOFF.

heading: C BOUNCING BALL

text:
The dataset consists of 50,000 training examples and 10,000 test examples showing ∼50 frames of either 4 solid balls bouncing in a confined square geometry (4Balls), 6-8 balls bouncing in a confined geometry (678Balls), 3 balls bouncing in a confined geometry with an occluded region (Curtain), or balls of different colors (Colored 678Balls). The balls have distinct states (and hence distinct object files) but share underlying procedures (schema), which we aim to capture using SCOFF.
We trained baselines as well as proposed model for about 100 epochs. . We also provide the rollouts predicted by the models in the Figures 8, 10, 9, 11, 12.
As part of future work, We also provide a scenario where instead of activating all the OF only a subset of the OF that are most relevant to the input based on attention scores are activated. We denote the original model as SCOFF-ab, and the ablation of only activating a subset of the OF as SCOFF.
Comparison to the RMC baseline is shown in fig. 7.

heading: D BABYAI: REINFORCEMENT LEARNING

text:
We use the GotoObjMaze environment (i.e MiniGrid-GotoObjMaze-v0) from (Chevalier-Boisvert et al., 2018). Here, the agent has to navigate to an object, the object may be in another room. We use exactly the same RL setup as in (Chevalier-Boisvert et al., 2018) except we extend the setup in BabyAI to only apply RGB images of the world rather than symbolic state representations, and hence making the task much more difficult. Hyper-parameters for this task are listed in Tab. 5.
In this environment, the agent is expected to navigate a 3x3 maze of 6x6 rooms, randomly interconnected by doors to find an object like "key". Here we use only one object file, but different number
of schemas (4 in this example). If we look at the object files (vs) schemata, schemata 4 is being triggered when the "key" is in agent’s view as shown in fig. 13.
E INTUITIVE PHYSICS
We use the similar training setup as (Riochet et al., 2019). Hyper-parameters related to the proposed method are listed in Tab. 6.

heading: F SWITCHING DYNAMICS

text:
We consider a scenario where a ball follows one of the two dynamics, horizontal and vertical oscillations at a given time. We limit the number of switches between the dynamics followed to be one. An example ground truth trajectory is given in figure 14. Here we run another experiment, where we have more number of OFs as compared to number of entities or objects, and then we investigate if we are still able to have factorization of different dynamics in different schemata.
Table 4: Hyperparameters for the bouncing balls task
Parameter Value Number of object files (nf ) 4 Number of schemata (ns) 2/4/6 Size of Hidden state of object file 100 Optimizer Adam(Kingma and Ba, 2014) learning rate 1 · 10−4 batch size 64 Inp keys 64 Inp Values 100 Inp Heads 1 Inp Dropout 0.1 Comm keys 32 Comm Values 32 Comm heads 4 Comm Dropout 0.1
Table 5: Hyperparameters for BabyAI
Parameter Value Number of object files (nf ) 1 Number of schemata (ns) 2/4/6 Size of Hidden state of object file 510 Optimizer Adam(Kingma and Ba, 2014) Learning rate 3 · 10−4 Inp keys 64 Inp Values 256 Inp Heads 4 Inp Dropout 0.1 Comm keys 16 Comm Values 32 Comm heads 4 Comm Dropout 0.1
F.1 EXPERIMENT SETUP
The dataset consists of 10,000 trajectories of 51 length with the switching between dynamics happening at the middle of the trajectory.
We use the same architecture for encoder as well as decoder as in (Van Steenkiste et al., 2018). At each time step, we give the last five frames stacked across the channels as input to the encoder.
F.2 THE FACTORISATION OF DYNAMICS INTO SCHEMATA
As shown in figure 15, SCOFF is effective in factorising the two different dynamics into two different schemata, even if the order of dynamics followed is different.
Timesteps along a trajectory
T=0 T=5 T=15T=10
Sc he
m at
a
(a) Object Files (nf = 1) (vs) Schemata (ns = 4) Activation

heading: G PSEUDOCODE FOR SCOFF ALGORITHM


summaries: This paper introduces the Visio-Linguistic Neural Module Network (VilNMN) consisting of a pipeline of dialogue and video understanding neural modules. Motivated by Hu et al. (2017), Kottur et al (2017), this paper extends the NMNs on video tasks for interpretable neural models. The model explicitly resolves entity references (dialog understanding) and detects actions from videos (video understanding) for response generation. Experiments show that NMNs achieve competitive results on AVSD (video-dialog) and TGIF-QA (video-QA) benchmarks.

summaries: This paper studies the language grounding aspect of video-language problems. It proposes a Neural Module Network (NMN) for explicit reasoning of visually-grounded object/action entities and their relationships. The proposed method is demonstrated to be somewhat effective in the audio-visual dialogue task and has been shown superior to existing works on video QA. Overall, the paper is motivated clearly and is delivered with good clarity. The followings need to be clarified.

summaries: The paper studies the application of neural module network to video-grounded language tasks. They propose a method dubbed Visio-Linguistic Neural Module Network (VilNMN) to retrieve spatio-temporal information in a video through a linguistic-based parsed program. In particular, VilNMN first extracts entity references and their corresponding actions in linguistic cues. This information is then being used to locate relevant information in the visual cue to arrive at the correct answer. The proposed method is evaluated on two large scales benchmarks AVSD and TGIF-QA, demonstrating competitive performance with state-of-the-art methods.

abstractText: Neural module networks (NMN) have achieved success in image-grounded tasks such as Visual Question Answering (VQA) on synthetic images. However, very limited work on NMN has been studied in the video-grounded language tasks. These tasks extend the complexity of traditional visual tasks with the additional visual temporal variance. Motivated by recent NMN approaches on image-grounded tasks, we introduce Visio-Linguistic Neural Module Network (VilNMN) to model the information retrieval process in video-grounded language tasks as a pipeline of neural modules. VilNMN first decomposes all language components to explicitly resolve any entity references and detect corresponding action-based inputs from the question. The detected entities and actions are used as parameters to instantiate neural module networks and extract visual cues from the video. Our experiments show that VilNMN can achieve promising performance on two video-grounded language tasks: video QA and video-grounded dialogues.

authors:
Aishwarya Agrawal
Dhruv Batra
Devi Parikh

title: Analyzing the behavior of visual question answering models

venue: In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,

authors:
Huda Alamri
Vincent Cartillier
Abhishek Das
Jue Wang
Stefan Lee
Peter Anderson
Irfan Essa
Devi Parikh
Dhruv Batra
Anoop Cherian
Tim K. Marks
Chiori Hori

title: Audio-visual sceneaware dialog

authors:
Jacob Andreas
Marcus Rohrbach
Trevor Darrell
Dan Klein

title: Learning to compose neural networks for question answering

venue: In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,

title: Neural module networks

authors:
Stanislaw Antol
Aishwarya Agrawal
Jiasen Lu
Margaret Mitchell
Dhruv Batra
C Lawrence Zitnick
Devi Parikh

title: Vqa: Visual question answering

venue: In Proceedings of the IEEE international conference on computer vision,

authors:
Dzmitry Bahdanau
Kyunghyun Cho
Yoshua Bengio

title: Neural machine translation by jointly learning to align and translate

venue: In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations,

authors:
Satanjeev Banerjee
Alon Lavie

title: Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization

authors:
Joao Carreira
Andrew Zisserman

title: Quo vadis, action recognition? a new model and the kinetics dataset

venue: In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Yun-Wei Chu
Kuan-Yen Lin
Chao-Chun Hsu
Lun-Wei Ku

title: Multi-step joint-modality attention network for scene-aware dialogue system

venue: DSTC Workshop @ AAAI,

authors:
Kevin Clark
Christopher D. Manning

title: Deep reinforcement learning for mention-ranking coreference models

authors:
Mostafa Dehghani
Stephan Gouws
Oriol Vinyals
Jakob Uszkoreit
Lukasz Kaiser

title: Universal transformers

venue: In International Conference on Learning Representations,

authors:
Chenyou Fan
Xiaofan Zhang
Shu Zhang
Wensheng Wang
Chi Zhang
Heng Huang

title: Heterogeneous memory enhanced multimodal attention model for video question answering

authors:
Shi Feng
Eric Wallace
Alvin Grissom II
Mohit Iyyer
Pedro Rodriguez
Jordan Boyd-Graber

title: Pathologies of neural models make interpretations difficult

venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,

authors:
Akira Fukui
Dong Huk Park
Daylen Yang
Anna Rohrbach
Trevor Darrell
Marcus Rohrbach

title: Multimodal compact bilinear pooling for visual question answering and visual grounding

authors:
Jiyang Gao
Runzhou Ge
Kan Chen
Ram Nevatia

title: Motion-appearance co-memory networks for video question answering

authors:
Lianli Gao
Pengpeng Zeng
Jingkuan Song
Yuan-Fang Li
Wu Liu
Tao Mei
Heng Tao Shen

title: Structured two-stream attention network for video question answering

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Xavier Glorot
Yoshua Bengio

title: Understanding the difficulty of training deep feedforward neural networks

venue: In Proceedings of the thirteenth international conference on artificial intelligence and statistics,

authors:
Yash Goyal
Tejas Khot
Douglas Summers-Stay
Dhruv Batra
Devi Parikh

title: Making the v in vqa matter: Elevating the role of image understanding in visual question answering

venue: In CVPR,

authors:
Nitish Gupta
Kevin Lin
Dan Roth
Sameer Singh
Matt Gardner

title: Neural module networks for reasoning over text

authors:
Chi Han
Jiayuan Mao
Chuang Gan
Josh Tenenbaum
Jiajun Wu

title: Visual concept-metaconcept learning

venue: In Advances in Neural Information Processing Systems,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Shawn Hershey
Sourish Chaudhuri
Daniel PW Ellis
Jort F Gemmeke
Aren Jansen
R Channing Moore
Manoj Plakal
Devin Platt
Rif A Saurous
Bryan Seybold

title: Cnn architectures for large-scale audio classification

venue: In Acoustics, Speech and Signal Processing (ICASSP),

authors:
C. Hori
H. Alamri
J. Wang
G. Wichern
T. Hori
A. Cherian
T.K. Marks
V. Cartillier
R.G. Lopes
A. Das
I. Essa
D. Batra
D. Parikh

title: End-to-end audio visual scene-aware dialog using multimodal attention-based video features

venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),

authors:
Chiori Hori
Anoop Cherian
Tim K Marks
Takaaki Hori

title: Joint student-teacher learning for audio-visual scene-aware dialog

venue: Proc. Interspeech 2019,

authors:
Ronghang Hu
Huazhe Xu
Marcus Rohrbach
Jiashi Feng
Kate Saenko
Trevor Darrell

title: Natural language object retrieval

authors:
Ronghang Hu
Jacob Andreas
Marcus Rohrbach
Trevor Darrell
Kate Saenko

title: Learning to reason: End-to-end module networks for visual question answering

venue: In Proceedings of the IEEE International Conference on Computer Vision,

authors:
Ronghang Hu
Jacob Andreas
Trevor Darrell
Kate Saenko

title: Explainable neural computation via stack neural module networks

venue: In Proceedings of the European conference on computer vision (ECCV),

authors:
Deng Huang
Peihao Chen
Runhao Zeng
Qing Du
Mingkui Tan
Chuang Gan

title: Location-aware graph convolutional networks for video question answering

venue: In The AAAI Conference on Artificial Intelligence (AAAI),

authors:
Yunseok Jang
Yale Song
Youngjae Yu
Youngjin Kim
Gunhee Kim

title: Tgif-qa: Toward spatiotemporal reasoning in visual question answering

authors:
Robin Jia
Percy Liang

title: Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

authors:
Jianwen Jiang
Ziqiang Chen
Haojie Lin
Xibin Zhao
Yue Gao

title: Divide and conquer: Questionguided spatio-temporal contextual attention for video question answering

authors:
Pin Jiang
Yahong Han

title: Reasoning with heterogeneous graph alignment for video question answering

authors:
Yichen Jiang
Mohit Bansal

title: Self-assembling modular networks for interpretable multi-hop reasoning

venue: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP),

authors:
Justin Johnson
Bharath Hariharan
Laurens van der Maaten
Li Fei-Fei
C Lawrence Zitnick
Ross Girshick

title: Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

authors:
Justin Johnson
Bharath Hariharan
Laurens Van Der Maaten
Judy Hoffman
Li Fei-Fei
C Lawrence Zitnick
Ross Girshick

title: Inferring and executing programs for visual reasoning

venue: In Proceedings of the IEEE International Conference on Computer Vision, pp. 2989–2998,

title: Convolutional neural networks for sentence classification

venue: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP),

authors:
Diederick P Kingma
Jimmy Ba

venue: In International Conference on Learning Representations (ICLR),

authors:
Satwik Kottur
José MF Moura
Devi Parikh
Dhruv Batra
Marcus Rohrbach

title: Visual coreference resolution in visual dialog using neural module networks

venue: In Proceedings of the European Conference on Computer Vision (ECCV),

authors:
Shachi H Kumar
Eda Okur
Saurav Sahay
Jonathan Huang
Lama Nachman

title: Leveraging topics and audio features with multimodal attention for audio visual scene-aware dialog

authors:
Hung Le
Steven C.H. Hoi

title: Video-grounded dialogues with pretrained generation language models

venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5842–5848,

authors:
Hung Le
S Hoi
Doyen Sahoo
N Chen

title: End-to-end multimodal dialog systems with hierarchical multimodal attention on video features

venue: In DSTC7 at AAAI2019 workshop,

authors:
Hung Le
Doyen Sahoo
Nancy Chen
Steven Hoi

title: Multimodal transformer networks for end-to-end video-grounded dialogue systems

authors:
Thao Minh Le
Vuong Le
Svetha Venkatesh
Truyen Tran

title: Learning to reason with relational video representation for question answering

venue: arXiv preprint arXiv:1907.04553,

title: Hierarchical conditional relation networks for video question answering

venue: arXiv preprint arXiv:2002.10698,

authors:
Hwanhee Lee
Seunghyun Yoon
Franck Dernoncourt
Doo Soon Kim
Trung Bui
Kyomin Jung

title: Dstc8-avsd: Multimodal semantic transformer network with retrieval style word generator

venue: DSTC Workshop

authors:
Chenyi Lei
Lei Wu
Dong Liu
Zhao Li
Guoxin Wang
Haihong Tang
Houqiang Li

title: Multiquestion learning for visual question answering

authors:
Jie Lei
Licheng Yu
Mohit Bansal
Tamara Berg

title: TVQA: Localized, compositional video question answering

authors:
Jiwei Li
Michel Galley
Chris Brockett
Georgios Spithourakis
Jianfeng Gao
Bill Dolan

title: A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

authors:
Xiangpeng Li
Jingkuan Song
Lianli Gao
Xianglong Liu
Wenbing Huang
Xiangnan He
Chuang Gan

title: Beyond rnns: Positional self-attention with co-attention for video question answering

venue: In The 33rd AAAI Conference on Artificial Intelligence,

authors:
Zekang Li
Zongjia Li
Jinchao Zhang
Yang Feng
Cheng Niu
Jie Zhou

title: Bridging text and video: A universal multimodal transformer for video-audio scene-aware dialog

authors:
Chen Liang
Jonathan Berant
Quoc Le
Kenneth D. Forbus
Ni Lao

title: Neural symbolic machines: Learning semantic parsers on Freebase with weak supervision. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

title: Rouge: A package for automatic evaluation of summaries

venue: Text Summarization Branches Out,

authors:
Dat Tien Nguyen
Shikhar Sharma
Hannes Schulz
Layla El Asri

title: From film to video: Multi-turn question answering with multi-modal context

venue: In AAAI 2019 Dialog System Technology Challenge (DSTC7) Workshop,

authors:
Kishore Papineni
Salim Roukos
Todd Ward
Wei-Jing Zhu

title: Bleu: a method for automatic evaluation of machine translation

venue: In Proceedings of the 40th annual meeting on association for computational linguistics,

authors:
Mengye Ren
Ryan Kiros
Richard Zemel

title: Exploring models and data for image question answering

authors:
Shaoqing Ren
Kaiming He
Ross Girshick
Jian Sun

title: Faster R-CNN: Towards real-time object detection with region proposal networks

venue: In Advances in Neural Information Processing Systems (NIPS),

authors:
Ramon Sanabria
Shruti Palaskar
Florian Metze

title: Cmu sinbad’s submission for the dstc7 avsd challenge

authors:
Idan Schwartz
Seunghak Yu
Tamir Hazan
Alexander G Schwing

title: Factor graph attention

authors:
Iulian V Serban
Alessandro Sordoni
Yoshua Bengio
Aaron Courville
Joelle Pineau

title: Building end-to-end dialogue systems using generative hierarchical neural network models

venue: In Thirtieth AAAI Conference on Artificial Intelligence,

authors:
Sofia Serrano
Noah A. Smith

title: Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2931–2951, Florence, Italy, July 2019

venue: Association for Computational Linguistics. doi: 10.18653/v1/P19-1282. URL https: //www.aclweb.org/anthology/P19-1282

authors:
Gunnar A Sigurdsson
Gül Varol
Xiaolong Wang
Ali Farhadi
Ivan Laptev
Abhinav Gupta

title: Hollywood in homes: Crowdsourcing data collection for activity understanding

venue: In European Conference on Computer Vision,

authors:
Nitish Srivastava
Geoffrey Hinton
Alex Krizhevsky
Ilya Sutskever
Ruslan Salakhutdinov

title: Dropout: a simple way to prevent neural networks from overfitting

venue: The Journal of Machine Learning Research,

authors:
Christian Szegedy
Vincent Vanhoucke
Sergey Ioffe
Jon Shlens
Zbigniew Wojna

title: Rethinking the inception architecture for computer vision

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Ł ukasz Kaiser
Illia Polosukhin

title: Attention is all you need

authors:
Ramakrishna Vedantam
C Lawrence Zitnick
Devi Parikh

title: Cider: Consensus-based image description evaluation

authors:
Youngjae Yu
Hyungjin Ko
Jongwook Choi
Gunhee Kim

title: End-to-end concept word detection for video captioning, retrieval, and question answering

heading: 1 INTRODUCTION

text:
Vision-language tasks have been studied to build intelligent systems that can perceive information from multiple modalities, such as images, videos, and text. Extended from imaged-grounded tasks, e.g. (Antol et al., 2015), recently Jang et al. (2017); Lei et al. (2018) propose to use video as the grounding features. This modification poses a significant challenge to previous image-based models with the additional temporal variance through video frames. Recently Alamri et al. (2019) further develop video-grounded language research into the dialogue domain. In the proposed task, videogrounded dialogues, the dialogue agent is required to answer questions about a video over multiple dialogue turns. Using Figure 1 as an example, to answer questions correctly, a dialogue agent has to resolve references in dialogue context, e.g. “he” and “it”, and identify the original entity, e.g. “a boy" and “a backpack". In addition, the dialogue agent also needs to identify the actions of these entities, e.g. “carrying a backpack” to retrieve information along the temporal dimension of the video.
Current state-of-the-art approaches to video-grounded language tasks, e.g. (Le et al., 2019b; Fan et al., 2019) have achieved remarkable performance through the use of deep neural networks to retrieve grounding video signals based on language inputs. However, these approaches often assume the reasoning structure, including resolving references of entities and detecting the corresponding actions to retrieve visual cues, is implicitly learned. An explicit reasoning structure becomes more beneficial as the tasks complicates in two scenarios: video with complex spatial and temporal dynamics, and language inputs with sophisticated semantic dependencies, e.g. questions positioned in a dialogue context. In these cases, it becomes challenging to interpret model outputs, assess model reasoning capability, and identify errors in neural network models.
Similar challenges have been observed in image-grounded tasks in which deep neural networks often exhibit shallow understanding capability as they simply exploit superficial visual cues (Agrawal et al., 2016; Goyal et al., 2017; Feng et al., 2018; Serrano & Smith, 2019). Andreas et al. (2016b) propose neural model networks (NMNs) by decomposing a question into sub-sequences called program and assembling a network of neural operations. Motivated by this line of research, we propose an NMN approach to video-grounded language tasks. Our approach benefits from integrating neural networks with a compositional reasoning structure to exploit low-level information signals in video. An example of the reasoning structure can be seen on the right side of Figure 1.
We propose Visio-Linguistic Neural Module Network (VilNMN) for video-grounded language tasks. VilNMN leverages entity-based dialogue representations as inputs to neural operations on spatial and temporal-level visual features. Previous approaches exploit question-level and token-level representations to extract question-dependent information from video (Jang et al., 2017; Fan et al., 2019; Le et al., 2019b). In complex videos with many entities or actions, these approaches might not be optimal to locate the right features. To exploit object-level features, VilNMN is trained to identify relevant entities first, and then to extract the temporal steps using detected actions of these entities.
VilNMN is also trained to resolve any co-references in language inputs, e.g. questions in a dialogue context, to identify the original entities. Previous approaches to video-grounded dialogues often obtain question global representations in relation to dialogue context. These approaches might be suitable to represent general semantics in open-domain or chit-chat dialogues (Serban et al., 2016; Li et al., 2016) but they are not ideal to detect fine-grained entity-based information as the dialogue context evolves over time.
In summary, we introduce a neural module network approach to video-grounded language tasks through a reasoning pipeline with entity and action representations applied on the spatio-temporal dynamics of video. To cater to complex semantic inputs in language inputs, e.g. dialogues, our approach also allows models to resolve entity references to incorporate question representations with fine-grained entity information. In our evaluation, we achieve competitive performance on the large-scale benchmark Audio-visual Scene-aware Dialogues (AVSD) (Alamri et al., 2019). We also adapt VilNMN for video QA and obtain the state-of-the-art on the TGIF-QA benchmark (Jang et al., 2017) across all tasks. Our experiments and ablation analysis indicate a potential direction to develop compositional and interpretable neural models for video-grounded language tasks.

heading: 2 RELATED WORK

text:
Video QA has been a proxy for evaluating a model’s understanding capability of language and video and the task is treated as a visual information retrieval task. Jang et al. (2017); Gao et al. (2018); Jiang et al. (2020) propose to learn attention guided by question global representation to retrieve spatial-level and temporal-level visual features. Li et al. (2019); Fan et al. (2019); Jiang & Han (2020) model interaction between all pairs of question token-level representations and temporal-level features of input video. Extended from video QA, video-grounded dialogue is an emerging task that combines dialogue response generation and video-language understanding research. Nguyen et al. (2018); Hori et al. (2019); Hori et al. (2019); Sanabria et al. (2019); Le et al. (2019a;b) extend traditional QA models by adding dialogue history neural encoders. Kumar et al. (2019) enhances dialogue features with topic-level representations to express the general topic in each dialogue. Sanabria et al. (2019) considers the task as a video summary task and concatenates question and dialogue history into a single sequence and proposes to transfer parameter weights from a large-scale video summary model. Different from prior work, we dissect the question sequence and explicitly detect and decode any entities and their references. Our models also benefit from the additional insights on how models learn to use component linguistic inputs for extraction of visual information.
Extending from the line of research on neural semantic parsing (Jia & Liang, 2016; Liang et al., 2017), Andreas et al. (2016b;a) introduce NMNs to address visual QA by decomposing questions into linguistic sub-structures, known as programs, to instantiate a network of neural modules. NMN models have achieved significant success in synthetic image domains where complex reasoning process is required (Johnson et al., 2017b; Hu et al., 2018; Han et al., 2019). Our work is related to the recent work that extends NMN models to real data domains. For instance, Kottur et al. (2018); Jiang & Bansal (2019); Gupta et al. (2020) extend NMNs to visual dialogues and reading comprehension tasks. In this paper, we introduce a new approach that exploits NMN to learn dependencies between the lexical composition in language inputs and the spatio-temporal dynamics in videos. This is not present in prior NMN models which are designed to apply on a two-dimensional image input without temporal variance. In video represented as sequence of images, each represented by object-level features, applying prior NMN models require aggregating frame-level features, e.g. through average pooling, resulting in potential loss of information. An alternative solution is a late-fusion method in which an NMN model performs reasoning structure programs on sampled video frames only. An object tracking mechanism or attention mechanism is then used to fuse the output representations. Instead, we propose to construct a reasoning structure with multi-step interaction between the space-time information in video with entity-action detected in text.

heading: 3 METHOD

text: In this section, we present the design of our model, called Visio-Linguistic Neural Module Networks (VilNMN). An overview of the model can be seen in Figure 2. The input to the model consists of a dialogue D which is grounded on a video V . The input components include the question of current dialogue turnQ, dialogue historyH, and the features of input video, including visual and audio input. The output is a dialogue response, denoted asR. Each text input component is a sequence of words w1, ..., wm ∈ Vin, the input vocabulary. Similarly, the output response R is a sequence of tokens w1, ..., wn ∈ Vout, the output vocabulary. To learn compositional programs, we follow Johnson et al. (2017a); Hu et al. (2017) and consider program generation as a sequence-to-sequence task. Different from prior approaches, our models are trained to fully generate the parameters of component modules in text. This approach is appropriate as reasoning programs in real data domains such as current video-grounded dialogues are usually shorter than those for synthetic data (Johnson et al., 2017a) and thus, program generation takes less computational cost. However, module parameters, i.e. entities and actions, contain much higher semantic variance than synthetic data, and our approach facilitates better transparency and interpretability. We adopt a simple template “〈param1〉〈module1〉〈param2〉〈module2〉...” as the target sequence. The resulting target sequences for dialogue and video understanding programs are sequences Pdial and Pvid respectively.

heading: 3.1 ENCODERS

text: Text Encoder. A text encoder is shared to encode text inputs, including dialogue history, questions, and captions. The text encoder converts each text sequence X = w1, ..., wm into a sequence of embeddings X ∈ Rm×d. We use a trainable embedding matrix to map token indices to vector representations of d dimensions through a mapping function φ. These vectors are then integrated with ordering information of tokens through a positional encoding function with layer normalization (Ba et al., 2016; Vaswani et al., 2017). The embedding and positional representations are combined through element-wise summation. The encoded dialogue history and question of the current turn are defined as H = Norm(φ(H) + PE(H)) ∈ RLH×d and Q = Norm(φ(Q) + PE(Q)) ∈ RLQ×d. To decode program and response sequences auto-repressively, a special token “_sos” is concatenated as the first token w0. The decoded token w1 is then appended to w0 as input to decode w2 and so on. Similarly to input source sequences, at decoding time step j, the input target sequence is encoded to obtain representations for dialogue understanding program Pdial|j−10 , video understanding program Pvid|j−10 , and system response R| j−1 0 . We combine vocabulary of input and output sequences and share the embedding matrix E ∈ R|V|×d where V = Vin ∩ Vout. Video Encoder. To encode video input, we use pre-trained models to extract visual features and audio features. We denote F as the sampled video frames or video clips. For object-level visual features, we denote O as the maximum number of objects considered in each frame. The resulting output from a pretrained object detection model is Zobj ∈ RF×O×dvis . We concatenate each object representation with the corresponding coordinates projected to dvis dimensions. We also make use of a CNN-based pre-trained model to obtain features of temporal dimension Zcnn ∈ RF×dvis . The audio feature is obtained through a pretrained audio model, Zaud ∈ RF×daud . We passed all video features through a linear transformation layer with ReLU activation to the same embedding dimension d.

heading: 3.2 NEURAL MODULES

text:
We introduce neural modules that are used to assemble an executable program constructed by the generated sequence from question parsers. We provide an overview of neural modules in Table 1 and demonstrate dialogue understanding and video understanding modules in Figure 3 and 4 respectively. Each module parameter, e.g. “a backpack”, is extracted from the parsed program. For each parameter, we denote P ∈ Rd as the average pooling of component token embeddings. find(P,H)→Hent. This module handles entity tracing by obtaining a distribution over tokens in dialogue history. We use an entity-to-dialogue-history attention mechanism applied from an entity Pi to all tokens in dialogue history. Any neural network that learn to generate attention between two tensors is applicable .e.g. (Bahdanau et al., 2015; Vaswani et al., 2017). The attention matrix normalized by softmax, Afind,i ∈ RLH , is used to compute the weighted sum of dialogue history token representations. The output is combined with entity embedding Pi to obtain contextual entity representation Hent,i ∈ Rd. summarize(Hent,Q)→Qctx. For each contextual entity representation Hent,i, i = 1, ..., Nent, it is projected to LQ dimensions and is combined with question token embeddings through elementwise summation to obtain entity-aware question representation Qent,i ∈ RLQ×d. It is fed to a one-dimensional CNN with max pooling layer (Kim, 2014) to obtain a contextual entity-aware question representation. We denote the final output as Qctx ∈ RNent×d. While previous models usually focus on global or token-level dependencies (Hori et al., 2019; Le et al., 2019b) to encode question features, our modules compress fine-grained question representations
at entity level. Specifically, find and summarize modules can generate entity-dependent local and global representations of question semantics. We show that our modularized approach can achieve better performance and transparency than traditional approaches to encode dialogue context (Serban et al., 2016; Vaswani et al., 2017) (Section 4).
where(P,V)→Vent. Similar to the find module, this module handle entity-based attention to the video input. However, the entity representation P in this case is parameterized by the original entity in dialogue rather than in question (See Section 3.3 for more description). Each entity Pi is stacked to match the number of sampled video frames/clips F . An attention network is used to obtain entity-to-object attention matrix Awhere,i ∈ RF×O. The attended feature are compressed through weighted sum pooling along the spatial dimension, resulting in Vent,i ∈ RF×d, i = 1, ..., Nent. when(P,Vent)→Vent+act. This module follows a similar architecture as the where module. However, the action parameter Pi is stacked to match Nent dimensions. The attention matrix Awhen,i ∈ RF is then used to compute the visual entity-action representations through weighted sum along the temporal dimension. We denote the output for all actions Pi as Vent+act ∈ RNent×Nact×d
describe(P,Vent+act)→Vctx. This module is a linear transformation to compute Vctx = Wdesc
T [Vent+act;Pstack] ∈ RNent×Nact×d where Wdesc ∈ R2d×d, Pstack is the stacked representations of parameter embedding P to Nent ×Nact dimensions, and [; ] is the concatenation operation. The exist module is a special case of describe module where the parameter P is the average pooled question embeddings. The above where module is applied to object-level features. For temporal-based features such as CNN-based and audio features, the same neural operation is applied along the temporal dimension. Each resulting entity-aware output is then incorporated to frame-level features through element-wise summation (Please refer to Appendix A.1).
An advantage of our architecture is that it separates dialogue and video understanding. We adopt a transparent approach to solve linguistic entity references during the dialogue understanding phase. The resolved entities are fed to the video understanding phase to learn entity-action dynamics in video. We show that our approach is robust when dialogue evolves to many turns and video extends over time (Please see Section 4 and Appendix C).

heading: 3.3 DECODERS

text:
Question parsers. The parsers decompose questions into sub-sequences to construct compositional reasoning programs for dialogue and video understanding. Each parser is an attention-based Transformer decoder. Given the encoded question Q, to decode program for dialogue understanding, the contextual signals are integrated through 2 attention layers: one attention on previously generated tokens, and the other on question tokens.
To generate programs for video understanding, the contextual signals are learned and incorporated in a similar manner. However, to exploit dialogue contextual cues, the execution output of dialogue
understanding neural modules Qctx (See Section 3.2) is incorporated to each vector in Pvid through an additional attention layer. This layer integrates the entity-dependent contextual representations from Qctx to explicitly decode the original entities for video understanding programs.
Response Decoder. System response is decoded by incorporating the dialogue context and video context outputs from the corresponding reasoning programs to target token representations. We follows a vanilla Transformer decoder architecture (Le et al., 2019b), which consists of 3 attention layers: self-attention to attend on existing tokens, attention to Qctx from dialogue understanding program execution, and attention to Vctx from video understanding program execution.
Optimization. We use the standard cross-entropy losses for prediction of dialogue and video understanding programs and output responses. We optimize models by joint training to minimize: L = αLdial + βLvid + Lres = α ∑ j − log(Pdial(Pdial,j)) + β ∑ l− log(Pvideo(Pvideo,l)) + ∑ n− log(Pres(Rn))
where P is the probability distribution of an output token. The probability is computed by passing output representations from the parsers and decoder to a linear layer W ∈ Rd×V with softmax activation. We share the parameters between W and embedding matrix E. The hyper-parameters α ≥ 0 and β ≥ are fine-tuned during training.

heading: 4 EXPERIMENTS

text:
Datasets. We use the AVSD benchmark from the 7th Dialogue System Technology Challenge (DSTC7) (Hori et al., 2019). In the experiments with AVSD, we consider two settings: one with video summary and one without video summary as input. In the setting with video summary, the summary is concatenated to the dialogue history before the first dialogue turn. We also adapt VilNMN to the video QA benchmark TGIF-QA (Jang et al., 2017). Different from AVSD, TGIF-QA contains a diverse set of tasks, which address different visual aspects in video.
Training Procedure. We follow prior approaches (Hu et al., 2017; 2018; Kottur et al., 2018) by obtaining the annotations of the programs through a language parser (Hu et al., 2016) and a reference resolution model (Clark & Manning, 2016). During training, we directly use these soft labels of programs and the given ground-truth responses to train the models. The labels are augmented with label smoothing technique (Szegedy et al., 2016). During inference time, we generate all programs and responses from given dialogues and videos. We run beam search to enumerate programs for dialogue and video understanding and dialogue responses. (Please see Appendix B for more details).
AVSD Results. We evaluate model performance by the objective metrics based on word overlapping, including BLEU (Papineni et al., 2002), METEOR (Banerjee & Lavie, 2005), ROUGE-L (Lin, 2004), and CIDEr (Vedantam et al., 2015), between each generated response and 6 reference gold responses. As seen in Table 2, our models outperform most of existing approaches. In particular, the performance of our model in the setting without video summary input is comparable to the GPT-based RLM (Li et al., 2020) with much smaller model size. The Student-Teacher baseline (Hori et al., 2019) specifically focuses on the performance gap between models with and without textual signals from video summary through a dual network of expert and student models. Instead, VilNMN reduces this performance gap through efficiently extracting relevant visual/audio information based on fine-grained entity and action signals. We also found that VilNMN applied on object-level features is competitive to the model applied on CNN-based features. The flexibility of VilNMN neural programs can also be seen in the experiment when the video understanding program is applied on the caption input as a visual feature.
Ablation Analysis. We experiment with several variants of VilMNM (either NMN or non-NMNbased) in the setting with CNN based features and video summary input As can be seen in Table 3, our approach to video and dialogue understanding through compositional reasoning programs exhibits better performance than non-compositional approaches. Compared to the approaches that directly process frame-level features in videos (Row B) or token-level features in dialogues (Row C, D), our full VilNMN (Row A) considers entity-level and action-level information extraction and thus, avoids unnecessary and possibly noisy extraction. Compared to the approaches that obtain dialogue contextual cues through a hierarchical encoding architecture (Row E, F) such as (Serban et al., 2016; Hori et al., 2019), VilNMN directly addresses the challenge of entity references in dialogues. As mentioned, we hypothesize that the hierarchical encoding architecture is more appropriate for less
entity-sensitive dialogues such as chit-chat and open-domain dialogues. Please see Appendix C for additional analysis of performance breakdown by dialogue turns and video lengths.
Intepretability. A difference of VilNMN from previous approaches to video-grounded dialogues is the model interpretability based on the predicted dialogue and video programs. From Figure 5, we observe that in cases where predicted dialogue programs and video program match or are close to the gold labels, the model can generate generally correct responses. In cases of wrong predicted responses, we can further look at how the model understands the questions based on predicted programs. In the 3rd turn of example 1, the output response is missing a minor detail as compared to the label response because the video program fails to capture the parameter “rooftop”. These subtle yet important details can determine whether output responses can fully address user queries. Similarly, in example 2, the model answers the question “what room” instead of question about “an object”. For additional qualitative analysis, please see Appendix D.
TGIF-QA Results. In TGIF-QA experiments, we report the result using the L2 loss in Count task and accuracy in other tasks. From Table 4, VilNMN outperforms all baseline models in all tasks by a large margin. Compared to AVSD experiments, the TGIF-QA experiments emphasize video understanding ability of the models, removing the requirement for dialogue understanding and natural

heading: Turn Question Dialogue Understanding

text: language generation. This is demonstrated through higher performance gaps between VilNMN with generated programs and soft label programs as compared to ones in AVSD experiments. We also observe that an attention layer attending to question is important during the response decoding phase in TGIF-QA as there is no dialogue context Qctx in Video QA tasks.

heading: 5 CONCLUSION

text: While conventional neural network approaches have achieved notable successes in video-grounded dialogues and video QA, they often rely on superficial pattern learning principles between contextual cues from questions/dialogues and videos. In this work, we introduce Visio-Linguistic Neural Module Network (VilNMN). VilNMN consists of dialogue and video understanding neural modules, each of which performs entity and action-level operations on language and video components. Our comprehensive experiments on AVSD and TGIF-QA benchmarks show that our models can achieve competitive performance while promoting a compositional and interpretable learning approach.

heading: A ADDITIONAL MODEL DETAILS

heading: A.1 NEURAL MODULES ON TEMPORAL FEATURES

text:
To adapt our neural modules to temporal features, we apply the same neural architectures in all modules except for the where module. In object-level features, this module operates on object-based or spatial-based level. We can apply this module to temporal-based features similarly simply by not stacking the parameter and pooling the attended features along the temporal dimension. For an entity parameter Pi, the attention matrix in this case is an entity-to-temporal-step matrix Awhere,i ∈ RF and the resulting pooled feature is Vent,i ∈ Rd. Before feeding this representation to a when module, we incorporate each Vent,i into feature of each temporal step through an MLP layer and element-wise summation, resulting in V stackent,i ∈ RF×d where F is the number of sampled video frames/clips. An overview of the where module with temporal features can be seen in Figure 6.
We adapt this module in a similar manner to other temporal-level features such as audio and textual features such as video caption. We keep the same architecture in the when module. We denote the resulting output from the when module for all actions Pi is Vact ∈ RNact×d. We concatenate this to the output from the previous where module Vent to obtain Vent+act ∈ R(Nent+Nact)×d. This is used as input to the describe or exist module.

heading: A.2 QUESTION PARSER

text:
The parsers decompose questions into sub-sequences to construct compositional reasoning programs for dialogue and video understanding. Each parser is an attention-based Transformer decoder. The Transformer attention is a multi-head attention on query, key, and value tensors, denoted as Attention(Query,Key,Value). For each token in the Query sequence , the distribution over tokens in the Key sequence is used to obtain the weighted sum of the corresponding representations in the Value sequence.
Attention(Query,Key, V alue) = softmax( QueryKeyT√
dkey )V alue ∈ RLquery×dquery
Each attention is followed by a feed-forward network applied on each position identically. We exploit the multi-head and feed-forward architecture, which show good performance in NLP tasks such as NMT and QA (Vaswani et al., 2017; Dehghani et al., 2019), to efficiently incorporate contextual cues from dialogue components to parse question into reasoning programs. Given the encoded question Q, to decode program for dialogue understanding, the contextual signals are integrated through 2 attention layers: one attention on previously generated tokens, and the other on question tokens. At time step j, we denote the output from an attention layer as Adial,j.
A (1) dial = Attention(Pdial| j−1 0 , Pdial| j−1 0 , Pdial| j−1 0 ) ∈ Rj×d
A (2) dial = Attention(A (1) dial, Q,Q) ∈ R j×d
Similarly, to generate programs for video understanding, the contextual signals are learned and incorporated in a similar manner. However, to exploit dialogue contextual cues, the execution output of dialogue understanding neural modules Qctx is incorporated to each vector in Pdial through an additional attention layer. This layer integrates the resolved entity information to decode the original entities for video understanding. It is equivalent to a reasoning process that converts the question from its original multi-turn semantics to single-turn semantics.
A (3) vid = Attention(A (2) vid, Qctx, Qctx) ∈ R j×d

heading: A.3 NON-NMN MODELS

text:
For ablation analysis, we evaluate several variants of VilNMN, based on the following categories:
To test the contribution of our NMN approach for video understanding, we remove the parser for video understanding program and related neural modules and replace them with pure neural network architecture (Model B). Specifically, we remove neural modules where, when, describe, and exist. We then directly use video feature embeddings V as Vctx as input to the original attention layer in response decoder similarly to (Hori et al., 2019; Sanabria et al., 2019).
A(3)res = Attention(A (2) res , V, V ) ∈ Rj×d
To further test the contribution of NMN architecture for dialogue understanding, we similarly remove the question parser for dialogue understanding program and neural modules find and describe. We then directly use the dialogue history embeddings H and question embeddings Q as inputs to the response decoder in two different ways. First, we replace the original attention on dialogue context Qctx with two attention layers to attend on dialogue history and question sequentially (Model C). As noted by Le et al. (2019b), question input contains much more relevant signals than dialogue history and attention operation should be separated from the one on dialogue history.
A(2a)res = Attention(A (1) res , H,H) ∈ Rj×d A(2b)res = Attention(A (2a) res , Q,Q) ∈ Rj×d
Alternatively, we simply concatenate dialogue and question embeddings similarly to (Hori et al., 2019; Sanabria et al., 2019) and use it as input to the original attention layer (Model D).
A(2)res = Attention(A (1) res , [H;Q], [H;Q]) ∈ Rj×d
To use more sophisticated neural models for dialogue understanding, we further adopt the hierarchical encoding architecture with question attention (Li et al., 2016; Serban et al., 2016; Hori et al., 2019). Each dialogue turn Ht, including a pair of human utterance and system response, is processed separately by a word-level RNN such as LSTM (Model E) or GRU (Model F). A sentence-level RNN is used to sequentially process the last hidden states obtained previously turn by turn. The output in each recurrent step is fed to an attention layer such as (Bahdanau et al., 2015; Vaswani et al., 2017) to obtain question-aware representations of dialogue history.
Hwordt = RNN(Ht) ∈ Rd
Hsentt = RNN(H word t ) ∈ Rd
H = [Hsentt ]|T−1t=1 ∈ Rd×(T−1)
Qctx = Attention(Q,H,H) ∈ RLQ×d
where T is the current dialogue turn. The output is treated as Qctx and is fed to the corresponding attention layer in the response decoder.

heading: B ADDITIONAL EXPERIMENT DETAILS

heading: B.1 DATASETS

text:
We use the AVSD benchmark from DSTC7 (Hori et al., 2019) which consists of dialogues grounded on the Charades videos (Sigurdsson et al., 2016). Each dialogue contains up to 10 dialogue turns, each turn consists of a question and expected response about a given video. For visual features, we use the 3D CNN based features from a pretrained I3D model (Carreira & Zisserman, 2017) and object-level features from a pretrained FasterRNN model (Ren et al., 2015b). The audio features are obtained from a pretrained VGGish model (Hershey et al., 2017). In the experiments with AVSD, we consider two settings: one with video summary and one without video summary as input. In the setting with video summary, the summary is concatenated to the dialogue history before the first dialogue turn. We also adapt VilNMN to the video QA benchmark TGIF-QA (Jang et al., 2017). Different from AVSD, TGIF-QA contains a diverse set of QA tasks:
• Count: open-ended task which counts the number of repetitions of an action
• Action: multiple-choice (MC) task which asks about a certain action occurring for a fixed number of times
• Transition: MC task which emphasizes temporal transition in video • Frame: open-ended task which can be answered from visual contents of one of video frames
For the TGIF-QA benchmark, we use the extracted features from a pretrained ResNet model (He et al., 2016).

heading: B.2 TRAINING DETAILS

text: We use a training batch size of 32 and embedding dimension d = 128 in all experiments. Where Transformer attention is used, we fix the number of attention heads to 8 in all attention layers. In neural modules with MLP layers, the MLP network is fixed to 2 linear layers with a ReLU activation in between. In neural modules with CNN, we adopt a vanilla CNN architecture for text classification (without the last MLP layer) where the number of input channels is 1, the kernel sizes are {3, 4, 5}, and the number of output channels is d. We initialize models with uniform distribution (Glorot & Bengio, 2010). During training, we adopt the Adam optimizer (Kingma & Ba, 2015) and a decaying learning rate Vaswani et al. (2017) where we fix the warm-up steps to 15K training steps. We employ dropout (Srivastava et al., 2014) of 0.2 at all networks except the last linear layers of question parsers and response decoder. We train models up to 50 epochs and select the best models based on the average loss per epoch in the validation set.

heading: C ADDITIONAL RESULTS

text:
To evaluate model robustness, we report the relative performance by calculating the difference of CIDEr in experimental settings against the most basic setting. Specifically, we compare against performance of output responses in the first dialogue turn position (i.e. 2nd-10th turn vs. the 1st turn), or responses grounded on the shortest video length range (video ranges are intervals of 0-10th, 10-20th percentile and so on). We report the results of the model variants A, B, and E (See the Ablation Analysis section in the main paper and Appendix A.3 for model description). First, as can be seen in Figure 7, for various dialogue turn positions, we observe that the original VilNMN (model A) suffers less than model E when dialogues extend over time up the 8th turn. This explains the contribution of dialogue understanding modules in solving entities even when the dialogues grow longer. Secondly, as compared to model B, we observe that the Full VilNMN (model A) is less affected as the videos grounding the dialogues grow longer. The difference is clear when the video length increases up to 33 seconds.
We also report the absolute scores and compare model variants. In Table 6a, we compare model variants B and E. We observe that model B generally performs better than model E in overall, especially in higher turn positions, i.e. from the 4th turn to 8th turn. Interestingly, we note some mixed results in very low turn position, i.e. the 2nd and 3rd turn, and very high turn position, i.e. the 10th turn. Potentially, in very high turn position, the neural based approach such as hierarchical RNN can better capture the global dependencies within dialogue context than the entity-based compositional NMN method.
In Table 6b, we compare model variants A and B. We note that the performance gap between model A and B is quite distinct, with 7/10 cases of video ranges in which model A outperforms. However, similarly to our prior observations in experiments by dialogue turn, in lower ranges (i.e. 1-23
seconds) and higher ranges (37-75 seconds), model A performs not as well as model B. There are additional factors that we will need to examine further to explain the results, such as the complexity of the questions for these short and long-range videos. Potentially, our question parser for video understanding program needs more sophisticated composition method to retrieve information from these video ranges.

heading: D QUALITATIVE ANALYSIS

text:
We extract the predicted programs and responses for some example dialogues in Figure 8, 9, 10, and 11 and report our observations:
• We observe that when the predicted programs are correct, the output responses generally match the ground-truth (See the 1st and 2nd turn in Figure 8, and the 1st and 4th turn in Figure 10) or close to the ground-truth responses (1st turn in Figure 9).
• When the output responses do not match the ground truth, we can understand the model mistakes by interpreting the predicted programs. For example, in the 3rd turn in Figure 8, the output response describes a room because the predicted video program focuses on the entity “what room” instead of the entity “an object” in the question. Another example is the 3rd turn in Figure 10 where the entity “rooftop” is missing in the video program. These mismatches can deviate the information retrieved from the video during video program execution, leading to wrong output responses with wrong visual contents.
• We also note that in some cases, one or both of the predicted programs are incorrect, but the predicted responses still match the ground-truth responses. This might be explained as the predicted module parameters are not exactly the same as the ground truth but they are close enough (e.g. 4th turn in Figure 8). Sometimes, our model predicted programs that are more appropriate than the ground truth. For example, in the 2nd turn in Figure 9, the program is added with a where module parameterized by the entity “the shopping bag” which was solved from the reference “them” mentioned in the question.
• We observe that for complex questions that involve more than one queries (e.g. the 3rd turn in Figure 10), it becomes more challenging to decode an appropriate video understanding program and generate responses that can address all queries.
• In Figure 11, we demonstrate some output examples of VilNMN and compare with two baselines: Baseline (Hori et al., 2019) and MTN (Le et al., 2019b). We noted that VilNMN can include important entities relevant to the current dialogue turn to construct output responses while other models might miss some entity details, e.g. “them/dishes” in example A and “the magazine” in example B. These small yet important details can determine the correctness of dialogue responses.

title: VILNMN: A NEURAL MODULE NETWORK APPROACH TO VIDEO-GROUNDED LANGUAGE TASKS