over time.
Long range reasoning | integration of the object relations over time is recurrent by using a RNN for f(). Since reasoning from a full sequence cannot
be done by inferring the relations between two frames, f() allows long range
reasoning on sequences of variable length.
3.2 Object instance features
The object features Ot=
ok
t	K
k=1for each frame tused for the ORN module
described above are computed and collected from local regions predicted by
a mask predictor. Independently for each frame Xtof the input data block,
we predict object instances as binary masks Bk
tand associated object class
predictions ck
t, a distribution over Cclasses. We use Mask-RCNN [15], which
is able to detect objects in a frame using region proposal networks [30] and
produces a high quality segmentation mask for each object instance.
The objective is to collect features for each object instance, which jointly
describe its appearance, the change in its appearance over time, and its shape,
i.e. the shape of the binary mask. In theory, appearance could also be described
by pooling the feature representation learned by the mask predictor (Mask RCNN). However, in practice we choose to pool features from the dedicated object
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.
Fabien Baradel Natalia Neverova Christian Wolf Julien Mille Greg Mori