## Prediction and Description of Near-Future Activities in Video

Tahmida Mahmud a , Mohammad Billah a , Mahmudul Hasan b , Amit K. Roy-Chowdhury a a Department of Electrical and Computer Engineering, University of California, Riverside, CA 92521, USA b Comcast AI, Washington, DC-20005, USA

## ABSTRACT

Most of the existing works on human activity analysis focus on recognition or early recognition of the activity labels from complete or partial observations. Similarly, almost all of the existing video captioning approaches focus on the observed events in videos. Predicting the labels and the captions of future activities where no frames of the predicted activities have been observed is a challenging problem, with important applications that require anticipatory response. In this work, we propose a system that can infer the labels and the captions of a sequence of future activities. Our proposed network for label prediction of a future activity sequence has three branches where the first branch takes visual features from the objects present in the scene, the second branch takes observed sequential activity features, and the third branch captures the last observed activity features. The predicted labels and the observed scene context are then mapped to meaningful captions using a sequence-to-sequence learning-based method. Experiments on four challenging activity analysis datasets and a video description dataset demonstrate that our label prediction approach achieves comparable performance with the state-of-the-arts and our captioning framework outperform the state-of-the-arts.

## 1. Introduction

Activity analysis is a widely studied problem in the computer vision community. A large number of the existing works focus on recognition of observed activities or early recognition of partially observed activities. Predicting the labels of future activities which have not yet been observed is di GLYPH&lt;11&gt; erent from the recognition problem, where inferences need to be made on activity features which have been observed. The word 'prediction' has been used in [1-5], referring to the early recognition task, i.e., predicting the label of the ongoing activity where the first few frames have already been observed. However, in the prediction problem we are addressing, no observation is available beforehand . Predicting the future activity labels is critical in real life scenarios, where anticipatory response is required based on an observed segment of the video, e.g., driver intent prediction [6, 7] in Advanced Driver Assistance Systems (ADAS) where a description of which lane the driver might move into in the near future is necessary to predict the likelihood of potential collisions in complex tra GLYPH&lt;14&gt; c scenarios, or Hu- man Intent Prediction (HIP) [8, 9] in human-robot collaboration where the robot may need to predict what the human may do in the future to ensure safety and e GLYPH&lt;14&gt; ciency. There are a number of approaches [10-22] which perform label prediction on real-life activity datasets.

1 The paper has been accepted for publication in Computer Vision and Image Understanding.

Generating description of visual content is an active research area in both computer vision and natural language processing community. Since vision and language are two of the richest interaction modalities available to humans, it is crucial to understand the relationship between them. Language is the most natural way to make information from any semantic representation meaningful. In the last few years, this problem has received significant attention for image captioning [23-26] as well as video captioning [27-43]. Unlike image description, video description has to deal not only with the appearance of the objects but also with motion over time. There has been significant work in the multimedia community on joint image and text processing, e.g., for retrieval, captioning etc. [44-51]; however, none of them except [51] address the problem of predicting captions for the near-future events in videos. To the best of our knowledge, almost all of the existing works on video captioning focus on the observed portion of the video, i.e., describe events which have already happened or happening at the moment. Ours is one

Fig. 1. There are k activities in the observed portion of a video with starting times ( t 1 s , t 2 s , ..., t ks ) and ending times ( t 1 e , t 2 e , ..., t ke ). We want to predict the labels and the captions of ( k + 1) th , ( k + 2) th , ... activities. 1

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00943v5-figure-03c4b7fd-0e23-4d25-bd58-fa1c08a245a3.jpg)

of the earliest works where we look into the problem of providing captions for a sequence of near-future unobserved events in videos . Generating the labels of future unobserved activities can be considered as the first step towards describing the future. But it may be desirable to o GLYPH&lt;11&gt; er a richer description than a simple one-word phrase label for specific applications like assistive / systems [52, 53] for the visually impaired. There has been work on generating future frames [54], which are much richer in content, but the approach is constrained to only a few such frames. Our work lies in between these two extremes: it can generate semantically meaningful captions that describe changes in activities and thus able to predict much further in time than the frame generation work [54], while at the same time, provides a much richer description than label prediction [10-13].

takes place immediately after 'cut slices'; but if there is a delay, then 'take out from drawer' happens before. Once the labels are available, we map them along with the scene context of the last observed portion to generate meaningful captions for a sequence of future activities. Instead of using a ruleor template-based natural language generation approach, we are motivated by the data driven domain-independent learningbased approach [59] which replaced rule based methods in statistical machine translation. Instead of performing the mapping between two language spaces, we are doing a mapping from labels to captions. This sequence-to-sequence learning-based approach as used in other captioning works [30, 39] makes minimal assumptions on the sequence structure.

## 1.1. Problem Definition

For a video observed up to a certain time, we want to predict the labels of the future activity sequence and provide a caption describing these future activities in the context of the observed video. This is illustrated in Fig. 1. Here, we have only observed up to k th activity. Now we want to predict the labels and the captions of the future activity sequence, i.e., the labels and the captions of the ( k + 1) th , ( k + 2) th , GLYPH&lt;1&gt; GLYPH&lt;1&gt; GLYPH&lt;1&gt; activities and the starting time of that sequence, i.e., t ( k + 1) s . In our experiments, each predicted sequence contains three unobserved future events (chosen empirically because predicting beyond three activities is very error prone).

## 1.2. Overview of the Approach

In this paper, we present an integrated approach to answer two important questions regarding the unobserved portion of a video observed up to a particular time: what activities will happen next , and what captions describe them best . We predict the labels of a sequence of future unobserved activities in both coarse (VIRAT Ground Dataset [55]) and fine grained activity datasets (MPII-Cooking Dataset [56], MPII-Cooking 2 Dataset [57] and EPIC-KITCHENS Dataset [58]). This is posed as a joint label and starting time prediction task because intuitively the problem of predicting the label and the starting time of unobserved activities are closely related. For example, in MPIICooking Dataset, 'cut slices' can be followed by two probable activities: 'spice' or 'take out from drawer'. Usually, 'spice'

An overview of our proposed framework is illustrated in Fig. 2. To infer the future unobserved activity sequence, we use context information (relationship between actors, objects, and activities) from the observed portion of the videos. We incorporate sequential activity context (temporal ordering among activities), scene context (objects present in the last observed portion of the scene) and inter-activity time (di GLYPH&lt;11&gt; erence between the starting time of the observed activity and the future activity) context. The details of the context information are provided in Section 3.1.1. We develop a deep network for label prediction by merging three branches: one with two fully connected layers, another with two LSTM layers and the last one with another two fully connected layers. The network is trained on the previous activity features and the features of the objects present in the last observed portion of the scene. For captioning a sequence of future activities, we use a multi-layered LSTM to map the predicted labels and observed scene context to a fixed dimensional vector. Another deep LSTM conditioned on the input sequence is used for extracting the target caption from that vector. The ability of LSTM layers to incorporate long term sequential dependencies makes it a suitable choice for this application.

1 Please note that although the captions are generated for future unobserved events, they are in past tense since the reference human descriptions are given in past tense in the dataset which we used during training to generate coherent results for meaningful comparison with [60] and [37].

Fig. 2. Overview of our approach. The label prediction network is trained on both the sequential activity features from previously observed activities and the object features present in the last observed portion of the scene. The sequence-to-sequence learning-based mapping network finally maps the sequential labels and observed scene context to a sequence of captions. A detailed version of this figure is given in Fig. 3.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00943v5-figure-7263ff69-550d-4ad8-98f2-da5d710f77d5.jpg)

## 1.3. Main Contributions

## 2. Relation to Existing Works

Our work focuses on providing a description of what activities may happen in the near-future from a sequence of current observations. We first predict the labels of the future activities, which is then followed by captioning. To the best of our knowledge, almost all of the existing works on video captioning focus on the observed portion of the video, and ours is one of the earliest works for captioning a sequence of near-future unobserved events in videos. It provides a richer description than just predicting the label of the next activity and predicts over a longer duration than works on frame prediction. The main contributions of this work are:

- 1. We jointly model the sequential relationships of the activities, scene context and the last observed activity features in order to predict the labels of a future activity sequence.
- 2. We solve a novel and relevant problem of captioning a sequence of future unobserved events of a video using a sequence-to-sequence learning-based approach.
- 3. We perform extensive experiments that show the e GLYPH&lt;11&gt; ectiveness of the proposed framework.

## 1.4. Organization of the Paper

The rest of the paper is organized as follows. Section 2 gives a description of the state-of-the-art approaches in applications of LSTMs, activity analysis, and video captioning. The role of di GLYPH&lt;11&gt; erent context attributes is provided in Section 3.1.1 and Section 3.2.1. The network architectures are provided in Section 3.1.2 and Section 3.2.2 and model training details are provided in Section 3.1.3 and Section 3.2.3. A test case scenario is explained in Section 3.3. Experimental results and comparisons with state-of-the-art methods are shown in Section 4 and conclusions are drawn in Section 5.

Our work involves the following areas of interest: Long Short-Term Memory (LSTM) network, future activity label prediction, and video captioning. We will review some relevant papers from these areas.

Long Short-Term Memory (LSTM) Network. Unlike traditional neural networks, Recurrent Neural Network (RNN) has the capability of allowing information to be passed from one step of the network to the next using the loops inherent to their structure. However, in practice, RNNs cannot handle long-term dependencies, primarily because of the vanishing and exploding gradient problem. To overcome the challenge of handling long-term dependency, a special type of RNN called LSTM (Long Short-Term Memory) was introduced in [61]. LSTMs have achieved impressive performance in di GLYPH&lt;11&gt; erent sequence learning problems [30, 59, 62-64]. Its ability to capture longrange dependencies makes it a perfect tool for long-term context incorporation.

Future Activity Label Prediction. There have been a few works which predict the future unobserved activity such as approaches using Hierarchical state space Markov Chains [65], semantic scene labeling [11], Probabilistic Su GLYPH&lt;14&gt; x Tree (PST) [2], augmented- Hidden Conditional Random Field (a-HCRF) [66], Markov Random Field (MRF) [10], kernel-based reinforcement learning [67], max-margin learning [68], and deep network [12-22, 54, 69]. Among these, [10-22] perform prediction, without any observation of the activity to be predicted, in the label space. In [54], where visual representation of images is predicted and then recognition algorithm is applied, actions can be anticipated only up to one second in the future. The focus of [69] is forecasting behavior goal where the fundamen-/ tal state variables involved are di GLYPH&lt;11&gt; erent than the label space. [13] infers about the labels of a future activity sequence using a CNN-based and a RNN-based approach. However, they predict the labels of a future unobserved activity sequence only; whereas the main focus of this work is predicting the captions

of a future activity sequence. Our previous work on activity prediction [12] has achieved the highest accuracy on two challenging activity datasets incorporating di GLYPH&lt;11&gt; erent context attributes but did not perform sequence prediction.

Video Captioning. The initial works on video captioning [34, 70-74] focus on rule-based systems where sentences are generated using predefined templates following certain linguistic rules. Later, learning-based data driven approaches [2729, 36, 60, 75-77] became popular. As the methods started becoming free from manual engineering, the problem became more scalable providing flexibility to work with larger datasets. Recently, Recurrent Neural Network (RNN)-based approaches [30-33, 78] have achieved promising performance in video captioning. One of the earliest works [32] using RNNs extends the image captioning methods by average pooling the video frames which only works for short video clips containing just one event. To overcome this shortcoming, recurrent encoderbased models [30], [78], [31] and attention models [33] have been proposed. [37] uses a hierarchical RNN to generate a paragraph for richer description. Another paper [38] performs dense-captioning of events in videos using context information. There is only one other previous work [51] for predicting the captions of future unobserved activities. We want to point out a major di GLYPH&lt;11&gt; erence between their work and ours. Their work uses the caption from the observed portion of the video to predict the caption for the future unobserved events. This setting can be restrictive for some real-world scenarios like autonomous driving, assistive systems for the visually impaired or humanrobot interaction where only visual data captured by the camera is available. Our method is based on a two-step prediction approach where the action labels are predicted first using only the visual spatio-temporal context from the observed events and then the predicted labels and the observed scene context are used for caption prediction. We do not need any captions on the observed portion of the video. Thus, our approach tackles a more challenging and generic problem.

Extension to Previous Works. The goal of this work is to predict the captions of a sequence of future activities which is, to the best of our knowledge, one of the earliest works in this area. This is leveraged on our previously published paper on activity label and starting time prediction [12]. Instead of predicting the label of one future activity at a time, here we are predicting the labels of a sequence of future activities and finally captioning the future activity sequence using the predicted label information. We show results on two additional datasets called MPII-Cooking 2 Dataset [57] and EPIC-KITCHENS Dataset [58] for label prediction and conduct experiments on a video description dataset called TACoS Multi-Level Corpus [60] built on MPII-Cooking 2 [57] demonstrating the e GLYPH&lt;11&gt; ectiveness of our captioning method.

## 3. Methodology

In this section, we discuss the motivation behind the choice of our networks explaining the importance of di GLYPH&lt;11&gt; erent context attributes for the task, the network architectures in details, the training scheme and the way we obtained the final results in the test phase.

## 3.1.1. Role of Di GLYPH&lt;11&gt; erent Context Attributes

Activities usually have some temporal ordering, e.g., a person will get into a car and then it starts to move, or vegetables will be washed, peeled, and then put on the pan. Therefore, previous activities can provide useful information about the upcoming ones which can be referred to as sequential activity context . It is true that not all the steps in a complex activity will always occur exactly in the same sequence, which is why our prediction results are probabilistic and di GLYPH&lt;11&gt; erent outputs are possible. Activities are also characterized by the objects present in the last observed portion of the scene during the time of their occurrence which can be referred to as scene context . For many activities, predicting the future has multiple plausible options. To reduce this specific ambiguity, we take scene context into account along with the sequential information. Thus, combining the information obtained from these two di GLYPH&lt;11&gt; erent context attributes (temporal sequence and spatial objects), we infer the sequence of future unobserved activities. For example, if three sequential activities in a video are 'wash objects', 'peel' and 'cut slices', then there may be two probable future activity sequences: 'screw open', 'take out from spice holder', and 'spice' or 'put in bowl', 'puree' and 'smell' (based on two di GLYPH&lt;11&gt; erent training instances). But a bowl present in the scene increases the possibility of the latter sequence.

Several research works on activity recognition [79-85] and prediction [10, 13] have shown significant performance improvement by using such context information which are also known as context-aware approaches. Most of the existing works have graphical model-based approaches for context incorporation. However, they are not very suitable to handle the context of long-term dependency. As mentioned before, LSTM is a popular choice for sequential context incorporation. LSTM networks are straightforward to fine-tune end-to-end and can handle sequential data of varying lengths. So, we use LSTM to incorporate sequential activity context. However, for including the scene context from the last observed frame, there is no need for handling such sequential dependency (taking longterm scene context into account can increase ambiguity) and fully connected layers can capture this e GLYPH&lt;14&gt; ciently.

The inter-activity time between di GLYPH&lt;11&gt; erent activities depends on their labels. For example, we know from experience that 'peel' or 'cut slices' takes more time than 'wash objects'. Thus, by observing the previous activity features we can infer the inter-activity time (di GLYPH&lt;11&gt; erence between the starting time of the observed activity and the future activity) which can be referred to as inter-activity time context .

## 3.1.2. Network Architecture

Our proposed architecture for jointly predicting the labels and the starting time of a future activity sequence is shown in Fig. 3. The LSTM is used to solve a sequential input, sequential output problem. We use the activity features extracted from three (chosen empirically) previously observed activities as the LSTM input. Increasing the sequence length does not improve the prediction accuracy significantly (see Section 4.3.3 for details). We use a two-layer (chosen empirically) LSTM with 256

Fig. 3. Proposed architecture for future activity label and caption prediction. In the top figure, the first two fully connected layers (yellow) incorporate the scene context (extracted from objects in the last frame) which use object features as input. The two LSTM layers (green) are used to incorporate the sequential activity context ('take out', 'wash', 'put on' in this example) which use motion-based features as inputs. The last two fully connected layers (peach) are used to incorporate inter-activity time context which use the last observed activity features (motion-based) ('put on' in this example) as input. There is a fully connected layer (blue) where all these layers are merged together. The output layer (gray) performs the final prediction, where for each element of the future activity sequence, the first few nodes (green) are used as the logistic regression nodes for label prediction. The last node (blue) of the output layer is used as the regression node for starting time prediction. All of the layers have 256 nodes. In the bottom figure, the predicted label ('cut o GLYPH&lt;11&gt; ends' in this example) and the scene context ('cucumber' in this example) are then used as input to the encoder LSTM layers and finally the decoder LSTM layers generate the captions. Here, EOS denotes End of Sentence.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00943v5-figure-274278bf-303e-4a9d-b46b-150fa663303b.jpg)

memory units in each layer. The input of the two (chosen empirically) fully connected layers in the first branch are the visual features extracted from the objects present in the last observed portion of the scene and there are 256 nodes in each layer. The input of the two (chosen empirically) fully connected layers in the third branch are the activity features extracted from the last observed activity and have 256 nodes in each layer too. Parameter sensitivity analysis for the number of nodes used are provided in the supplementary material.

Finally, the outputs from these three branches are combined together and another fully connected layer is added on top of it. The merging combines the e GLYPH&lt;11&gt; ect of di GLYPH&lt;11&gt; erent context attributes. In the output layer, for each of the three future activities in the sequence, the first few (equal to the number of activity classes) nodes are used as the logistic regression nodes for sequential label prediction and the last node is used as a regression node for predicting the starting time of the future activity sequence.

( k GLYPH&lt;0&gt; 2) th activities are used. For predicting the labels of the future sequence containing the ( k + 2) th , ( k + 3) th and ( k + 4) th activities, activity features extracted from the ( k + 1) th , k th and ( k GLYPH&lt;0&gt; 1) th activities are used and so on. We have not used the ground-truth labels of the observations during either training or testing for any of the experiments. The two fully connected layers in the first branch use visual object features from the last observed portion of the scene as input. Another two fully connected layers in the third branch use activity features extracted from the last observed activity as input. We use ReLU activation function for all the fully connected layers. In the output layer, we use softmax activation function in the logistic regression nodes for predicting the label of each activity in the sequence and ReLU activation function in the regression node for predicting the starting time of the sequence. The parameters of the entire network are jointly optimized.

## 3.1.3. Model Training Approach

This training method di GLYPH&lt;11&gt; ers from our previous approach [12] in terms of the training procedure. We use the popular open-source deep learning package Keras [86] with TensorFlow [87] in the backend which has ready-to-use implementations of LSTM and fully connected layers. The input sequences for the LSTM are chosen in a sliding window manner with a stride of one for data augmentation. For example, to predict the labels of the future sequence containing the ( k + 1) th , ( k + 2) th and ( k + 3) th activities, activity features extracted from the k th , ( k GLYPH&lt;0&gt; 1) th and

We take the summation of the following two losses to compute the final loss. One is the cross-entropy loss function which is defined as follows:

$$\text{proach} \quad \mathcal { L } ( \mathbf X, \mathbf Y ) \ = - \frac { 1 } { n } \sum _ { i = 1 } ^ { n } \sum _ { j = 1 } ^ { c } \mathbf 1 ( y ^ { ( i ) } = j ) \\ \text{popular} \quad \text{$\times\log p(y^{(i)} = j| \mathbf x^{(i)} )$}$$

Here, X = f x (1) ; :::; x ( n ) g is the set of input feature vectors (activity features of the last three observed activities and features of the objects present in the last observed portion of the scene) in the training dataset, Y = f y (1) ; :::; y ( n ) g is the corresponding set of labels for those input features, and j = f 1 ; :::; c g is the set of class labels. 1 ( ) is an indicator function. : x ( ) i is the sequential

activity features extracted from the previous three activities.

Another is the mean squared loss between the ground truth inter-activity time and the predicted inter-activity time which is defined as follows:

$$\mathcal { L } ( \mathbf P, \mathbf Q ) & = \frac { 1 } { n } \sum _ { i = 1 } ^ { n } ( q ^ { ( i ) } - \hat { q } ^ { ( i ) } ) ^ { 2 } & ( 2 ) & \text{tionary} \\ & & \text{fixed-d}$$

Here, P = f p (1) ; :::; p ( n ) g is the set of input feature vectors (activity feature of the last observed activity) in the training dataset, and Q = f q (1) ; :::; q n ( ) g is the corresponding set of inter-activity times for those input features. ˆ q i ( ) represents the predicted inter-activity time given input p i ( ) where the ground truth interactivity time is q i ( ) . ˆ q i ( ) is a function of the input features P . The outputs of the training are the labels of the three future activities and the starting time of that activity sequence.

The parameters of the network are jointly optimized by minimizing both of these losses. To optimize the network, we use a stochastic gradient descent with an adaptive sub-gradient method (Adam) [88] which is popular for its impressive history of empirical success. We also tested with Adagrad [89], Adamax [88], Nadam [90] and RMSProp [91] but empirically chose Adam. We use Dropout layer [92] with a probability of 0 2 after each layer to prevent overfitting. : The batch size is set to 128. We use a learning rate of 0 001. :

## 3.2. Caption Generation for Activity Sequences

## 3.2.1. Role of Scene Context for Label to Caption Mapping

Motivated by the inspiring performance of the sequence-tosequence models in [59] for machine translation and in [31] for video to text mapping, we use a similar model for label to sentence mapping where both the input ( a1 ; a2 ; GLYPH&lt;1&gt; GLYPH&lt;1&gt; GLYPH&lt;1&gt; ; am ) and the output ( b1 ; b2 ; GLYPH&lt;1&gt; GLYPH&lt;1&gt; GLYPH&lt;1&gt; ; bn ) are sequences of words of variable length. Since the labels do not contain any object information, it is hard to predict the object in the caption only from the label. For example, it is di GLYPH&lt;14&gt; cult to map from wash to A person washed carrots . So, we use the scene context from the observed portion along with the label in the encoder LSTM input for meaningful mapping of the objects.

## 3.2.2. Network Architecture

The input to the encoder LSTM is text e.g., cut apart cucumber , take out egg fridge , cut o GLYPH&lt;11&gt; ends carrot , etc. obtained by concatenating the corresponding predicted labels and scene context. In the captions, verbs are followed by objects. To maintain this order, scene context follows the label in the text input. So, sequence-to-sequence learning via encoder LSTM is important here to incorporate this sequential information efficiently and maintain meaningful structure between subject, verb and objects. We do not provide subject as the text input since the subject is constant ( the person ) throughout the dataset. However, for any other dataset where di GLYPH&lt;11&gt; erent subjects exist e.g., man , woman , boy , girl etc., our network would take the text input in subject-verb-object order as a natural structure. An encoder-decoder LSTM pair is the best option for maintaining meaningful structure between subject, verb and object to incorporate this information correctly. Both the encoder LSTM and the decoder LSTM have 3 layers with 1000 memory units in each layer.

## 3.2.3. Model Training Approach

At first, we perform embedding by generating a dictionary using all the words in the input of the training set and then convert these words to one hot vectors according to that dictionary. We choose a vocabulary size of 20000. We use one LSTM layer to encode the label and the scene context to a fixed-dimensional vector and use another LSTM layer to generate a sentence from that vector. We estimate the conditional probability p ( b1 ; b2 ; GLYPH&lt;1&gt; GLYPH&lt;1&gt; GLYPH&lt;1&gt; ; bn a1 j ; a2 ; GLYPH&lt;1&gt; GLYPH&lt;1&gt; GLYPH&lt;1&gt; ; am ) given the input ( a1 ; a2 ; GLYPH&lt;1&gt; GLYPH&lt;1&gt; GLYPH&lt;1&gt; ; am ). In our case, since the caption is always longer than the combination of label and scene context, n is always bigger than m .

During encoding, the first LSTM generates a sequence of hidden states ( h1 ; h2 ; GLYPH&lt;1&gt; GLYPH&lt;1&gt; GLYPH&lt;1&gt; ; hm ) given the label and the scene context ( a1 ; a2 ; GLYPH&lt;1&gt; GLYPH&lt;1&gt; GLYPH&lt;1&gt; ; am ). Then a fixed-dimensional vector z corresponding to the label is generated by the last hidden state of the LSTM. The decoder LSTM computes the conditional probability of the output sentence given the input label and the scene context as follows:

$$p ( b _ { 1 }, b _ { 2 }, \cdots, b _ { n } | a _ { 1 }, a _ { 2 }, \cdots, a _ { m } ) \\ = \prod _ { d = 1 } ^ { n } p ( b _ { d } | z, b _ { 1 }, \cdots, b _ { d - 1 } ) \quad ( 3 )$$

where the distribution of p ( bd z j ; b1 ; GLYPH&lt;1&gt; GLYPH&lt;1&gt; GLYPH&lt;1&gt; ; bd 1 GLYPH&lt;0&gt; ) is represented by a softmax over all the words in the vocabulary.

During training, the log probability of a correct caption (sentence) is maximized given the label and the scene context. Cross-entropy loss function is used in this model. The batch size we use is 1000. Keras [86] with TensorFlow [87] is the library we use for this work.

## 3.3. Test Case Scenario

For predicting the labels of a future activity sequence, the activity features of the last three observed activities are used in the LSTM input, the features of the objects present in the last observed portion of the scene are used as the input of the first fully connected layers and the activity features of the last observed activity are used as the input of another two fully connected layer. Based on the learned sequence-to-sequence relationship in the training phase, the network predicts the labels of the next three unobserved activities. Using this predicted sequence of labels and observed scene context, the most likely captions for the future activity sequence are generated by the encoder-decoder LSTM pair.

## 4. Experiments

We conduct experiments on four challenging datasets: MPIICooking Dataset [56], MPII-Cooking 2 Dataset [57], EPICKITCHENS Dataset [58] (fine grained indoor activities) and VIRAT Ground Dataset [55] (coarse outdoor activities) to evaluate the performance of our label prediction framework for a future activity sequence. We provide the starting time prediction performance for a future sequence from [12] in the supplementary material. To evaluate the performance of our proposed captioning framework, we conduct experiments on the

challenging video description dataset TACoS Multi-Level Corpus [60] built on MPII-Cooking 2 [57]. The goal of the experiments is to compare our predictions with ground truth values as well as the state-of-the-arts, and perform an ablation analysis of the methods.

## 4.1. Datasets

MPII-Cooking Dataset. MPII-Cooking Dataset is a fine grained complex activity dataset where the participants interact with di GLYPH&lt;11&gt; erent tools, ingredients, and containers to complete a recipe. It has 65 di GLYPH&lt;11&gt; erent cooking activities recorded from 12 participants. There are 44 videos with a length of more than 8 hours. The dataset contains a total of 5 ; 609 annotations [56].

MPII-Cooking 2 Dataset. MPII-Cooking 2 Dataset is a fine grained complex activity dataset where the participants interact with di GLYPH&lt;11&gt; erent tools, ingredients, and containers to complete a recipe. It has 67 di GLYPH&lt;11&gt; erent cooking activities recorded from 30 participants. In total there are 273 videos with a length of more than 27 hours [57].

EPIC-KITCHENS Dataset. EPIC-KITCHENS Dataset is a large-scale egocentric video benchmark dataset comprised of cooking activities performed by 32 di GLYPH&lt;11&gt; erent subjects in their native kitchen environments. The dataset contains 55 hours of video consisting of 11 5 : M frames, 272 training videos, 39596 annotated segments, and 454 3 : K object bounding boxes. Considering all unique pairs of verb and noun in the public training set, there are total 2513 unique actions [58].

VIRAT Ground Dataset. VIRAT Ground Dataset is a challenging human activity dataset which consists of 11 di GLYPH&lt;11&gt; erent activities recorded in natural outdoor scenes with background clutter. There are total 329 videos with a length of around 5 hours [55]. However, we use only 275 of them as some videos have incomplete annotations.

TACoS Multi-Level Corpus. This video description dataset consists of 185 long indoor videos which contain di GLYPH&lt;11&gt; erent actors, fine-grained activities, and small objects in daily cooking scenarios. Each video is annotated by multiple turkers. For each video, there are detailed descriptions with at most 15 sentences, a short description (3-5 sentences), and a single sentence. Since, workers could describe videos without aligning each sentence to the video, the descriptions are natural and have a complex sentence structure [60].

Detailed description of these datasets are available in the supplementary material. These datasets are untrimmed, unlike the trimmed datasets popularly used for recognition tasks in activity analysis, and have context information. Since we are captioning unobserved future activities, we need untrimmed datasets containing natural sequences of activities with annotated video description. Because of these requirements, the choice of datasets on which our method can be demonstrated is limited. For example, we cannot use MPII-Cooking Dataset [56] or VIRAT Ground Dataset [55] for captioning evaluation as used in [12] for label prediction evaluation since they do not have human descriptions. We cannot use YouCookII Dataset [97] or Tasty Dataset [51] since they do not have the action label annotation required for training the label prediction network. We cannot use Activity Net Captions [38] because there are only 1 5 activity instances on average in each video which is : not enough to incorporate the sequential context for label prediction.

## 4.2. Features

We use C3D features pre-trained on the Sports-1M dataset [98] as activity features for MPII-Cooking Dataset, MPIICooking 2 Dataset, and VIRAT dataset. The C3D features are of size 4096 and extracted for each 16 frames with a temporal stride of 8 frames. Then we perform max pooling to get a fixedlength feature vector for the video. For EPIC-KITCHENS, we use the pre-extracted RGB and flow features for the sequential activity context and the faster R-CNN-based object features for the scene context provided by [17, 99]. We claim that our method is independent of any particular choice of feature. This is mentioned in Section 4.3 where using bag-of-word based Motion Boundary Histograms (MBH) [100] features gives similar label prediction results for MPII-Cooking Dataset. According to [101], MBH features are extracted around densely sampled points and a codebook is generated using k-means clustering for these 4000 words long features.

## 4.3. Label Prediction Results for Activity Sequences

Objective. The main objective of these experiments is to analyze how well our framework can predict the labels of a future unobserved activity sequence.

Performance Measures. The evaluation metrics we use are: 1. multi-class precision (Pr), 2. multi-class recall (Rc), and 3. overall accuracy for top-1 match, top-3 matches and top-5 matches. For all these metrics, the higher value indicates better prediction performance.

Compared Methods. We compare with existing recognition approaches by using our method to determine the labels of the observed activities. For recognition of the observed activities, we use the observed activity features from the ( i GLYPH&lt;0&gt; 2) th , ( i GLYPH&lt;0&gt; 1) th and i th activities to predict the label of the i th activity and so on. For MPII-Cooking Dataset, we compare with [93] which estimates the labels of the observed activities. We show that the performance of our prediction of observed activities, is better than that of the recognition method using a combination of CNN and LSTM [93]. For label prediction, we compare with a convolutional LSTM-based approach [95] and achieve better performance. For MPII-Cooking 2 Dataset, we compare with [57] which estimates the labels of the observed activities and show that the precision we achieve for prediction of the observed activities, is higher than that of the recognition method using a combination of dense trajectories and hand trajectories [57]. For EPIC-KITCHENS Dataset, we compare with [22] which estimates the labels of the observed activities and show that the accuracy we achieve for prediction of the observed activities, is higher than that of the recognition method using a model constrained to reason about the present [22]. For label prediction, we compare with the state-of-the-art rollingunrolling LSTM-based approach [17] and achieve comparable performance. Please note that we cannot compare recognition performance with [17] since they reported results only on the test set of EPIC-KITCHENS Dataset for which the ground

Table 1. Label recognition performance for all of the datasets. The best numerical results are marked in bold.

| Dataset             | Method                                  | Precision   | Recall    | Accuracy %(Top-1)   | Accuracy %(Top-3)   | Accuracy %(Top-5)   |
|---------------------|-----------------------------------------|-------------|-----------|---------------------|---------------------|---------------------|
| MPII-Cooking [56]   | CNN + LSTM [93] Proposed Method         | 34.8 72.1   | 51.7 69.3 | - 81.3              | - 93.3              | - 96.2              |
| MPII-Cooking 2 [57] | Dense trajectories [57] Proposed Method | 52.2 61.6   | - 55.6    | - 69.0              | - 88.3              | - 92.7              |
| EPIC-KITCHENS [58]  | Fusion [22] Proposed Method             | -           | - 3.5     | 6.0 11.4            | - 21.4              | - 27.2              |
| VIRAT [55]          | Sparse Autoencoder [94] Proposed Method | 3.9 - 52.1  | - 25.2    | 54.2 71.9           | - 86.5              | - 95.4              |

Table 2. Label prediction performance for all of the datasets. The best numerical results are marked in bold.

| Dataset             | Method                                  | Precision   | Recall   | Accuracy %(Top-1)   | Accuracy %(Top-3)   | Accuracy %(Top-5)   |
|---------------------|-----------------------------------------|-------------|----------|---------------------|---------------------|---------------------|
| MPII-Cooking [56]   | Convolutional LSTM [95] Proposed Method | -           | - 67.6   | 60.4 79.9           | - 93.0              | -                   |
|                     |                                         | 72.1        |          |                     |                     | 96.2                |
| MPII-Cooking 2 [57] | - Proposed Method                       | - 58.8      | - 53.3   | - 65.5              | - 82.4              | - 88.5              |
| EPIC-KITCHENS [58]  | RU-LSTM [17]                            | -           | -        | -                   | -                   | 35.3                |
|                     | Proposed Method                         | 6.6         | 3.7      | 11.3                | 23.6                | 29.6                |
| VIRAT [55]          | Graphical Model [10]                    | -           | -        | 68.5                | -                   | -                   |
|                     | SCR-Graph [96]                          | 21.3        | -        | -                   | -                   | -                   |
|                     | Proposed Method                         | 49.6        | 22.2     | 71.8                | 86.4                | 94.4                |

Table 3. Sequence prediction performance comparisons for all of the datasets. The best numerical results are marked in bold.

| Dataset             | Method                                | Accuracy % Next-to-Next Activity   | Accuracy % Next-to-Next-to-Next Activity   |
|---------------------|---------------------------------------|------------------------------------|--------------------------------------------|
| MPII-Cooking [56]   | Multi-step Prediction Proposed Method | 78.1 79.1                          | 77.5 78.1                                  |
| MPII-Cooking 2 [57] | Multi-step Prediction Proposed Method | 63.7 64.4                          | 62.6 63.5                                  |
| EPIC-KITCHENS [58]  | Proposed Method                       | 8.1                                | 6.6                                        |
| VIRAT [55]          | Multi-step Prediction Proposed Method | 70.7 71.5                          | 68.5 69.2                                  |

truth is not available since the test evaluation server is closed now. For VIRAT Dataset, there is a graphical model-based approach [10], a semantic scene labeling-based approach [11], and a graph reasoning network-based approach [96] for prediction of future unobserved activities. We compare our method with [10] and [96] and achieve higher accuracy for label prediction. We cannot compare with [11] since they use scene specific customized set of labels which are not annotated in the original dataset. We also compare with a state-of-the-art active learningbased recognition approach which uses sparse autoencoder [94] and achieve higher accuracy.

To evaluate our label prediction results for further activities in the future sequence, we compare with our previous multistep prediction baseline [12] where we predicted the next-tonext activity i.e., 2-step prediction (using activity features from the ( i GLYPH&lt;0&gt; 3) th , ( i GLYPH&lt;0&gt; 2) th and ( i GLYPH&lt;0&gt; 1) th activities, we predicted the label of the ( i + 1) th activity) and the next-to-next-to-next activity i.e., 3-step prediction. Multi-step prediction is di GLYPH&lt;11&gt; erent from sequence prediction. In multi-step prediction, each prediction step is treated as uncorrelated with the others, while in sequence prediction, the correlations are accounted for. For example, in multi-step prediction, we predict the label of the ( i + 1) th unob-

served activity using context information from the i th , ( i GLYPH&lt;0&gt; 1) th , and ( i GLYPH&lt;0&gt; 2) th activities one step at a time. But in sequence prediction, we predict the labels of the i th , ( i + 1) th , and ( i + 2) th activities together using context information from the ( i GLYPH&lt;0&gt; 1) th , ( i GLYPH&lt;0&gt; 2) th , and ( i GLYPH&lt;0&gt; 3) th activities. Since these individual activities are part of a single sequence and thus correlated with each other, the network learns the temporal ordering of the activities better when a sequence of activities is used as the output during training. We also compare our sequence prediction results with [13].

Experimental Setup. For experiments on MPII-Cooking Dataset, we use five fold leave-one-person-out cross validation approach. Among 12 subjects, we use 7 for training and 5 for testing. For each of the five training instances, we use 7 training subjects and 4 testing subjects for training, leaving 1 from that set for testing. This is done 5 times leaving 1 testing subject out and then averaging the results known as 'five-fold leave-one-person-out' cross validation. For experiments on MPII-Cooking 2 Dataset, we use the experimental setup (same train-test split) of [57]. For activity recognition experiments on EPIC-KITCHENS Dataset, we use the experimental setup (same train-test split) of [22] i.e. report results on the same validation set composed of the following kitchens: P03, P14, P23, and P30 from the public training set since the labels of the test sets (S1 and S2) are not available in the evaluation server anymore. For activity prediction experiments on EPIC-KITCHENS Dataset, we use the experimental setup (232 training videos and 40 validation videos) and protocol (1 sec anticipation time) of [17]. For experiments on VIRAT Ground Dataset, we use the first 170 videos for training and the rest of them for testing. The network is trained on a NVIDIA Tesla K80 GPU.

Recognition Performance. Comparisons of our recognition performance (for observed activities) on all of the datasets with the state-of-the-art recognition methods are shown in Table 1. The methods we compare to did not report all of the evaluation metrics we use - hence the missing values. Our method outperforms the state-of-the-arts for all of the datasets.

Prediction Performance. Our label prediction results (for unobserved future activities) for all of the datasets and comparison with other prediction methods for MPII-Cooking Dataset, EPIC-KITCHENS Dataset, and VIRAT Ground Dataset are shown in Table 2. It is seen that our method outperforms the prediction method proposed in [95] for MPII-Cooking, achieves comparable prediction performance with [17] for EPIC-KITCHENS Dataset, and outperforms the prediction method proposed in [10] and [96] for VIRAT. The methods we compare to did not report all of the evaluation metrics we use -hence the missing values. We achieve similar label prediction accuracy of 79 9% and 80 7% for MPII-Cooking Dataset : : using C3D and MBH features respectively which justifies the claim that our method is independent of the choice of features. Sequence prediction result comparisons with the baseline multi-step prediction method are shown in Table 3. For EPICKITCHENS Dataset, there might be other actions which are not annotated between two annotated actions. Since the dataset is not densely annotated, it is not meaningful to use the dataset for multi-step prediction because of the lack of reliable ground truth. Therefore, we do not report multi-step prediction results for EPIC-KITCHENS Dataset. As the prediction horizon increases, there is a gradual accumulation of error. It is to be noted that even for sequence prediction, prediction results for the first activity in the future sequence have higher accuracy than that of the next activities as we are still using scene context from the last observed portion of the scene which is related to the immediate future activity label. Using the RNN-based anticipation approach of [13] with ground truth observations, we achieve a Top-1 accuracy of only 26 3% for MPII-Cooking : Dataset. This is expected since [13] does not incorporate scene context or inter-activity context and the e GLYPH&lt;11&gt; ect of these context attributes are shown in Table 4 in Section 4.3.2.

## 4.3.1. Multiple Possibilities for Future Activity Labels

One particular activity sequence can have multiple possible outcomes. For example, 'wash objects' and 'peel' can be followed by either 'cut apart' and 'cut slices'. As the network has been trained on both of these possible sequences (in one case the network has probably seen 'cut apart' as the next activity and in another case 'cut slices' as the next activity), it is hard to say precisely which is the next activity. Earlier we mentioned that in case of multiple possibilities, such as while choosing between 'spice' or 'put in bowl' after 'wash objects', 'peel' and 'cut slices', a bowl in the scene increases the probability of the activity label being the latter one. But in these types of closely related activities ('cut apart' and 'cut slices'), scene context cannot contribute much as both of the activities require a knife. Therefore, we present the topk choices with the associated probabilities for each of them. In spite of having many closely related ambiguous activities ('cut dice', 'cut slices', 'cut apart') in the dataset, our top-1 match outperforms the baseline in terms of accuracy. Our method can also handle an 'unknown' label (never seen in training) when the probability of none of the predicted future activities crosses a threshold.

## 4.3.2. E ect of Di GLYPH&lt;11&gt; GLYPH&lt;11&gt; erent Context Attributes.

We perform an ablation study to justify the choice of our network. Using only sequential activity context and scene context (eliminating inter-activity time context), we get relatively lower label prediction accuracy for all of the datasets than that of our proposed network. Similarly, using only sequential activity context and inter-activity time context (eliminating scene context), we get lower label prediction accuracy than that of our proposed network for all of the datasets as shown in shown in Table 4.

## 4.3.3. Analysis of Observation Horizon

Here, we justify the choice of the observation horizon. We empirically chose a sequence length of 3 for preceding activity features as sequence length of 2, 5, 7 and 9 give relatively lower accuracy for MPII-Cooking Dataset as shown in Table 5. We believe that this is because the longer horizon brings in the e GLYPH&lt;11&gt; ect of activities that are not closely related.

Table 4. Ablation study for label prediction for all of the datasets. The best numerical results are marked in bold.

|                 | Top-1 Accuracy%   | Top-1 Accuracy%                      | Top-1 Accuracy%        |
|-----------------|-------------------|--------------------------------------|------------------------|
| Dataset         | Proposed Network  | Removing inter-activity time context | Removing scene context |
| MPII- Cooking   | 79.9              | 75.7                                 | 33.7                   |
| MPII- Cooking 2 | 65.5              | 60.2                                 | 45.7                   |
| EPIC- KITCHENS  | 11.3              | 8.7                                  | 4.1                    |
| VIRAT           | 71.8              | 69.2                                 | 61.0                   |

Table 5. Sequence length sensitivity analysis for MPII-Cooking Dataset. The best numerical result is marked in bold.

|   Top- 1 Accuracy % Sequence Length |   Top- 1 Accuracy % Sequence Length |   Top- 1 Accuracy % Sequence Length |   Top- 1 Accuracy % Sequence Length |   Top- 1 Accuracy % Sequence Length |
|-------------------------------------|-------------------------------------|-------------------------------------|-------------------------------------|-------------------------------------|
|                                 2   |                                 3   |                                 5   |                                 7   |                                 9   |
|                                78.6 |                                79.9 |                                79.1 |                                77.5 |                                76.9 |

## 4.4. Captioning Results for Activity Sequences

Objective. The objective of these experiments is to evaluate the quality of our generated captions against the ground truth captions annotated by the human annotators. More results are presented in the supplementary material.

Performance Measure. The evaluation metrics we use are BLEU (Bilingual Evaluation Understudy) [102], CIDEr (Consensus-based Image Description Evaluation) [103], METEOR (Metric for Evaluation of Translation with Explicit ORdering) [104], ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [105], and SPICE (Semantic Propositional Image Caption Evaluation) [106]. BLEU is a weighted average of variable length phrase matches against the reference translations in machine translation. CIDEr evaluates how well a candidate sentence matches the consensus of a set of image descriptions. METEOR uses the generalized concept of unigram matching between the machine produced translation and human-produced reference translations. ROUGE compares an automatically produced caption against a human-produced reference. We use ROUGE-L which naturally considers sentence level structure similarity and automatically identifies longest co-occurring in sequence n-grams. SPICE is an automatic image caption evaluation metric which compares semantic propositional content. In our case, the number of word matches is compared between the generated captions and the reference captions annotated by the human annotators. For all of the metrics, higher value indicates better performance.

Comparisons. To the best of our knowledge, no existing method for generating captions for future unobserved events in

Table 6. Comparisons of BLEU@ 4 (in percent) (B), CIDEr (C), METEOR (M), ROUGE-L (R), and SPICE (S) scores per sentence for short descriptions in TACoS Multi-Level Corpus. The best numerical results are marked in bold.

| Method                |    B | C     | M     | R     | S     |
|-----------------------|------|-------|-------|-------|-------|
| SR Based [60]         | 24.7 | -     | -     | -     | -     |
| Hierarchical RNN[37]  | 30.5 | 1.602 | 0.287 | -     | -     |
| Proposed (Observed)   | 47.6 | 1.774 | 0.352 | 0.725 | 0.162 |
| Baseline (Unobserved) | 10.4 | 0.821 | 0.194 | 0.319 | 0.083 |
| Proposed (Unobserved) | 39.2 | 1.493 | 0.302 | 0.631 | 0.132 |

videos has reported performance on TACoS Multi-Level Corpus. Therefore, as a baseline, we compare with the input text of the encoder LSTM with the ground truth captions in order to measure the improvement achieved by our caption generation module. We also compare with existing captioning approaches for the observed portion of the video by using our method to determine the captions of the observed activities. We compare with [60] which first predicts a semantic representation (SR) of the observed portion and then generates detailed captions. We compare against their per sentence BLEU@4 score for short descriptions. We also compare with the BLEU@4, CIDEr and METEOR scores reported in [37] which exploits hierarchical RNNs to generate captions for the observed portion.

Similar to label prediction, since none of the existing methods perform sequence prediction for captions, we can only compare our captioning result for the first unobserved activity with di GLYPH&lt;11&gt; erent state-of-the-art methods. However, to evaluate our captioning results for further activities in the future sequence, we compare with multi-step prediction baseline where we predict the next-to-next caption i.e., 2-step caption prediction and the next-to-next-to-next caption i.e., 3-step caption prediction. Multi-step captioning yields di GLYPH&lt;11&gt; erent results than sequence captioning because of the same reason as in label prediction.

Experimental Setup. For experiments on TACoS MultiLevel Corpus, we use the experimental setup (same test split) of [36] which has also been used in [60]. This information is provided with MPII-Cooking 2 Dataset [57]. We train our network on a NVIDIA Tesla K80 GPU.

Quantitative Evaluation. Comparisons of our video caption generation results on TACoS Multi-Level Corpus for observed activities with the state-of-the-art methods are shown in Table 6. Not all of the metrics are reported in [37, 60] - hence the missing values. Our method outperforms the state-of-the-arts [37, 60] for observed events. Our method outperforms the baseline for the unobserved events which shows that the generated captions are richer in context than the captions simply obtained by the estimated labels and objects. Quantitative comparisons for captioning a future sequence with the baseline multi-step caption prediction method for TACoS Multi-Level Corpus are shown in Table 7. As the prediction horizon increases, there is a gradual accumulation of error.

Qualitative Evaluation. Qualitatively, our generated cap-

Table 7. Sequence captioning performance comparisons for TACoS MultiLevel Corpus in terms of BLEU@ 4 (in percent) (B), CIDEr (C), METEOR (M), ROUGE-L (R), and SPICE (S) scores. The best numerical results are marked in bold.

| Method                                        |    B |     C |     M |     R |     S |
|-----------------------------------------------|------|-------|-------|-------|-------|
| Multi-step Captioning (Next-to-Next)          | 29.9 | 0.56  | 0.274 | 0.609 | 0.086 |
| Proposed Method (Next-to-Next)                | 30.2 | 0.588 | 0.291 | 0.614 | 0.112 |
| Multi-step Captioning (Next-to-Next-to- Next) | 19.8 | 0.548 | 0.254 | 0.593 | 0.079 |
| Proposed Method (Next-to-Next- to-Next)       | 20.6 | 0.557 | 0.264 | 0.598 | 0.103 |

tions like 'The person poured the co GLYPH&lt;11&gt; ee into a cup' is much more meaningful than simple texts like 'pour carafe co GLYPH&lt;11&gt; ee cup' used in the input of the encoder LSTM obtained from the predicted labels and objects. Qualitative Comparisons for captioning a future sequence with the baseline multi-step caption prediction method for TACoS Multi-Level Corpus are shown in Table 8. Fig. 4 depicts an example sequence showing both of our label prediction and captioning results.

Table 8. Qualitative comparisons of the generated erroneous captions for multi-step caption generation vs. proposed sequential captioning. Mistakes in the captions are marked in bold. Please note that the more we try to predict ahead, the more erroneous the generated captions become.

| No. of Steps   | Generated Captions (Multi-Step)                    | Generated Captions (Proposed Method)             | Reference Captions                                    |
|----------------|----------------------------------------------------|--------------------------------------------------|-------------------------------------------------------|
| 2 3            | The person sliced the leek The person took out egg | The person peeled the leek The person peeled egg | The person peeled the leek The person peeled the leek |

## 4.4.1. E ect of the Performance of Label Prediction GLYPH&lt;11&gt;

We show the BLEU@4, CIDEr, METEOR, ROUGE-L, and SPICE scores of our generated captions when generated from the ground truth activity labels and when generated from the predicted labels in Table 9. The corresponding qualitative comparisons for erroneous results are shown in Table 10. The type of mistakes made in the generated captions with predicted labels is mostly related to wrong verbs. This is expected since the information regarding the verbs comes from the labels. We get a label prediction accuracy of 65 5% with precision 58 8 : : and recall 53 3 for MPII-Cooking 2 Dataset [57] which gives : an idea about its e GLYPH&lt;11&gt; ect on the evaluation metrics in Table 9 obtained using ground truth labels and predicted labels.

Table 9. Comparisons of BLEU@ 4 (in percent) (B), CIDEr (C), METEOR (M), ROUGE-L (R), and SPICE (S) scores per sentence for short descriptions using ground truth labels vs. predicted labels for caption generation in TACoS Multi-Level Corpus.

| Labels Used                          |    B |     C |     M |     R |     S |
|--------------------------------------|------|-------|-------|-------|-------|
| Ground Truth Labels Predicted Labels | 44   | 1.615 | 0.351 | 0.729 | 0.155 |
|                                      | 39.2 | 1.493 | 0.302 | 0.631 | 0.132 |

Table 10. Qualitative comparisons of generated erroneous captions using ground truth labels vs. predicted labels for caption generation in TACoS Multi-Level Corpus. Mistakes in the captions are marked in bold.

| Human Description                                               | Generated Captions with Predicted Labels              | Generated Captions with Ground Truth Labels               |
|-----------------------------------------------------------------|-------------------------------------------------------|-----------------------------------------------------------|
| 1. The person sliced the carrot 2. The person chopped the herbs | The person peeled the carrot The person cut the herbs | The person sliced the carrot The person chopped the herbs |

## 4.4.2. E ect of Scene Context GLYPH&lt;11&gt;

MPII-Cooking 2 Dataset [57] has many small objects with similar shapes and appearances. Detecting and recognizing these small objects (sometimes with occlusion) in complex videos is a di GLYPH&lt;14&gt; cult problem itself. The performance of the object recognition method is crucial to the quality of the generated captions. The error of the object recognition method is propagated in two steps: first during label prediction using predicted scene context and then during the mapping from predicted scene context to objects in the captions.

Using the predicted scene context obtained by the object recognition method used in [57] (combining dense trajectories, hand trajectories and hand cSift features), we compute the BLEU@4, CIDEr, METEOR, ROUGE-L, and SPICE scores for TACoS Multi-Level Corpus. We show the evaluation metrics of our generated captions when generated from the ground truth scene context and from the predicted scene context using the above-mentioned object recognition method in Table 11. Please note that the evaluation metrics using our caption generation method with predicted scene context is higher than that of the compared methods as well. The corresponding qualitative comparisons for erroneous results are shown in Table 12. The type of mistakes made in the generated captions with predicted scene context is mostly related to wrong objects. This is expected since the information regarding the objects comes from the scene context. The mean AP using the above-mentioned

Fig. 4. An example activity sequence showing our label prediction and captioning results on TACoS Multi-Level Corpus.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00943v5-figure-1c1d4618-a3a9-4888-9edf-3eb1b100718e.jpg)

Table 11. Comparisons of BLEU@ 4 (in percent) (B), CIDEr (C), METEOR (M), ROUGE-L (R), and SPICE (S) scores per sentence for short descriptions using ground truth scene context vs. predicted scene context for caption generation in TACoS Multi-Level Corpus.

Table 13. Comparisons of BLEU@ 4 (in percent) (B), CIDEr (C), METEOR (M), ROUGE-L (R), and SPICE (S) scores per sentence for short descriptions using di GLYPH&lt;11&gt; erent length of observed activity sequences in TACoS MultiLevel Corpus. The best numerical results are marked in bold.

| Scene Context Used   |    B |     C |     M |     R |     S |
|----------------------|------|-------|-------|-------|-------|
| Ground Truth         | 39.2 | 1.493 | 0.302 | 0.631 | 0.132 |
| Predicted            | 30.8 | 1.033 | 0.292 | 0.623 | 0.126 |

Table 12. Qualitative comparisons of generated captions using ground truth scene context vs. using predicted scene context for caption generation in TACoS Multi-Level Corpus. Mistakes in the captions are marked in bold.

| Human Description                                                                     | Generated Captions with Predicted Scene Context                                 | Generated Captions with Ground Truth Scene Context                              |
|---------------------------------------------------------------------------------------|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------|
| 1. The person cut an orange in half 2. The person took a plum out of the refrigerator | The person cut the lime in half The person took a onion out of the refrigerator | The person cut the orange in half The person took plums out of the refrigerator |

object recognition method for MPII-Cooking 2 Dataset [57] is 43 7% which gives an idea about the relation between the per-: formance of the object recognition method and the performance of label prediction. This in turn shows the e GLYPH&lt;11&gt; ect of scene context on the performance of caption generation. A better object recognition method will lead to better captioning performance.

## 4.4.3. Analysis of Observation Horizon

We empirically find that a sequence length of 3 for preceding activity features provides best accuracy for label prediction in MPII-Cooking 2 Dataset [57]. While working with TACoS Multi-Level Corpus, we use observed sequence lengths of 2, 3, 5, 7 and 9 and achieved the highest BLEU@4, CIDEr, METEOR, ROUGE-L, and SPICE scores for caption generation

|   Observed Sequence Length |    B |     C |     M |     R |     S |
|----------------------------|------|-------|-------|-------|-------|
|                          2 | 22   | 1.117 | 0.257 | 0.584 | 0.104 |
|                          3 | 39.2 | 1.493 | 0.302 | 0.631 | 0.132 |
|                          5 | 24   | 1.078 | 0.262 | 0.597 | 0.107 |
|                          7 | 31.9 | 1.142 | 0.284 | 0.614 | 0.117 |
|                          9 | 38.5 | 1.156 | 0.297 | 0.628 | 0.128 |

in TACoS Multi-Level Corpus [60] with an observed sequence length of 3 as shown in Table 13. The corresponding qualitative analysis is given in Table 14. Although the number of wrong words in each sentence is similar, there is reasonable di GLYPH&lt;11&gt; erence in the values of the evaluation metrics in Table 13. This is because the label prediction performance changes as we change the observed sequence length and this in turn changes the number of such erroneously generated captions.

## 5. Conclusions

In this work, we proposed a solution to a novel problem of predicting the labels and captions of a sequence of future unobserved activities. We took advantage of the combination of LSTM and fully connected layers to exploit the contextual relationship among activities and objects for label prediction. For mapping the predicted labels and scene context to meaningful captions, we incorporated a sequence-to-sequence learningbased approach using an encoder-decoder LSTM pair. Rigorous experimental analysis on challenging datasets proves the robustness of our framework. In the future, we plan to extend our prediction method to a multi-camera environment and investigate how to predict new unseen activity classes. Another interesting direction for future work could be tackling the domain adaptation problem where we predict the labels and the

Table 14. Qualitative comparisons of the generated erroneous captions using di GLYPH&lt;11&gt; erent length of observed activity sequences in TACoS Multi-Level Corpus. Mistakes in the captions are marked in bold. Please note that in most of these erroneous examples, the verbs are incorrect as a result of incorrectly predicted labels.

|   Obs. Seq. Length | Predicted Labels   | Generated Captions            | Reference Captions         |
|--------------------|--------------------|-------------------------------|----------------------------|
|                  2 | cut apart          | The person cut apart the leek | The person peeled the leek |
|                  3 | peel               | The person peeled the leek    | The person peeled the leek |
|                  5 | slice              | The person sliced the leek    | The person peeled the leek |
|                  7 | peel               | The person peeled the leek    | The person peeled the leek |
|                  9 | slice              | The person sliced the leek    | The person peeled the leek |

captions of a dataset by training on the labels and the captions of a di GLYPH&lt;11&gt; erent dataset.

- [17] A. Furnari, G. Farinella, Rolling-unrolling lstms for action anticipation from first-person video, IEEE Transactions on Pattern Analysis and Machine Intelligence.

## 6. Acknowledgments

This work was partially supported by NSF grant 1724341 and ONRgrant N00014-19-1-2264. We gratefully acknowledge the support of NVIDIA with the donation of the Tesla K80 GPU used for this research.

## References

- [1] S. Cao, K. Chen, R. Nevatia, Activity recognition and prediction with pose based discriminative patch model, in: WACV, 2016, pp. 1-9.
- [2] K. Li, Y. Fu, Prediction of human activity by discovering temporal sequence patterns, IEEE TPAMI 36 (8) (2014) 1644-1657.
- [3] W. Li, M. Fritz, Recognition of ongoing complex activities by sequence prediction over a hierarchical label space, in: WACV, 2016, pp. 1-9.
- [4] M. S. Ryoo, Human activity prediction: Early recognition of ongoing activities from streaming videos, in: ICCV, 2011, pp. 1036-1043.
- [5] L. Wang, X. Zhao, Y. Si, L. Cao, Y. Liu, Context-associative hierarchical memory model for human activity recognition and prediction, IEEE TMM.
- [6] A. Zyner, S. Worrall, J. Ward, E. Nebot, Long short term memory for driver intent prediction, in: Intelligent Vehicles Symposium (IV), 2017, pp. 1484-1489.
- [7] B. Morris, A. Doshi, M. Trivedi, Lane change intent prediction for driver assistance: On-road design and evaluation, in: Intelligent Vehicles Symposium (IV), 2011, pp. 895-901.
- [8] C. L. McGhan, A. Nasir, E. M. Atkins, Human intent prediction using markov decision processes, Journal of Aerospace Information Systems 12 (5) (2015) 393-397.
- [9] E. C. Townsend, E. A. Mielke, D. Wingate, M. D. Killpack, Estimating human intent for physical human-robot co-manipulation, arXiv preprint arXiv:1705.10851.
- [10] A. Chakraborty, A. Roy-Chowdhury, Context-aware activity forecasting, in: ACCV, 2014, pp. 21-36.
- [11] K. M. Kitani, B. D. Ziebart, J. A. Bagnell, M. Hebert, Activity forecasting, in: ECCV, 2012, pp. 201-214.
- [12] T. Mahmud, M. Hasan, A. K. Roy-Chowdhury, Joint prediction of activity labels and starting times in untrimmed videos, in: ICCV, 2017, pp. 5773-5782.
- [13] Y. Abu Farha, A. Richard, J. Gall, When will you do what?-anticipating temporal occurrences of activities, in: CVPR, 2018, pp. 5343-5352.
- [14] N. Mehrasa, A. A. Jyothi, T. Durand, J. He, L. Sigal, G. Mori, A variational auto-encoder model for stochastic point processes, in: CVPR, 2019, pp. 3165-3174.
- [15] C. Sun, A. Shrivastava, C. Vondrick, R. Sukthankar, K. Murphy, C. Schmid, Relational action forecasting, in: CVPR, 2019, pp. 273-283.
- [16] Y. Abu Farha, J. Gall, Uncertainty-aware anticipation of activities, in: ICCV Workshops, 2019, pp. 0-0.
- [18] H. Gammulle, S. Denman, S. Sridharan, C. Fookes, Forecasting future action sequences with neural memory networks, arXiv preprint arXiv:1909.09278.
- [19] H. Gammulle, S. Denman, S. Sridharan, C. Fookes, Predicting the future: A jointly learnt model for action anticipation, in: ICCV, 2019, pp. 5562-5571.
- [20] Q. Ke, M. Fritz, B. Schiele, Time-conditioned action anticipation in one shot, in: CVPR, 2019, pp. 9925-9934.
- [21] J. Liang, L. Jiang, J. C. Niebles, A. G. Hauptmann, L. Fei-Fei, Peeking into the future: Predicting future person activities and locations in videos, in: CVPR, 2019, pp. 5725-5734.
- [22] A. Miech, I. Laptev, J. Sivic, H. Wang, L. Torresani, D. Tran, Leveraging the present to anticipate the future in videos, in: CVPR Workshops, 2019, pp. 0-0.
- [23] A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, D. Forsyth, Every picture tells a story: Generating sentences from images, in: ECCV, 2010, pp. 15-29.
- [24] G. Kulkarni, V. Premraj, V. Ordonez, S. Dhar, S. Li, Y. Choi, A. C. Berg, T. L. Berg, Babytalk: Understanding and generating simple image descriptions, IEEE TPAMI 35 (12) (2013) 2891-2903.
- [25] O. Vinyals, A. Toshev, S. Bengio, D. Erhan, Show and tell: A neural image caption generator, in: CVPR, 2015, pp. 3156-3164.
- [26] J. Johnson, A. Karpathy, L. Fei-Fei, Densecap: Fully convolutional localization networks for dense captioning, in: CVPR, 2016, pp. 45654574.
- [27] P. Das, C. Xu, R. F. Doell, J. J. Corso, A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching, in: CVPR, 2013, pp. 2634-2641.

[28]

S. Guadarrama, N. Krishnamoorthy, G. Malkarnenkar, S. Venugopalan,

R. Mooney, T. Darrell, K. Saenko, Youtube2text: Recognizing and de- scribing

arbitrary activities

using semantic

hierarchies and zero-shot

recognition, in: ICCV, 2013, pp. 2712-2719.

- [29] N. Krishnamoorthy, G. Malkarnenkar, R. J. Mooney, K. Saenko, S. Guadarrama, Generating natural-language video descriptions using text-mined knowledge., in: AAAI, Vol. 1, 2013, p. 2.
- [30] J. Donahue, L. A. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, T. Darrell, Long-term recurrent convolutional networks for visual recognition and description, in: CVPR, 2015, pp. 26252634.
- [31] S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, K. Saenko, Sequence to sequence-video to text, in: ICCV, 2015, pp. 4534-4542.
- [32] S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. Mooney, K. Saenko, Translating videos to natural language using deep recurrent neural networks, arXiv preprint arXiv:1412.4729.
- [33] L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, A. Courville, Describing videos by exploiting temporal structure, in: ICCV, 2015, pp. 4507-4515.
- [34] A. Barbu, A. Bridge, Z. Burchill, D. Coroian, S. Dickinson, S. Fidler, A. Michaux, S. Mussman, S. Narayanaswamy, D. Salvi, et al., Video in sentences out, arXiv preprint arXiv:1204.2742.
- [35] M. U. G. Khan, Y. Gotoh, Describing video contents in natural language, in: Proceedings of the Workshop on Innovative Hybrid Approaches to

the Processing of Textual Data, 2012, pp. 27-35.

- [36] M. Rohrbach, W. Qiu, I. Titov, S. Thater, M. Pinkal, B. Schiele, Translating video content to natural language descriptions, in: ICCV, 2013, pp. 433-440.
- [37] H. Yu, J. Wang, Z. Huang, Y. Yang, W. Xu, Video paragraph captioning using hierarchical recurrent neural networks, in: CVPR, 2016, pp. 45844593.
- [38] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, J. C. Niebles, Dense-captioning events in videos, arXiv preprint arXiv:1705.00754.
- [39] A. Rohrbach, M. Rohrbach, B. Schiele, The long-short story of movie description, in: German conference on pattern recognition, Springer, 2015, pp. 209-221.
- [40] X. Liang, Z. Hu, H. Zhang, C. Gan, E. P. Xing, Recurrent topic-transition gan for visual paragraph generation, in: ICCV, 2017, pp. 3362-3371.
- [41] A. Shin, K. Ohnishi, T. Harada, Beyond caption to narrative: Video captioning with multiple sentences, in: ICIP, IEEE, 2016, pp. 3364-3368.
- [42] H. Yu, S. Cheng, B. Ni, M. Wang, J. Zhang, X. Yang, Fine-grained video captioning for sports narrative, in: CVPR, 2018, pp. 6006-6015.
- [43] Y. Xiong, B. Dai, D. Lin, Move forward and tell: A progressive generator of video descriptions, in: ECCV, 2018, pp. 468-483.
- [44] R. Shetty, J. Laaksonen, Frame-and segment-level features and candidate pool evaluation for video caption generation, in: ACM International Conference on Multimedia, 2016, pp. 1073-1076.
- [45] J. Xu, T. Yao, Y. Zhang, T. Mei, Learning multimodal attention lstm networks for video captioning, in: ACM International Conference on Multimedia, 2017, pp. 537-545.
- [46] S. Wang, D. Guo, X. Xu, L. Zhuo, M. Wang, Cross-modality retrieval by joint correlation learning, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 15 (2s) (2019) 1-16.
- [47] M. Chowdhury, P. Rameswar, E. Papalexakis, A. K. Roy-Chowdhury, Webly supervised joint embedding for cross-modal image-text retrieval, in: ACM International Conference on Multimedia, 2018, pp. 18561864.
- [48] C. Yan, Y. Tu, X. Wang, Y. Zhang, X. Hao, Y. Zhang, Q. Dai, Stat: spatial-temporal attention mechanism for video captioning, IEEE Transactions on Multimedia.
- [49] L. Gao, Z. Guo, H. Zhang, X. Xu, H. T. Shen, Video captioning with attention-based lstm and semantic consistency, IEEE Transactions on Multimedia 19 (9) (2017) 2045-2055.
- [50] J. Dong, X. Li, C. G. Snoek, Predicting visual features from text for image and video caption retrieval, IEEE Transactions on Multimedia 20 (12) (2018) 3377-3388.
- [51] F. Sener, A. Yao, Zero-shot anticipation for instructional activities, in: ICCV, 2019, pp. 862-871.
- [52] W. Elmannai, K. Elleithy, Sensor-based assistive devices for visuallyimpaired people: current status, challenges, and future directions, Sensors 17 (3) (2017) 565.
- [53] M. Owayjan, A. Hayek, H. Nassrallah, M. Eldor, Smart assistive navigation system for blind and visually impaired individuals, in: International Conference on Advances in Biomedical Engineering (ICABME), 2015, pp. 162-165.
- [54] C. Vondrick, H. Pirsiavash, A. Torralba, Anticipating the future by watching unlabeled video, arXiv preprint arXiv:1504.08023.
- [55] S. Oh, A. Hoogs, A. Perera, N. Cuntoor, C. Chen, J. T. Lee, S. Mukherjee, J. Aggarwal, H. Lee, L. Davis, et al., A large-scale benchmark dataset for event recognition in surveillance video, in: CVPR, 2011, pp. 3153-3160.
- [56] M. Rohrbach, S. Amin, M. Andriluka, B. Schiele, A database for fine grained activity detection of cooking activities, in: CVPR, 2012, pp. 1194-1201.
- [57] M. Rohrbach, A. Rohrbach, M. Regneri, S. Amin, M. Andriluka, M. Pinkal, B. Schiele, Recognizing fine-grained and composite activities using hand-centric features and script data, IJCV 119 (3) (2016) 346-373.
- [58] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, M. Wray, Scaling egocentric vision: The epic-kitchens dataset, in: ECCV, 2018.
- [59] I. Sutskever, O. Vinyals, Q. V. Le, Sequence to sequence learning with neural networks, in: NIPS, 2014, pp. 3104-3112.
- [60] A. Rohrbach, M. Rohrbach, W. Qiu, A. Friedrich, M. Pinkal, B. Schiele, Coherent multi-sentence video description with variable level of detail, in: GCPR, 2014, pp. 184-195.
- [61] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural computation 9 (8) (1997) 1735-1780.
- [62] A. Graves, N. Jaitly, Towards end-to-end speech recognition with recurrent neural networks., in: ICML, Vol. 14, 2014, pp. 1764-1772.
- [63] P. H. Pinheiro, R. Collobert, Recurrent convolutional neural networks for scene labeling., in: ICML, 2014, pp. 82-90.
- [64] F. Visin, M. Ciccone, A. Romero, K. Kastner, K. Cho, Y. Bengio, M. Matteucci, A. Courville, Reseg: A recurrent neural network-based model for semantic segmentation, in: CVPR Workshops, 2016, pp. 4148.
- [65] P. Lade, N. C. Krishnan, S. Panchanathan, Task prediction in cooking activities using hierarchical state space markov chain and object based task grouping, in: IEEE International Symposium on Multimedia, IEEE, 2010, pp. 284-289.
- [66] X. Wei, P. Lucey, S. Vidas, S. Morgan, S. Sridharan, Forecasting events using an augmented hidden conditional random field, in: ACCV, 2014, pp. 569-582.
- [67] D. A. Huang, K. M. Kitani, Action-reaction: Forecasting the dynamics of human interaction, in: ECCV, 2014, pp. 489-504.
- [68] T. Lan, T. C. Chen, S. Savarese, A hierarchical representation for future action prediction, in: ECCV, 2014, pp. 689-704.
- [69] N. Rhinehart, K. M. Kitani, First-person activity forecasting with online inverse reinforcement learning, in: ICCV, 2017, pp. 3696-3705.
- [70] A. Kojima, T. Tamura, K. Fukunaga, Natural language description of human activities from video images based on concept hierarchy of actions, IJCV 50 (2) (2002) 171-184.
- [71] M. W. Lee, A. Hakeem, N. Haering, S.-C. Zhu, Save: A framework for semantic annotation of visual events, in: CVPR Workshop, 2008, pp. 1-8.
- [72] M. U. G. Khan, L. Zhang, Y. Gotoh, Human focused video description, in: ICCV Workshop, 2011, pp. 1480-1487.
- [73] M. U. G. Khan, L. Zhang, Y. Gotoh, Towards coherent natural language description of video streams, in: ICCV Workshop, 2011, pp. 664-671.
- [74] P. Hanckmann, K. Schutte, G. J. Burghouts, Automated textual descriptions for a wide range of video events with 48 human actions, in: ECCV, 2012, pp. 372-380.
- [75] C. Sun, R. Nevatia, Semantic aware video transcription using random forest classifiers, in: ECCV, 2014, pp. 772-786.
- [76] R. Xu, C. Xiong, W. Chen, J. J. Corso, Jointly modeling deep video and compositional text to bridge vision and language in a unified framework., in: AAAI, Vol. 5, 2015, p. 6.
- [77] H. Yu, J. M. Siskind, Learning to describe video with weak supervision by exploiting negative sentential information., in: AAAI, 2015, pp. 3855-3863.
- [78] H. Xu, S. Venugopalan, V. Ramanishka, M. Rohrbach, K. Saenko, A multi-scale multiple instance video description network, arXiv preprint arXiv:1505.05914.
- [79] W. Choi, K. Shahid, S. Savarese, Learning context for collective activity recognition, in: CVPR, 2011, pp. 3273-3280.
- [80] Z. Deng, M. Zhai, L. Chen, Y. Liu, S. Muralidharan, M. Roshtkhari, J. Mehrsan, G. Mori, Deep structured models for group activity recognition, arXiv preprint arXiv:1506.04191.
- [81] M. Ibrahim, S. Muralidharan, Z. Deng, A. Vahdat, G. Mori, A hierarchical deep temporal model for group activity recognition, in: CVPR, 2016, pp. 1971-1980.
- [82] T. Lan, Y. Wang, W. Yang, G. Mori, Beyond actions: Discriminative models for contextual group activities, in: NIPS, 2010, pp. 1216-1224.
- [83] X. Wang, Q. Ji, Video event recognition with deep hierarchical context model, in: CVPR, 2015, pp. 4418-4427.
- [84] B. Yao, L. Fei-Fei, Modeling mutual context of object and human pose in human-object interaction activities, in: CVPR, 2010, pp. 17-24.
- [85] Y. Zhu, N. M. Nayak, A. Roy-Chowdhury, Context-aware modeling and recognition of activities in video, in: CVPR, 2013, pp. 2491-2498.
- [86] F. Chollet, Keras (2015).
- [87] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al., Tensorflow: Large-scale machine learning on heterogeneous distributed systems, arXiv preprint arXiv:1603.04467.
- [88] D. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980.
- [89] J. Duchi, E. Hazan, Y. Singer, Adaptive subgradient methods for online learning and stochastic optimization, Journal of Machine Learning

Research 12 (Jul) (2011) 2121-2159.

- [90] T. Dozat, Incorporating nesterov momentum into adam.(2016), Dostupn´ e z: http: // cs229.stanford.edu proj2015 054 report.pdf. / /
- [91] T. Tieleman, G. Hinton, Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude, COURSERA: Neural networks for machine learning 4 (2) (2012) 26-31.
- [92] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting., Journal of Machine Learning Research 15 (1) (2014) 1929-1958.
- [93] B. Ni, X. Yang, S. Gao, Progressively parsing interactional objects for fine grained action detection, in: CVPR, 2016, pp. 1020-1028.
- [94] M. Hasan, A. Roy-Chowdhury, Continuous learning of human activity models using deep nets, in: ECCV, 2014, pp. 705-720.
- [95] F. Rezazadegan, S. Shirazi, M. Baktashmotlagh, L. Davis, On encoding temporal evolution for real-time action prediction, International Journal of Computer Vision (2019) 1-7.
- [96] B. Chen, D. Li, Y. He, C. Hua, Scr-graph: Spatial-causal relationships based graph reasoning network for human action prediction, arXiv preprint arXiv:1912.05003.
- [97] L. Zhou, C. Xu, J. J. Corso, Towards automatic learning of procedures from web instructional videos, arXiv preprint arXiv:1703.09788.
- [98] D. Tran, L. Bourdev, R. F. L. Torresani, M. Paluri, Learning spatiotemporal features with 3d convolutional networks, in: ICCV, 2015, pp.

4489-4497.

- [99] A. Furnari, G. M. Farinella, What would you expect? anticipating egocentric actions with rolling-unrolling lstms and modality attention, in: ICCV, 2019, pp. 6252-6261.
- [100] N. Dalal, B. Triggs, C. Schmid, Human detection using oriented histograms of flow and appearance, in: ECCV, 2006, pp. 428-441.
- [101] H. Wang, A. Kl¨ aser, C. Schmid, C. Liu, Action recognition by dense trajectories, in: CVPR, 2011, pp. 3169-3176.
- [102] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th annual meeting on association for computational linguistics, 2002, pp. 311-318.
- [103] R. Vedantam, C. Lawrence Zitnick, D. Parikh, Cider: Consensus-based image description evaluation, in: CVPR, 2015, pp. 4566-4575.
- [104] S. Banerjee, A. Lavie, Meteor: An automatic metric for mt evaluation with improved correlation with human judgments, in: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and or summarization, 2005, pp. 65-72. /
- [105] C. Lin, Recall-oriented understudy for gisting evaluation (rouge), Retrieved August 20.
- [106] P. Anderson, B. Fernando, M. Johnson, S. Gould, Spice: Semantic propositional image caption evaluation, in: ECCV, Springer, 2016, pp. 382-398.
Tahmida Mahmud Mohammad Billah Mahmudul Hasan Amit K. Roy-Chowdhury
Prediction and Description of Near-Future Activities in Video
Most of the existing works on human activity analysis focus on recognition or
early recognition of the activity labels from complete or partial observations.
Similarly, almost all of the existing video captioning approaches focus on the
observed events in videos. Predicting the labels and the captions of future
activities where no frames of the predicted activities have been observed is a
challenging problem, with important applications that require anticipatory
response. In this work, we propose a system that can infer the labels and the
captions of a sequence of future activities. Our proposed network for label
prediction of a future activity sequence has three branches where the first
branch takes visual features from the objects present in the scene, the second
branch takes observed sequential activity features, and the third branch
captures the last observed activity features. The predicted labels and the
observed scene context are then mapped to meaningful captions using a
sequence-to-sequence learning-based method. Experiments on four challenging
activity analysis datasets and a video description dataset demonstrate that our
label prediction approach achieves comparable performance with the
state-of-the-arts and our captioning framework outperform the
state-of-the-arts.