## MPT-PAR:Mix-Parameters Transformer for Panoramic Activity Recognition

Wenqing Gan, Yan Sun, Feiran Liu, Xiangfeng Luo

Abstract -The objective of the panoramic activity recognition task is to identify behaviors at various granularities within crowded and complex environments, encompassing individual actions, social group activities, and global activities. Existing methods generally use either parameter-independent modules to capture task-specific features or parameter-sharing modules to obtain common features across all tasks. However, there is often a strong interrelatedness and complementary effect between tasks of different granularities that previous methods have yet to notice. In this paper, we propose a model called MPT-PAR that considers both the unique characteristics of each task and the synergies between different tasks simultaneously, thereby maximizing the utilization of features across multi-granularity activity recognition. Furthermore, we emphasize the significance of temporal and spatial information by introducing a spatio-temporal relationenhanced module and a scene representation learning module, which integrate the the spatio-temporal context of action and global scene into the feature map of each granularity. Our method achieved an overall F 1 score of 47.5% on the JRDB-PAR dataset, significantly outperforming all the state-of-the-art methods.

a unified framework capable of recognizing behaviors at different granularities simultaneously in crowded and complex scenes, including individual actions, social group activities, and global activities. This task does not impose restrictions on the number of people in the scene and allows for the subdividing groups. Recognizing behaviors of different granularities within a unified framework leverages the complementary relationships between tasks, as different granularities introduce diverse inductive biases that broaden the model's perspective, enhancing its generalization ability to unknown data distributions.

Index Terms -Human Activity Understanding, Group Activity, Panoramic Activity Recognition.

## I. INTRODUCTION

D UE to the extensive application of human activity recognition tasks in various fields such as video surveillance [1], [2], [45], healthcare [7], [32], [33] and sports analysis [3], [4], [46], this task has become crucial in video understanding. Previously, human activity recognition primarily focused on single-granularity tasks, such as recognizing individual actions [27], [28], [48] or group activities [29]-[31]. These tasks require a fixed number of people in the scene and assume that, in multi-person scenes, all individuals belong to a single group without any subdividable subgroups. These assumptions are often unrealistic in real-world scenarios, thereby limiting the utility of these methods. Moreover, focusing on singlegranularity tasks makes it difficult to understand the entire scene, while tasks of different granularities can mutually reinforce each other. For instance, accurate individual actions recognition can help the model learn the commonalities of individual behaviors, thereby aiding in more accurate group activity recognition. To address these issues, the panoramic activity recognition task, as proposed by [5], aims to establish

This work was supported in part by the National Natural Science Foundation of China under Grant 62002215, and in part by the Shanghai Pujiang Program under Grant 20PJ1404400.

Wenqing Gan, Yan Sun, Feiran Liu and Xiangfeng Luo are with the School of Computer Engineering and Science, Shanghai University, Shanghai 200444, China (e-mail: ganwenqing@shu.edu.cn; yansun@shu.edu.cn; shirleyllfr@shu.edu.cn; luoxf@shu.edu.cn).

To establish a unified framework for simultaneous multitask learning, the most critical factor is the structure design-specifically, determining which parts are task-specific and which are shared across tasks. This significantly impacts performance, training efficiency, and generalization. Current methods utilize parameter-sharing backbones and individual feature relationship learning modules to learn the foundational features common to multi-granularity activity recognition tasks, and then use multiple task-specific parameterindependent classification heads for multitask recognition. For cross-granularity aggregation modules, JRDB-PAR [5] uses parameter-independent modules to better capture specific features and nuances of each task, avoiding optimization conflicts of different task supervision signals; on the other hand, MUP [6] uses parameter-sharing modules to extract common relationships and motion features, promoting information sharing and enhancing overall performance. However, these methods do not effectively to simultaneously leverage the advantages of both aggregation strategies and overlook their complementary effects. Besides, since social groups and global activities are aggregated from individual actions, leveraging temporal information of individuals can lead to a better understanding of the scenarios. However, this requires multi-object tracking and group evolution detection, and they are very difficult to implement in challenging panoramic scenes. Therefore, previous methods ignored temporal information and relied on single-frame recognition, resulting in suboptimal performance. Furthermore, previous methods performed panoramic activity recognition using only cropped individual features, limiting the features to a local receptive field. As a result, they inevitably neglect the advantages of the global contextual information.

In this work, we propose a novel network called MixParameters Transformer for Panoramic Activity Recognition (MPT-PAR) to address those challenges. A cross-granularity aggregation module is introduced to integrate features from

individuals to social groups and from individuals to the global level. This module, comprising structurally identical parameter-independent and parameter-sharing encoders, models relationships between individuals through multi-head selfattention and aggregates individual behavior information via a learnable cls token. The fused features from both encoders are combined, to capture common relationships and motion features while retaining specific task features and nuances. To exploit temporal information, we follow the conventional approach for multi-object tracking and group evolution detection in group activity recognition, which is closely related to panoramic activity recognition. We also propose spatiotemporal relation-enhanced module, which model spatial relationships between different individuals in the same frame and temporal relationships of the same individual across frames. To provide comprehensive global context, we expand the receptive field by modeling scene information of the entire video frame, capturing behavior-related elements through spatial attention, flexibly integrating scene representations and activity features through cross-attention. Extensive experiments are conducted on the JRDB-PAR dataset to validate the effectiveness of the proposed network. Our method achieved an overall F 1 score of 47.5%, significantly improving by 6.0% compared to state-of-the-art methods. Additionally, it shows the greatest improvement in the sub-task of global activity recognition compared to previous methods, with an F 1 score of 61.1%. We also compared our approach with SOTA methods that perform single-granularity tasks, the significantly improvement demonstrating the mutual reinforcement between tasks of different granularities.

The main contributions of this paper are:

- · We propose a novel network MPT-PAR, it uses both parameter-sharing and parameter-independent aggregation encoders for cross-granularity feature integration, leveraging the complementary advantages to improve panoramic activity recognition.
- · We introduce spatio-temporal relation-enhanced module to utilize temporal information by enhancing spatial and temporal relationships for individual features.
- · We explicitly model scene information to provide comprehensive global context, captures behavior-related elements through spatial attention, and integrate scene representations with activity features through cross-attention.
- · Conducting extensive experiments on the JRDB-PAR dataset, we demonstrate the effectiveness of proposed method with an overall F 1 score of 47.5%, showing a significant improvement of 6.0% over state-of-the-art methods.

## II. RELATED WORK

## A. Single Granularity Activity Recognition

1) Human Action Recognition: Human action recognition (HAR) is a critical research area in computer vision that aims to automatically identify various human actions from video data. The advancement of deep learning has led to the development of numerous models based on Convolutional Neural Networks (CNN) [34]-[36] and Recurrent Neural

Networks (RNN) [37]-[39] for this purpose. A milestone in this field is the Two-Stream Convolutional Network [34], which utilizes static frame images and optical flow images to separately capture spatial and temporal features, resulting in significant performance gains. Additionally, RNN architectures like Long Short-Term Memory (LSTM) [37], [38] and Gated Recurrent Units (GRU) [39] have been extensively applied to video sequence data, further improving action recognition accuracy. In recent years, Transformer-based frameworks [40], [41], [49] have been incorporated to enhance the model's ability to focus on key frames and regions, thereby boosting recognition performance. These techniques capture intricate spatio-temporal relationships and feature interactions, achieving state-of-the-art performance on several public datasets. Moreover, multimodal fusion approaches, such as networks combining depth information [47] and skeletal data [42], have shown superior performance in complex scenarios.

2) Social Group Detection and Activity Recognition: The task of social group detection aims to divide people into subgroups based on their social activities or relationships. Building on this concept, social activity recognition aims to simultaneously identify the social activities of each subgroup. Early methods focused primarily on social group detection due to hardware limitations and small datasets. Solera et al. [43] manually designed features representing body and social identity to understand relationships among group members and used trajectory clustering to detect social groups. Wang et al. [20] introduced a benchmark for social group detection with a 'global to local magnification' framework that encodes global trajectories and local interactions, yielding promising results. Recently, the focus has shifted towards social activity recognition. Ehsanpour et al. [23] developed an end-to-end trainable network using integrated self-attention and graph attention modules to recognize social subgroups and their behaviors. However, this method oversimplifies social activities by treating them as the dominant individual actions within each group, which is not practical in many real-world scenarios. To address this, Kim et al. [44] created a large-scale highdefinition dataset specifically for social behavior detection and proposed a new Transformer-based model capable of handling an unknown number of groups and potential group members without relying on clustering algorithms.

3) Group Activity Recognition: It is a subfield of human activity recognition, focuses on identifying activities involving multiple individuals. The early work in this field primarily relied on manually designed features to extract information from video frames. Hierarchical graphical models [8] and dynamic Bayesian networks [9] were commonly used to interpret group activities in video environments. With the advent of deep learning, recent approaches have favored using deep neural networks for group activity recognition tasks. Yuan et al. [10] proposed a GCN-based method through a Dynamic Relation module and Dynamic Walk module for spatio-temporal individual inference. Xie et al. [11] introduced an actor-centric causality graph that focuses on analyzing the impact of two actors on asynchronous causality detection, complementing the learned asynchronous relationships with the synchronous ones derived from a Transformer model.

GroupFormer [12] employs a clustering attention mechanism, leveraging a clustering space-time Transformer to enhance individual and group representations by integrating spatial and temporal dependencies. Addressing the reliance on bounding box annotations for group activity recognition, a detectorfree Transformer-based model called DFWSGAR [13] was proposed, which locates and encodes parts of the group activity context through attention mechanisms, aggregating them into a single group representation without requiring bounding box labels or object detectors.

Overall, these methods focus solely on single-granularity tasks and they impose some unrealistic limitations. For instance, human action recognition often requires videos to contain only one or a few individuals, with the actors occupying the main part of the frame in each scene. In group activity recognition, the number of people in the video is usually fixed, and all individuals are assumed to belong to a single group without subdivisions. Those unrealistic assumptions significantly limit the application of these methods. Additionally, single-granularity task models did not realize the mutual reinforcement between tasks at different levels. In contrast, the objective of this work is to simultaneously achieve individual action recognition, social group activity recognition, and global activity recognition in crowded scenes, making it more practical for many real-world applications.

## B. Panoramic Activity Recognition

Panoramic activity recognition aims to identify individual actions, social group activities, and global activities concurrently. As a newly proposed and challenging problem, this field remains underexplored with limited current work. Han et al. [5] proposed a novel hierarchical graph neural network to progressively represent and model multi-granularity activities and mutual social relationships within crowds. By modeling each individual as a graph node, a graph network is constructed. Multiple parameter-independent GCN-based cross-granularity aggregation modules aggregate individual feature nodes into social group nodes bottom-up. Similarly, social group nodes are further aggregated into global nodes. This parameterindependent aggregation approach better captures the specific features and nuances of each task, avoiding optimization conflicts in supervising signals on the same parameters and improving individual task performance. In contrast, MUP [6] uses a unified cross-granularity aggregation module to encode behaviors at different granularity levels with the same parameters in an end-to-end manner. This approach learns potentially generic motion patterns across multi-granularity human behaviors, facilitating information sharing among tasks and enhancing the performance of panoramic activity recognition sub-tasks.

However, these methods only use either parameterindependent or parameter-sharing modules to integrate crossgranularity activity features, overlooking the complementary strengths of these two methods. Besides, these methods perform activity recognition at each granularity based solely on individual features extracted from a single key frame, without considering temporal information and the overall scene context.

## III. PROPOSED APPROACH

## A. Overview

The architecture of our network is illustrated in Fig. 1. The backbone is a ResNet-18 network [14] pre-trained on ImageNet. For the input video frame sequence X img ∈ R T × × 3 H × W ( T indicates the number of frames), the backbone extracts feature maps resulting in scene features X scene ∈ R T × × C H ′ × W ′ ( C means the number of channels). Each individual's features X ∈ R T × N × D ( N , D represent the number of individuals and the dimension of individual's features respectively) are then cropped using RoIAlign [15]. The scene features X scene are passed through a scene representation learning module to obtain the scene representation. The cropped individual features X are enhanced by spatio-temporal relation-enhanced module (STRE), which enrich individual features with spatio-temporal relational context. Subsequently, parameter-sharing and independent crossgranularity aggregation module (PSICGA) is employed to integrate individual features into social group and global representations. Before classifying the individual, social group, and global features, they are fused with scene representations through cross-attention to fully exploit the scene context.

## B. Spatio-Temporal Relation-Enhanced Module (STRE)

Previous methods ignored temporal information and relied solely on single frames for panoramic human activity recognition. However, actions are typically executed over a sequence of continuous frames. Ignoring temporal information can result in the loss of action details, leading to inaccurate understanding of actions by the model, which in turn affects the recognition of social group activity and global activity. Moreover, in real-world scenarios, interpersonal relationships are dynamically changing, making it unreliable to infer groupings of people based on a single image.

To address this issue, we introduce temporal information and use spatio-temporal relation-enhanced module to obtain features enriched with temporal and spatial context. Our STRE module consist of a transformer-based [17] temporal encoder and a spatial encoder, which learn temporal and spatial individual contextual features, respectively. Inspired by TimeSformer [16], we connect the temporal and spatial encoders serially to reduce computational complexity. Experiments demonstrated that the serial structure is simpler and more effective (details in Section IV-D2).

1) Spatial Encoder: In crowded multi-person scenes, individual actions are not entirely independent but are significantly influenced by interactions with others and the behaviors of those present in the scene. Therefore, considering spatial context is crucial for panoramic human activity recognition. We employ a transformer-based spatial encoder to learn spatial contextual information of individuals. The spatial encoder captures interaction relationships between the target individual and surrounding individuals by assigning relative importance weights.

Given the input individual representation X ∈ R T × N × D , we treat the temporal dimension as the batch dimension and use different learnable projection matrix to map X to

Fig. 1. Overview of our proposed model.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00420v1-figure-4deafc8e-e1c6-49b7-af54-c6b784fc3f49.jpg)

Q K , and V . The spatial relationship attention weights of neighboring individuals are then computed using softmax , and the weights are multiplied by V to obtain the spatial context-enhanced feature X S ∈ R T × N × D . For the t -th frame, this process can be expressed as:

$$Q ^ { t } = X ^ { t } W _ { t q }, K ^ { t } = X ^ { t } W _ { t k }, V ^ { t } = X ^ { t } W _ { t v } \quad ( 1 ) \ \ C. \ P a \$$

$$X _ { s } ^ { t } = F F N ( s o f t m a x ( \frac { Q ^ { t } K ^ { t } } { \sqrt { D } } ) V ^ { t } + V ^ { t } ) \quad ( 2 ) \quad \text{para}$$

where W tq , W tk and W tv are learnable projection matrices, and FFN is a feedforward network.

2) Temporal Encoder: Considering the temporal correlation of panoramic activities, the temporal encoder captures temporal contextual information by computing the adaptive importance of individuals across consecutive frames, enriching the temporal features of individuals. The structure of the temporal encoder is identical to that of the spatial encoder. The difference lies in treating the spatial dimension as the batch dimension for the temporal encoder. We transpose the temporal and spatial dimensions of X s , reshaping it to X ′ s ∈ R N × × T D , and feed it into the temporal encoder, resulting in the timecontext-enhanced individual representation X t ∈ R N × × T D .

As temporal information is only considered at the individual level, we average the time dimension of the enhanced individual features to obtain X st ∈ R N × D , serving as the foundation for constructing social group and global features.

## C. Parameter-Sharing and Independent Cross-Granularity Aggregation Module (PSICGA)

As a multi-granularity and multi-task learning task, panoramic activity recognition requires simultaneously addressing individual action recognition, social group activity recognition, and global activity recognition. Using parameter-independent aggregation modules for bottom-up cross-granularity feature integration helps capture task-specific features relevant to each task and avoids optimization conflicts in shared parameters, thereby enhancing multi-task learning efficiency. However, as indicated by MUP [6], different parameter modules model activities at various granularities separately, making it impossible for the model to learn potential general motion patterns across different granularities of human behavior. Therefore, we design a module that integrates parameter-sharing and parameter-independent mechanisms to learn task-specific features independently and alleviate optimization conflicts between tasks, while capturing common be-

Fig. 2. Structure of PSICGA module.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00420v1-figure-c40b1799-f5f5-4b84-a805-9c9a89c85098.jpg)

havioral patterns during cross-granularity aggregation through shared parameters.

As shown in Fig. 2, the proposed PSICGA module is constructed from two encoders: a parameter-sharing aggregation encoder (PSA) and a parameter-independent aggregation encoder (PIA). Although these two encoders share the same structure, which incorporates layer normalization, multiheaded self-attention and FFN, the parameters in them are different. The PSICGA module employs a parallel structure, where the input features are fed into both the PSA and PIA encoder. The network then combines the cls tokens from the outputs of the two modules to obtain the fused features.

For individual to global aggregation, we first add a learnable cls token X cls ∈ R 1 × D to the individual feature sequence X st ∈ R N × D . It summarizes the individual behavior context and the interactions learned by the multi-headed self-attention, representing the aggregated global activity features. To better model the relationships within the aggregation module, we add learnable spatial position embedding to provide positional information. The individual feature sequence X ′ st ∈ R ( N +1) × D with the added cls token and spatial position encoding is fed into the PIA encoder PIA global () and PSA encoder PSA () . The cls tokens representing global activity features from the two modules are combined to obtain the cross-granularity aggregated global activity feature X global . The process is defined as:

$$X ^ { \prime } _ { s t } = [ X _ { c l s }, x ^ { 1 } _ { s t }, x ^ { 2 } _ { s t } \dots x ^ { n } _ { s t } ] + P \quad \quad ( 3 ) \ \text{head} \,.$$

$$X _ { c l s } ^ { g l o b a l } = P I A _ { g l o b a l } ( X _ { s t } ^ { \prime } ), X _ { c l s } ^ { s h a r e } = P S A ( X _ { s t } ^ { \prime } ) \ \ ( 4 ) \ \ R ^ { N \times 1 \geqslant } _ { \cdot \cdot \cdot \cdot }$$

$$X _ { g l o b a l } = X _ { c l s } ^ { g l o b a l } + \lambda X _ { c l s } ^ { s h a r e } \quad \quad ( 5 ) \quad \text{pre} \colon$$

where P represents spatial position embedding, and λ is a weight parameter controlling the addition.

For individual to social group integration, we first use clustering to obtain social group detection results, then perform cross-granularity aggregation for each social group. The

PSICGA module used in this aggregation is similar to the one used in individual to social group integration, with the only difference being that the PIA encoder with different parameters PIA social () is used to obtain the corresponding X social cls , while PSA () encoder is reused for both integrations.

## D. Scene Representation Learning

The global scene contextual information is crucial for panoramic activity recognition. Our scene representation learning module consists of scene representation generation and scene representation fusion. First, visual scene tokens are generated from the scene features extracted by the backbone, and they are aggregated to form the scene representation. Then, cross-attention is employed to fuse the scene representation with activity features at different granularities.

1) Scene Representation Generation: Inspired by Visual Transformers [18], the global scene information in video frames can be summarized using a compact set of visual tokens. We use a set of visual scene tokens to summarize the global scene information of the current video frame and aggregate them into a scene representation. For the scene feature map X scene ∈ R T × D H × ′ × W ′ extracted by the backbone, we first add learnable positional encodings to retain positional information. Next, pointwise convolution is applied to assign each pixel to K visual scene tokens X ′ scene ∈ R T × K × ( H W ′ ′ ) . A spatial attention matrix A ∈ R T × K × ( H W ′ ′ ) is generated using softmax , and the spatial attention weights are used to compute the weighted sum of each pixel. 2D convolution is applied to aggregate the scene features along the spatial dimension ( H W ′ ′ ) to channel D , generating K visual scene tokens Z ∈ R T × K × D . This process can be described as:

$$Z = c o n v ( s o f t m a x ( c o n v ( X _ { s c e n e } ) ) X _ { s c e n e } ) \quad ( 6 )$$

Through the spatial attention mechanism, each visual scene token adaptively focuses on notable parts of the entire scene. Finally, we perform AvgPooling on the first two dimensions of the visual scene tokens, merging the K visual scene tokens into the scene representation X S ∈ R 1 × D .

2) Scene Representation Fusion: To fully integrate the global context information contained in the scene representation with features at various granularities, we use a multihead cross-attention and a feedforward network for fusion. For individual-level features, the inputs to the multi-head crossattention are the enhanced individual representation X t ∈ R N × × 1 D and the scene representation X S ∈ R 1 × D . The individual representation is used as the query, and the scene representation as the key in the multi-head cross-attention to fuse individual and scene representations, yielding the global scene context enhanced individual representation. This operation is similarly performed at the social group level. In the global activities recognition task, we directly concatenate the global-level features with the scene representation and use a learnable MLP to fuse them.

TABLE I

COMPARATIVE RESULTS OF THE PANORAMIC ACTIVITY RECOGNITION (%). THE BEST RESULTS ARE MARKED IN BOLD AND THE SECOND BEST ONES ARE UNDERLINED. 'AR', 'GAR', AND 'PAR' REPRESENT ACTION RECOGNITION, GROUP ACTIVITY RECOGNITION AND PANORAMIC ACTIVITY RECOGNITION SEPARATELY. DFWSGAR [13] AND DECOMPL [24] DO NOT EMPLOY INDIVIDUAL ACTION RECOGNITION AS AN AUXILIARY TASK, SO THEIR RESULTS FOR INDIVIDUAL ACTION RECOGNITION CANNOT BE COMPARED.

|    | Method         | Individual   | Individual   | Act.   | Social Group Act.   | Social Group Act.   | Social Group Act.   | Global Act.   | Global Act.   | Global Act.   | Overall   |
|----|----------------|--------------|--------------|--------|---------------------|---------------------|---------------------|---------------|---------------|---------------|-----------|
|    | Method         | P i          | R i          | F i    | P p                 | R p                 | F p                 | P g           | R g           | F g           | F a       |
| A  | UniFormer [25] | 51.0         | 40.5         | 43.7   | -                   | -                   | -                   | -             | -             | -             | -         |
| R  | vid-TLDR [26]  | 53.0         | 43.4         | 45.7   | -                   | -                   | -                   | -             | -             | -             | -         |
|    | ARG [22]       | 39.9         | 30.7         | 33.2   | -                   | -                   | -                   | 63.6          | 44.3          | 50.7          | -         |
| G  | SA-GAT [23]    | 44.8         | 40.4         | 40.3   | -                   | -                   | -                   | 36.7          | 29.9          | 31.4          | -         |
| A  | JRDB-Base [19] | 19.1         | 34.4         | 23.6   | -                   | -                   | -                   | 44.6          | 46.8          | 45.1          | -         |
| R  | DFWSGAR [13]   | -            | -            | -      | -                   | -                   | -                   | 57.2          | 66.4          | 60.1          | -         |
|    | DECOMPL [24]   | -            | -            | -      | -                   | -                   | -                   | 64.6          | 49.5          | 54.5          | -         |
| P  | JRDB-PAR [5]   | 51.0         | 40.5         | 43.4   | 24.7                | 26.0                | 24.8                | 52.8          | 31.8          | 38.8          | 35.6      |
| A  | MUP [6]        | 55.4         | 44.8         | 47.7   | 25.4                | 26.6                | 25.1                | 58.0          | 49.0          | 51.8          | 41.5      |
| R  | Ours           | 59.2         | 58.6         | 56.0   | 25.5                | 27.3                | 25.4                | 69.1          | 57.6          | 61.1          | 47.5      |

## E. Multi-Task Loss Function

Following JRDB-PAR [5], we sum the losses of multiple tasks to serve as supervision for panoramic activity recognition. The total loss is defined as:

$$\mathcal { L } & = \mathcal { L } _ { i } + \mathcal { L } _ { s } + \mathcal { L } _ { g } + \mathcal { L } _ { d } \\ & = \sum _ { u } L ( \mathbf a _ { u } ^ { I }, \mathbf a _ { u } ^ { I } ) + \sum _ { k } L ( \mathbf a _ { k } ^ { S }, \mathbf a _ { k } ^ { S } ) + L ( \mathbf a ^ { G }, \mathbf a ^ { G } ) + L ( \mathbf R, \mathbf \tilde { R } ) \quad \text{$0.5. If $t$} \\ & = \sup _ { u } L ( \mathbf a _ { u } ^ { I }, \mathbf a _ { u } ^ { I } ) + \sum _ { k } L ( \mathbf a _ { k } ^ { S }, \mathbf a _ { k } ^ { S } ) + L ( \mathbf a ^ { G }, \mathbf a ^ { G } ) + L ( \mathbf R, \mathbf \tilde { R } ) \quad \text{$0.5. If $t$}$$

(7)

where L i , L s , L g , and L d represent the losses for individual, social group, global activity recognition, and social group detection, respectively. u and k denote each individual's features and social group features in the scene. L ( ) · represents binary cross-entropy loss, and ˜ · denotes the ground truth labels. R is the interpersonal relationship matrix with values of 0 or 1, where 1 indicates that the corresponding two individuals belong to the same group.

## IV. EXPERIMENT

## A. Datasets and Metrics

The JRDB-PAR [5] dataset is a novel benchmark specifically created for the panoramic human activity recognition task. This dataset extends the JRDB-Act [19] dataset by manually annotating social group activities and global activities, and adding annotations for social group detection. The JRDBPAR dataset comprises 360-degree RGB videos captured by a mobile robot in various densely populated areas such as campuses, cafeterias, and classrooms, providing 628,000 annotated personal bounding boxes with IDs. The dataset employs multiclass labels for activity annotation, meaning each individual, group, or frame can have one or multiple activity labels. It includes 27 individual action classes (e.g., walking, talking), 11 social group activities (e.g., chatting, working together), and 7 global activities (e.g., commuting, conversing). The dataset consists of 27 video segments, totaling 27,920 frames, with 20 segments used for training and 7 for testing. Key frames are selected every 15 frames for annotation and evaluation.

For individual actions recognition evaluation, we utilize common metrics in multi-label classification tasks: precision, recall, and F 1 score, to measure the accuracy of activity classification for each instance in the test dataset. Precision, recall, and F 1 score are denoted as P i , R i , and F i respectively. Social group activity recognition involves both group detection and activity category recognition. Group detection uses the classical Half metric [20] (IoU &gt; 0.5), where a detected group is considered correct if the intersection-overunion between predicted and actual group members exceeds 0.5. If the activity category of a detected group is correctly predicted, it is regarded as a correct social group activity prediction. Precision, recall, and F 1 score for social group activity recognition are denoted as P p , R p , and F p respectively. For global activity recognition, we also evaluate using precision, recall, and F 1 score, denoted as P g , R g , and F g . The overall evaluation metric for panoramic activity detection is the average of these three F 1 scores, denoted as the comprehensive F 1 score F a = ( F i + F p + F g ) / 3 . This approach allows for a comprehensive assessment of individual actions detection, group activity recognition, and global activity recognition.

## B. Implementation Details

1) Model Details: We use a ResNet-18 [14] pretrained on the ImageNet dataset as the network backbone. In STRE, both the spatial and temporal encoders share the same structure, consisting of two layers with eight heads each. In PSICGA, the parameter-independent aggregation encoder has four layers with 12 heads, while the parameter-sharing aggregation encoder has two layers with 12 heads. For individual-to-social group aggregation, the weight parameter in Equation (5) is λ = 0 8 . . For individual-to-global aggregation, λ = 1 . The number of visual tokens for scene representation generation is set to 16.

2) Training: We implemented our network using the PyTorch framework and trained it on two GTX 3090 GPUs. Consistent with JRDB-PAR [5], we use ground truth social group detection labels during training and spectral clustering [21] for social group detection during evaluation. We set the batch size to 2 and sample three frames. We use the Adam optimizer with an initial learning rate of 7 × 10 -6 and a weight decay of 10 -2 , training for 30 epochs.

## C. Comparison with State-of-the-Art Methods

To demonstrate the superiority of our method, we compared it with state-of-the-art methods. Given the scarcity of existing

Fig. 3. The STRE structures studied in this work. (a) The serial structure. (b) The parallel structure. (c) The parallel-then-serial structure. (d) The structure with one cross-attention module. (e) The structure with two cross-attention modules.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00420v1-figure-4fcca7f2-b479-49d1-a8fb-20cf64e81a5e.jpg)

TABLE II ABLATION STUDIES ON STRE, PSICGA AND SRL.

| STRE   | PSICGA   | SRL   | Overall F a              |
|--------|----------|-------|--------------------------|
| ✓      | ✓        | ✓     | 39.0 44.3 43.6 41.7 46.9 |
| ✓      | ✓        | ✓     | 45.9                     |
| ✓      | ✓        | ✓ ✓   | 44.4                     |
|        | ✓        |       |                          |
| ✓      |          |       | 47.5                     |

panoramic activity recognition methods, we adapted the leading group activity recognition methods ARG [22], SA-GAT [23], JRDB-Base [19], DFWSGAR [13], and DECOMPL [24] for comparison. Since DFWSGAR [13] and DECOMPL [24] do not employ individual action recognition as an auxiliary task, their results for individual action recognition and social group activity recognition cannot be compared. Additionally, we applied the state-of-the-art individual activity recognition methods such as UniFormer [25] and Vid-TLDR [26] to the JRDB-PAR dataset for comparison.

As shown in Table I, our method outperforms the state-ofthe-art panoramic activity recognition methods in all three subtasks and overall scores. Furthermore, compared to methods that focus on single-granularity tasks, our method exhibits superior performance across all levels of tasks, highlighting the mutual reinforcement among multiple tasks in panoramic activity recognition.

## D. Ablation Studies

1) Effectiveness of Individual Modules: We conducted experiments with different combinations of modules to verify their effectiveness. The experiment uses temporal average pooling on the collected individual features instead of STRE to evaluate the effectiveness of the module. Similarly for PSICGA, we replaced it with max pooling for crossgranularity aggregation. Additionally, we removed the scene representation learning (SRL) module for comparison. As shown in Table II, when all the modules were removed, our baseline method still achieved a score of 39.0%. This result surpasses some methods that only use single-frame information, indicating that even simple temporal average pooling

TABLE III

ABLATION STUDIES ON STRE STRUCTURE IN FIGURE 3.

| Structure                       |   Overall F a |
|---------------------------------|---------------|
| Parallel (b)                    |          44.1 |
| Parallel-then-serial (c)        |          46.7 |
| One cross-attention module (d)  |          39.2 |
| Two cross-attention modules (e) |          45.7 |
| Serial (a)                      |          47.5 |

TABLE IV ABLATION STUDIES ON PSICGA MODULE STRUCTURE.

| Parameter-independent   | Parameter-sharing   |   Overall F a |
|-------------------------|---------------------|---------------|
| ✓                       |                     |          42.5 |
|                         | ✓                   |          43.3 |
| ✓                       | ✓                   |          47.5 |

can significantly enhance model performance. This highlights the importance of temporal information in panoramic activity recognition tasks. When the STRE was added, the model's performance further improved significantly, reaching a score of 44.3%. Adding the PSICGA module and the SRL module individually also resulted in substantial improvements, demonstrating the effectiveness of our proposed modules. Subsequently, we combined the modules in pairs, and the synergy between the modules further improves the performance compared to using each module individually.

2) STRE Module Structure: As shown in Fig. 3, we evaluated different structures of STRE, including serial structure (a), parallel structure (b), parallel-then-serial structure (c), and using one (d) and two (e) cross-attention mechanisms to fuse the temporal encoder and spatial encoder. Table III shows that compared to the serial structure, the parallel structure's performance decreased significantly. The decline in performance may be due to the interference between the individual features caused by simultaneous temporal and spatial encoding, reducing overall performance. Additionally, incorporating additional cross-attention to fuse parallel computation results was less effective than the serial structure. Our experimental results show that the serial structure used in STRE, despite its simplicity, better captures the spatio-temporal relationships of individual features, demonstrates its outstanding effectiveness.

3) Effect of PSICGA Module: To verify the effectiveness of this module, the network either uses parameter-sharing

TABLE V ABLATION STUDIES ON THE NUMBER OF VISUAL SCENE TOKENS.

|   Numbers of token | Overall F a   |
|--------------------|---------------|
|                  4 | 40.6%         |
|                  8 | 47.1%         |
|                 16 | 47.5%         |
|                 24 | 46.3%         |
|                 32 | 45.3%         |

(a) GT

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00420v1-figure-fc250229-3dff-4110-84f0-487dd9072b5d.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00420v1-figure-39347535-62b7-4100-9a44-1c614b443b22.jpg)

Fig. 4. The visualization of individual action granularity.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00420v1-figure-f290694d-df41-4222-b57f-6171c0397fb7.jpg)

(b) vid-TLDR [11]

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00420v1-figure-a6e22c89-5f0e-4d8b-adf5-7009cfa89bd3.jpg)

aggregation encoder or parameter-independent aggregation encoder individually for cross-granularity aggregation separately. As shown in Table IV, using the parameter-sharing aggregation encoder alone outperformed using the parameterindependent aggregation encoder alone. This might be because parameter sharing can better capture latent common behaviors across different levels, enhancing the model's generalization capability. Furthermore, when both parameter-sharing and parameter-independent aggregation encoders were used for cross-granularity aggregation, the network's performance significantly improved. This indicates that our proposed crossgranularity aggregation module can fully utilize the complementary effects of parameter sharing and independence. It can effectively learn task-specific features, alleviate parameter optimization conflicts, and capture latent common behaviors across different levels.

4) Investigation of the number of visual scene tokens: We evaluated the impact of the number of visual scene tokens on experimental results in scene representation learning. The global scene information in video frames can be summarized using a small and compact set of visual scene tokens, with each token adaptively focusing on the important parts of the scene. Table V shows that the model's performance is gradually improved with the increasing number of tokens from 4 to 16. More tokens help the model capture global scene information more effectively, thereby enhancing recognition accuracy. However, when the number of tokens exceeds 16,

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00420v1-figure-77848d47-e6a6-4b52-be0d-b52bc09b14f3.jpg)

(a) GT

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00420v1-figure-6f0f9a72-8e44-4e04-808c-89b5a05b6992.jpg)

(b) DFWSGAR [23]

(c) MUP[8]

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00420v1-figure-57ca9fe2-8adc-40d1-afc6-3e82c30fbe5e.jpg)

Fig. 5. The visualization of global activity granularity.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00420v1-figure-422a9ae0-fef6-4d5a-afb9-67a661a05542.jpg)

the model's performance starts to decline. This decline is likely due to information redundancy, which makes it harder for the model to extract useful information, thus negatively affecting performance.

## E. Qualitative Analysis

1) Visualization of Results: We visualized the prediction results of the MPT-PAR model and MUP [6] at different activity granularities. Besides, we compared our method with the vid-TLDR [26] and DECOMPL [24] at individual and global activity granularities. Fig. 4 shows the results for individual action predictions. Since vid-TLDR [26] relies only on cropped individual features, it failed to correctly identify the 'interaction with door' label. Additionally, MUP [6], which overlooks temporal information, might struggle with occlusions, leading to incorrect identification of the label as 'standing'. The comparison indicates that our model, utilizing temporal and scene information, accurately recognizes indi-

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00420v1-figure-4a4a21c8-280b-44de-a816-fbe3efdc1d82.jpg)

(b) MUP[8]

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00420v1-figure-ecaa8c5f-8d0d-44a1-8439-fbaaedc75bdf.jpg)

(c) Ours

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00420v1-figure-0869e698-8c35-4486-b9d9-99bc2e3879d7.jpg)

Fig. 6. The visualization of social group activity granularity.

Fig. 7. The visualization of scene attention maps in the scene representation generation module.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00420v1-figure-66b8ebca-df0f-414d-9cc7-789a906c707a.jpg)

vidual action even in crowded scenes with occlusions. The incorporation of global scene context provides additional scene information, enhancing recognition accuracy. As illustrated in Fig. 5, our model perceives scene semantics, such as computers, desks, and chairs, enabling more precise global behavior recognition. However, the proposed model exhibits certain limitations. The direct application of spectral clustering [21] for social group detection diminishes its accuracy. In Fig. 6, three individual pedestrians, who are in close proximity and performing the same 'walking' action, are considered to belong to the same social group, therefore mistakenly thinking them 'walking together'.

2) Visualization of Scene Attention Maps: We visualized the spatial attention matrices computed in the scene representation learning module to better observe the distribution of attention within scenes. Fig. 7 demonstrates that attention predominantly focuses on individuals within the scene, aligning with our expectations as individuals are the primary subjects of panoramic activity recognition. Additionally, significant attention is observed on other scene elements such as stairs, trees, railings, and chairs. These scene elements influence human behavior within the environment. For example, the presence of stairs correlates with actions of ascending or descending. The visualization results indicate that our scene representation module effectively captures contextual information relevant to behaviors, providing additional cues to enhance activity recognition in complex environments.

## V. CONCLUSION

This paper introduces a novel network architecture, named MPT-PAR, for panoramic activity recognition. The network leverages complementary advantages of parameter sharing and independent cross-granularity aggregation modules, enhancing the effectiveness of activity recognition at all levels. Furthermore, temporal relationships enhanced by transformers and explicit modeling of scene information further help the model better understand the video content. Experiments on the JRDBPAR dataset validate the efficacy of the proposed method, significantly improving performance across all granularity tasks. In future work, we will improve the reliability of social group aggregation and further enhance the practicality of the method.

## REFERENCES

- [1] Z. Sun, Q. Ke, H. Rahmani, M. Bennamoun, G. Wang, and J. Liu, 'Human action recognition from various data modalities: A review,' IEEE transactions on pattern analysis and machine intelligence , vol. 45, no. 3, pp. 3200-3225, 2022.
- [2] L. Yu, Y. Qian, W. Liu, and A. G. Hauptmann, 'Argus++: Robust realtime activity detection for unconstrained video streams with overlapping cube proposals,' in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2022, pp. 112-121.

- [3] Y. Li, Y. Liu, R. Yu, H. Zong, and W. Xie, 'Dual attention based spatialtemporal inference network for volleyball group activity recognition,' Multimedia Tools and Applications , vol. 82, no. 10, pp. 15 515-15 533, 2023.
- [4] F. Askari, R. Ramaprasad, J. J. Clark, and M. D. Levine, 'Interaction classification with key actor detection in multi-person sports videos,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 3580-3588.
- [5] R. Han, H. Yan, J. Li, S. Wang, W. Feng, and S. Wang, 'Panoramic human activity recognition,' in European Conference on Computer Vision . Springer, 2022, pp. 244-261.
- [6] M. Cao, R. Yan, X. Shu, J. Zhang, J. Wang, and G.-S. Xie, 'Mup: Multi-granularity unified perception for panoramic activity recognition,' in Proceedings of the 31st ACM International Conference on Multimedia , 2023, pp. 7666-7675.
- [7] N. Zouba, F. Bremond, M. Thonnat, A. Anfosso, E. Pascual, P. Mallea, ´ V. Mailland, and O. Guerin, 'Assessing computer systems for monitoring elderly people living at home,' in Proceedings of the 19th IAGG World Congress of Gerontology and Geriatrics, Paris, France , 2009, pp. 5-9.
- [8] M. R. Amer, P. Lei, and S. Todorovic, 'Hirf: Hierarchical random field for collective activity recognition in videos,' in Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13 . Springer, 2014, pp. 572-585.
- [9] Y. Zhu, N. M. Nayak, and A. K. Roy-Chowdhury, 'Context-aware modeling and recognition of activities in video,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2013, pp. 2491-2498.
- [10] H. Yuan, D. Ni, and M. Wang, 'Spatio-temporal dynamic inference network for group activity recognition,' CoRR , vol. abs/2108.11743, 2021. [Online]. Available: https://arxiv.org/abs/2108.11743
- [11] Z. Xie, T. Gao, K. Wu, and J. Chang, 'An actor-centric causality graph for asynchronous temporal inference in group activity,' in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023, pp. 6652-6661.
- [12] S. Li, Q. Cao, L. Liu, K. Yang, S. Liu, J. Hou, and S. Yi, 'Groupformer: Group activity recognition with clustered spatial-temporal transformer,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 13 668-13 677.
- [13] D. Kim, J. Lee, M. Cho, and S. Kwak, 'Detector-free weakly supervised group activity recognition,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 20 083-20 093.
- [14] K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770-778.
- [15] K. He, G. Gkioxari, P. Doll´ ar, and R. Girshick, 'Mask r-cnn,' in Proceedings of the IEEE international conference on computer vision , 2017, pp. 2961-2969.
- [16] G. Bertasius, H. Wang, and L. Torresani, 'Is space-time attention all you need for video understanding?' in ICML , vol. 2, no. 3, 2021, p. 4.
- [17] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al. , 'An image is worth 16x16 words: Transformers for image recognition at scale,' arXiv preprint arXiv:2010.11929 , 2020.
- [18] B. Wu, C. Xu, X. Dai, A. Wan, P. Zhang, Z. Yan, M. Tomizuka, J. Gonzalez, K. Keutzer, and P. Vajda, 'Visual transformers: Token-based image representation and processing for computer vision,' arXiv preprint arXiv:2006.03677 , 2020.
- [19] M. Ehsanpour, F. Saleh, S. Savarese, I. Reid, and H. Rezatofighi, 'Jrdb-act: A large-scale dataset for spatio-temporal action, social group and activity detection,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 20 983-20 992.
- [20] X. Wang, X. Zhang, Y. Zhu, Y. Guo, X. Yuan, L. Xiang, Z. Wang, G. Ding, D. Brady, Q. Dai et al. , 'Panda: A gigapixel-level human-centric video dataset,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 3268-3278.
- [21] L. Zelnik-Manor and P. Perona, 'Self-tuning spectral clustering,' Advances in neural information processing systems , vol. 17, 2004.
- [22] J. Wu, L. Wang, L. Wang, J. Guo, and G. Wu, 'Learning actor relation graphs for group activity recognition,' in Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , 2019, pp. 99649974.
- [23] M. Ehsanpour, A. Abedin, F. Saleh, J. Shi, I. Reid, and H. Rezatofighi, 'Joint learning of social groups, individuals action and sub-group activities in videos,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16 . Springer, 2020, pp. 177-195.
- [24] B. Demirel and H. Ozkan, 'Decompl: Decompositional learning with attention pooling for group activity recognition from a single volleyball image,' arXiv preprint arXiv:2303.06439 , 2023.
- [25] K. Li, Y. Wang, P. Gao, G. Song, Y. Liu, H. Li, and Y. Qiao, 'Uniformer: Unified transformer for efficient spatiotemporal representation learning,' arXiv preprint arXiv:2201.04676 , 2022.
- [26] J. Choi, S. Lee, J. Chu, M. Choi, and H. J. Kim, 'vid-tldr: Training free token merging for light-weight video transformer,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 18 771-18 781.
- [27] Z. Xing, Q. Dai, H. Hu, J. Chen, Z. Wu, and Y.-G. Jiang, 'Svformer: Semi-supervised video transformer for action recognition,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2023, pp. 18 816-18 826.
- [28] W. Wu, Y. Song, Z. Sun, J. Wang, C. Xu, and W. Ouyang, 'What can simple arithmetic operations do for temporal modeling?' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 13 712-13 722.
- [29] D. Pei, D. Huang, L. Kong, and Y. Wang, 'Key role guided transformer for group activity recognition,' IEEE Transactions on Circuits and Systems for Video Technology , vol. 33, no. 12, pp. 7803-7818, 2023.
- [30] H. Zhou, A. Kadav, A. Shamsian, S. Geng, F. Lai, L. Zhao, T. Liu, M. Kapadia, and H. P. Graf, 'Composer: compositional reasoning of group activity in videos with keypoint-only modality,' in European Conference on Computer Vision . Springer, 2022, pp. 249-266.
- [31] N. V. Chappa, P. Nguyen, A. H. Nelson, H.-S. Seo, X. Li, P. D. Dobbs, and K. Luu, 'Spartan: Self-supervised spatiotemporal transformers approach to group activity recognition,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 51585168.
- [32] A. Haque, M. Guo, A. Alahi, S. Yeung, Z. Luo, A. Rege, J. Jopling, L. Downing, W. Beninati, A. Singh et al. , 'Towards vision-based smart hospitals: a system for tracking and monitoring hand hygiene compliance,' in Machine Learning for Healthcare Conference . PMLR, 2017, pp. 75-87.
- [33] D. W. Bates, D. Levine, A. Syrowatka, M. Kuznetsova, K. J. T. Craig, A. Rui, G. P. Jackson, and K. Rhee, 'The potential of artificial intelligence to improve patient safety: a scoping review,' NPJ digital medicine , vol. 4, no. 1, p. 54, 2021.
- [34] K. Simonyan and A. Zisserman, 'Two-stream convolutional networks for action recognition in videos,' Advances in neural information processing systems , vol. 27, 2014.
- [35] D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri, 'A closer look at spatiotemporal convolutions for action recognition,' in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , 2018, pp. 6450-6459.
- [36] M. E. Kalfaoglu, S. Kalkan, and A. A. Alatan, 'Late temporal modeling in 3d cnn architectures with bert for action recognition,' in Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16 . Springer, 2020, pp. 731-747.
- [37] J. Liu, A. Shahroudy, D. Xu, and G. Wang, 'Spatio-temporal lstm with trust gates for 3d human action recognition,' in Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14 . Springer, 2016, pp. 816-833.
- [38] M. Majd and R. Safabakhsh, 'Correlational convolutional lstm for human action recognition,' Neurocomputing , vol. 396, pp. 224-229, 2020.
- [39] F. Baradel, C. Wolf, and J. Mille, 'Human action recognition: Pose-based attention draws focus to hands,' in Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops , Oct 2017.
- [40] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Luˇ ci´ c, and C. Schmid, 'Vivit: A video vision transformer,' in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 6836-6846.
- [41] H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik, and C. Feichtenhofer, 'Multiscale vision transformers,' in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 6824-6835.
- [42] X. Bruce, Y. Liu, X. Zhang, S.-h. Zhong, and K. C. Chan, 'Mmnet: A model-based multimodal network for human action recognition in rgb-d videos,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 3, pp. 3522-3538, 2022.
- [43] F. Solera, S. Calderara, and R. Cucchiara, 'Socially constrained structural learning for groups detection in crowd,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 38, no. 5, pp. 995-1008, 2016.
- [44] D. Kim, Y. Song, M. Cho, and S. Kwak, 'Towards more practical group activity detection: A new benchmark and model,' 2023. [Online]. Available: https://arxiv.org/abs/2312.02878

- [45] Y. Xian, X. Rong, X. Yang, and Y. Tian, 'Evaluation of low-level features for real-world surveillance event detection,' IEEE Transactions on Circuits and Systems for Video Technology , vol. 27, no. 3, pp. 624634, 2017.
- [46] R. Li and B. Bhanu, 'Energy-motion features aggregation network for players' fine-grained action analysis in soccer videos,' IEEE Transactions on Circuits and Systems for Video Technology , vol. 34, no. 2, pp. 955972, 2024.
- [47] Y. Ji, Y. Yang, F. Shen, H. T. Shen, and W.-S. Zheng, 'Arbitrary-view human action recognition: A varying-view rgb-d action dataset,' IEEE Transactions on Circuits and Systems for Video Technology , vol. 31, no. 1, pp. 289-300, 2020.
- [48] Y. Chen, H. Ge, Y. Liu, X. Cai, and L. Sun, 'Agpn: Action granularity pyramid network for video action recognition,' IEEE Transactions on Circuits and Systems for Video Technology , vol. 33, no. 8, pp. 39123923, 2023.
- [49] Y. Mou, X. Jiang, K. Xu, T. Sun, and Z. Wang, 'Compressed video action recognition with dual-stream and dual-modal transformer,' IEEE Transactions on Circuits and Systems for Video Technology , vol. 34, no. 5, pp. 3299-3312, 2024.
Wenqing Gan Yan Sun Feiran Liu Xiangfeng Luo
MPT-PAR:Mix-Parameters Transformer for Panoramic Activity Recognition
The objective of the panoramic activity recognition task is to identify
behaviors at various granularities within crowded and complex environments,
encompassing individual actions, social group activities, and global
activities. Existing methods generally use either parameter-independent modules
to capture task-specific features or parameter-sharing modules to obtain common
features across all tasks. However, there is often a strong interrelatedness
and complementary effect between tasks of different granularities that previous
methods have yet to notice. In this paper, we propose a model called MPT-PAR
that considers both the unique characteristics of each task and the synergies
between different tasks simultaneously, thereby maximizing the utilization of
features across multi-granularity activity recognition. Furthermore, we
emphasize the significance of temporal and spatial information by introducing a
spatio-temporal relation-enhanced module and a scene representation learning
module, which integrate the the spatio-temporal context of action and global
scene into the feature map of each granularity. Our method achieved an overall
F1 score of 47.5\% on the JRDB-PAR dataset, significantly outperforming all the
state-of-the-art methods.