title: SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal
    Perturbation and Learning Stabilization
summary: Human action understanding is crucial for the advancement of multimodal
  systems. While recent developments, driven by powerful large language models
  (LLMs), aim to be general enough to cover a wide range of categories, they
  often overlook the need for more specific capabilities. In this work, we
  address the more challenging task of Fine-grained Action Recognition (FAR),
  which focuses on detailed semantic labels within shorter temporal duration
  (e.g., "salto backward tucked with 1 turn"). Given the high costs of annotating
  fine-grained labels and the substantial data needed for fine-tuning LLMs, we
  propose to adopt semi-supervised learning (SSL). Our framework, SeFAR,
  incorporates several innovative designs to tackle these challenges.
  Specifically, to capture sufficient visual details, we construct Dual-level
  temporal elements as more effective representations, based on which we design a
  new strong augmentation strategy for the Teacher-Student learning paradigm
  through involving moderate temporal perturbation. Furthermore, to handle the
  high uncertainty within the teacher model's predictions for FAR, we propose the
  Adaptive Regulation to stabilize the learning process. Experiments show that
  SeFAR achieves state-of-the-art performance on two FAR datasets, FineGym and
  FineDiving, across various data scopes. It also outperforms other
  semi-supervised methods on two classical coarse-grained datasets, UCF101 and
  HMDB51. Further analysis and ablation studies validate the effectiveness of our
  designs. Additionally, we show that the features extracted by our SeFAR could
  largely promote the ability of multimodal foundation models to understand
  fine-grained and domain-specific semantics.
paper_authors._id: 67776b23b2a6ac5157bafa60
paper_authors.name: Yongle Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:05:56.463Z
paper_authors._id: 655b39f5a296e5c74c72cc6d
paper_authors.avatarUrl: /avatars/48aa1874526cda3035bce919fd459e69.svg
paper_authors.fullname: Huang
paper_authors.type: user
paper_authors.user: KyleHuang9
paper_authors._id: 67776b23b2a6ac5157bafa61
paper_authors.name: Haodong Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:03:10.538Z
paper_authors._id: 6570450a78d7aca0c361a177
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6570450a78d7aca0c361a177/z0GrnXEsjK2_G-hFfQhKv.jpeg
paper_authors.fullname: Harold Chen
paper_authors.type: user
paper_authors.user: Harold328
paper_authors._id: 67776b23b2a6ac5157bafa62
paper_authors.name: Zhenbang Xu
paper_authors._id: 67776b23b2a6ac5157bafa63
paper_authors.name: Zihan Jia
paper_authors._id: 67776b23b2a6ac5157bafa64
paper_authors.name: Haozhou Sun
paper_authors._id: 67776b23b2a6ac5157bafa65
paper_authors.name: Dian Shao
paper_title: SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal
    Perturbation and Learning Stabilization
paper_summary: Human action understanding is crucial for the advancement of multimodal
  systems. While recent developments, driven by powerful large language models
  (LLMs), aim to be general enough to cover a wide range of categories, they
  often overlook the need for more specific capabilities. In this work, we
  address the more challenging task of Fine-grained Action Recognition (FAR),
  which focuses on detailed semantic labels within shorter temporal duration
  (e.g., "salto backward tucked with 1 turn"). Given the high costs of annotating
  fine-grained labels and the substantial data needed for fine-tuning LLMs, we
  propose to adopt semi-supervised learning (SSL). Our framework, SeFAR,
  incorporates several innovative designs to tackle these challenges.
  Specifically, to capture sufficient visual details, we construct Dual-level
  temporal elements as more effective representations, based on which we design a
  new strong augmentation strategy for the Teacher-Student learning paradigm
  through involving moderate temporal perturbation. Furthermore, to handle the
  high uncertainty within the teacher model's predictions for FAR, we propose the
  Adaptive Regulation to stabilize the learning process. Experiments show that
  SeFAR achieves state-of-the-art performance on two FAR datasets, FineGym and
  FineDiving, across various data scopes. It also outperforms other
  semi-supervised methods on two classical coarse-grained datasets, UCF101 and
  HMDB51. Further analysis and ablation studies validate the effectiveness of our
  designs. Additionally, we show that the features extracted by our SeFAR could
  largely promote the ability of multimodal foundation models to understand
  fine-grained and domain-specific semantics.
paper_ai_summary: SeFAR, a semi-supervised learning framework using dual-level temporal elements and adaptive regulation, achieves state-of-the-art performance in fine-grained action recognition and enhances multimodal foundation models' understanding of fine-grained semantics.
paper_ai_keywords: fine-grained action recognition semi-supervised learning dual-level temporal elements teacher-student learning paradigm temporal perturbation adaptive regulation FineGym FineDiving UCF101 HMDB51