title: F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactions
summary: Existing 3D human object interaction (HOI) datasets and models simply align
  global descriptions with the long HOI sequence, while lacking a detailed
  understanding of intermediate states and the transitions between states. In
  this paper, we argue that fine-grained semantic alignment, which utilizes
  state-level descriptions, offers a promising paradigm for learning semantically
  rich HOI representations. To achieve this, we introduce Semantic-HOI, a new
  dataset comprising over 20K paired HOI states with fine-grained descriptions
  for each HOI state and the body movements that happen between two consecutive
  states. Leveraging the proposed dataset, we design three state-level HOI tasks
  to accomplish fine-grained semantic alignment within the HOI sequence.
  Additionally, we propose a unified model called F-HOI, designed to leverage
  multimodal instructions and empower the Multi-modal Large Language Model to
  efficiently handle diverse HOI tasks. F-HOI offers multiple advantages: (1) It
  employs a unified task formulation that supports the use of versatile
  multimodal inputs. (2) It maintains consistency in HOI across 2D, 3D, and
  linguistic spaces. (3) It utilizes fine-grained textual supervision for direct
  optimization, avoiding intricate modeling of HOI states. Extensive experiments
  reveal that F-HOI effectively aligns HOI states with fine-grained semantic
  descriptions, adeptly tackling understanding, reasoning, generation, and
  reconstruction tasks.
paper_authors._id: 66a05e60007a749299a6efb9
paper_authors.name: Jie Yang
paper_authors._id: 66a05e60007a749299a6efba
paper_authors.name: Xuesong Niu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-24T08:02:02.291Z
paper_authors._id: 64620ca8a489ecb6b6b5b4e4
paper_authors.avatarUrl: /avatars/9bf8060cb7bc59c0369ddf14c50aa480.svg
paper_authors.fullname: Xuesong Niu
paper_authors.type: user
paper_authors.user: nxsEdson
paper_authors._id: 66a05e60007a749299a6efbb
paper_authors.name: Nan Jiang
paper_authors._id: 66a05e60007a749299a6efbc
paper_authors.name: Ruimao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-24T08:04:18.088Z
paper_authors._id: 65324848220f490dbaa3b8c9
paper_authors.avatarUrl: /avatars/cec7dcadf4657bb4f61e2c9a8075521b.svg
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: Ruimao
paper_authors._id: 66a05e60007a749299a6efbd
paper_authors.name: Siyuan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-24T08:05:05.841Z
paper_authors._id: 63c7a33121bd95f80ed74652
paper_authors.avatarUrl: /avatars/7dd59afea785a2bff0ec2b757abd474e.svg
paper_authors.fullname: Siyuan Huang
paper_authors.type: user
paper_authors.user: thuhsy
paper_title: F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactions
paper_summary: Existing 3D human object interaction (HOI) datasets and models simply align
  global descriptions with the long HOI sequence, while lacking a detailed
  understanding of intermediate states and the transitions between states. In
  this paper, we argue that fine-grained semantic alignment, which utilizes
  state-level descriptions, offers a promising paradigm for learning semantically
  rich HOI representations. To achieve this, we introduce Semantic-HOI, a new
  dataset comprising over 20K paired HOI states with fine-grained descriptions
  for each HOI state and the body movements that happen between two consecutive
  states. Leveraging the proposed dataset, we design three state-level HOI tasks
  to accomplish fine-grained semantic alignment within the HOI sequence.
  Additionally, we propose a unified model called F-HOI, designed to leverage
  multimodal instructions and empower the Multi-modal Large Language Model to
  efficiently handle diverse HOI tasks. F-HOI offers multiple advantages: (1) It
  employs a unified task formulation that supports the use of versatile
  multimodal inputs. (2) It maintains consistency in HOI across 2D, 3D, and
  linguistic spaces. (3) It utilizes fine-grained textual supervision for direct
  optimization, avoiding intricate modeling of HOI states. Extensive experiments
  reveal that F-HOI effectively aligns HOI states with fine-grained semantic
  descriptions, adeptly tackling understanding, reasoning, generation, and
  reconstruction tasks.
paper_ai_keywords: Semantic-HOI HOI states state-level descriptions F-HOI multimodal instructions Multi-modal Large Language Model fine-grained textual supervision HOI sequence understanding reasoning generation reconstruction