TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through
  Task Tokenization
Synthesizing diverse and physically plausible Human-Scene Interactions (HSI)
is pivotal for both computer animation and embodied AI. Despite encouraging
progress, current methods mainly focus on developing separate controllers, each
specialized for a specific interaction task. This significantly hinders the
ability to tackle a wide variety of challenging HSI tasks that require the
integration of multiple skills, e.g., sitting down while carrying an object. To
address this issue, we present TokenHSI, a single, unified transformer-based
policy capable of multi-skill unification and flexible adaptation. The key
insight is to model the humanoid proprioception as a separate shared token and
combine it with distinct task tokens via a masking mechanism. Such a unified
policy enables effective knowledge sharing across skills, thereby facilitating
the multi-task training. Moreover, our policy architecture supports variable
length inputs, enabling flexible adaptation of learned skills to new scenarios.
By training additional task tokenizers, we can not only modify the geometries
of interaction targets but also coordinate multiple skills to address complex
tasks. The experiments demonstrate that our approach can significantly improve
versatility, adaptability, and extensibility in various HSI tasks. Website:
https://liangpan99.github.io/TokenHSI/
67eac6433755a17e3cbff585
Liang Pan
claimed_verified
2025-04-01T07:47:34.754Z
6630cc7e9ee8861dd0b9bdbd
/avatars/01da7d60cbb107d58329bbb80d924eb4.svg
Liang Pan
user
lianganimation
67eac6433755a17e3cbff586
Zeshi Yang
admin_assigned
2025-04-01T08:21:03.851Z
649c178950b4be74229d680f
/avatars/941dec90fdfc46b9ae23378e3a3113f4.svg
Zeshi Yang
user
Zeshi209
67eac6433755a17e3cbff587
Zhiyang Dou
admin_assigned
2025-04-01T08:21:09.328Z
645223fb01d7bd9555ea399a
https://cdn-avatars.huggingface.co/v1/production/uploads/645223fb01d7bd9555ea399a/fVR7XmGg6pMSRKx9sEdvT.png
Zhiyang Dou
user
frankzydou
67eac6433755a17e3cbff588
Wenjia Wang
admin_assigned
2025-04-01T08:21:22.428Z
6437a813fac5ea753f1c72d2
/avatars/69e60e60497e404149a1dad46649dad4.svg
wenjia Wang
user
WenjiaWang
67eac6433755a17e3cbff589
Buzhen Huang
67eac6433755a17e3cbff58a
Bo Dai
admin_assigned
2025-04-01T08:21:59.939Z
635f93577c05eb9f59966209
/avatars/add48d7e3790a6b500d6c451ef8b0f75.svg
Intelligent Digital Creation
user
BoDai
67eac6433755a17e3cbff58b
Taku Komura
67eac6433755a17e3cbff58c
Jingbo Wang
admin_assigned
2025-04-01T08:21:43.455Z
669cb638666901f41dae51bf
/avatars/a7cc19e3db84bd86bee3eb6fd4897959.svg
Jingbo Wang
user
jingbocuhk
TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through
  Task Tokenization
Synthesizing diverse and physically plausible Human-Scene Interactions (HSI)
is pivotal for both computer animation and embodied AI. Despite encouraging
progress, current methods mainly focus on developing separate controllers, each
specialized for a specific interaction task. This significantly hinders the
ability to tackle a wide variety of challenging HSI tasks that require the
integration of multiple skills, e.g., sitting down while carrying an object. To
address this issue, we present TokenHSI, a single, unified transformer-based
policy capable of multi-skill unification and flexible adaptation. The key
insight is to model the humanoid proprioception as a separate shared token and
combine it with distinct task tokens via a masking mechanism. Such a unified
policy enables effective knowledge sharing across skills, thereby facilitating
the multi-task training. Moreover, our policy architecture supports variable
length inputs, enabling flexible adaptation of learned skills to new scenarios.
By training additional task tokenizers, we can not only modify the geometries
of interaction targets but also coordinate multiple skills to address complex
tasks. The experiments demonstrate that our approach can significantly improve
versatility, adaptability, and extensibility in various HSI tasks. Website:
https://liangpan99.github.io/TokenHSI/
A unified transformer-based policy, TokenHSI, improves versatility in Human-Scene Interactions by integrating multiple interaction skills through shared and task-specific tokens.
transformer-based policy TokenHSI humanoid proprioception shared token task tokens masking mechanism multi-task training variable length inputs