title: Synchronize Dual Hands for Physics-Based Dexterous Guitar Playing
summary: We present a novel approach to synthesize dexterous motions for physically
  simulated hands in tasks that require coordination between the control of two
  hands with high temporal precision. Instead of directly learning a joint policy
  to control two hands, our approach performs bimanual control through
  cooperative learning where each hand is treated as an individual agent. The
  individual policies for each hand are first trained separately, and then
  synchronized through latent space manipulation in a centralized environment to
  serve as a joint policy for two-hand control. By doing so, we avoid directly
  performing policy learning in the joint state-action space of two hands with
  higher dimensions, greatly improving the overall training efficiency. We
  demonstrate the effectiveness of our proposed approach in the challenging
  guitar-playing task. The virtual guitarist trained by our approach can
  synthesize motions from unstructured reference data of general guitar-playing
  practice motions, and accurately play diverse rhythms with complex chord
  pressing and string picking patterns based on the input guitar tabs that do not
  exist in the references. Along with this paper, we provide the motion capture
  data that we collected as the reference for policy training. Code is available
  at: https://pei-xu.github.io/guitar.
paper_authors._id: 66f4bf2adc51f8cd91384cd8
paper_authors.name: Pei Xu
paper_authors._id: 66f4bf2adc51f8cd91384cd9
paper_authors.name: Ruocheng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:41:22.170Z
paper_authors._id: 62abb35b26adcd4a3845b41b
paper_authors.avatarUrl: /avatars/928720c54d947cc8ebcfa9a007f272ea.svg
paper_authors.fullname: Ruocheng Wang
paper_authors.type: user
paper_authors.user: rcwang
paper_title: Synchronize Dual Hands for Physics-Based Dexterous Guitar Playing
paper_summary: We present a novel approach to synthesize dexterous motions for physically
  simulated hands in tasks that require coordination between the control of two
  hands with high temporal precision. Instead of directly learning a joint policy
  to control two hands, our approach performs bimanual control through
  cooperative learning where each hand is treated as an individual agent. The
  individual policies for each hand are first trained separately, and then
  synchronized through latent space manipulation in a centralized environment to
  serve as a joint policy for two-hand control. By doing so, we avoid directly
  performing policy learning in the joint state-action space of two hands with
  higher dimensions, greatly improving the overall training efficiency. We
  demonstrate the effectiveness of our proposed approach in the challenging
  guitar-playing task. The virtual guitarist trained by our approach can
  synthesize motions from unstructured reference data of general guitar-playing
  practice motions, and accurately play diverse rhythms with complex chord
  pressing and string picking patterns based on the input guitar tabs that do not
  exist in the references. Along with this paper, we provide the motion capture
  data that we collected as the reference for policy training. Code is available
  at: https://pei-xu.github.io/guitar.
paper_ai_summary: The approach uses cooperative learning of individual hand policies for bimanual control, enhancing training efficiency and demonstrated in guitar-playing tasks.
paper_ai_keywords: bimanual control cooperative learning individual policies latent space manipulation joint state-action space policy training motion capture data