Synchronize Dual Hands for Physics-Based Dexterous Guitar Playing
We present a novel approach to synthesize dexterous motions for physically
simulated hands in tasks that require coordination between the control of two
hands with high temporal precision. Instead of directly learning a joint policy
to control two hands, our approach performs bimanual control through
cooperative learning where each hand is treated as an individual agent. The
individual policies for each hand are first trained separately, and then
synchronized through latent space manipulation in a centralized environment to
serve as a joint policy for two-hand control. By doing so, we avoid directly
performing policy learning in the joint state-action space of two hands with
higher dimensions, greatly improving the overall training efficiency. We
demonstrate the effectiveness of our proposed approach in the challenging
guitar-playing task. The virtual guitarist trained by our approach can
synthesize motions from unstructured reference data of general guitar-playing
practice motions, and accurately play diverse rhythms with complex chord
pressing and string picking patterns based on the input guitar tabs that do not
exist in the references. Along with this paper, we provide the motion capture
data that we collected as the reference for policy training. Code is available
at: https://pei-xu.github.io/guitar.
66f4bf2adc51f8cd91384cd8
Pei Xu
66f4bf2adc51f8cd91384cd9
Ruocheng Wang
admin_assigned
2024-09-26T09:41:22.170Z
62abb35b26adcd4a3845b41b
/avatars/928720c54d947cc8ebcfa9a007f272ea.svg
Ruocheng Wang
user
rcwang
Synchronize Dual Hands for Physics-Based Dexterous Guitar Playing
We present a novel approach to synthesize dexterous motions for physically
simulated hands in tasks that require coordination between the control of two
hands with high temporal precision. Instead of directly learning a joint policy
to control two hands, our approach performs bimanual control through
cooperative learning where each hand is treated as an individual agent. The
individual policies for each hand are first trained separately, and then
synchronized through latent space manipulation in a centralized environment to
serve as a joint policy for two-hand control. By doing so, we avoid directly
performing policy learning in the joint state-action space of two hands with
higher dimensions, greatly improving the overall training efficiency. We
demonstrate the effectiveness of our proposed approach in the challenging
guitar-playing task. The virtual guitarist trained by our approach can
synthesize motions from unstructured reference data of general guitar-playing
practice motions, and accurately play diverse rhythms with complex chord
pressing and string picking patterns based on the input guitar tabs that do not
exist in the references. Along with this paper, we provide the motion capture
data that we collected as the reference for policy training. Code is available
at: https://pei-xu.github.io/guitar.
The approach uses cooperative learning of individual hand policies for bimanual control, enhancing training efficiency and demonstrated in guitar-playing tasks.
bimanual control cooperative learning individual policies latent space manipulation joint state-action space policy training motion capture data