title: FürElise: Capturing and Physically Synthesizing Hand Motions of Piano
    Performance
summary: Piano playing requires agile, precise, and coordinated hand control that
  stretches the limits of dexterity. Hand motion models with the sophistication
  to accurately recreate piano playing have a wide range of applications in
  character animation, embodied AI, biomechanics, and VR/AR. In this paper, we
  construct a first-of-its-kind large-scale dataset that contains approximately
  10 hours of 3D hand motion and audio from 15 elite-level pianists playing 153
  pieces of classical music. To capture natural performances, we designed a
  markerless setup in which motions are reconstructed from multi-view videos
  using state-of-the-art pose estimation models. The motion data is further
  refined via inverse kinematics using the high-resolution MIDI key-pressing data
  obtained from sensors in a specialized Yamaha Disklavier piano. Leveraging the
  collected dataset, we developed a pipeline that can synthesize
  physically-plausible hand motions for musical scores outside of the dataset.
  Our approach employs a combination of imitation learning and reinforcement
  learning to obtain policies for physics-based bimanual control involving the
  interaction between hands and piano keys. To solve the sampling efficiency
  problem with the large motion dataset, we use a diffusion model to generate
  natural reference motions, which provide high-level trajectory and fingering
  (finger order and placement) information. However, the generated reference
  motion alone does not provide sufficient accuracy for piano performance
  modeling. We then further augmented the data by using musical similarity to
  retrieve similar motions from the captured dataset to boost the precision of
  the RL policy. With the proposed method, our model generates natural, dexterous
  motions that generalize to music from outside the training dataset.
paper_authors._id: 670773181771817ab426144d
paper_authors.name: Ruocheng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:57:56.927Z
paper_authors._id: 62abb35b26adcd4a3845b41b
paper_authors.avatarUrl: /avatars/928720c54d947cc8ebcfa9a007f272ea.svg
paper_authors.fullname: Ruocheng Wang
paper_authors.type: user
paper_authors.user: rcwang
paper_authors._id: 670773181771817ab426144e
paper_authors.name: Pei Xu
paper_authors._id: 670773181771817ab426144f
paper_authors.name: Haochen Shi
paper_authors._id: 670773181771817ab4261450
paper_authors.name: Elizabeth Schumann
paper_authors._id: 670773181771817ab4261451
paper_authors.name: C. Karen Liu
paper_title: FürElise: Capturing and Physically Synthesizing Hand Motions of Piano
    Performance
paper_summary: Piano playing requires agile, precise, and coordinated hand control that
  stretches the limits of dexterity. Hand motion models with the sophistication
  to accurately recreate piano playing have a wide range of applications in
  character animation, embodied AI, biomechanics, and VR/AR. In this paper, we
  construct a first-of-its-kind large-scale dataset that contains approximately
  10 hours of 3D hand motion and audio from 15 elite-level pianists playing 153
  pieces of classical music. To capture natural performances, we designed a
  markerless setup in which motions are reconstructed from multi-view videos
  using state-of-the-art pose estimation models. The motion data is further
  refined via inverse kinematics using the high-resolution MIDI key-pressing data
  obtained from sensors in a specialized Yamaha Disklavier piano. Leveraging the
  collected dataset, we developed a pipeline that can synthesize
  physically-plausible hand motions for musical scores outside of the dataset.
  Our approach employs a combination of imitation learning and reinforcement
  learning to obtain policies for physics-based bimanual control involving the
  interaction between hands and piano keys. To solve the sampling efficiency
  problem with the large motion dataset, we use a diffusion model to generate
  natural reference motions, which provide high-level trajectory and fingering
  (finger order and placement) information. However, the generated reference
  motion alone does not provide sufficient accuracy for piano performance
  modeling. We then further augmented the data by using musical similarity to
  retrieve similar motions from the captured dataset to boost the precision of
  the RL policy. With the proposed method, our model generates natural, dexterous
  motions that generalize to music from outside the training dataset.
paper_ai_keywords: markerless setup pose estimation models inverse kinematics MIDI key-pressing data Yamaha Disklavier piano physically-plausible hand motions imitation learning reinforcement learning physics-based bimanual control diffusion model natural reference motions high-level trajectory fingering finger order finger placement musical similarity