  Recent deep learning approaches have achieved impressive performance on
visual sound separation tasks. However, these approaches are mostly built on
appearance and optical flow like motion feature representations, which exhibit
limited abilities to find the correlations between audio signals and visual
points, especially when separating multiple instruments of the same types, such
as multiple violins in a scene. To address this, we propose "Music Gesture," a
keypoint-based structured representation to explicitly model the body and
finger movements of musicians when they perform music. We first adopt a
context-aware graph network to integrate visual semantic context with body
dynamics, and then apply an audio-visual fusion model to associate body
movements with the corresponding audio signals. Experimental results on three
music performance datasets show: 1) strong improvements upon benchmark metrics
for hetero-musical separation tasks (i.e. different instruments); 2) new
ability for effective homo-musical separation for piano, flute, and trumpet
duets, which to our best knowledge has never been achieved with alternative
methods. Project page: http://music-gesture.csail.mit.edu.

Computer Vision and Pattern Recognition;Machine Learning;Multimedia;Sound;Audio and Speech Processing