Sound Recovery Using Vibration Modes of the Object in a Video
When a sound hits an object, it causes the surface of the object to vibrate. Some research has been carried out on the recovering of sounds by extracting the vibrations seen on video images. This research is expected to be applied in the field of surveillance and security because sounds can be recorded from relatively far away. The vibration of objects due to sound is so fast and minute that it is invisible. However, it is possible to observe such changes in objects by using the high speed video as the movement of each pixel by using a complex steerable pyramid. In the conventional method, the movements of all pixels are added together to recover the sound. So it is possible that some noise source vibrations are mixed because there are some pixels that move independently of the sound source being focused upon. In this paper, we propose a sound recovery method focusing on the vibration modes of the object associated with the frequency. The vibrating parts of objects are different depending on the material, shape and frequency. The vibration is composed of some normal vibrations, and each has different loops and nodes. We confirm which part of the object is vibrating for each frequency of the sound, and recover the sound using a filter based on the response of the object. Which part is vibrating is confirmed from the amplitude response of each pixel when the signal of that frequency is the largest. This response and the reliability of the signal of each pixel are multiplied to each pixel as a filter. We recovered sounds from several objects in videos and ascertained the effectiveness of the method.
2018 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)