Music has become an integral part of modern movies after early silent films, and is important in conveying the mood and setting of the film at any given moment, expressing character emotion, heightening drama, advancing the story, or even sometimes misleading the viewer. To enhance the personal media experience, music is also often inserted in the sound track during media authoring. For example, users typically choose an appropriate music for slide shows of photos, or add music to videos, primarily to help set the mood and emotion suitable for the content in the photos and video.
Given the aforementioned purpose, it is important to select the thematically representative music. At present, such music selection is manually performed by the users for a given media, or retrieved from an indexed music library to match the theme of the media. An example of the latter is described by Luo et al., “Photo-centric Multimedia Authoring Enhanced by Cross-Media Retrieval,” in Proceedings of 2005 SPIE International Symposium on Visual Communication and Image Processing, where the theme (e.g., beach, mountain, city, fall foliage, etc.) of a photo collection is determined automatically by image classification, and a music of the matching theme is retrieved from a repertoire manually pre-indexed by related themes. Tiling slide shows by Chen et al., “Tiling slideshow,” in Proceedings of 2006 ACM International Conference on Multimedia, describes a method for creating slide shows with matching beat music to improve user experience.
A theme in music composition is the material on which the composition is based. It can be a repeating musical expression such as found in Beethoven. The Encyclopedic Fasquelle (Michel 1958-61) defines a theme as “Any element, motif, or small musical piece that has given rise to some variation becomes thereby a theme.”
However, the music theme contained in media production and media experience is different; it is semantic. Broadly speaking, these themes can be ethnographic (related to ethnic groups and other ethnic formations, their ethnogenesis, composition, resettlement, social welfare characteristics, as well as their material and spiritual culture), geographic (related to a geographic region or location), demographic (related to characteristics of a human population, including gender, race, age, income, disabilities, mobility (in terms of travel time to work or number of vehicles available), educational attainment, home ownership, employment status, and even location), or based on an event or activity.
Music selection based on semantic themes has been studied in the prior art. U.S. Patent Application Publication 20110054646 describes an entertainment system that has a music storage system storing a plurality of music pieces, a playback system coupled with the music storage system, a navigation system providing current map information including a present location, wherein a current map has a plurality of zones each being assigned to one of a plurality of zone types, and a controller for controlling playback of selected music pieces, wherein the controller maintains a plurality of playlists, each having a plurality of music pieces and being assigned to at least one zone type. The controller receives information of a present location and a current zone type and selects an assigned playlist, wherein the navigation system further provides information about a distance and/or time to a next zone. The controller modifies the assigned playlist such that a transition to the next zone is timely synchronized with the ending of a music piece of the assigned playlist.
U.S. Patent Application Publication 20100273610 describes systems and techniques for generating an athletic training program and selecting music for playing during the training program are described. Based on specified parameters, a training program module can generate a customized training program intended to help an athlete reach a goal. In conjunction therewith or independently thereof, a music selection module can generate a music playlist for playing during a training program. Music selection parameters can include training intensity, user speed, user location, user mood, a user's current performance (e.g., as compared to an expected performance) and the like. The music selection module can select songs from a personal library or a public database of music. Music selection can be made to maximize user motivation and inspiration.
U.S. Patent Application Publication 20100070057 relates to a system that automatically associates background music with an image slideshow. The system can receive a selected image collection, extract metadata from the image collection, and associate audio files with the image collection based on the metadata. The system will then prompt concurrent playing of the audio file while the image collection is displayed. The metadata identifies a theme for the image collection which can form the basis for associating the audio file with the image collection. This system is similar to Luo et al., “Photo-centric Multimedia Authoring Enhanced by Cross-Media Retrieval,” in Proceedings of 2005 SPIE International Symposium on Visual Communication and Image Processing.
However, none of the prior art treats the music as already pre-indexed but does not disclose how to obtain music suitable for a given semantic theme.
Music theme classification is related to but different from music genre classification. Music can be divided into many genres in many different ways. These classifications are often arbitrary and controversial, and closely related styles often overlap. Many do not believe that generic classification of musical styles is possible in any logically consistent way, and also argue that doing so sets limitations and boundaries that hinder the development of music. While no one doubts that it is possible to note similarities between musical pieces, there are often exceptions and caveats associated. Labeling music with genres often does not reflect a specific culture, race, or time period. Larger genres consist of more specific subgenres. Common music genres include classic music, contemporary music, folk music, jazz, rock, country, and so on. For a survey on this topic, please see “Automatic genre classification of music content: a survey”, N. Scaringella, G. Zoia, and D. Mlynek, Signal Processing Magazine, IEEE, Vol. 23, Nr. 2 (2006), p. 133-141.
In addition and very importantly, all of the above mentioned prior art assume that a collection of music already pre-exists for the purpose of music selection or music classification.
There is therefore a need for a system to first collect a set of potentially usefully music and then select from such a set of candidate music any music that is thematically representative.