How did they evaluate the quality of annotations?
They manually check the captions and employ extra annotators to further revise the annotations.
Audio captioning is the novel task of general audio content description using free text. It is an intermodal translation task (not speech-to-text), where a system accepts as an input an audio signal and outputs the textual description (i.e. the caption) of that signal. In this paper we present Clotho, a dataset for audio captioning consisting of 4981 audio samples of 15 to 30 seconds duration and 24 905 captions of eight to 20 words length, and a baseline method to provide initial results. Clotho is built with focus on audio content and caption diversity, and the splits of the data are not hampering the training or evaluation of methods. All sounds are from the Freesound platform, and captions are crowdsourced using Amazon Mechanical Turk and annotators from English speaking countries. Unique words, named entities, and speech transcription are removed with post-processing. Clotho is freely available online (this https URL).
Captioning is the intermodal translation task of describing the human-perceived information in a medium, e.g. images (image captioning) or audio (audio captioning), using free text BIBREF0, BIBREF1, BIBREF2, BIBREF3. In particular, audio captioning was first introduced in BIBREF3, it does not involve speech transcription, and is focusing on identifying the human-perceived information in an general audio signal and expressing it through text, using natural language. This information includes identification of sound events, acoustic scenes, spatiotemporal relationships of sources, foreground versus background discrimination, concepts, and physical properties of objects and environment. For example, given an audio signal, an audio captioning system would be able to generate captions like “a door creaks as it slowly revolves back and forth”. The dataset used for training an audio captioning method defines to a great extent what the method can learn BIBREF0, BIBREF4. Diversity in captions allows the method to learn and exploit the perceptual differences on the content (e.g. a thin plastic rattling could be perceived as a fire crackling) BIBREF0. Also, the evaluation of the method becomes more objective and general by having more captions per audio signal BIBREF4. Recently, two different datasets for audio captioning were presented, Audio Caption and AudioCaps BIBREF5, BIBREF6. Audio Caption is partially released, and contains 3710 domain-specific (hospital) video clips with their audio tracks, and annotations that were originally obtained in Mandarin Chinese and afterwards translated to English using machine translation BIBREF5. The annotators had access and viewed the videos. The annotations contain description of the speech content (e.g. “The patient inquired about the location of the doctor’s police station”). AudioCaps dataset has 46 000 audio samples from AudioSet BIBREF7, annotated with one caption each using the crowdsourcing platform Amazon Mechanical Turk (AMT) and automated quality and location control of the annotators BIBREF6. Authors of AudioCaps did not use categories of sounds which they claimed that visuals were required for correct recognition, e.g. “inside small room”. Annotators of AudioCaps were provided the word labels (by AudioSet) and viewed the accompanying videos of the audio samples. The perceptual ambiguity of sounds can be hampered by providing contextual information (e.g. word labels) to annotators, making them aware of the actual source and not letting them describe their own perceived information. Using visual stimuli (e.g. video) introduces a bias, since annotators may describe what they see and not what they hear. Also, a single caption per file impedes the learning and evaluation of diverse descriptions of information, and domain-specific data of previous audio captioning datasets have an observed significant impact on the performance of methods BIBREF5. Finally, unique words (i.e. words appearing only once) affect the learning process, as they have an impact on the evaluation process (e.g. if a word is unique, will be either on training or on evaluation). An audio captioning dataset should at least provide some information on unique words contained in its captions. In this paper we present the freely available audio captioning dataset Clotho, with 4981 audio samples and 24 905 captions. All audio samples are from Freesound platform BIBREF8, and are of duration from 15 to 30 seconds. Each audio sample has five captions of eight to 20 words length, collected by AMT and a specific protocol for crowdsourcing audio annotations, which ensures diversity and reduced grammatical errors BIBREF0. During annotation no other information but the audio signal was available to the annotators, e.g. video or word tags. The rest of the paper is organized as follows. Section SECREF2 presents the creation of Clotho, i.e. gathering and processing of the audio samples and captions, and the splitting of the data to development, evaluation, and testing splits. Section SECREF3 presents the baseline method used, the process followed for its evaluation using Clotho, and the obtained results. Section SECREF4 concludes the paper.