We explore the problem of audio captioning 1 : generating natural language description for any kind of audio in the wild, which has been surprisingly unexplored in previous research. We contribute a large-scale dataset of 46K audio clips with human-written text pairs collected via crowdsourcing on the AudioSet dataset (Gemmeke et al., 2017). Our thorough empirical studies not only show that our collected captions are indeed loyal to the audio inputs but also discover what forms of audio representation and captioning models are effective for audio captioning. From extensive experiments, we also propose two novel components that are integrable with any attention-based captioning model to help improve audio captioning performance: the top-down multi-scale encoder and aligned semantic attention. Captioning , the task of translating a multimedia input source into natural language, has been substantially studied over the past few years. The vast majority of the journey has been through the visual senses ranging from static images to videos. Yet, the exploration into the auditory sense has been circumscribed to human speech transcription (Panayotov et al., 2015; Nagrani et al., 2017), leaving the basic natural form of sound in an uncharted territory of the captioning research. Recently, sound event detection has gained much attention such as DCASE challenges (Mesaros et al., 2017) along with the release of a large scale AudioSet dataset (Gemmeke et al., 2017). However, sound classification ( e.g . predicting multiple labels for a given sound) and event detection ( e.g . localizing the sound of interest in a clip) may not be sufficient for a full understanding of the sound. Instead, a natural sen-1 For a live demo and details, https://audiocaps.github.io [Audio Classification] rumble | vehicle | speech | car | outside [Video Captioning] A bus passing by with some people walking by in the afternoon. [Audio Captioning] A muffled rumble with man and woman talking in the background while a siren blares in the distance. tence offers a greater freedom to express a sound, because it allows to characterize objects along with their states, properties, actions and interactions. For example, suppose that suddenly sirens are ringing in the downtown area. As a natural reaction, people may notice the presence of an emergency vehicle, even though they are unable to see any flashing lights nor feel the rush of wind from a passing vehicle. Instead of simply tagging this sound as ambulance or siren , it is more informative to describe which direction the sound is coming from or whether the source of the sound is moving closer or further away, as shown in Figure 1. To that end, we address the audio captioning problem for audios in the wild, which has not been studied yet, to the best of our knowledge. This work focuses on one of the most important bases toward this research direction, contributing a large-scale dataset . The overarching sources of in-the-wild sounds are grounded on the AudioSet (Gemmeke et al., 2017), so far the largest collection of sound events collected from Youtube videos. We newly collect human-written sentences for a subset of AudioSet audio clips via crowdsourcing on Amazon Mechanical Turk (sec-tion 3). We also develop two simple yet effective techniques to generate captions through the joint use of multi-level pretrained features and better attention mechanism named aligned-semantic attention (section 4). Lastly, we perform experiments contrasting between video-based captions and audio-focused captions by employing a variety of features and captioning models (section 5). The contributions of this work are as follows. 1. To the best of our knowledge, this work is the first attempt to address the audio captioning task for sound in the wild. We contribute its first large-scale dataset named AudioCaps , which consists of 46K pairs of audio clips and text description. 2. We perform thorough empirical studies not only to show that our collected captions are indeed true to the audio inputs and but also to discover what forms of audio representations and captioning models are effective. For example, we observe that the embeddings from large-scale pretrained VGGish (Her-shey et al., 2017) are powerful in describing the audio input, and both temporal and semantic attention are helpful to enhance captioning performance. 3. From extensive experiments, we propose two simple yet effective technical components that further improve audio captioning performance: the top-down multi-scale encoder that enables the joint use of multi-level features and aligned semantic attention that advances the consistency between semantic attention and spatial/temporal attention. Speech recognition and separation . One of the most eminent tasks for audio understanding may be speech recognition, the task of recognizing and translating human spoken language into text with less emphasis on background sound that may coexist. A multitude of datasets exist for such task e.g . Speech Commands dataset (Warden, 2018), Common Voice dataset (Mozilla, 2017), Librispeech (Panayotov et al., 2015), LS Speech (Ito, 2017). As one of similar lineage, automatic speech separation forks an input audio signal into several individual speech sources (Hershey et al., 2016; Ephrat et al., 2018). To most of these tasks, in the wild sound is deemed as background noise to be removed as an obstructer of speech recognition. On the other hand, our work puts the spotlight on these neglected sounds and express them through natural language. Audio classification and sound event detection . This line of tasks emphasizes categorizing a sound into a set of predefined classes. There exist a number of datasets to aid in achieving this goal, including DCASE series (Stowell et al., 2015; Mesaros et al., 2016, 2017), UrbanSound8k (Salamon et al., 2014), ESC (Piczak, 2015). AudioSet (Gemmeke et al., 2017) is an audio event dataset collected from Youtube that is unsurpassed in terms of coverage and size, structured with an ontology containing 527 classes. Another predominant large-scale dataset is Freesound (Fon-seca et al., 2017). It consists of audio samples from freesound.org recordings based on the preceding AudioSet ontology. In contrast to audio classification, which uniquely map the audio to a set of labels, our task generates a descriptive sentence. Hence, it needs to not only detect salient sounds of classes but also explores their states, properties, actions or interactions. Captioning tasks and datasets . The vast majority of captioning tasks and datasets focus on the visual domain. Image captioning generates text description of an image, and numerous datasets are proposed, such as Flickr 8k (Rashtchian et al., 2010), Flickr 30k (Young et al., 2014), MS COCO (Lin et al., 2014), DenseCap (Johnson et al., 2016) and Conceptual Captions (Sharma et al., 2018). Akin to the image captioning is video captioning, for which there are many datasets too, including MSVD (Guadarrama et al., 2013), MSR-VTT (Xu et al., 2016), LSMDC (Rohrbach et al., 2017) and ActivityNet Captions (Krishna et al., 2017).Com-pared to previous captioning tasks and datasets, our work confines the problem by focusing on in the wild audio inputs. Recently, there have been some efforts to solve video captioning with audio input (Hori et al., 2017, 2018; Wang et al., 2018). However, the audio input merely serves as auxiliary features for video captioning, and as a result, it only marginally improves the performance ( e.g . BLEU-4 score: 39.6 (video only) vs. 40.3 (video + MFCC) (Wang et al., 2018)). These results are partly culpable to dataset collection, where the annotators mostly rely on the video input. On the contrary, our collection induces the annotators to mainly abide to audio, hence, increasing the dependency of written text on the audio input as can be shown in our survey analysis in Figure 5. Our AudioCaps dataset entails 46K audio caption pairs. Table 1 outlines its key statistics. The audio sources are rooted in AudioSet (Gemmeke et al., 2017), a large-scale audio event dataset, from which we draft the AudioCaps , as discussed below. We present more details of data collection and statistics in the Appendix. It is important to select qualified audio clips as the first step of dataset collection. The chosen categories of clips must be well-rounded in coverage of naturally occurring audios, be relevant to practical applications and appear with high frequency. To that end, we tailor the AudioSet dataset (Gemmeke et al., 2017) that comprises 1,789,621 human-labeled 10 second YouTube excerpts with an ontology of 527 audio event categories. However, an immediate collection of captions from these audios pose several difficulties: (i) too many audio clips, (ii) inconsistent level of abstraction among the classes, (iii) distribution bias of some labels and (iv) noisy labels that are only noticeable from visual cues. We circumvent these issues through a controlled sampling process as described below. Among 527 audio event categories of AudioSet, we first exclude all the labels whose number of clips are less than 1,000 to promote a balanced distribution within the dataset. We also remove all 151 labels in the music super-category, because they are often indiscernible even for a human. For example, a human with no expertise can hardly discriminate the sound of Guitar from Banjo . Thus, we set aside the musical territory for future exploration. We further discard categories if they do not satisfy the following two constraints. The word labels should be identifiable solely from sound (i) without requiring visuals ( e.g . remove the category inside small room ) and (ii) without requiring any expertise ( e.g . remove power windows and electric windows because their distinction may be possible only for car experts). Fi-Split # clips # captions # words/caption # labels/clip Train 38,118 38,118 8.79 (8) 4.25 (4) Val 500 2,500 10.12 (9) 4.06 (3) Test 979 4,895 10.43 (9) 4.03 (3) Total 39,597 45,513 9.03 (9) 4.22 (4) Table 1: Some statistics of AudioCaps dataset. nally, we select 75 word labels derived from 7 augmented super-categories as avoiding the sharp skewness in the word labels ( e.g . 48.5% clips include speech label). We limit the number of instances per category to 2,000 by sampling with preference to audio clips associated with more word labels to prioritize the audios with diverse content. The final number of audio clips is about 115K, from which we obtain captions for 46K as the first version. The collected captions should be precise, spe-cific, diverse, expressive, large-scale and correlated with the paired audios with minimal visual presumptions. Such complex nature of our requirements necessitates employing crowdworkers through Amazon Mechanical Turk (AMT). Some qualification measures are set for the crowdworkers, such as they should hold a +95% HIT approval rate and the total number of approved HITs that are greater than 1,000 and be located at one of [AU, CA, GB, NZ, US]. In total, 108 caption writing workers and 3 caption reviewing workers participate and are compensated at 10 cents per clip. Annotation Interface . Figure 2 shows our annotation interface, which is designed to minimize the visual presumption while maintaining diversity. Each task page consists of an audio clip of about 10 seconds, word hints and video hints. The word hints are the word labels that are provided by AudioSet for the clip and are employed A train is approaching with a low rumble and rhythmic click and squeal Below officers creep toward the entrance the door and points a gun as hints to the crowdworkers. Even to humans, recognizing the true identity of a sound can be ambiguous, and thus the word hints act as a precursor to accurately guide the crowdworkers during the description process, while staying aloof from visual bias. Another benefit is that the diversity of the word labels may also enrich the expressiveness of the description. Also derived from AudioSet, the video hints are provided as a stronger hint for sounds that are too difficult even to the human ear or for clips associated with some erroneous or missing word hints (weak labels). We advise the workers to use them as a last resort measure. Some instructions 2 are also provided to demarcate crowdworkers' descriptions as follows. (i) Do not include the words for visuals in the video that are not present in the sound. (ii) Ignore speech semantics. (iii) When applicable, be detailed and expressive. (iv) Do not be imaginative and be literal and present with the descriptions. Quality Control . We use a qualification test to discern many crowdworkers who frequently violate the given instructions ( e.g . transcribing instead of describing, just enumerating provided word hints or writing visual captions). Interested crowdworkers must participate in the test and submit a response, which the authors manually check and approve if they are eligible. We employ three additional workers to verify the data in accordance to our guidelines. In order to maintain high approval rates, we periodically blacklist malicious crowdworkers while granting reasonable incentives to benevolent workers. We exclude the period symbol from all the captions, convert numbers to words using num2words 3 and correct grammar errors by languagetool 4 . We then tokenize words with spacy 5 . Finally, we build a dictionary V with a size of 4506 by choosing all the unique tokens. Figure 3 qualitatively compares some caption examples between our AudioCaps and two captioning datasets with audio: LSMDC (Rohrbach et al., 2017) and MSR-VTT (Xu et al., 2016). Since both LSMDC and MSR-VTT focus more on describing videos than audios, their captions are characterized by visually grounded vocabularies (blue). On the other hand, the captions of AudioCaps accompany sound-based vocabularies (red). We present a hierarchical captioning model that can attend to the fine details of the audio. The backbone of our model is an LSTM (Hochreiter and Schmidhuber, 1997) that we fortify with two novel components which are easily integrable with any attention-based captioning model. The topdown multi-scale encoder enables the contextual use of multi-level features, and the aligned semantic attention enhances the consistency between semantic attention and temporal attention (see Figure 4). Our experiments in section 5.3 show that these two techniques lead to non-trivial performance improvement. The input to our model are mel-frequency cepstral coefficient (MFCC) audio features (Davis and Mermelstein, 1980) and the output is a sequence of words { y m } Mm =1 , each of which is a symbol from the dictionary. For text representation, we use fastText (Bojanowski et al., 2016) trained on the Common Crawl corpus to initialize the word embedding matrix W emb , which is fine-tuned with the model during training. We represent word sequences ( e.g . attribute words for semantic attention and output words for answer captions) in a distributional space as { d n } Nn =1 with d n = W emb w n where w n is a one-hot vector for n -th word in the word sequence and d n  R 300 . Unlike speech data, sound in the wild is not always continuous. It can be often brief, noisy, occluded, in-the-distance and randomly sparsed throughout the audio. Hence, the lower-level features can be useful to capture such characteristics of natural sound, although they may lack the semantics of the higher-level features. Thus, the joint use of these two levels of features can be mutually beneficial. The top-down multi-scale encoder takes as input the two-level audio embedding { f t } Tt =1 , { c t } Tt =1 and generates the fused encoding vector, where T is the sequence length of the audio. For input, we use the features from the two layers of the pretrained VGGish network (Hershey et al., 2017): the fc2 vector { f t } Tt =1 as a high-level semantic feature, and the conv4 vector { c t } Tt =1 as a mid-level feature. The first level of hierarchy encodes high-level features { f t } Tt =1 using a bi-directional LSTM. We regard the last hidden state as the global audio embedding h ctxt  RI :  h a 1 t = biLSTM ( f t ,  h a 1 t  1 ,  h a 1 t +1 ) , (1) h ctxt = W c [  h a 1 T ;  h a 11 ] + b c , (2) where W c  RI  D 1 and b c  RI are parameters, I is the dimension of input to the next layer and D 1 is the dimension of the first layer hidden states. We then reshape and encode mid-level features { c t } Tt =1  R 512 using another bi-directional LSTM. In order to inject the global semantics, we perform an element-wise addition of h ctxt to the mid-level feature along the time axis, and feed them into the bi-directional LSTM one at a time,  "#  $# a cat meows  and cry VGGish cry baby cat c4 fc2 Aligned Semantic Attention Decoder Semantic Encoder TopDown Encoder f $ f " & $ & "  '()( * $ * +  $,  -, attention flow temporal attention Figure 4: The audio captioning model with top-down multi-scale encoder and aligned semantic attention . In many captioning models (You et al., 2016; Yu et al., 2017; Laokulrat et al., 2018; Long et al., 2018), semantic attention has been independently used from temporal/spatial attention. However, it can be troublesome because there may exist some discrepancies between the two attentions i.e . they do not attend to the same part of the input. For instance, given an audio of a cat meowing and a baby crying, temporal attention may attend to the crying baby while semantic attention attends to the word cat . We propose a simple yet effective approach that implicitly forces both semantic and tempo-ral/spatial attention to be correctly aligned to one another to maximize the mutual consistency. For semantic attention, we extract a set of N attribute words for each audio: following You et al. (2016), we retrieve the nearest training audio from the subset of AudioSet and transfer its labels as attribute words. We encode each attribute word vector using a bi-directional LSTM (named semantic encoder ):  h wn = biLSTM ( d n ,  h wn  1 ,  h wn +1 ) , (4) where d n is the input text representation of the attribute word sequence. We then align these semantic word features  h wn to the temporal axis of the audio features  h a 2 t via the attention flow layer (Seo et al., 2017). For notational simplicity, we omit the bidirectional arrow in the following. Attention flow layer . We first compute the similarity matrix, S  RT  N between each pair of audio and word features using the score function  ( h a 2 t , h wn )  R :  ( h a 2 t , h wn ) = W  [ h a 2 t ; h wn ; h a 2 t  h wn ] , (5) S tn =  ( h a 2 t , h wn ) , (6) where  is element-wise multiplication. We then use S to obtain the attentions and the attended vectors in two directions: word-to-audio {  h wt } Tt =1  RD 2 and audio-to-word  h a 2  RD 2 : a t = softmax ( S t : ) ,  h wt = (cid:88) n a tn h wn , (7) b = softmax (max row ( S )) ,  h a 2 = (cid:88) t b t h a 2 t , (8) where a t  RN , b  RT . Lastly, we concatenate them into { h flowt } Tt =1  R 4 D 2 , while keeping the temporal axis intact: h flowt = [ h a 2 t ;  h wt ; h a 2 t   h wt ; h a 2 t   h a 2 ] . (9) Temporal attention over attention flow . We now have an embedding that aligns the semantic features of words with the time steps of audio features. Subsequently, we apply temporal attention over it; the attention weight is calculated as in Lu-ong et al. (2015). Specifically, we use the global method for each t in { h flowt } Tt =1 :  m = align ( h decm , h flowt ) , (10) c m = (cid:88) t  mt h flowt , (11) a m = tanh ( W dec [ c m ; h decm ]) , (12) where h decm  RD o is the state of the decoder LSTM, c m  R 4 D 2 is the context vector,  m  RT is the attention mask, and W dec  RD o  (4 D 2 + D o ) is a parameter. Next, we obtain the output word probability: s m = softmax ( W o a m ) (13) where W o  RV  D o . Finally, we select the output word as y m +1 = argmax s V ( s m ) . We repeat this process until y m +1 reaches an EOS token. The model is trained to maximize the log-likelihood assigned to the target labels via the softmax as done in most captioning models. We perform several quantitative evaluations to provide more insights about our AudioCaps dataset. Specifically, our experiments are designed to answer the following questions: 1. Are the collected captions indeed faithful to the audio inputs? 2. Which audio features are useful for audio captioning on our dataset? 3. What techniques can improve the performance of audio captioning? We present further implementation details and more experimental results in the Appendix. Some resulting audio-caption pairs can be found at https://audiocaps.github.io/supp. Evaluation metrics . Audio captioning can be quantitatively evaluated by the language similarity between the predicted sentences and the ground-truths (GTs) such as BLEU (Papineni et al., 2002), CIDEr (Vedantam et al., 2015), METEOR (Baner-jee and Lavie, 2005), ROUGE-L (Lin, 2004) and SPICE (Anderson et al., 2016). In all metrics, higher scores indicate better performance. Audio features . Audios are resampled to 16kHz, and stereo is converted into mono by averaging both channels. We zero-pad clips that are shorter than 10 seconds and extract three levels of audio features. For the low-level audio feature, the lengthy raw audios are average-pooled by the WaveNet encoder as in Engel et al. (2017). For the mid-level feature, mel-frequency cepstral co-efficients (MFCC) (Davis and Mermelstein, 1980) are extracted using librosa (McFee et al., 2015) with a window size of 1024, an overlap of 360 and the number of frames at 240, and encoded further with a bi-directional LSTM followed by a gated convolutional encoder (Xu et al., 2018). Lastly, we use two high-level features: the 24th output layer of SoundNet 6 (Aytar et al., 2016) with a (10  1024) dimension and the final output embedding of VGGish 7 (Hershey et al., 2017) with a (10  128) dimension of (time  embedding). Video features . To contrast with video captioning datasets, we also extract video features at the frame-level and at the sequence-level from YouTube clips. For frame features, we use VGG16 (Simonyan and Zisserman, 2015) pretrained on the ILSVRC-2014 dataset (Rus-sakovsky et al., 2015). For sequence features, we use C3D 8 (Tran et al., 2015) pretrained on the Sport1M dataset (Karpathy et al., 2014). We extract subsequent frames with 50% overlap centered at each time step on the input clips for AudioSet videos, while proceeding with no overlap for MSR-VTT clips as in the original paper. We sample videos at 25fps. Retrieval methods . As straightforward baselines, we test the 1-nearest search with audio features, denoted by 1NN-MFCC , 1NN-SoundNet and 1NN-VGGish . For a query audio, we find its closest training audio using the (cid:96) 2 distance on the features and return its text as a prediction. We mean-pool all the audio features over time, because it empirically leads to a strong performance. LSTM methods . As simple generative baselines, we test with the LSTM decoder, denoted by -LSTM postfix, where the encoded audio feature is set as the initial state of the LSTM. For instance, WaveNet-LSTM is the model with the WaveNet encoder and the LSTM decoder. We use a single-layer LSTM with dropout (Srivastava et al., 2014) and layer normalization (Ba et al., 2016). Attention models . We test two popular attention models developed in video captioning research: (i) TempAtt (Luong et al., 2015; Yao et al., 2016) generates captions by selectively attending to audio features over time, and (ii) SemAtt (You et al., 2016) creates text attending to attribute words as secondary information. Our models . We denote our top-down multi-scale encoder as the prefix TopDownand aligned semantic attention as AlignedAtt. Upper-bounds . Given that each test data has five human-generated captions, we perform cross validation on the five GT captions as an upper-bound of performance denoted as Human . We regard one of five human annotations as model prediction and compute the performance metric with the other four as ground-truths. After doing this on each of five, we then average the scores. We discuss experimental results in response to the three questions regarding the AudioCaps dataset. We first evaluate whether the collected audio-based captions are indeed loyal to the audio clips. As one possible method to validate it, we perform comparative experiments with the video-oriented MSR-VTT dataset (Xu et al., 2016). Note that MSR-VTT and AudioCaps both provide pairs of audio clips and its corresponding videos, allowing us to perform this comparative study. We hypothesize that the captions from MSR-VTT would not coherently map to audio features, because they are written mainly based on the visual information. In contrast, AudioCaps captions would be better aligned to audio features than visual features. The results in Table 4 support our hypothesis. In MSR-VTT, the video-based captioning model C3D-LSTM attains better scores than the preceding three audio-captioning models *-LSTM , while in AudioCaps the video-based model performs far worse than the audio models. This may be due to our collection method of AudioCaps, which encourages turkers to submit the descriptions based on the audio rather than the visual. Vocabulary comparison . We also make comparisons between AudioCaps and MSR-VTT in terms of vocabulary usage in the captions. We select the 1,800 most frequent vocabularies of verbs, adjectives and adverbs from each dataset, and run a user study in which three different workers are asked to categorize each sampled word into one of ( Audio, Visual, Both, Not Applicable ). The category label per word is decided by a majority vote of three workers' opinions. We use AMT once more to collect the unbiased opinions. In or-Methods B-1 B-2 B-3 B-4 METEOR CIDEr ROUGE-L SPICE 1NN-MFCC 34.1 17.8 10.0 5.3 9.9 8.7 23.4 4.7 1NN-SoundNet (Aytar et al., 2016) 39.1 22.0 12.9 7.6 12.0 16.4 27.2 6.9 1NN-VGGish (Hershey et al., 2017) 44.2 26.5 15.8 9.0 15.1 25.2 31.2 9.2 WaveNet-LSTM (Engel et al., 2017) 48.9 31.5 20.2 13.0 13.8 29.6 35.5 9.0 MFCC-LSTM (Xu et al., 2018) 57.3 40.0 26.8 16.4 18.4 44.8 41.1 11.5 SoundNet-LSTM (Aytar et al., 2016) 54.0 38.0 26.4 17.6 16.5 43.2 39.2 10.8 VGGish-LSTM (Hershey et al., 2017) 58.7 42.3 29.8 20.4 18.7 50.4 42.6 13.0 TempAtt-WaveNet-LSTM (Luong et al., 2015) 50.7 34.3 22.9 14.8 14.8 28.2 36.4 8.6 TempAtt-MFCC-LSTM (Luong et al., 2015) 57.7 40.7 27.6 17.9 18.2 49.3 41.8 12.4 TempAtt-SoundNet-LSTM (Luong et al., 2015) 55.5 37.4 24.8 15.8 17.0 43.4 40.0 11.6 TempAtt-VGGish(FC2)-LSTM (Luong et al., 2015) 61.3 43.2 29.6 19.5 19.3 50.9 43.5 13.5 TempAtt-VGGish(C4)-LSTM (Luong et al., 2015) 61.8 44.5 30.7 20.4 19.4 55.3 44.0 13.2 TempAtt-VGGish(C3)-LSTM (Luong et al., 2015) 61.2 44.1 30.3 20.9 19.0 52.3 43.7 13.0 TopDown-VGGish(FC2,C4)-LSTM 62.9 45.1 31.5 21.4 19.9 57.7 44.8 14.3 TopDown-VGGish(FC2,C4,C3)-LSTM 60.9 43.7 30.7 20.8 20.0 55.8 43.7 13.6 TopDown-SemTempAtt(1NN) (You et al., 2016) 62.2 44.9 31.3 20.9 20.2 58.1 44.9 13.6 TopDown-AlignedAtt(1NN) 61.4 44.6 31.7 21.9 20.3 59.3 45.0 14.4 Human 65.4 48.9 37.3 29.1 28.8 91.3 49.6 21.6 Table 2: Captioning results of different methods on AudioCaps measured by language similarity metrics. der to guarantee thoughtful submissions, we ask the workers to provide a description using the word. We compensate $0.05 per word to English-speaking workers with a 95% approval rate. Figure 5 shows that AudioCaps has more vocabularies tagged as Audio ( e.g . neighs, rustling ) by 18.9%p more than MSR-VTT. Furthermore, 56.3% of the total vocabularies in AudioCaps are categorized as audio-related, that is, labeled as Audio or Both ( e.g . vibrating, applauds ). Hence, this vocabulary comparison result reassures that AudioCaps is more audio-oriented than MSR-VTT. are more suitable for captioning on AudioCaps. The best results are obtained by VGGish-LSTM . This may be because VGGish is pretrained on YouTube audio clips, similar to AudioCaps. Although the topics of YouTube are extremely diverse, the domain proximity may help VGGish learn more utilizable features for AudioCaps. SoundNet-LSTM shows inferior performance compared to VGGish-LSTM , one possible reason being because it is pretrained with Flickr videos, which are rather distant in domain from the source of our dataset, in terms of topic diversity and the amount of possible noise. MFCC-LSTM does not perform as well as VGGish-LSTM , even with the similar convolutional recurrent encoder. This result hints that pretraining with a proper dataset is essential for audio captioning. A comparison between MFCC-LSTM and WaveNet-LSTM reveals that using MFCC is better than directly taking raw waveform as input. The raw waveform is relatively long ( > 500  longer than MFCC); hence, it may pose a difficulty for RNN-based encoders to precisely represent the whole audio context. Temporal attention consistently boosts the captioning performance of the LSTM decoder in all audio features, as shown in the models with TempAttprefix in Table 2. No-(Ours) tably, a large performance gain is observed for TempAtt-MFCC-LSTM . This may be because MFCC features are transformed to temporally longer features than SoundNet and VGGish features (240 > 10), and thus allow temporal attention to better aid the model and bypass the vanishing gradient problem. The semantic attention is also favorable for captioning performance, as SemTempAtt(1NN)-VGGish-LSTM in Table 3 slightly outperforms TempAtt-VGGish(FC2)-LSTM in Table 2. That is, the additional use of semantic attention enhances the temporal attention model. Obviously, when using GT labels instead of 1NN retrieved labels as attribute words, the performance increases much, hinting that better semantic attributes are more synergetic with the aligned attention. The comparison between different layers (C4, C3, FC2) confirms the effectiveness of jointly using multi-level features. The fused features by the top-down multi-scale encoder ( i.e . TopDown) prove the most beneficial as they outperform their counterparts in Table 2. However, a stack of (FC2,C4) layers performs the best, while the three layer stack is slightly inferior, presumably due to overfitting and weak information flow between the upper and lower levels of the stacks. Finally, our best performing model is TopDown-AlignedAtt where both the topdown multi-scale encoder and aligned semantic attention are jointly used. We postulate that the two techniques synergize well thanks to rich information provided by TopDown allowing for better attention alignment. Figure 6 shows selected examples of audio captioning. In each set, we show a video frame, GT and text descriptions generated by our method and baselines. Many audio clips consist of sounds with multiple sources in sequence, for which baselines often omit some details or mistakenly order the event sequence, whereas our model is better at capturing the details in the correct order. We addressed a new problem of audio captioning for sound in the wild. Via Amazon Mechanical Turk, we contributed a large-scale dataset named AudioCaps , consisting of 46K pairs of audio clips and human-written text. In our experiments, we showed that the collected captions were indeed faithful to the audio inputs as well as improve the captions by two newly proposed components: the top-down multi-scale encoder and aligned semantic attention. There are several possible directions beyond this work. First, we can further expand the scope of AudioCaps . Second, our model is integrable with speech counterparts to achieve more complete auditory captioning tasks. We would like to thank SNU Vision & Learning Lab members and Yunseok Jang for the helpful comments and discussions. This work is supported by Kakao and Kakao Brain corporations and the international cooperation program by the NRF of Korea (NRF-2018K2A9A2A11080927). Gunhee Kim is the corresponding author.
objective abstain result objective abstain abstain abstain abstain abstain other abstain abstain abstain abstain result objective abstain method objective objective objective objective objective result result objective other other other other other other abstain other other other other other other objective other other other other other method other other other other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method method abstain abstain method method abstain method method method method abstain abstain abstain abstain abstain method abstain method method method abstain abstain abstain abstain method method method method method abstain method method method abstain method method method method method method abstain method method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain result abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain method other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method objective objective objective abstain objective result other other other