Department of Computer Science and Engineering & Center for Superintelligence Seoul National University, Seoul, Korea We address the problem of abstractive summarization in two directions: proposing a novel dataset and a new model. First, we collect Reddit TIFU dataset, consisting of 120K posts from the online discussion forum Reddit. We use such informal crowd-generated posts as text source, in contrast with existing datasets that mostly use formal documents as source such as news articles. Thus, our dataset could less suffer from some biases that key sentences usually locate at the beginning of the text and favorable summary candidates are already inside the text in similar forms. Second, we propose a novel abstractive summarization model named multilevel memory networks (MMN), equipped with multi-level memory to store the information of text from different levels of abstraction. With quantitative evaluation and user studies via Amazon Mechanical Turk, we show the Reddit TIFU dataset is highly abstractive and the MMN outperforms the state-of-the-art summarization models. The code and dataset are available at http://vision. snu.ac.kr/projects/reddit-tifu . Abstractive summarization methods have been under intensive study, yet they often suffer from inferior performance compared to extractive methods (Allahyari et al., 2017; Nallapati et al., 2017; See et al., 2017). Admittedly, by task defini-tion, abstractive summarization is more challenging than extractive summarization. However, we argue that such inferior performance is partly due to some biases of existing summarization datasets. The source text of most datasets (Over et al., 2007; Hermann et al., 2015; Cohan et al., 2018; Grusky et al., 2018; Narayan et al., 2018a) originates from formal documents such as news articles, which have some structural patterns of which extractive methods better take advantage. In formal documents, there could be a strong tendency that key sentences locate at the beginning of the text and favorable summary candidates are already inside the text in similar forms. Hence, summarization methods could generate good summaries by simply memorizing keywords or phrases from particular locations of the text. Moreover, if abstractive methods are trained on these datasets, they may not show much abstraction (See et al., 2017), because they are implicitly forced to learn structural patterns (Kedzie et al., 2018). Grusky et al. (2018) and Narayan et al. (2018a) recently report similar extractive bias in existing datasets. They alleviate this bias by collecting articles from diverse news publications or regarding intro sentences as gold summary. Different from previous approaches, we propose to alleviate such bias issue by changing the source of summarization dataset. We exploit user-generated posts from the online discussion forum Reddit , especially TIFU subreddit, which are more casual and conversational than news articles. We observe that the source text in Reddit does not follow strict formatting and disallows models to simply rely on locational biases for summarization. Moreover, the passages rarely contain sentences that are nearly identical to the gold summary. Our new large-scale dataset for abstractive summarization named as Reddit TIFU contains 122,933 pairs of an online post as source text and its corresponding long or short summary sentence. These posts are written by many different users, but each pair of post and summary is created by the same user. Another key contribution of this work is to propose a novel memory network model named multilevel memory networks (MMN). Our model is equipped with multi-level memory networks, storing the information of source text from different levels of abstraction ( i.e . word-level, sentence-level, paragraph-level and document-level). This design is motivated by that abstractive summarization is highly challenging and requires not only to understand the whole document, but also to find salient words, phrases and sentences. Our model can sequentially read such multiple levels of information to generate a good summary sentence. Most abstractive summarization methods (See et al., 2017; Li et al., 2017; Zhou et al., 2017; Liu et al., 2018; Cohan et al., 2018; Paulus et al., 2018) employ sequence-to-sequence (seq2seq) models (Sutskever et al., 2014) where an RNN encoder embeds an input document and another RNN decodes a summary sentence. Our MMN has two major advantages over seq2seq-based models. First, RNNs accumulate information in a few fixed-length memories at every step regardless of the length of an input sequence, and thus may fail to utilize far-distant information due to vanishing gradient. It is more critical in summarization tasks, since input text is usually very long ( > 300 words). On the other hand, our convolutional memory explicitly captures long-term information. Second, RNNs cannot build representations of different ranges, since hidden states are sequentially connected over the whole sequence. This still holds even with hierarchical RNNs that can learn multiple levels of representation. In contrast, our model exploits a set of convolution operations with different receptive fields; hence, it can build representations of not only multiple levels but also multiple ranges ( e.g . sentences, paragraphs, and the whole document). Our experimental results show that the proposed MMN model improves abstractive summarization performance on both our new Reddit TIFU and existing Newsroom-Abs (Grusky et al., 2018) and XSum (Narayan et al., 2018a) datasets. It outperforms several state-of-the-art abstractive models with seq2seq architecture such as (See et al., 2017; Zhou et al., 2017; Li et al., 2017). We evaluate with quantitative language metrics ( e.g . perplexity and ROUGE (Lin, 2004)) and user studies via Amazon Mechanical Turk (AMT). The contributions of this work are as follows. 1. We newly collect a large-scale abstractive summarization dataset named Reddit TIFU . As far as we know, our work is the first to use non-formal text for abstractive summarization. 2. We propose a novel model named multi-level memory networks (MMN). To the best of our knowledge, our model is the first attempt to leverage memory networks for the abstractive summarization. We discuss the unique updates of the MMN over existing memory networks in Section 2. 3. With quantitative evaluation and user studies via AMT, we show that our model outperforms state-of-the-art abstractive summarization methods on both Reddit TIFU, Newsroom abstractive subset and XSum dataset. Our work can be uniquely positioned in the context of the following three topics. Neural Abstractive Summarization . Many deep neural network models have been proposed for abstractive summarization. One of the most dominant architectures is to employ RNN-based seq2seq models with attention mechanism such as (Rush et al., 2015; Chopra et al., 2016; Nallapati et al., 2016; Cohan et al., 2018; Hsu et al., 2018; Gehrmann et al., 2018).In addition, recent advances in deep network research have been promptly adopted for improving abstractive summarization. Some notable examples include the use of variational autoencoders (VAEs) (Miao and Blunsom, 2016; Li et al., 2017), graph-based attention (Tan et al., 2017), pointer-generator models (See et al., 2017), self-attention networks (Liu et al., 2018), reinforcement learning (Paulus et al., 2018; Pasunuru and Bansal, 2018), contextual agent attention (Celikyilmaz et al., 2018) and integration with extractive models (Hsu et al., 2018; Gehrmann et al., 2018). Compared to existing neural methods of abstractive summarization, our approach is novel to replace an RNN-based encoder with explicit multi-level convolutional memory. While RNN-based encoders always consider the whole sequence to represent each hidden state, our multilevel memory network exploits convolutions to control the extent of representation in multiple levels of sentences, paragraphs, and the whole text. Summarization Datasets . Most existing summarization datasets use formal documents as source text. News articles are exploited the most, including in DUC (Over et al., 2007), Gigaword (Napoles et al., 2012), CNN/DailyMail (Nal-lapati et al., 2016; Hermann et al., 2015), News-TIFU by forgetting my chemistry textbook and all of my notes in a city five hours away () So the past three days I was at a sporting event in Windsor. room (Grusky et al., 2018) and XSum (Narayan et al., 2018a) datasets. Cohan et al. (2018) introduce datasets of academic papers from arXiv and PubMed. Hu et al. (2015) propose the LC-STS dataset as a collection of Chinese microblog's short text each paired with a summary. However, it selects only formal text posted by verified organizations such as news agencies or government institutions. Compared to previous summarization datasets, our dataset is novel in that it consists of posts from the online forum Reddit. Rotten Tomatoes and Idebate dataset (Wang and Ling, 2016) use online text as source, but they are relatively small in scale: 3.7K posts of RottenTomatoes compared to 80K posts of TIFU-short as shown in Table 1. Moreover, Rotten Tomatoes use multiple movie reviews written by different users as single source text, and one-sentence consensus made by another professional editor as summary. Thus, each pair of this dataset could be less coherent than that of our TIFU, which is written by the same user. The Idebate dataset is collected from short arguments of debates on controversial topics, and thus the text is rather formal. On the other hand, our dataset contains the posts of interesting stories happened in daily life, and thus the text is more unstructured and informal. Neural Memory Networks . Many memory network models have been proposed to improve memorization capability of neural networks (Kaiser et al., 2017; Na et al., 2017; Yoo et al., 2019). Weston et al. (2014) propose one of early memory networks for language question answering (QA); since then, many memory networks have been proposed for QA tasks (Sukhbaatar Dataset # posts # words/post # words/summ RottenTomatoes 3,731 2124.7 (1747) 22.2 (22) Idebate 2,259 178.3 (160) 11.4 (10) TIFU-short 79,949 342.4 (269) 9.33 (8) TIFU-long 42,984 432.6 (351) 23.0 (21) Table 1: Statistics of the Reddit TIFU dataset compared to existing opinion summarization corpora, RottenTomatoes and Idebate (Wang and Ling, 2016). et al., 2015; Kumar et al., 2016; Miller et al., 2016). Park et al. (2017) propose a convolutional read memory network for personalized image captioning. One of the closest works to ours may be Singh et al. (2017), which use a memory network for text summarization. However, they only deal with extractive summarization by storing embeddings of individual sentences into memory. Compared to previous memory networks, our MMN has four novel features: (i) building a multi-level memory network that better abstracts multi-level representation of a long document, (ii) employing a dilated convolutional memory write mechanism to correlate adjacent memory cells, (iii) proposing normalized gated tanh units to avoid covariate shift within the network, and (iv) generating an output sequence without RNNs. We introduce the Reddit TIFU dataset whose key statistics are outlined in Table 1. We collect data from Reddit, which is a discussion forum platform with a large number of subreddits on diverse topics and interests. Specifically, we crawl all the posts from 2013-Jan to 2018-Mar in the TIFU subreddit, where every post should strictly follow the posting rules, otherwise they are removed. Thanks to the following rules 1 , the posts in this subreddit can be an excellent corpus for abstractive summarization: Rule 3: Posts and titles without context will be removed. Your title must make an attempt to encapsulate the nature of your f***up. Rule 11: All posts must end with a TL;DR summary that is descriptive of your f***up and its consequences . Thus, we regard the body text as source, the title as short summary, and the TL;DR summary as long summary. As a result, we make two sets of datasets: TIFU-short and TIFU-long . Figure 1 shows an example post of the TIFU subreddit. PG Lead Ext-Oracle PG/Lead PG/Oracle Dataset R-1 R-2 R-L R-1 R-2 R-L R-1 R-2 R-L Ratio (R-L) Ratio (R-L) CNN/DM (Nallapati et al., 2016) 36.4 15.7 33.4 39.6 17.7 36.2 54.7 30.4 50.8 0.92x 0.66x NY Times (Sandhaus, 2008) 44.3 27.4 40.4 31.9 15.9 23.8 52.1 31.6 46.7 1.70x 0.87x Newsroom (Grusky et al., 2018) 26.0 13.3 22.4 30.5 21.3 28.4 41.4 24.2 39.4 0.79x 0.57x Newsroom-Abs (Grusky et al., 2018) 14.7 2.2 10.3 13.7 2.4 11.2 29.7 10.5 27.2 0.92x 0.38x XSum (Narayan et al., 2018a) 29.7 9.2 23.2 16.3 1.6 12.0 29.8 8.8 22.7 1.93x 1.02x TIFU-short 18.3 6.5 17.9 3.4 0.0 3.3 8.0 0.0 7.7 5.42x 2.32x TIFU-long 19.0 3.7 15.1 2.8 0.0 2.7 6.8 0.0 6.6 5.59x 2.29x Table 2: Comparison of F1 ROUGE scores between different datasets (row) and methods (column). We build a vocabulary dictionary V by choosing the most frequent V (=15K) words in the dataset. We exclude any urls, unicodes and special characters. We lowercase words, and normalize digits to 0. Subreddit names and user ids are replaced with @subreddit and @userid token, respectively. We use markdown 2 package to strip markdown format, and spacy 3 to tokenize words. Common prefixes of summary sentences ( e.g . tifu by, tifu-, tl;dr, etc) are trimmed. We do not take OOV words into consideration, since our vocabulary with size 15K covers about 98% of word frequencies in our dataset. We set the maximum length of a document as 500. We exclude the gold summaries whose lengths are more than 20 and 50 for TIFU-short and TIFU-long , respectively. They amount to about 0.6K posts in both datasets ( i.e . less than 1% and 3%). We use these maximum lengths, based on previous datasets ( e.g . 8, 31, 56 words on average per summary in Gigaword, DUC, and CNN/DailyMail datasets, respectively). We randomly split the dataset into 95% for training, 5% for test. We discuss some abstractive characteristics found in Reddit TIFU dataset, compared to existing sum-2 Weak Lead Bias . Formal documents including news articles tend to be structured to emphasize key information at the beginning of the text. On the other hand, key information in informal online text data are more spread across the text. Figure 2 plots the density histogram of the relative locations of bigrams of gold summary in the source text. In the CNN/DailyMail and Newsroom, the bigrams are highly concentrated on the front parts of documents. Contrarily, our Reddit TIFU dataset shows rather uniform distribution across the text. This characteristic can be also seen from the ROUGE score comparison in Table 2. The Lead baseline simply creates a summary by selecting the first few sentences or words in the document. Thus, a high score of the Lead baseline implicates a strong lead bias. The Lead scores are the lowest in our TIFU dataset, in which it is more difficult for models to simply take advantage of locational bias for the summary. Strong Abstractness . Besides the locational bias, news articles tend to contain wrap-up sentences that cover the whole article, and they often have resemblance to its gold summary. Its existence can be measured by the score of the Ext-Oracle baseline, which creates a summary by selecting the sentences with the highest average score of F1 ROUGE-1/2/L. Thus, it can be viewed as an upper bound for extractive models (Narayan et al., 2018a,b; Nallapati et al., 2017). In Table 2, the ROUGE scores of the Ext-Oracle are the lowest in our TIFU dataset. It means that the sentences that are similar to gold summary scarcely exist inside the source text in our dataset. This property forces the model to be trained to focus on comprehending the entire text instead of simply finding wrap-up sentences. Finally, PG/Lead and PG/Oracle in Table 2 are the ROUGE-L ratios of PG with Lead and Ext-Oracle , respectively. These metrics can quantify the dataset according to the degree of difficulty for extractive methods and the suitability for abstractive methods, respectively. High scores of the TIFU dataset in both metrics show that it is potentially an excellent benchmark for evaluation of abstractive summarization systems. Figure 3 shows the proposed multi-level memory network (MMN) model. The MMN memorizes the source text with a proper representation in the memory and generates a summary sentence one word at a time by extracting relevant information from memory cells in response to previously generated words. The input of the model is a source text { x i } = x 1 , ..., x N , and the output is a sequence of summary words { y t } = y 1 , ..., y T , each of which is a symbol from the dictionary V . Online posts include lots of morphologically similar words, which should be closely embedded. Thus, we use the fastText (Bojanowski et al., 2016) trained on the Common Crawl corpus, to initialize the word embedding matrix W emb . We use the same embedding matrix W emb for both source text and output sentences. That is, we represent a source text { x i } Ni =1 in a distributional space as { d 0 i } Ni =1 by d 0 i = W emb x i where x i is a one-hot vector for i -th word in the source text. Likewise, output words { y t } Tt =1 is embedded as { o 0 t } Tt =1 , and d 0 i and o 0 t  R 300 . As shown in Figure 3(a), the multi-level memory network takes the source text embedding { d 0 i } Ni =1 as an input, and generates S number of memory tensors { M a/cs } Ss =1 as output, where superscript a and c denote input and output memory representation, respectively. The multi-level memory network is motivated by that when human understand a document, she does not remember it as a single whole document but ties together several levels of abstraction ( e.g . word-level, sentence-level, paragraph-level and document-level). That is, we generate S sets of memory tensors, each of which associates each cell with different number of neighboring word embeddings based on the level of abstraction. To build memory slots of such multi-level memory, we exploit a multi-layer CNN as the write network, where each layer is chosen based on the size of its receptive field. However, one issue of convolution is that large receptive fields require many layers or large filter sizes. For example, stacking 6 layers with a filter size of 3 results in a receptive field size of 13, i.e . each output depends on 13 input words. In order to grow the receptive field without increasing the computational cost, we exploit the dilated convolution (Yu and Koltun, 2016; Oord et al., 2016a) for the write network. Memory Writing with Dilated Convolution . In dilated convolution, the filter is applied over an area larger than its length by skipping input values with a certain gap. Formally, for a 1-D n -length input x  R n  300 and a filter w : { 1 , ..., k }  R 300 , the dilated convolution operation F on s elements of a sequence is defined as F ( x , s ) = k (cid:88) i =1 w ( i )  x s + d  ( i (cid:98) k/ 2 (cid:99) ) + b , (1) where d is the dilation rate, k is the filter size, s  d  ( i  (cid:98) k/ 2 (cid:99) ) accounts for the direction of dilation and w  R k  300  300 and b  R 300 are the parameters of the filter. With d = 1 , the dilated convolution reduces to a regular convolution. Using a larger dilation enables a single output at the top level to represent a wider range of input, thus effectively expanding the receptive field. To the embedding of a source text { d 0 i } Ni =1 , we recursively apply a series of dilated convolutions F ( d 0 )  RN  300 . We denote the output of the l -th convolution layer as { d li } Ni =1 . Normalized Gated Tanh Units . Each convolution is followed by our new activation of normalized gated tanh unit (NGTU), which is illustrated in Figure 4(b): GTU ( d l ) = tanh ( F lf ( d l ))   ( F lg ( d l )) , (2) d l +1 = LayerNorm ( d l + GTU ( d l )) , (3) where  is a sigmoid,  is the element-wise multiplication and F lf and F lg denote the filter and gate for l -th layer dilated convolution, respectively. The NGTU is an extension of the existing gated tanh units (GTU) (Oord et al., 2016a,b) by applying weight normalization (Salimans and Kingma, 2016) and layer normalization (Ba et al., 2016). This mixed normalization improves earlier work of Gehring et al. (2017), where only weight normalization is applied to the GLU. As in Figure 4(a), it tries to preserve the variance of activations throughout the whole network by scaling the output of residual blocks by  0 . 5 . However, we observe that this heuristic does not always preserve the variance and does not empirically work well in our dataset. Contrarily, the proposed NGTU not only guarantees preservation of activation variances but also significantly improves the performance. Multi-level Memory . Instead of using only the last layer output of CNNs, we exploit the outputs of multiple layers of CNNs to construct S sets of memories. For example, memory constructed from the 4-th layer, whose receptive field is 31, may have sentence-level embeddings, while memory from the 8-th layer, whose receptive field is 511, may have document-level embeddings. We obtain each s -th level memory M a/cs by resembling key-value memory networks (Miller et al., 2016): M as = d m ( s ) , M cs = d m ( s ) + d 0 . Recall that M as and M cs  RN  300 are input and output memory matrix, respectively. m ( s ) indicates an index of convolutional layer used for the s -th level memory. For example, if we set S = 3 and m = { 3 , 6 , 9 } , we make three-level memories, each of which uses the output of the 3-rd, 6-th, and 9-th convolution layer, respectively. To output memory representation M cs , we add the document embedding d 0 as a skip connection. We discuss how to predict the next word y t +1 at time step t based on the memory state and previously generated words y 1: t . Figure 3(b) visualizes the overall procedure of decoding. We first apply max-pooling to the output of the last layer of the encoder network to build a whole document embedding d whole  R 300 : d whole = maxpool ([ d L 1 ; ... ; d LN ]) . The decoder is designed based on WaveNet (Oord et al., 2016a) that uses a series of causal dilated convolutions, denoted by  F ( o l 1: t )  R t  300 . We globally condition d whole to obtain embeddings of previously generated words o l 1: t as: h lf/g =  F lf/g ( o l 1: t + W lf/g d whole ) , (6) h la = tanh ( h lf )   ( h lg ) , (7) o l +11: t = LayerNorm ( o l 1: t + h la ) , (8) where h lf/g are the filter and gate hidden state respectively, and learnable parameters are W lf and W lg  R 300  300 . We initialize o 0 t = W emb y t . We set the level of the decoder network to L = 3 for TIFU-short and L = 5 for TIFU-long. Next, we generate S number of query vectors { q st } Ss =1 at time t to our memory network as q st = tanh ( W sq o Lt + b sq ) , (9) where W sq  R 300  300 and b sq  R 300 . Next, we obtain the output word probability: s t = softmax ( W o [ M 1 o t ; ... ; M So t ; o Lt ]) , (11) where W o  R (300  ( S +1))  V . Each of these query vectors { q st } Ss =1 is fed into the attention function of each level of memory. As in (Vaswani et al., 2017), the attention function is M so t = softmax ( q st ( M as ) T  d emb ) M cs , (10) where we set d emb = 300 for the embedding dimension and M so t  R 300 . Finally, we select the word with the highest probability y t +1 = argmax s V ( s t ) . Unless y t +1 is an EOS token, we repeat generating the next word by feeding y t +1 into the output convolution layer of Eq.(8). We use the softmax cross-entropy loss from estimated y t to its target y GT,t . However, it forces the model to predict extremes (zero or one) to distinguish among the ground truth and alternatives. The label smoothing alleviates this issue by acting as a regularizer that makes the model less confident in its prediction. We smooth the target distribution with a uniform prior distribution u (Pereyra et al., 2017; Edunov et al., 2017; Vaswani et al., 2017). Thus, the loss over the training set D is L =  (cid:88) log p  ( y | x )  DKL ( u || p  ( y | x )) . We implement label smoothing by modifying the ground truth distribution for word y GT,t to be p ( y GT,t ) = 1  (cid:15) and p ( y (cid:48) ) = (cid:15)/ V for y (cid:48) (cid:54) = y GT,t where (cid:15) is a smoothing parameter set to 0.1. Further details can be found in the Appendix. Evaluation Metrics . We evaluate the summarization performance with two language metrics: perplexity and standard F1 ROUGE scores (Lin, 2004). We remind that lower perplexity and higher ROUGE scores indicate better performance. Datasets . In addition to Reddit TIFU, we also evaluate on two existing datasets: abstractive subset of Newsroom (Grusky et al., 2018) and XSum (Narayan et al., 2018a). These are suitable benchmarks for evaluation of our model in two aspects. First, they are specialized for abstractive summarization, which meets well the goal of this work. Second, they have larger vocabulary size (40K, 50K) than Reddit TIFU (15K), and thus we can evaluate the learning capability of our model. Baselines . We compare with three abstractive summarization methods, one basic seq2seq model, two heuristic extractive methods and variants of our model. We choose PG (See et al., 2017), SEASS (Zhou et al., 2017), DRGD (Li et al., 2017) as the state-of-the-art methods of abstractive summarization. We test the attention based seq2seq model denoted as s2s-att (Chopra et al., 2016). As heuristic extractive methods, the Lead-1 uses the first sentence in the text as summary, and the Ext-Oracle takes the sentence with the highest average score of F1 ROUGE-1/2/L with the gold summary in the text. Thus, Ext-Oracle can be viewed as an upper-bound for extractive methods. We also test variants of our method MMN-* . To validate the contribution of each component, we exclude one of key components from our model as follows: (i) -NoDilated with conventional convolutions instead, (ii) -NoMulti with no multi-level memory (iii) -NoNGTU with existing gated linear units (Gehring et al., 2017). That is, -NoDilated quantifies the improvement by the dilated convolution, -NoMulti assesses the effect of multi-level memory, and -NoNGTU validates the normalized gated tanh unit. Please refer to the Appendix for implementation details of our method. Table 3 compares the summarization performance of different methods on the TIFU-short/long dataset. Our model outperforms the state-of-the-art abstractive methods in both ROUGE and perplexity scores. PG utilizes a pointer network to copy words from the source text, but it may not be a good strategy in our dataset, which is more abstractive as discussed in Table 2. SEASS shows strong performance in DUC and Gigaword dataset, in which the source text is a single long sentence and the gold summary is its shorter version. Yet, it may not be sufficient to summarize much longer articles of our dataset, even with its second-level representation. DRGD is based on the variational autoencoder with latent variables to capture the structural patterns of gold summaries. This idea can be useful for the similarly structured formal documents but may not go well with di-TIFU-short TIFU-long vs. Baselines Win Lose Tie Win Lose Tie s2s-att 43.0 28.3 28.7 32.0 24.0 44.0 PG 38.7 28.0 33.3 42.3 33.3 24.3 SEASS 35.7 28.0 36.3 47.0 37.3 15.7 DRGD 46.7 17.3 15.0 61.0 23.0 16.0 Gold 27.0 58.0 15.0 22.3 73.7 4.0 Table 5: AMT results on the TIFU-short/long between our MMN and four baselines and gold summary. These state-of-the-art abstractive methods are not as good as our model, but still perform better than extractive methods. Although the Ext-Oracle heuristic is an upper-bound for extractive methods, it is not successful in our highly abstractive dataset; it is not effective to simply retrieve existing sentences from the source text. Moreover, the performance gaps between abstractive and extractive methods are much larger in our dataset than in other datasets (See et al., 2017; Paulus et al., 2018; Cohan et al., 2018), which means too that our dataset is highly abstractive. Table 4 compares the performance of our MMN on Newsroom-Abs and XSum dataset. We report the numbers from the original papers. Our model outperforms not only the RNN-based abstractive methods but also the convolutional-based methods in all ROUGE scores. Especially, even trained on single end-to-end training procedure, our model outperforms T-ConvS2S , which necessitates two training stages of LDA and ConvS2S . These results assure that even on formal documents with large vocabulary sizes, our multi-level memory is effective for abstractive datasets. We perform two types of qualitative evaluation to complement the limitation of automatic language metrics as summarization evaluation. User Preferences . We perform Amazon Mechanical Turk (AMT) tests to observe general users' preferences between the summarization of different algorithms. We randomly sample 100 test examples. At test, we show a source text and two summaries generated by our method and one baseline in a random order. We ask turkers to choose the more relevant one for the source text. We obtain answers from three different turkers for each test example. We compare with four abstractive baselines ( s2s-att , PG , SEASS and DRGD ) () I decided to go over to my friends house to a small party at 1 in the morning. I knew my parents would say no so I snuck out of the house. () I had been talking to my mom about how sad even hearing the theme song made me. Also she had seen me watching a bunch of sad anime theme songs and tearing up a little so she must have thought I was depressed. When I got home today my mom was practically in tears. () [Source Text] (GT) sneaking out of my friends house last night (Ours) sneaking out of my friends house (s2s-att) sneaking out of town (PG) not watching my friends (SEASS) accidentally spoiling my mom song (DRGD) watching a movie [Short Summary] () Saturday was on my way to a party and this dog was walking in the road. () Since it was a holiday I couldn't get her scanned for a chip but she was obviously neglected. Missing fur from flea infestation, () Yesterday I was able to go get her scanned for a chip. No chip. So I get ready to take her home and deflea her. () Anyway a third party today starts accusing me of stealing () and talking about pressing charges. () [Source Text] (GT) Saved a dog. Had to give dog back to possible abusers. Being accused of stealing the fucking dog. No good deed goes unpunished. (Ours) tried to help a dog got a bit and got accused of stealing (s2s-att) got accused of being a dog by stealing a _UNK bit the dog and accused of stealing dog to the police (SEASS) called a dog a _UNK might get charged with _UNK [Long Summary] (PG) _EOS (DRGD) i was a _UNK dog and I wasn't playing attention and got arrested for being a _UNK _UNK Figure 5: Examples of abstractive summary generated by our model and baselines. Table 5 summarizes the results of AMT tests, which validate that human annotators significantly prefer our results to those of baselines. As expected, the gold summary is voted the most. Summary Examples . Figure 5 shows selected examples of abstractive summarization. Baselines often generate the summary by mostly focusing on some keywords in the text, while our model produces the summary considering both keywords and the whole context thanks to multi-level memory. We present more examples in the Appendix. We introduced a new dataset Reddit TIFU for abstractive summarization on informal online text. We also proposed a novel summarization model named multi-level memory networks (MMN). Experiments showed that the Reddit TIFU dataset is uniquely abstractive and the MMN model is highly effective. There are several promising future directions. First, ROUGE metrics are limited to correctly capture paraphrased summaries, for which a new automatic metric of abstractive summarization may be required. Second, we can explore the data in other online forums such as Quora, Stackoverflow and other subreddits. We thank Chris Dongjoo Kim, Yunseok Jang and the anonymous reviewers for their helpful comments. This work was supported by Kakao and Kakao Brain corporations and IITP grant funded by the Korea government (MSIT) (No. 2017-0-01772, Development of QA systems for Video Story Understanding to pass the Video Turing Test). Gunhee Kim is the corresponding author.
abstain objective objective objective abstain objective result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method result abstain objective abstain objective abstain abstain method other abstain abstain abstain abstain abstain abstain objective abstain abstain method abstain objective method objective objective objective result method other other other other other objective method other other other other other other other objective other abstain other abstain other other other other other other other objective other other other other method method abstain abstain abstain abstain abstain abstain other method abstain abstain abstain method abstain method abstain abstain abstain method method result abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain other abstain abstain method abstain abstain objective other abstain abstain abstain abstain abstain result abstain abstain method abstain result abstain abstain method method method abstain abstain abstain abstain other method abstain method abstain abstain method abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain result method method abstain method abstain abstain abstain abstain abstain abstain result abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method method objective objective abstain abstain abstain objective other other other