Abstract Opinion summarization is the task of automatically creating summaries that reflect subjective information expressed in multiple documents, such as product reviews. While the majority of previous work has focused on the extractive setting, i.e., selecting fragments from input reviews to produce a summary, we let the model generate novel sentences and hence produce abstractive summaries. Recent progress in summarization has seen the development of supervised models which rely on large quantities of document-summary pairs. Since such training data is expensive to acquire, we instead consider the unsupervised setting, in other words, we do not use any summaries in training. We define a generative model for a review collection which capitalizes on the intuition that when generating a new review given a set of other reviews of a product, we should be able to control the amount of novelty going into the new review or, equivalently, vary the extent to which it deviates from the input. At test time, when generating summaries, we force the novelty to be minimal, and produce a text reflecting consensus opinions. We capture this intuition by defining a hierarchical variational autoencoder model. Both individual reviews and the products they correspond to are associated with stochastic latent codes, and the review generator (decoder) has direct access to the text of input reviews through the pointer-generator mechanism. Experiments on Amazon and Yelp datasets, show that setting at test time the review's latent code to its mean, allows the model to produce fluent and coherent summaries reflecting common opinions. Summarization of user opinions expressed in online resources, such as blogs, reviews, social media, or internet forums, has drawn much attention due to its potential for various information access applications, such as creating digests, search, and report generation (Hu and Liu, 2004; Angelidis and Lapata, 2018; Medhat et al., 2014). Although there has been significant progress recently in summarizing non-subjective context (Rush et al., 2015; Nallapati et al., 2016; Paulus et al., 2017; See et al., 2017; Liu et al., 2018), modern deep learning methods rely on large amounts of annotated data that are not readily available in the opinion-summarization domain and expensive to produce. Moreover, annotation efforts would have to be undertaken for multiple domains as online reviews are inherently multi-domain (Blitzer et al., 2007) and summarization systems highly domain-sensitive (Isonuma et al., 2017). Thus, perhaps unsurprisingly, there is a long history of applying unsupervised and weakly-supervised methods to opinion summarization (e.g., Mei et al. 2007; Titov and McDonald 2008; Angelidis and Lapata 2018), however, these approaches have primarily focused on extractive summarization, i.e., producing summaries by copying parts of the input reviews. In this work, we instead consider abstractive summarization which involves generating new phrases, possibly rephrasing or using words that were not in the original text. Abstractive summaries are often preferable to extractive ones as they can synthesize content across documents avoiding redundancy (Barzilay et al., 1999; Carenini and Cheung, 2008; Di Fabbrizio et al., 2014). In addition, we focus on the unsupervised setting and do not use any summaries for training. Unlike aspect-based summarization (Liu, 2012), which rewards the diversity of opinions, we aim to generate summaries that represent consensus (i.e., dominant opinons in reviews). We argue that such summaries can be useful for quick decision making, and to get an overall feel for a product or business (see the example in Table 1). More specifically, we assume we are provided with a large collection of reviews for various products and businesses and define a generative model of this collection. Intuitively, we want to design such a model that, when generating a review for a product 1 relying on a set of other reviews, we can control the amount of novelty going into the new review or, equivalently, vary the extent to which it deviates from the input. At test time, we can force the novelty to be minimal, and generate summaries representing consensus opinions. We capture this intuition by defining a hierarchical variational autoencoder (VAE) model. Both products and individual reviews are associated with latent representations. Product representations can store, for example, overall sentiment, common top-ics, and opinions expressed about the product. In contrast, latent representations of reviews depend on the product representations and capture the content of individual reviews. While at training time the latent representations are random variables, we fix them to their respective means at test time. As desired for summarization, these average' (or copycat') reviews differ in writing style from a typical review. For example, they do not contain irrelevant details that are common in customer reviews, such as mentioning the occasion or saying how many family members accompanied the reviewer. In order to encourage the summaries to include spe-1 For simplicity, we refer to both products (e.g., iPhone X) and businesses (e.g., a specific Starbucks branch) as products . cific details, the review generator (decoder') has direct access to the text of input reviews through the pointer-generator mechanism (See et al., 2017). In the example in Table 1, the model included specific information about the restaurant type and its location in the generated summary. As we will see in ablation experiments, without this conditioning, model performance drops substantially, as the summaries become more generic. We evaluate our approach on two datasets, Amazon product reviews and Yelp reviews of businesses. The only previous method dealing with unsupervised multi-document opinion summarization, as far as we are aware of, is MeanSum (Chu and Liu, 2019). Similarly to our work, they generate consensus summaries and consider the Yelp benchmark. Whereas we rely on continuous latent representations, they treat the summary itself as a discrete latent representation of a product. Although this captures the intuition that a summary should relay key information about a product, using discrete latent sequences makes optimization challenging; (Miao and Blunsom, 2016; Baziotis et al., 2019; Chu and Liu, 2019) all have to use an extra training loss term and biased gradient estimators. Our contributions can be summarized as follows:  we introduce a simple end-to-end approach to unsupervised abstractive summarization;  we demonstrate that the approach substantially outperforms the previous method, both when measured with automatic metrics and in human evaluation;  we provide a dataset of abstractive summaries for Amazon products. As discussed above, we approach the summarization task from a generative modeling perspective. We start with a high level description of our model, then, in Sections 2.2 and 2.3, we describe how we estimate the model and provide extra technical details. In Section 3, we explain how we use the model to generate summaries. Our text collection consists of groups of reviews, with each group corresponding to a single product. Conditional independence of the reviews given the group representation c . (b) The r i 's decoder accesses other reviews of the group ( r 1 , ..., r i  1 , r i +1 , ..., r N ). Our latent summarization model (which we call COPYCAT ) captures this hierarchical organization and can be regarded as an extension of the vanilla text-VAE model (Bowman et al., 2016). COPYCAT uses two sets of latent variables as shown in Figure 1a. Namely, we associate each review group (equivalently, each product) with a continuous variable c , which captures the group's latent seman-tics'. In addition, we associate each individual review ( r i ) with a continuous variable z i , encoding the semantics of that review. The information stored in z i is used by the decoder p  ( r i | z i ) to produce review text r i . The marginal log-likelihood of one group of reviews r 1: N = ( r 1 , . . . , r N ) is given by log p  ( r 1: N ) = log (cid:90) (cid:34) p  ( c ) N (cid:89) i =1 (cid:20)(cid:90) p  ( r i | z i ) p  ( z i | c )d z i (cid:21) d c (cid:35) , where we marginalize over variables c and z 1: N . When generating a new review r i , given the set of previous reviews r 1: i , the information about these reviews has to be conveyed through the latent representations c and z i . This bottleneck is undesirable, as it will make it hard for the model to pass fine-grain information. For example, at generation time, the model should be reusing named entities (e.g., product names or technical characteristics) from other reviews rather than hallucinating' or avoiding generating them at all, resulting in generic and non-informative text. We alleviate this issue by letting the decoder directly access other reviews. We can formulate this as an autoregressive model: p  ( r 1: N | c ) = N (cid:89) i =1 p  ( r i | r 1 , ..., r i  1 , c ) . As we discuss in Section 2.3, the conditioning is instantiated using the pointer-generator mechanism (See et al., 2017) and, thus, will specifically help in generating rare words (e.g., named entities). We want our summarizer to equally rely on every review, without imposing any order (e.g., temporal) on the generation process. Instead, as shown in Figure 1b, when generating r i , we let the decoder access all other reviews within a group, r  i = ( r 1 , . . . , r i  1 , r i +1 , . . . , r N ) . This is closely related to pseudolikelihood estimation (Besag, 1975) or Skip-Thought's objective (Kiros et al., 2015). The final objective that we maximize for each group of reviews r 1: N : log (cid:90) p  ( c ) N (cid:89) i =1 (cid:20)(cid:90) p  ( r i | z i , r i ) p  ( z i | c )d z i (cid:21) d c (2) We will confirm in ablation experiments that both hierarchical modeling (i.e., using c ) and the direct conditioning on other reviews are beneficial. As standard with VAEs and variational inference in general (Kingma and Welling, 2013), instead of directly maximizing the intractable marginal likelihood in Equation 2, we maximize its lower bound: 3 L ( ,  ; r 1: N ) = E c  q  ( c | r 1: N ) (cid:34) N (cid:88) i =1 E z i  q  ( z i | r i ,c ) [log p  ( r i | z i , r i )]  N (cid:88) i =1 DKL [ q  ( z i | r i , c ) || p  ( z i | c )] (cid:35)  DKL [ q  ( c | r 1: N ) || p  ( c )] . 3 See the derivations in Appendix A.1. The lower bound includes two inference networks', q  ( c | r 1: N ) and q  ( z i | r i , c ) , which are neural networks parameterized with  and will be discussed in detail in Section 2.3. They approximate the corresponding posterior distributions of the model. The first term is the reconstruction error: it encourages the quality reconstruction of the reviews. The other two terms are regularizers. They control the amount of information encoded in the latent representation by penalizing the deviation of the estimated posteriors from the corresponding priors, the deviation is measured in terms of the Kullback-Leibler (KL) divergence. The bound is maximized with respect to both the generative model's parameters  and inference networks' parameters  . Due to Gaussian assumptions, the Kullback-Leibler (KL) divergence terms are available in closed form, while we rely on the reparam-eterization trick (Kingma and Welling, 2013) to compute gradients of the reconstruction term. The inference network predicting the posterior for a review-specific variable q  ( z i | r i , c ) is needed only in training and is discarded afterwards. In contrast, we will exploit the inference network q  ( c | r 1: N ) when generating summaries, as discussed in Section 3. A GRU encoder (Cho et al., 2014) embeds review words w to obtain hidden states h . Those representations are reused across the system, e.g., in the inference networks and the decoder. The full architecture used to produce the latent codes c and z i is shown in Figure 2. We make Gaussian assumptions for all distributions (i.e. posteriors and priors). As in Kingma and Welling (2013), we use separate linear projections (LPs) to compute the means and diagonal log-covariances. 2.3.2 Prior p ( c ) and posterior q  ( c | r 1: N ) We set the prior over group latent codes to the standard normal distribution, p ( c ) = N ( c ; 0 , I ) . In order to compute the approximate posterior q  ( c | r 1: N ) , we first predict the contribution (im-portance') of each word in each review  ti to the code of the group:  ti = exp( f  ( m ti )) (cid:80) Nj =1 (cid:80) T j k exp( f  ( m kj )) , where T i is the length of r i and f  is a feed-forward neural network (FFNN) 4 which takes as input concatenated word embeddings and hidden states of the GRU encoder, m ti = [ h ti  w ti ] , and returns a scalar. Next, we compute the intermediate representation with the weighted sum:  h = (cid:80) Ni =1 (cid:80) T i t  ti m ti . Finally, we compute the Gaussian's parameters using the affine projections:   ( r 1: N ) = L  h + b L log   ( r 1: N ) = G  h + b G 2.3.3 Prior p  ( z i | c ) and posterior q  ( z i | r i ,c ) To compute the prior on the review code z i , p  ( z i | c ) = N ( z i ;   ( c ) , I  ( c )) , we linearly project the product code c . Similarly, to compute the parameters of the approximate posterior q  ( z | r i , c ) = N ( z ;   ( r i , c ) , I  ( r i , c )) , we concatenate the last encoder's state h T i i of the review r i and c , and perform affine transformations. | To compute the distribution p  ( r i | z i , r i ) , we use an auto-regressive GRU decoder with the attention mechanism (Bahdanau et al., 2015) and a pointer-generator network. We compute the context vector c ti = att ( s ti , h i ) by attending to all the encoder's hidden states h i of the other reviews r i of the group, where the decoder's hidden state s ti is used as a query. The 4 We use FFNNs with the tanh non-linearity in several model components. hidden state of the decoder is computed using the GRU cell as s ti = GRU  ( s t  1 i , [ w ti  c t  1 i  z i ]) . The pointer-generator network computes two internal word distributions that are hierarchically aggregated into one distribution (Morin and Bengio, 2005). One distribution assigns probabilities to words being generated using a fixed vocabulary, and another one probabilities to be copied directly from the other reviews r i . In our case, the network helps to preserve details and, especially, to generate rare tokens. Given reviews r 1: N , we generate a summary that reflects common information using trained components of the model. Formally, we could sample a new review from p  ( r | r 1: N ) = E c  q  ( c | r 1: N ) (cid:20) E z  p  ( z | c ) [ p  ( r | z, r 1: N )] (cid:21) . As we argued in the introduction and will revisit in experiments, a summary or summarizing review, should be generated relying on the mean of the reviews' latent code. Consequently, instead of sampling z from p  ( z | c ) = N ( z ;   ( c ) , I  ( c )) , we set it to   ( c ) . We also found beneficial, in terms of evaluation metrics, not to sample c but instead to rely on the mean predicted by the inference network q  ( c | r 1: N ) . Our experiments were conducted on business customer reviews from the Yelp Dataset Challenge and Amazon product reviews (He and McAuley, 2016). These were pre-processed similarly to Chu and Liu (2019), and the corresponding data statistics are Dataset Training Validation Yelp 38,776/1,012,280 4,311/113,373 Amazon 183,103/4,566,519 9,639/240,819 Table 2: Data statistics after pre-processing. shown in Table 2. Details of the pre-processing are available in Appendix A.2. These datasets present different challenges to abstractive summarization systems. Yelp reviews contain much personal information and irrelevant details which one may find unnecessary in a summary. Our summarizer, therefore, needs to distill important information in reviews while abstracting away from details such as a listing of all items on the menu, or mentions of specific dates or occasions upon which customers visited a restaurant. On the contrary, in Amazon reviews, we observed that users tend to provide more objective information and specific details that are useful for decision making (e.g., the version of an electronic product, its battery life, its dimensions). In this case, it would be desirable for our summarizer to preserve this information in the output summary. For evaluation, we used the same 100 human-created Yelp summaries released by Chu and Liu (2019). These were generated by Amazon Mechanical Turk (AMT) workers, who summarized 8 input reviews. We created a new test for Amazon reviews following a similar procedure (see Appendix A.6 for details). We sampled 60 products and 8 reviews for each product, and they were shown to AMT workers who were asked to write a summary. We collected three summaries per product, 28 products were used for development and 32 for testing. We used GRUs (Cho et al., 2014) for sequential encoding and decoding we used GRUs. We randomly initialized word embeddings that were shared across the model as a form of regularization (Press and Wolf, 2017). Further, optimization was performed using Adam (Kingma and Ba, 2014). In order to overcome the posterior collapse (Bow-man et al., 2016), both for our model and the vanilla VAE baseline, we applied cyclical annealing (Fu et al., 2019). The reported ROUGE scores are based on F1 (see Appendix A.3 for details on hy-perparameters). Opinosis is a graph-based abstractive summarizer (Ganesan et al., 2010) designed to generate short opinions based on highly redundant texts. Although it is referred to as abstractive, it can only select words from the reviews. LexRank is an unsupervised algorithm which selects sentences to appear in the summary based on graph centrality (sentences represent nodes in a graph whose edges have weights denoting similarity computed with tf-idf). A node's centrality can be measured by running a ranking algorithm such as PageRank (Page et al., 1999). MeanSum 5 is the unsupervised abstractive summarization model (Chu and Liu, 2019) discussed in the introduction. We also trained a vanilla text VAE model (Bow-man et al., 2016) with our GRU encoder and decoder. When generating a summary for r 1 , ..., r N , we averaged the means of q  ( z i | r i ) . Finally, we used a number of simple summarization baselines. We computed the clustroid review for each group as follows. We took each review from a group and computed ROUGE-L with respect to all other reviews. The review with the highest ROUGE score was selected as the clustroid review. Furthermore, we sampled a random review from each group as the summary, and constructed the summary by selecting the leading sentences from each review of a group. Additionally, as an upper bound, we report the performance of an oracle review, i.e., the highest-scoring review in a group when computing ROUGE-L against reference summaries. 5 For experiments on Yelp, we used the checkpoint provided by the authors, as we obtained very similar ROUGE scores when retraining the model. As can be seen in Tables 3 and 4, our model, Copycat, yields the highest scores on both Yelp and Amazon datasets. We observe large gains over the vanila VAE. We conjecture that the vanilla VAE struggles to properly represent the variety of categories under a single prior p ( z ) . For example, reviews about a sweater can result in a summary about socks (see example summmaries in Appendix). This contrasts with our model which allows each group to have its own prior p  ( z | c ) and access to other reviews during decoding. The gains are especially large on the Amazon dataset, which is very broad in terms of product categories. Our model also substantially outperforms MeanSum. As we will confirm in human evaluation, MeanSum's summaries are relatively fluent at the sentence level but often contain hallucinations, i.e., information not present in the input reviews. Best-Worst Scaling We performed human evaluation using the AMT platform. We sampled 50 businesses from the human-annotated Yelp test set and used all 32 test products from the Amazon set. We recruited 3 workers to evaluate each tuple containing summaries from MeanSum, our model, LexRank, and human annotators. The reviews and summaries were presented to the workers in random order and were judged using Best-Worst Scaling (Louviere and Woodworth, 1991; Louviere et al., 2015). BWS has been shown to produce more reliable results than ranking scales (Kiritchenko and Mohammad, 2016). Crowdwork-ers were asked to judge summaries according to the Fluency Coherence Non Red. criteria listed below (we show an abridged version below, the full set of instructions is given in Appendix A.5). The non-redundancy and coherence criteria were taken from Dang (2005). Fluency : the summary sentences should be grammatically correct, easy to read and understand; Coherence : the summary should be well structured and well organized; Non-redundancy : there should be no unnecessary repetition in the summary; Opinion consensus : the summary should reflect common opinions expressed in the reviews; Overall : based on your own criteria (judgment) please select the best and the worst summary of the reviews. For every criterion, a system's score is computed as the percentage of times it was selected as best minus the percentage of times it was selected as worst (Orme, 2009). The scores range from -1 (unanimously worst) to +1 (unanimously best). On Yelp, as shown in Table 5, our model scores higher than the other models according to most criteria, including overall quality. The differences with other systems are statistically significant for all the criteria at p < 0 . 01 , using post-hoc HD Tukey tests. The difference in fluency between our system and gold summaries is not statistically significant. The results on Amazon are shown in Table 6. Our system outperforms other methods in terms of fluency, coherence, and non-redundancy. As with Yelp, it trails LexRank according to the opinion consensus criterion. Additionally, LexRank is slightly preferable overall. All pairwise differences between our model and comparison systems are statistically significant at p < 0 . 05 . Opinion consensus (OC) is a criterion that captures the coverage of common opinions, and it seems to play a different role in the two datasets. On Yelp, LexRank has better coverage compared to our model, as indicated by the higher OC score, but is not preferred overall . In contrast, on Amazon, while the OC score is on the same par, LexRank is preferred overall . We suspect that presenting a breadth of exact details on Amazon is more important than on Yelp. Moreover, LexRank tends to produce summaries that are about 20 tokens longer than ours resulting in better coverage of input details. Content Support The ROUGE metric relies on unweighted n-gram overlap and can be insensitive to hallucinating facts and entities (Falke et al., 2019). For example, referring to a burger joint as a veggie restaurant is highly problematic from a user perspective but yields only marginal differences in ROUGE. To investigate how well the content of the summaries is supported by the input reviews, we performed a second study. We used the same sets as in the human evaluation in Section 5.2, and split MeanSum and our system's summaries into sentences. Then, for each summary sentence, we assigned 3 AMT workers to assess how well the sentence is supported by the reviews. Workers were advised to read the reviews and rate sentences using one of the following three options. Full support : all the content is reflected in the reviews; Partial support : only some content is reflected in the reviews; No support : content is not reflected in the reviews. The results in Table 7 indicate that our model is better at preserving information than MeanSum. Ablations To investigate the importance of the model's individual components, we performed ablations by removing the latent variables ( z i and c , one at a time), and attention over the other reviews. The models were re-trained on the Amazon dataset. The results are shown in Table 8. They indicate that all components play a role, yet the most significant drop in ROUGE was achieved when the variable z was removed, and only c remained. Summaries obtained from the latter system were wordier and looked more similar to reviews. Dropping the attention (w/o r i ) results in more generic summaries as the model cannot copy details from the input. Finally, the smallest quality drop in terms of ROUGE-L was observed when the variable c was removed. In the introduction, we hypothesized that using the mean of latent variables would result in more grounded summaries reflecting the content of the input reviews, whereas sampling would yield texts with many novel and potentially irrelevant details. To empirically test this hypothesis, we sampled the latent variables during summary generation, as opposed to using mean values (see Section 3). We indeed observed that the summaries were wordier, less fluent, and less aligned to the input reviews, as is also reflected in the ROUGE scores (Table 8). Copy Mechanism Finally, we analyzed which words are copied by the full model during summary generation. Generally, the model copies around 3-4 tokens per summary. We observed a tendency to copy product-type specific words (e.g., shoes ) as well as brands and names. Extractive weakly-supervised opinion summarization has been an active area of research. A recent example is Angelidis and Lapata (2018). First, they learn to assign sentiment polarity to review segments in a weakly-supervised fashion. Then, they induce aspect labels for segments relying on R1 R2 RL w/o r i 0.2866 0.0454 0.1863 w/o c 0.2767 0.0507 0.1919 w/o z 0.2926 0.0416 0.1739 Sampling 0.2563 0.0434 0.1716 Full 0.3197 0.0581 0.2016 Table 8: Ablations, ROUGE scores on Amazon. a small sample of gold summaries. Finally, they use a heuristic to construct a summary of segments. Opinosis (Ganesan et al., 2010) does not use any supervision. The model relies on redundancies in opinionated text and PoS tags in order to generate short opinions. This approach is not well suited for the generation of coherent long summaries and although it can recombine fragments of input text, it cannot generate novel words and phrases. LexRank (Erkan and Radev, 2004) is an unsupervised extractive approach which builds a graph in order to determine the importance of sentences, and then selects the most representative ones as a summary. Isonuma et al. (2019) introduce an unsupervised approach for single review summarization, where they rely on latent discourse trees. Other earlier approaches (Gerani et al., 2014; Di Fabbrizio et al., 2014) relied on text planners and templates, while our approach does not require rules and can produce fluent and varied text. Finally, conceptually related methods were applied to unsupervised single sentence compression (West et al., 2019; Baziotis et al., 2019; Miao and Blunsom, 2016). The most related approach to ours is MeanSum (Chu and Liu, 2019) which treats a summary as a discrete latent state of an autoencoder. In contrast, we define a hierarchical model of a review collection and use continuous latent codes. In this work, we presented an abstractive summarizer of opinions, which does not use any summaries in training and is trained end-to-end on a large collection of reviews. The model compares favorably to the competitors, especially to the only other unsupervised abstractive multi-review summarization system. Furthermore, human evaluation of the generated summaries (by considering their alignment with the reviews) shows that those created by our model better reflect the content of the input. We would like to thank the anonymous reviewers for their valuable comments. Also, Stefanos Angelidis for help with the data as well as Jonathan Mallinson, Serhii Havrylov, and other members of Edinburgh NLP group for discussion. We gratefully acknowledge the support of the European Research Council (Titov: ERC StG BroadSem 678254; Lapata: ERC CoG TransModal 681760) and the Dutch National Science Foundation (NWO VIDI 639.022.518).
abstain objective abstain method objective method method abstain abstain abstain abstain abstain abstain abstain objective abstain method abstain result method objective method method abstain abstain abstain abstain abstain abstain method abstain abstain result method abstain method abstain abstain objective abstain method method abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain method abstain abstain abstain abstain other abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain other method method method method method method abstain abstain abstain method method abstain abstain method abstain method abstain abstain abstain abstain abstain method method method method abstain method method method method method abstain method abstain abstain abstain abstain abstain abstain method method method method method abstain method method method result result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain result abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other other other other other other other other other other other method other other objective method abstain result other other other