Abstract In our everyday chit-chat, there is a conversation initiator, who proactively casts an initial utterance to start chatting. However, most existing conversation systems cannot play this role. Previous studies on conversation systems assume that the user always initiates conversation, and have placed emphasis on how to respond to the given user's utterance. As a result, existing conversation systems become passive. Namely they continue waiting until being spoken to by the users. In this paper, we consider the system as a conversation initiator and propose a novel task of generating the initial utterance in open-domain non-task-oriented conversation. Here, in order not to make users bored, it is necessary to generate diverse utterances to initiate conversation without relying on boilerplate utterances like greetings. To this end, we propose to generate initial utterance by summarizing and chatting about news articles, which provide fresh and various contents everyday. To address the lack of the training data for this task, we constructed a novel large-scale dataset through crowd-sourcing. We also analyzed the dataset in detail to examine how humans initiate conversations (the dataset will be released to facilitate future research activ-ities). We present several approaches to conversation initiation including information retrieval based and generation based models. Experimental results showed that the proposed models trained on our dataset performed reasonably well and outperformed baselines that utilize automatically collected training data in both automatic and manual evaluation. Conversation 1 systems are becoming increasingly important as a means to facilitate human-computer (cid:3) Japan Corporation. 1 Conversation in this paper refers to open-domain non-task-oriented conversations and chit-chat. communication. However, most of the studies on conversation systems have been based on the as-sumption that a human always initiates conversation. As a result, the systems are designed to be passive ( Yan, 2018), meaning that they keep waiting until they are spoken to by the human and will never speak to the human proactively. For example, popular encoder-decoder models ( Sutskever et al., 2014; Vinyals and Le, 2015) are designed to respond to input utterances provided by humans, and it is difficult for them to proactively initiate the conversation. Although some systems are able to initiate conversations, they basically adopt template-based generation methods and thus lack diversity. This paper investigates generating the very first utterance in a conversation. We feel strongly that conversation systems should not always be passive; sometimes, they have to proactively initiate the conversation to enable more natural conversation. In addition, it is crucial to be able to initiate conversation in various ways in actual applications, since systems that initiate a conversation by always saying  Let's talk about something  or  Hello  are inherently boring. We propose a task setting in which the system initiates a conversation by talking about a news topic. In this task, the system is provided with 3989 a news post to talk about and uses it to generate the initial utterance of the conversation (Fig. 1). This task is referred to as conversation initiation in this paper. We have two primary reasons for using news posts. First, sharing and exchanging opinions about the latest news with friends is common in our daily conversations (Purcell et al., 2010) ( e.g, asking something like What do you think about today's news on Trump? ). Second, and more importantly, this task setting allows us to proactively generate diverse utterances to initiate conversations by simply using the latest news posts, which include a wide variety of content published daily. We created a large-scale dataset for training and evaluating conversation initiation models through a crowd-sourcing service. The crowd-sourcing workers were presented with news posts collected from Twitter and asked to create utterances to initiate a conversation about the post. The resulting dataset will be released to facilitate future studies at the time of publication. We developed several neural models, including retrieval-based and generation-based ones, to empirically compare their performances. We also compared the proposed models against baselines that utilize automatically constructed training dataset to investigate the effectiveness of our dataset. Both automatic and manual evaluation were used to assess not only the quality but also the diversity of the generated initial utterances. The results indicate that the proposed models successfully generated initial utterances for the given news posts, and significantly outperformed the baseline models. Our contributions are the following: (cid:15) We investigate the task of conversation initiation, which has been largely overlooked in previous studies. (cid:15) We construct and release a large-scale dataset for conversation initiation. (cid:15) We develop several neural models and empirically compare their effectiveness on our dataset. There are many existing studies on non-task-oriented conversation systems. Research started with rule-based methods (Weizenbaum, 1966; Wallace, 2009) and gradually shifted to statistical approaches (Ritter et al., 2011; Vinyals and Le, 2015), and many follow-up studies have since been undertaken to improve the quality of the generated responses (Hasegawa et al., 2013; Sordoni et al., 2015; Serban et al., 2016; Li et al., 2016b; Serban et al., 2017). However, the task of conversation initiation has been largely absent in these studies. There have also been efforts to develop systems that can chat with users about specific documents such as Wikipedia articles (Zhou et al., 2018) or reviews (Moghe et al., 2018). However, these studies did not investigate how to initiate such conversations, and as a result, their models assume that the initial utterance is always given by users. Also, their datasets are designed to be used to train models of multi-turn conversations about the given documents, rather than models of conversation initiation. For example, Moghe et al. ( 2018) utilized fixed templates to initiate conversations, and there are only a few (around 4k) utterances that can be used to train the model of conversation initiation in Zhou's dataset ( 2018). In contrast, we focus on the conversation initiation task, which those studies have largely overlooked, and develop a large-scale dataset that includes 109,460 utterances for this task (see Section 3). Therefore, our work can be considered complementary to the previous studies. In an approach that uses images rather than documents, ( Mostafazadeh et al., 2016) proposed a method of generating questions about an image to initiate conversation. Although, like us, they explored initiating conversation, they focused only on generating questions. In contrast, we investigate generating other types of initial utterances than questions. Also, they investigated a task setting in which users can see the images along with the conversation, while we do not present the news posts to users. This difference makes our generation task a bit more complicated (see Section 3). Some studies have attempted to make conversation systems more proactive rather than passively waiting for utterances from a user. (Li et al., 2016c) proposed a system that detects a stalemate in the conversation and then proactively 3990 casts a specific response for breaking the stalemate. They use the history of the user's utterances to select response candidates. (Yan et al., 2017; Yan and Zhao, 2018) proposed a method of proactively suggesting the user's next utterance. Although these methods have been successfully used in proactive conversation systems, the conversation initiation has not been investigated. A well-known problem of encoder-decoder-based conversational models is that they tend to generate generic responses such as  I don't know  (Vinyals and Le, 2015; Sordoni et al., 2015; Serban et al., 2016). Such responses understandably bore users, so there has been much research focus on generating more diverse responses (Li et al., 2016a; Xu et al., 2018; Baheti et al., 2018). We explore the problem of generating diverse initial utterances from a different perspective than other studies. In our problem setting, it is not obvious how to go beyond simple template-based systems, which cannot generate diverse utterances. We address this problem by generating initial utterances based on news posts, which feature various content and are updated every day. This study is complementary to previous attempts at diversification. Our method exploits existing neural conversation models, which tend to generate generic responses, as a component. The previous diversification methods can be used to improve the initial utterances in our method. Question Answering (QA) tasks have long been studied in the research community (Rajpurkar et al., 2016, 2018). In recent years, conversational variants of this task such as visual QA (Antol et al., 2015; Das et al., 2017) and conversational QA (Reddy et al., 2018) have been proposed. All of these tasks differ from our conversation initiation task since they focus on how to respond to questions. (Yoshino and Kawahara, 2014) proposed an information navigation system that presents users with the contents of news articles through conversation. Although this setting is similar to ours, their system always opens conversation by just presenting the news headline. Our study investigates initiating conversation in a more chatty way, and should contribute to making the systems more conversational and attractive. (Qin et al., 2018) proposed the task of generating comments about given news articles. Although this task is similar to ours, it is not designed to converse with users. Our task focuses on conversation and tries to generate initial utterances using news articles (posts). In this section, we explain how we constructed the dataset for the task of conversation initiation. We then analyze the constructed dataset to provide insights into its effectiveness. We first collected 104,960 Japanese news posts from the Twitter account @YahooNewsTopics, 2 which delivers the latest news in the world every day. The data were collected between December 31, 2013 and October 31, 2017. Some example posts collected from this account are listed in the third column of Table 1. 3 We investigate the task setting in which the system opens a conversation about a given news post. Here, we presume the post is not presented to the user during the conversation. Although letting users see the news posts would be possible, such a setting is not investigated here because our focus is a situation where users converse with the system only by voice. Such situations are growing more popular in recent years with the rise of voice-controlled conversation systems such as intelligent assistants ( e.g., Siri, Alexa, and Cortana) ( Jiang et al., 2015; Sano et al., 2016; Akasaki and Kaji, 2017) and smart speakers ( e.g., Amazon Echo and Google Home). Therefore, in our task setting, since the user does not always know about the news, it is preferred to first introduce the news summary so as to share the background knowledge before starting the conversation (see Fig. 1). In this sense, our task can be understood as a combination of summarization and chit-chat. Interestingly, the summarization subtask goes beyond the ordinary one in that we not only compress the content but also generate the text in a chit-chat-like style. To construct the dataset, we had cloud workers create the initial utterance of a conversation on the basis of a given news post. We instructed workers to not only chat about the news post but also to provide its brief summary. The workers were asked to use colloquial expressions because users feel strange when spoken to in literary expressions. We obtained a total of 104,960 pairs of news post and initial utterance 4 . Note that we created only the initial utterances (same as (Mostafazadeh et al., 2016)) because our focus is how to initiate conversation 5 . 4 Some news posts (typically emergency news such as earthquake) were posted more than once, and as the consequence the dataset includes 102,844 unique news posts. In the experiment, we took care so that the training and test datasets do not include the same news posts. 5 Of course it is necessary to continue successive conversation in an actual application, but we here leave this setting as a future work. Here we discuss our investigation of the 104,960 initial utterances. Some examples of the utterances are listed in Table 1. Most initial utterances first summarize the contents of the news post and then begin to chat about it, as we instructed. For subsequent analysis and model designing, we divided each initial utterance into sentences and then designated the one with the smallest edit distance from the input news post as summary part and the rest as chit-chat parts. The rationale behind the use of this heuristic is that the summary part shares more words with the original news post than the chit-chat part and consists of just one sentence in most cases. The statistics of the dataset are shown in Table 2. For the summary part, as seen in Tables 1 and 2, original news posts are compressed by 32.29% on average and are converted into a colloquial style. This indicates that the recruited cloud workers properly extracted the important contents from the input news posts and used them for making the summary part. Compared with the summary part, the number of words and vocabulary size for the chit-chat part are relatively small (Table 2). This is a natural phenomenon since the summary part uses more con-3992 Figure 2: Overview of initial utterance generation by our proposed approaches. tent words for summarization than the chit-chat part. To clarify how workers created these chitchats, we randomly sampled 10,000 utterances and manually classified them according to their dialogue acts, as shown in Table 1. We found that the majority (92% = (7929 + 273 + 1082) / 10000) are classified into three dialogue acts (IMPRESSION, URGING, and QUESTION). The remaining 8% miscellaneous utterances that do not belong to any of the three dialog acts. Most of the labeled initial utterances are the impressions and opinions of cloud workers about news posts (see the IMPRESSION act). Some of them are boilerplates ( e.g., Congrats ) while others show tremendous diversity ( e.g., It makes me want to drink a cold beer on a hot day ). It is interesting that some workers make an urging ( e.g., Let's evacuate quickly ) or ask a question ( e.g., Have you ever seen a handball game? ). These acts that attempt to solicit the user's response are important elements for conversation initiation. As described in Section 2, most of the initial utterances in the dataset can be divided into a summary part and a chit-chat part. Because it is possible to generate these two parts by two separate models or by a single joint model, we investigate both approaches and compare their performance in an experiment. The overview of our proposed approaches is given in Fig. The separate approach utilizes two different models to generate the summary part and the chit-chat part, respectively. The summary part is generated by the pointer-generator model, which allows both copying words by pointing to the input sentence and generating words from a fixed vocabulary (See et al., 2017). This model is suitable for generating the summary part because it can appropriately select the contents of the input sentence while compressing them to a proper length. To generate the chit-chat part, both generation-based and information retrieval (IR)-based methods are investigated. We use a common encoder-decoder model (Vinyals and Le, 2015) as the generation-based method (see Separate (Gen) in Fig. 2). Since this model tends to generate generic sentences that lack diversity (Vinyals and Le, 2015; Sordoni et al., 2015; Serban et al., 2016), we also adopt the MMI-antiLM method proposed by (Li et al., 2016a) to promote diversity. This method uses the following score function, instead of the commonly used log-likelihood, when decoding: log P ( T j S ) (cid:0) (cid:21) log U ( T ) ; (1) where T is an initial utterance and S is a news post. P ( T j S ) is the conditional likelihood of T 3993 given S , and U is a language model. In decoding, output candidates are generated using beam search and are then reranked by Eq. 1. This model penalizes generic sentences by U ( T ) . As the IR-based method, we utilize the embedding of an input news post to retrieve the closest news posts in the training data using cosine distance, and then extract the corresponding chitchat part (Ritter et al., 2011) (see Separate (IR) in Fig. 2). We adopt Smooth Inverse Frequency (SIF)-based embedding (Arora et al., 2017) for inducing news post embeddings. This method first calculates a weighted average of word embeddings in a news post s as: v s = 1 j s j  w 2 s a a + P ( w ) v w ; (2) where a is a hyperparameter and P ( w ) is the uni-gram probability calculated from the training data. Then, it reduces the influence of the first principal component by using the first singular vector u of the word vector matrix: v s = v s (cid:0) uu T v s ; (3) This method has demonstrated a competitive performance across various tasks ( Arora et al., 2017). We concatenate the summary part and the chitchat part of the training data and train only one pointer-generator model, as mentioned in Section 4.1 (see Joint in Fig. 2). Unlike the separate approach, this method can be considered multi-task learning of the summary and the chit-chat part generation. Thus, we expect it can generate the initial utterance precisely by considering the coherence between the summary and the chit-chat part. We examine the effectiveness of this approach through experiments in the following section. We empirically evaluate the performance of the proposed methods on the constructed dataset. In addition to the proposed methods, we implemented baselines that do not use labor-intensive labeled data, since carefully preparing the dataset is one of our contributions. These baselines generate summary and chit-chat parts separately in the following way and concatenate them as output. We gathered tweets (news posts) of major news accounts from Twitter and their corresponding replies (regarded as chit-chats). Those tweet-reply pairs can be used as pseudo training data to generate the chit-chat part. Since we cannot automatically acquire training data for generating the summary part, we output the first sentence of the input news post as the summary part. Overall, the following proposed and baseline methods were implemented for comparison: Baseline Generate the summary part and the chit-chat part by separate models using the pseudo-training data collected from Twitter. There are three variants of this method for generating the chit-chat part. Baseline (IR) and Baseline (Gen) use the IR-based method and the generation-based method, respectively. Baseline (Gen+MMI) uses MMI-antiLM (Li et al., 2016a) for decoding. Separate Generate the summary part and the chit-chat part separately using the approach described in Section 4.1 and the dataset described in Section 3.1. There are also three variants of this method, same as the baselines ( Separate (IR) , Separate (Gen), and Separate (Gen+MMI), respectively). Joint Generate the summary part and the chitchat part jointly using the approach described 3994 Model R-1 R-2 R-L D-1 D-2 D-S Baseline 70.2 59.1 67.5 17.7 60.1 99.8 Separate 66 : 5 50 : 6 63 : 8 15 : 7 52 : 4 99.8 Joint 68 : 8 54 : 1 66 : 3 15 : 2 51 : 8 99.8 Table 4: Results of summary part generation. We divided the 104,960 items of data (news post and initial utterance pairs) into 90,000, 10,000, and 4,960 for training data, development data, and test data, respectively. Input news posts that appear in the training data were removed from the test data. Consequently, 4,776 data were used as the final test data. To train the baseline model, we collected 277,813 tweets and their corresponding replies from six major Japanese news accounts 6 on Twitter. We then divided those pairs into 260,000 and 17,813 for training data and development data for the baselines. We performed tokenization using a Japanese morphological analyzer, MeCab, 7 with IPAdic dictionary, 8 and then removed usernames, URLs, and hashtags. We used OpenNMT-py (Klein et al., 2017) 9 for building the models described in Section 4. Their hyperparameter settings are given in Table 3. We used GloVe (Pennington et al., 2014) 10 to learn 300-dimensional word embeddings. We trained word embedding using a Japanese Wikipedia dump released on February 22nd, 2018. These embeddings were used for acquiring news post embeddings, as described in Section 4.1. As discussed in Section 3.2, since the initial utterance can be divided into separate parts that have different properties, we evaluated each part separately to examine the generated initial utterances. 6 @YahooNewsTopics, @livedoornews, @asahi, @mainichi, @mainichi jp, @nhk news 7 http://taku910.github.io/mecab/ 8 https://ja.osdn.net/projects/ipadic/ 9 https://github.com/OpenNMT/OpenNMT-py 10 https://nlp.stanford.edu/projects/glove/ We automatically divided the generated sentences and reference sentences into summary parts and chit-chat parts, as explained in Section 3.2. We used ROUGE-1, ROUGE-2, and ROUGE-L (Lin, 2004) for evaluating the summary part (de-noted as R-1 , R-2 , and R-L , respectively) and BLEU (Papineni et al., 2002) for evaluating the chit-chat part. We use different metrics for each part because ROUGE is often used for summarization tasks while BLEU is used for conversational tasks. Since these automatic metrics are insuf-ficient for evaluation (Novikova et al., 2017), we also perform a manual evaluation in Section 5.4. To evaluate diversity, we calculate the proportion of distinct unigrams, bigrams, and sentences ( D-1 , D-2 , and D-S , respectively) in the generated initial utterances (Li et al., 2016a). Table 4 lists the results of the summary part. The baseline method that outputs the first sentence of an input news post achieved higher ROUGE scores than the proposed methods. This does not necessarily mean that the proposed methods are poor because even the SOTA summarization system exceeds such a baseline by only a small margin (See et al., 2017). Also, our task has a requirement to convert sentences into colloquial expressions, and the ROUGE metric cannot capture such a subtle difference. We perform a deeper investigation into the quality of the generated initial utterance in the next section. Regarding the diversity, almost all of the generated initial utterances are distinct, as shown in Table 4. Table 5 shows the result of the chit-chat part. The proposed methods outperformed the baselines in terms of BLEU score. Although the baselines use two times as much training data as the proposed methods, their scores were quite low. This demonstrates the quality of our dataset. The score of Separate (IR) was relatively low among the proposed methods, presumably because the chitchat parts retrieved from the training data do not always match the content of the input news post. We also see that all the BLEU scores of the models are quite lower than ROUGE scores in Table 4. In general, both summarization and chat generation tasks often use automatic evaluation metrics to evaluate generated sentences, their scores tend to be much lower in the chat generation task. This is because the answer sentences (utterances) of the chat generation task have more diverse candidates than other generation tasks such as machine translation and summarization ( Li et al., 2016a,b; Baheti et al., 2018). We also examine the diversity of the chit-chat part in Table 5. Although the diversity of the IR-based methods was high, their BLEU scores deteriorated considerably. Among the generation-based methods, although Separate (Gen+MMI) achieved the highest BLEU score, it lacked diversity. In contrast, Joint achieved a reasonable BLEU score while maintaining diversity to some extent. Although diversity of utterances can be quantified automatically, ROUGE and BLEU scores do not always follow human intuition (Novikova et al., 2017; Lowe et al., 2017). Therefore, we evaluate the generated initial utterances manually. We picked the three proposed models with good performance in the automatic evaluation along with one baseline for this manual evaluation. 300 posts were sampled as the input news posts, and the outputs of the four methods were manually evaluated from two perspectives: 1) Naturalness : Does the utterance naturally initiate conversation? and 2) Coherency : Is the content of the utterance coherent with the given news post? We recruited crowd workers to score each utterance on a 4-point scale (Agree, Slightly Agree, Slightly Disagree, Dis-agree). Table 6 show the results of the manual evaluation for Naturalness and Coherency of the generated initial utterances. The proposed methods excluding Separate (IR) outperformed Baseline (Gen+MMI) in both perspectives and achieved reasonable scores compared to human upper-bound. The scores of Separate (IR) are quite low because the retrieval result does not follow the input news post in many cases. This reveals that although those sentences have high diversity, their quality is poor as initial utterances. Although Baseline (Gen+MMI) achieved high ROUGE scores in Table 4, its style is not colloquial. Thus, workers felt odd and lowered their scores. In conclusion, it is better to use the generation-based methods for conversation initiation. We also evaluated Dullness : Is the given utterance dull or boring? We used 15 manually created boilerplate utterances ( e.g., Hello., How are you?, Let's talk with me.) rather than Baseline (Gen+MMI) to confirm the effectiveness of utilizing news contents as the initial utterances. Table 7 show the results of the manual evaluation for Dullness of the generated initial utterances. We see that compared to our proposed methods, the score of the boilerplate baseline is quite high. This indicates that using boilerplate utterances for conversation initiation often bores users and possibly leads to early abandonment of the conversation. To determine the statistical significance of our results, we performed Wilcoxon signed-rank tests with Bonferroni correction (Wilcoxon, 1945). In Table 6, for all combinations except Baseline (Gen+MMI) vs. Separate (IR) and Separate (Gen+MMI) vs. Joint , there were significant differences (p-value < 0 : 005 (corrected)) in both perspectives. Similarly, in Table 7, there were statistically significant differences for all combinations except Separate (IR) vs. Separate (Gen+MMI) and Separate (Gen+MMI) vs. Joint . Finally, we investigated the initial utterances generated by Separate (Gen+MMI) and Joint . Ex-3996 news posts initial utterances A parade for the Rio Olympics and Paralympic medalists will be held in October. We found that Separate (Gen+MMI) tended to generate generic utterances ( e.g., That's amaz-ing, Get it together ) as the chit-chat part that fit any context, even though it uses a diversity-promoting function when decoding. In contrast, Joint could generate more diverse chit-chat parts by utilizing contents words such as parade and poaching . One possible reason for this phenomena is that the generated summary part acts like an additional condition of P ( T j S ) at the time of decoding the chit-chat part. This does not happen with Separate (Gen+MMI) , which simply concatenates the outputs of separate models. Interestingly, we found that there are some utterances giving a question (third example of Joint in Table 8) or making an urging (fourth example of Separate (Gen+MMI) in Table 8). Controlling utterances of the model by such dialogue acts (Wen et al., 2015; Zhao et al., 2017) can make the conversation initiation more diverse and attractive. We leave them as the future work at this time. We should note that although it is a problem common to all the generation-models, there is a possibility of transmitting false news contents (as in the second example of Separate (Gen+MMI) in Table 8) or ethically inappropriate contents to the users. Therefore, when adopting our method into an actual conversation application, we have to pay close attention to this problem. In this paper, we proposed the new task of conversation initiation. To generate diverse initial utterances that can improve user engagement, we utilized news articles that provide fresh and varied information every day and constructed a large-scale dataset using crowd workers. To perform the conversation initiation, we designed separate and joint approaches including both IR-based and generation-based methods. Empirical experiments showed that the proposed methods outperformed the baselines in both automatic and manual evaluation, and can generate diverse initial utterances that template-based methods cannot make. These results demonstrate the quality of our constructed dataset, that will be released for future studies 11 . As a natural next step, we plan to develop a more sophisticated conversation model, which can not only generate initial utterances but also continue the conversation for the given news contents (Yoshino and Kawahara, 2014). In that case, depending on the user's interest, the model needs to determine whether to do a usual chat or talk about the news contents. We also plan to improve the proposed method so that it can generate even better initial utterances. Since our task has two elements, summarization and chit-chat, the focus of our future work will be a more sophisticated multitask model that considers these relations. We thank Manabu Sassano for fruitful discussions and comments. We also thank the anonymous reviewers.
abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain objective abstain abstain method abstain method method abstain abstain objective objective abstain abstain objective objective method objective objective other other other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain other other objective other other objective other other method method method objective abstain abstain objective method objective abstain objective abstain method method method abstain result method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method method abstain abstain abstain abstain method method abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain objective result method abstain objective abstain abstain objective method other other