Zero-shot cross-domain dialogue state tracking (DST) enables us to handle task-oriented dialogue in unseen domains without the expense of collecting in-domain data. In this paper, we propose a slot description enhanced generative approach for zero-shot cross-domain DST. Specifically, our model first encodes dialogue context and slots with a pre-trained self-attentive encoder, and generates slot values in an auto-regressive manner. In addition, we incorporate Slot Type Informed Descriptions that capture the shared information across slots to facilitate cross-domain knowledge transfer. Experimental results on the MultiWOZ dataset show that our proposed method significantly improves existing state-of-the-art results in the zero-shot cross-domain setting. Task-oriented dialogue systems are designed to assist users in performing daily activities, such as restaurant booking, travel planning, and online shopping. These virtual assistants provide natural language interfaces to services and online APIs (Rastogi et al., 2020). Based on users' needs, these systems frequently require support for new domains. However, the current state-of-the-art systems require a substantial amount of in-domain data to properly model a new domain. The data-collection process is both expensive and time-consuming, and thus it is very important to study methods that can build robust and scalable dialogue systems using little to no in-domain data. The dialogue state tracking (DST) is an essential component of task-oriented dialogue systems that tracks users' requirements over multi-turn conversations. A popular formulation of the dialogue state is in the form of a list of slot-value pairs. In DST, tracking unseen slots in a new domain, a.k.a. zero-shot domain adaptation, is a significant challenge,  Work done during internship at Facebook Figure 1: High-level description of the T5DST. since the model has never seen in-domain training samples. There are two main lines of work to tackle this problem. The first proposes domain transferable models using copy mechanisms or ontology graph information (Wu et al., 2019; Zhou and Small, 2019). A limitation of such models is that they may not fully leverage pre-trained language models due to the specialized model architecture. The second line of work uses slot-descriptions as input to the model to facilitate the slot understanding (Rastogi et al., 2020). However, the provided slot descriptions are collected by crowd sourced human annotators and might be inconsistent among different domains. In general, the optimal approach for constructing slot descriptions in zero-shot settings remains unexplored. In this work, we tackle the challenge of zero-shot cross-domain DST via leveraging large scale pre-trained sequence-to-sequence (seq2seq) models and with effective encoding of slot descriptions. We first introduce a generative DST model called T5DST, which models the relation of a slot and its dialogue context with a self-attentive encoder, and generates the slot value with a decoder in an autoregressive manner. This simple design allows us to effectively incorporate a pre-trained Figure 2: Slot description examples. seq2seq model (e.g., T5 (Raffel et al., 2020)) without any task-specific modification. To further enhance the model's cross-domain transferability, we propose Slot Type Informed Descriptions that capture the shared information of different slots. Experimental results on the MultiWOZ benchmark (Budzianowski et al., 2018) suggest that 1) our model achieves significantly higher joint goal accuracy compared to existing results in zero-shot cross domain DST; 2) models using the proposed slot description formulation substantially outperform those using other slot description variants. Our contributions are summarized as the following:  We propose a simple yet novel generative DST model based on T5 that significantly improves existing zero-shot cross-domain DST results;  We investigate the effectiveness of different slot description formulations. To the best of our knowledge, this is the first work that comprehensively studies the effectiveness of slot descriptions in zero-shot cross-domain DST. Dialogue State Tracking has been of broad interest to the dialogue research community (Williams and Young, 2007; Williams et al., 2014; Heck et al., 2020; Liu et al., 2020; Wu et al., 2020; Madotto et al., 2020). Current state-of-the-art models (Chen et al., 2020; Lin et al., 2020; Heck et al., 2020; Hosseini-Asl et al., 2020; Ye et al., 2021; Li et al., 2020) trained with extensive annotated data have been shown promising performance in complex multi-domain conversations (Budzianowski et al., 2018). However, collecting large amounts of data for every Slot Type Slot Name Number hotel-book stay, hotel-book people, hotel-stars, train-book people, restaurant-book people Location train-destination, train-departure, taxi-destination, taxi-departure Time train-arriveby, train-leaveat, taxi-leaveat, restaurant-book time, taxi-arriveby Boolean hotel-parking, hotel-internet Name attraction-name, restaurant-name, hotel-name Day hotel-book day, train-day, restaurant-book day Table 1: Slot type of slots in MultiWOZ. domain is costly and inefficient. To address this issue, several methods (Wu et al., 2019; Zhou and Small, 2019) have proposed for transferring prior knowledge of existing domains to new ones. On the other hand, Campagna et al. (2020) proposed an abstract dialogue model that leverages the ontology and in-domain templates to generate a large amount of synthesized data for domain adaptation. Different from their method, in this paper, we utilize a pre-trained seq2seq model and slot descriptions for cross-domain DST without any in-domain data. Slot Description has been shown to be a promising technique in cross domain semantic parsing (Bapna et al., 2017; Shah et al., 2019; Namazi-far et al., 2020). To encourage this line of research in DST as well, MultiWOZ2.1 (Eric et al., 2019) provides a further annotation for slot descriptions. Rastogi et al. (2020) incorporated slot descriptions for facilitating cross domain DST, while Gao et al. (2019, 2020) formulated DST as a question answering problem by casting a slot name into questions. However, these works did not show the effectiveness of slot descriptions, by comparing the performance of models with and without them. There is no study on how to construct slot descriptions. In this paper, we aim to fill this research gap by providing an empirical study on the different slot description formulations. The design of our model follows the basis of generative question answering models. As illustrated in Figure 1, given a dialogue history which consists of an alternating set of utterances from two Model Joint Goal Accuracy Attraction Hotel Restaurant Taxi Train Average TRADE 19.87 13.70 11.52 60.58 22.37 25.76 SUMBT* 22.60 19.80 16.50 59.50 22.50 28.18 SimpleTOD++ 28.011.30 17.691.00 15.571.54 59.220.95 27.751.16 29.650.58 T5DST 32.66 0.10 18.731.67 20.55 0.96 64.62 0.24 31.27 0.47 33.56 0.54 w/ Human 31.921.42 20.720.35 20.090.67 64.120.28 28.831.28 33.140.17 w/ Naive 32.980.60 20.231.11 20.012.91 63.590.23 30.044.31 33.371.36 w/ Slot Value 32.860.56 20.030.87 16.650.37 65.09 0.12 29.662.75 32.860.48 w/ Question 32.450.39 19.791.18 21.82 0.91 64.400.27 32.611.38 34.210.63 w/ Slot Type 33.09 1.60 21.21 0.61 21.651.07 64.620.55 35.42 1.42 35.20 0.59 Table 2: Zero-shot cross-domain results in MultiWOZ 2.0. speakers, denoted as C t = { U 1 , R 1 , . . . , R t  1 , U t } , we add the "user:" and "system:" prefixes to the user and system utterance respectively. Then all the utterances and slot names s i are concatenated into a single sequence, i.e., user: U 1 . . . system: R t  1 user: U t [sep] s i . The sequence is used as the input to the encoder, and the decoder generates the corresponding slot value v i : v i = Seq 2 seq ( C t , s i ) . The learning objective of this generation process is minimizing the negative log-likelihood of v i given C t and s i , that is, where n is the number of slots to be tracked. We initialize the model parameters with T5 (Raf-fel et al., 2020), an encoder-decoder Transformer with relative position embeddings (Shaw et al., 2018) pre-trained on a massive amount of English text. We denote our model as T5DST . To incorporate slot descriptions into T5DST , we replace the slot name with its corresponding slot description as the model input. Although different slots may have distinguishing names, they can share the same slot type. As shown in Table 1, the slot type of hotel-stars and restaurant-book people are both number slots, while hotel-internet and hotel-parking are both boolean slots. In light of these observations, we hypothesize that adding slot type information to the slot description facilitates the knowledge transfer among different slots. We construct a template for each slot type that follows "[slot type] of [slot] of the [domain]" . We denote such a slot description as Slot Type . More details are available in Appendix A.1. We evaluate the proposed method on the MultiWOZ 2.0 dataset (Budzianowski et al., 2018), which has 7 domains. We use the pre-processing and evaluation setup from Wu et al. (2019), where restaurant, train, attraction, hotel, and taxi domains are used for training, as the test set only contains these 5 domains. In the zero-shot cross-domain experiments, the models are first trained with four domains and then evaluated on the test-set of the unseen domain. Joint goal accuracy is used to evaluate the performance of the models. The generated dialogue states are considered to be correct if and only if all of the predicted values exactly match the oracle values. We implement T5DST 1 based on the T5-small (60M parameters) model which has 6 encoder-decoder layers and the hidden size d model = 512 . All models are trained using an AdamW (Loshchilov and Hutter, 2018) optimizer with the initial learning rate of 0 . 0001 . In all cross-domain zero-shot experiments, we train the models with batch size 128 for 5 epochs. For the few-shot 1 Source code is available in https://github.com/ facebookresearch/Zero-Shot-DST Model Attraction Hotel Restaurant Taxi Train 1% 5% 10% 1% 5% 10% 1% 5% 10% 1% 5% 10% 1% 5% 10% TRADE 35.88 57.55 63.12 19.73 37.45 41.42 42.42 55.70 60.94 63.81 66.58 70.19 59.83 69.27 71.11 DSTQA N/A 70.47 71.60 N/A 50.18 53.68 N/A 58.95 64.51 N/A 70.90 74.19 N/A 70.35 74.50 T5DST w/ Slot Type 58.77 65.72 69.54 43.07 50.71 54.86 57.63 61.86 63.47 70.12 73.67 74.70 70.82 74.18 77.57 Table 3: Few-shot experimental results in MultiWOZ 2.0. experiments, the models are first trained on 4 domains for 5 epochs then fine-tuned with 1%, 5% and 10% of target domain data for 10 epochs. For full shot training, we train our model for at most 10 epochs with batch size 64 and early stop according to the loss in the validation set. Other hyper-prameters are same as zero-shot cross-domain setting. We use 8 NVIDIA V100 GPUs for all of our experiments. We use greedy decoding in test time. TRADE. Transferable dialogue state generator (Wu et al., 2019) which utilizes copy mechanism to facilitate domain knowledge transfer. DSTQA. Dialogue state tracking via question answering 2 over ontology graph (Zhou and Small, 2019). SimpleTOD++. SimpleTOD (Hosseini-Asl et al., 2020) uses a single causal language model GPT2 (Radford et al., 2019) to generate the dialogue states. To adapt this model to a zero-shot cross-domain setting, we also provide the slot name as the model input. We denote this model as SimpleTOD++. Human. Human annotated slot descriptions collected in MultiWOZ2.1 (Eric et al., 2019) and used in MultiWOZ2.2 (Zang et al., 2020). Naive. Simple transformation of the slot name from "domain-slot" to "[slot] of the [domain]" . Slot Value. Following recent works (Zhang et al., 2019; Rastogi et al., 2020), slots are divided into 2 We are aware of STARC (Gao et al., 2020). categorical and non-categorical slots. For categorical slots, we incorporate the candidate values into the slot description, i.e., "[slot] of the [domain] is [value-1] or [value-2]?" . The order of values is random. For non-categorical slots, their descriptions are the same as aforementioned Naive . Question. Similar to (Gao et al., 2019, 2020), we reformulate the slot into a natural language question, i.e., "What is the [slot] of the [domain] that is the user interested in?" . The results of the zero-shot cross domain experiments are shown in Table 2. Overall, T5DST achieves significantly higher performance in terms of averaged joint goal accuracy compared to the three baseline models TRADE, SUMBT, and Sim-pleTOD++. These results demonstrate that our model can effectively capture the slot-context relation, and thus generalize better in unseen domains. Replacing slot-names with human annotated slot descriptions does not bring improvement to the zero-shot performance. This might because of the diverse and inconsistent human descriptions among different domains. For example, the human descriptions of attraction-area and restaurant-area are "area to search for attractions" and "area or place of the restaurant" respectively. Such inconsistent descriptions increase the challenge on slot understanding in the zero-shot learning setting. the model using naive slot descriptions gives similar performance to the one that uses original slot names. The two approaches lead to similar semantic representation of the slots. In contrast, incorporating slot values hurts the learning, leading to a lower joint goal accuracy in the restaurant domain. We observe that even though adding value candidates improve some of the categorical slots (e.g., restaurant-area 68.35%  82.25% slot ac-curacy), it hurts the unseen non-categorical slots (e.g., restaurant-food 40.63%  26.10% slot accu-racy). These non-categorical slots are usually the Figure 3: Slot accuracy in attraction, taxi, and hotel domains of MultiWOZ 2.0. bottlenecks of joint goal accuracy. Finally, models trained with question style descriptions improves the performance in some domains, but fails in the others. Our proposed slot type informed descriptions consistently improves the zero-shot performance of T5DST in all the domains. It produced an average of 2% joint goal accuracy improvement compared to human labeled and naive description formulations. This result indicates that slot type information may better capture the shared property (e.g., time, location) among different slots, thus facilitating the domain knowledge transferring for DST. Figure 3 and 4 show the slot accuracy of models using Naive and Slot Type description. Compared to naive description, we obverse significant gain of time slots (e.g., arrive by and leave at), location slots (e.g., departure and destination), and number slots (e.g., book stay and book people) by adding slot type information. We conjecture that explicit information about the target value (i.e., slot type) is important in the low resource condition when the model does not have enough data to capture the semantic meaning of a new slot. We further conduct experiments in few-shot cross-domain settings, as in (Wu et al., 2019; Zhou and Small, 2019), where the models are first trained on 4 domains then fine-tuned with 1%, 5% and 10% of target domain data. As shown in Table 3, our model outperforms the DSTQA model in 4 out of 5 domains. Moreover, our approach is more practical in a real-world learning scenario as it does not require the supervision of a full ontology graph. We also conduct the full shot experiments and compare our model with previous methods. The reults are reported in Appendix A.2. In this paper, we propose leveraging large scale pre-trained models with an effective slot description formulation to tackle the zero-shot cross-domain DST challenge. Specifically, we propose T5DST, a novel generative DST model based on the T5 language model, and incorporate Slot Type Informed Descriptions to facilitate cross-domain knowledge transfer. In the evaluation on the MultiWOZ dataset, our approach substantially improves existing results in both the zero-shot and few-shot settings.
method objective objective abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective abstain abstain objective abstain objective abstain objective objective other other other other other other method other other other other other objective method other method abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain objective objective result