End-to-End Multi-Domain Task-Oriented Dialogue Systems with Multi-level Neural Belief Tracker
It has been an open research challenge for developing an end-to-end multi-domain task-oriented dialogue system, in which a human can converse with the dialogue agent to complete tasks in more than one domain. First, tracking belief states of multi-domain dialogues is difficult as the dialogue agent must obtain the complete belief states from all relevant domains, each of which can have shared slots common among domains as well as unique slots specifically for the domain only. Second, the dialogue agent must also process various types of information, including contextual information from dialogue context, decoded dialogue states of current dialogue turn, and queried results from a knowledge base, to semantically shape context-aware and task-specific responses to human. To address these challenges, we propose an end-to-end neural architecture for task-oriented dialogues in multiple domains. We propose a novel Multi-level Neural Belief Tracker which tracks the dialogue belief states by learning signals at both slot and domain level independently. The representations are combined in a Late Fusion approach to form joint feature vectors of (domain, slot) pairs. Following recent work in end-to-end dialogue systems, we incorporate the belief tracker with generation components to address end-to-end dialogue tasks. We achieve state-of-the-art performance on the MultiWOZ2.1 benchmark with 50.91% joint goal accuracy and competitive measures in task-completion and response generation.
[Contribution summary] Authors propose a new E2E model for the DST task that (1) implements a response generator that leverages DB query results as well as other dialog contexts with a late fusion approach for joining domain and slot pair representations, and (2) obtains the competitive results (50.91% for the joint task of DST & context-to-text generation, and 49.55% joint accuracy for the DST task, both on MultiWoZ 2.1). [Comments] The proposed model is reasonably structured. Empirical results show improvement over other baselines, with the main gain (among ablations) coming from incorporating the dialog states from previous turns, and the joint domain-slot module, etc. The empirical analysis section could improve and the novelty of the system over the previous literature be clarified -- some of the components used in this work are not entirely novel and previously utilized in other work, hence it is hard to discern where the main gain is really coming from, compared to the previous work (especially for the DST task). Empirical comparison of the joint DST & generation, which is one of the main results, is limited to Lei et al. There have been recent work on DST with new SOTA results (e.g. “Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset” by Rastogi et al.) -- please consider comparing the approaches.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
End-to-End Multi-Domain Task-Oriented Dialogue Systems with Multi-level Neural Belief Tracker
It has been an open research challenge for developing an end-to-end multi-domain task-oriented dialogue system, in which a human can converse with the dialogue agent to complete tasks in more than one domain. First, tracking belief states of multi-domain dialogues is difficult as the dialogue agent must obtain the complete belief states from all relevant domains, each of which can have shared slots common among domains as well as unique slots specifically for the domain only. Second, the dialogue agent must also process various types of information, including contextual information from dialogue context, decoded dialogue states of current dialogue turn, and queried results from a knowledge base, to semantically shape context-aware and task-specific responses to human. To address these challenges, we propose an end-to-end neural architecture for task-oriented dialogues in multiple domains. We propose a novel Multi-level Neural Belief Tracker which tracks the dialogue belief states by learning signals at both slot and domain level independently. The representations are combined in a Late Fusion approach to form joint feature vectors of (domain, slot) pairs. Following recent work in end-to-end dialogue systems, we incorporate the belief tracker with generation components to address end-to-end dialogue tasks. We achieve state-of-the-art performance on the MultiWOZ2.1 benchmark with 50.91% joint goal accuracy and competitive measures in task-completion and response generation.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



[Contribution summary] Authors propose a new E2E model for the DST task that (1) implements a response generator that leverages DB query results as well as other dialog contexts with a late fusion approach for joining domain and slot pair representations, and (2) obtains the competitive results (50.91% for the joint task of DST & context-to-text generation, and 49.55% joint accuracy for the DST task, both on MultiWoZ 2.1). [Comments] The proposed model is reasonably structured. Empirical results show improvement over other baselines, with the main gain (among ablations) coming from incorporating the dialog states from previous turns, and the joint domain-slot module, etc. The empirical analysis section could improve and the novelty of the system over the previous literature be clarified -- some of the components used in this work are not entirely novel and previously utilized in other work, hence it is hard to discern where the main gain is really coming from, compared to the previous work (especially for the DST task). Empirical comparison of the joint DST & generation, which is one of the main results, is limited to Lei et al. There have been recent work on DST with new SOTA results (e.g. “Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset” by Rastogi et al.) -- please consider comparing the approaches.
assistant
[Contribution summary] Authors propose a new E2E model for the DST task that (1) implements a response generator that leverages DB query results as well as other dialog contexts with a late fusion approach for joining domain and slot pair representations, and (2) obtains the competitive results (50.91% for the joint task of DST & context-to-text generation, and 49.55% joint accuracy for the DST task, both on MultiWoZ 2.1). [Comments] The proposed model is reasonably structured. Empirical results show improvement over other baselines, with the main gain (among ablations) coming from incorporating the dialog states from previous turns, and the joint domain-slot module, etc. The empirical analysis section could improve and the novelty of the system over the previous literature be clarified -- some of the components used in this work are not entirely novel and previously utilized in other work, hence it is hard to discern where the main gain is really coming from, compared to the previous work (especially for the DST task). Empirical comparison of the joint DST & generation, which is one of the main results, is limited to Lei et al. There have been recent work on DST with new SOTA results (e.g. “Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset” by Rastogi et al.) -- please consider comparing the approaches.