## Intermittent Semi-working Mask: A New Masking Paradigm for LLMs

Mingcong Lu, Jiangcai Zhu, Wang Hao, ZhengLi, Shusheng Zhang, Kailai Shao, Chao Chen, Nan Li,

Feng Wang, and Xin Lu ∗

Ant Group {lumingcong.lmc, jiangcai.zjc, zhangshusheng, kailai.skl,

qionghao.wh, qinzheng.lz, chixi.cc, linan.ln, zifan.wf, zsslx111333}@antgroup.com

## Abstract

Multi-turn dialogues are a key interaction method between humans and Large Language Models (LLMs), as conversations extend over multiple rounds, keeping LLMs' high generation quality and low latency is a challenge. Mainstream LLMs can be grouped into two categories based on masking strategy: causal LLM and prefix LLM. Several works have demonstrated that prefix LLMs tend to outperform causal ones in scenarios that heavily depend on historical context such as multi-turn dialogues or in-context learning, thanks to their bidirectional attention on prefix sequences. However, prefix LLMs have an inherent inefficient training problem in multi-turn dialogue datasets. In addition, the attention mechanism of prefix LLM makes it unable to reuse Key-Value Cache (KV Cache) across dialogue rounds to reduce generation latency. In this paper, we propose a novel masking scheme called Intermittent Semi-working Mask (ISM) to address these problems. Specifically, we apply alternate bidirectional and unidirectional attention on queries and answers in the dialogue history. In this way, ISM is able to maintain the high quality of prefix LLM and low generation latency of causal LLM, simultaneously. Extensive experiments illustrate that our ISM achieves significant performance.

## 1 Introduction

Recent developments in Large Language Models (LLMs) such as GPT-3.5 Ouyang et al. [2022] and GPT-4 Achiam et al. [2023] have attracted significant attention. Due to the powerful generation capability, LLMs have made remarkable achievements in different kinds of Natural Language Process (NLP) tasks through a unified generative paradigm. Specifically, the most natural and common way to interact with LLMs is through multi-turn dialogues. However, as the number of dialogue rounds increases, ensuring high quality and low latency of the generated answer by LLMs is a challenge.

Existing language models can be grouped into three categories according to framework architecture: Encoder-Decoder Vaswani et al. [2017], Raffel et al. [2020], Lewis et al. [2020], Encoder-Only Kenton and Toutanova [2019], Liu et al. [2019], Dong et al. [2019], and Decoder-Only Brown et al. [2020], Touvron et al. [2023a,b], Du et al. [2022]. Nowadays, most LLMs belong to decoder-only architecture, in this paper, our discourse is delimited to decoder-only architecture. In addition, based on the masking methods in various attention mechanisms, decoder-only category further includes causal decoders Brown et al. [2020], Touvron et al. [2023a] and prefix decoders Du et al. [2022]. The former employs unidirectional attention masking to restrict each token can only attend to preceding tokens and itself. Both the input and generated tokens are processed in a uniform manner within

∗ Corresponding author.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00539v1-figure-17ef601f-e54f-414c-a321-f385d21555ed.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00539v1-figure-eb0acf4b-fb58-40bf-a849-e92fc2c838c2.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00539v1-figure-02682146-8357-4ebf-8b73-bd11e92a0d8c.jpg)

c) Our Intermittent Semi-working Mask

Figure 1: Illustration of our Intermittent Semi-working Mask vs. existing Causal Mask and Prefix Mask in Multi-Turn Dialogues. Taking the second round of dialogue as an example, we show the mask difference between our method and existing works. The dialogue history ( P rompt+ Q uery 1 + A nswer 1 ) and current Q uery 2 serve as prefix sequences, LLMs should output A nswer 2 . Causal Mask employs unidirectional attention on prefix sequences, while Prefix Mask applies bidirectional attention. Our ISM utilizes alternate bidirectional and unidirectional attention on queries and answers in prefix sequences. All of them generate answer in auto-regressive.

the decoder. The latter applies bidirectional attention on prefix tokens and maintains unidirectional attention on generated tokens, it allows bidirectional encoding of prefix sequences.

Empirical studies Raffel et al. [2020], Tay et al. [2022], Anil et al. [2023] have revealed that restricting auto-regressive masks on the entire sequence is overly restrictive when LLM generates answers based on the given contexts. Other work Ding et al. [2023b] further proves that prefix LLM may be a better choice theoretically. However, the bidirectional attention in prefix LLM is a double-edged sword. In the bidirectional attention mechanism, each token of the prefix sequences can attend to all the tokens in the whole sequences. It means that the KV cache of the previous dialogue rounds cannot be reused when the prefix LLM generates the current response. When facing multi-turn dialogues, especially when the dialogue round reaches a large number such as 10 or more, prefix LLM will suffer from high generation latency. In addition, due to the bidirectional attention in prefix sequences, it's necessary to manually expand a multi-turn dialogue sample into several single-turn dialogue samples to train model for all the dialogue rounds. However, unidirectional attention enables to train causal LLMs for all rounds of dialogue generation through a single forward process.

In this paper, we propose a novel attention mask scheme called Intermittent Semi-working Mask (ISM), which preserves the strengths of the prefix decoder while concurrently addressing the issue of high generation latency in multi-turn dialogues. In other words, our ISM simultaneously possesses the high quality of prefix decoder and low latency of causal decoder. Specifically, as illustrated in Figure 1, given a prefix sequence includes the dialogue history and current query [ P Q , 1 , A 1 , · · · , Q n -1 , A n -1 , Q n ] , our ISM applies bidirectional and unidirectional attention on queries ( Q i , i = 1 , · · · , n ) and answers ( A i , i = 1 , · · · , n -1 ), respectively. Different from prefix mask, our ISM ensures queries or answers cannot attend to subsequent queries or answers in the dialogue history. Therefore, we can reuse the KV cache across dialogue rounds like causal decoder to prevent duplicate computation and reduce the generation latency. In addition, our ISM significantly improves the training efficiency for multi-turn dialogue datasets. Given a sample [ P Q , 1 , A 1 , · · · , Q n , A n ] , the usual training approach for a prefix LLM is to expand the data into n samples: [ P Q , 1 , A 1 ] , [ P Q , 1 , A 1 , Q 2 , A 2 ] , · · · , [ P Q , 1 , A 1 , · · · , Q n , A n ] . In contrast, our ISM can be trained on the original sample through a single forward process like causal LLMs.

To prevent the influence of the base model. We apply our ISM on prefix LLMs and causal LLMs and conduct extensive experiments in multi-turn dialogue benchmark datasets. We follow Zheng et al. [2024] to utilize GPT-4 as a stand-in for human raters to score them. The experimental results demonstrate that our ISM achieves state-of-the-art generation quality and latency in multi-turn dialogues. Overall, our contribution can be summarized as follows,

- · We propose a novel masking scheme called ISM, which applies alternate bidirectional and unidirectional attention on queries and answers in multi-turn dialogues. It maintains the high quality of prefix decoders and the low latency of causal decoders simultaneously.

- · We apply our ISM on causal LLMs and prefix LLMs and conduct extensive experiments to compare the generation quality and latency between our ISM and existing methods, the experimental results demonstrate the effectiveness of our method.
- · We deploy our ISM in a real-world online environment to test its practical performance, we observe our ISM brings significant improvements in generation latency and quality.

## 2 Related Work

## 2.1 LLMs for Multi-turn Dialogues

Recently, the advancements in LLMs have garnered lots of attention. To enhance the multi-turn capabilities of open-sourced LLMs, initial efforts begin with collecting human-ChatGPT dialogues, leading to the creation of Vicuna Chiang et al. [2023]. RealChat Zheng et al. [2023] later expands the data to 1 million conversations. To generate more sophisticated datasets, Baize Xu et al. [2023] and UltraChat Ding et al. [2023a] employ alternating GPT interactions. Further, Cue-CoT Wang et al. [2023] and ICL-AIF Fu et al. [2023b] enhance model capabilities for multi-turn interactions through In-Context-Learning (ICL) and Chain-of-Thought (CoT) algorithms.

To investigate LLMs' multi-turn ability, several works construct multi-turn dialogue benchmarks. Daily Dialogue Li et al. [2017] is a human-written high-quality multi-turn dialogue dataset. It reflects daily communication way and covers various topics about daily life. KdConv Zhou et al. [2020] is a Chinese human-labeled knowledge-grounded dialogue dataset for knowledge-driven conversation generation. NaturalConv Wang et al. [2021] is a Chinese multi-turn topic-driven conversation dataset, which contains conversations from six domains.

## 2.2 Evaluation of LLMs

The quality of the content generated by LLMs can be evaluated from both objective and subjective perspectives. The former aims to evaluate the capabilities of LLMs in an objective and quantitative way, usually by comparing LLMs' outputs with corresponding ground truth. Similarity metrics are calculated based on the outputs and references, such as BLEU Papineni et al. [2002], and ROUGE Lin [2004]. An alternative approach involves tasks as a classification task and forces LLMs to output closed options for accuracy calculation, such as MMLU Hendrycks et al. [2020] and CEVAL Huang et al. [2024]. The latter is applied to compare the performances of different LLMs by humans or other powerful LLM judges when facing more complicated scenarios. GPTScore Fu et al. [2023a] makes full use of GPT3 in achieving customized, multi-aspect and training-free evaluation. Chatbot Arena Zheng et al. [2024] showcases high agreement between GPT-4 and humans on making judgements. It indicates that leveraging LLM like GPT-4 for evaluating the quality of generated text could be a viable alternative to manual human assessment.

## 3 Methodology

In this section, we first give the mathematical description of causal LLM, prefix LLM, and our ISM. Then, we elaborate on the benefits of our ISM compared to causal and prefix LLM, respectively. In addition, we introduce how to implement our ISM on existing prefix and causal LLMs.

## 3.1 Mathematical description

## 3.1.1 Transformers: SSA, LSA

Given a sequence of input vectors X = ( x 1 , · · · , x n ) , the output of standard Softmax Self-Attention (SSA) layer is

$$x _ { j } \leftarrow x _ { j } + P V X s o f t m a x ( X ^ { \top } K ^ { \top } Q x _ { j } )$$

where P corresponds to the output projection, V K Q , , corresponds to the value, key, and query transformation, respectively.

Since the softmax attention of standard transformers is non-linear, its theoretical analysis becomes complicated even for a single layer. For this reason, theoretical approaches V on Oswald et al. [2023], Zhang et al. [2023] to analyze transformers have often dropped the softmax function from the attention, resorting to the Linear Self-Attention (LSA) layer,

$$x _ { j } \leftarrow x _ { j } + P V X ( X ^ { \top } K ^ { \top } Q x _ { j } ) = x _ { j } + P V \sum _ { i = 1 } ^ { n } x _ { i } ( x _ { i } ^ { \top } K ^ { \top } Q x _ { j } )$$

## 3.1.2 Causal LLM, Prefix LLM, and ISM for Multi-turn dialogue

Here, we briefly introduce the attentions of causal LM, prefix LM, and ISM in their LSA version for multi-turn dialogue. Given a multi-turn dialogue sample including the prompt ( P = ( x 1 , · · · , x m p ) ), queries ( Q i = ( x m p +1 or x m a i -1 +1 , · · · , x m q i ) , i = 1 , · · · , n ) and answers( A i = ( x m q i +1 , · · · , x m a i ) , i = 1 , · · · , n -1 ), that is [ P Q , 1 , A 1 , · · · , Q n -1 , A n -1 , Q n ] . The goal of the model is to predict A n of Q n using the context [ P Q , 1 , A 1 , · · · , Q n -1 , A n -1 ] .

The most classic form of attention is categorized as full (or bidirectional) attention, shown in equation 3.2, in which each input x j ( j = 1 , · · · , n ) can attend to all positions. Full attention is typically used in the transformer encoder.

Furthermore, another transformer decoder for in-context learning uses the auto-regressive attention

$$x _ { j } \leftarrow x _ { j } + P V \sum _ { i = 1 } ^ { j } x _ { i } ( x _ { i } ^ { \top } K ^ { \top } Q x _ { j } )$$

which restricts each token x j to attend only to previous positions (and itself) from 1 to j. This restriction is due to the role of the decoder as a causal language model (causal LLM) which predicts the next token in the context of the previously generated ones.

The original transformer uses a full attention based encoder and an auto-regressive attention based decoder. However, NLP researchers have a preference for models that are either encoder-only (e.g. BERTDevlin et al. [2019]) or decoder-only (e.g. GPTRadford et al. [2018], Brown et al. [2020], PaLMChowdhery et al. [2023], Anil et al. [2023]) according to the task at hand. One motivation for altering model is that it reduces the number of parameters by half.

A new form of attention, that lies intermediate to full attention and auto-regressive attention, emerged from the understanding that certain tasks can gain advantages from a prefix sequence, such as multi-turn dialogue and in-context learning. This attention in prefix LLM suggests the following:

$$x _ { j } \leftarrow x _ { j } + P V \sum _ { i = 1 } ^ { m a x ( j, n _ { q _ { n } } ) } x _ { i } ( x _ { i } ^ { \top } K ^ { \top } Q x _ { j } )$$

Where max j,n ( q n ) ensures each prefix token x j with j &lt; n q n can attend to all prefix tokens.

In our ISM, the form of which suggests the following:

$$x _ { j } \leftarrow x _ { j } + P V \sum _ { i = 1 } ^ { f ( j ) } x _ { i } ( x _ { i } ^ { \top } K ^ { \top } Q x _ { j } )$$

where f ( j ) is

$$f ( j ) = \begin{cases} m _ { q _ { 1 } } & \text{for $j<=m_{q_{1}}$} \\ m _ { q _ { k } } & \text{for $m_{a_{k-1}}<j<=m_{q_{k}},k>1$} \\ j & \text{for $m_{q_{k}}<j<=m_{a_{k}},k>=1$}, \end{cases}$$

## 3.2 Convergence in Multi-turn dialogue learning

To simplify the model in Multi-turn dialogue learning, we analyze Linear regression, which is a classical machine learning problem. Given a set of vectors QA i = (( q j , a j )) (where j = m i -1 + 1 , ..., m i , i = 1 , ..., n, m 0 = 0 ), which contains a set of input-label pairs, the goal is to find an optimal weight vector w that minimizes the l2-loss:

$$L ( \mathbf w ) = \frac { 1 } { 2 m _ { n } } \sum _ { k = 1 } ^ { m _ { n } } \| \mathbf w \mathbf q _ { k } - a _ { k } \| _ { 2 } ^ { 2 }$$

The gradient of the loss is ∇ w L = 1 m n ∑ m n k =1 ( wq k -a k ) q ⊤ k , and a gradient descent algorithm with step size α follows the update rule:

$$\mathbf w ^ { ( l ) } = \mathbf w ^ { ( l - 1 ) } + \frac { \alpha } { m _ { n } } \sum _ { k = 1 } ^ { m _ { n } } ( a _ { k } - \mathbf w ^ { ( l - 1 ) } \mathbf q _ { k } ) \mathbf q _ { k } ^ { \top }$$

According to Von Oswald et al. [2023], the input is formulated as

$$X = ( x _ { i } ^ { ( 0 ) }, \dots, x _ { m _ { n } } ^ { ( 0 ) } ), \text{ where } x _ { j } ^ { ( 0 ) } = \binom { \mathbf q _ { j } } { a _ { j } }$$

and the parameter matrices of are:

$$K = Q = \left ( \begin{array} { c c } \mathbf I _ { d \times d } & 0 \\ 0 & 0 \end{array} \right ), \mathbf V = \left ( \begin{array} { c c } 0 _ { d \times d } & 0 \\ \mathbf W ^ { ( 0 ) } & 0 \end{array} \right ), \mathbf P = \frac { \alpha } { m _ { n } } \mathbf I,$$

where w (0) is an initial weight vector. Furthermore, there is workDing et al. [2023b] proves that multi-layer LSA under the construction of progresses identically to multi-step gradient descent. The prososition is that for a multi-layer LSA satisfying the construction 7 and with w (0) = 0 , if its input Z is formatted as 8, then its l-th layer output is x ( ) l j = ( q ⊤ j , δ ( ) l j ) ⊤ , where δ ( ) l j = a j -w ( ) l q j and w ( ) l is the l-th updated weight from the gradient descents update rule in 6. Since q j never changes , we can simplify it and focus only on δ ( ) l j , which is the last output coordinate of the j-th LSA-layer for l &gt; 0 ,

$$\delta _ { j } ^ { ( l ) } = \delta _ { j } ^ { ( l - 1 ) } - \frac { \alpha } { m _ { n } } \sum _ { k = 1 } ^ { m _ { n } } \delta _ { j } ^ { ( l - 1 ) } \mathbf q _ { k } \mathbf q _ { j } ^ { \top }$$

with δ (0) j = a j . Defining ˜ a ( ) l j = a j -δ (0) j and rearranging 9, for ˜ a (0) j = 0 and ∀ l &gt; 0 , we have:

$$\tilde { a } _ { j } ^ { ( l ) } = \tilde { a } _ { j } ^ { ( l - 1 ) } - \frac { \alpha } { m _ { n } } \sum _ { k = 1 } ^ { m _ { n } } ( a _ { k } - \tilde { a } _ { k } ^ { ( l - 1 ) } ) \mathbf q _ { k } \mathbf q _ { j } ^ { \top }$$

For prefix LM, 9 and 10 are the the update rules. Mainwhile, a causal LM applies auto-regressive attention throughout the entire sequence. Therefore, plugging the same K Q P V , , , into 2, the update rules of 9, and 10 become:

$$\delta _ { j } ^ { ( l ) } = \delta _ { j } ^ { ( l - 1 ) } - \frac { \alpha } { m _ { n } } \sum _ { k = 1 } ^ { j } \delta _ { j } ^ { ( l - 1 ) } \mathbf q _ { k } \mathbf q _ { j } ^ { \top }$$

$$\tilde { a } _ { j } ^ { ( l ) } = \tilde { a } _ { j } ^ { ( l - 1 ) } - \frac { \alpha } { m _ { n } } \sum _ { k = 1 } ^ { j } ( a _ { k } - \tilde { a } _ { k } ^ { ( l - 1 ) } ) \mathbf q _ { k } \mathbf q _ { j } ^ { \top }$$

For ISM, the update rules of 9, and 10 become:

Figure 2: Comparison between our ISM and ChatGLM when facing multi-turn dialogue data. ChatGLM applies bidirectional attention (blue frame) to prefix tokens (dialogue history) when generating answer (orange frame), our ISM applies alternate bidirectional and unidirectional attention to previous queries and answers when generating an answer. [M]:=[MASK], [S]:=[START].

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00539v1-figure-a24e20ab-12be-4411-960e-e05d70bdf3df.jpg)

$$\delta _ { j } ^ { ( l ) } = \delta _ { j } ^ { ( l - 1 ) } - \frac { \alpha } { m _ { n } } \sum _ { k = 1 } ^ { f ( j ) } \delta _ { j } ^ { ( l - 1 ) } \mathbf q _ { k } \mathbf q _ { j } ^ { \top }$$

$$\tilde { a } _ { j } ^ { ( l ) } = \tilde { a } _ { j } ^ { ( l - 1 ) } - \frac { \alpha } { m _ { n } } \sum _ { k = 1 } ^ { f ( j ) } ( a _ { k } - \tilde { a } _ { k } ^ { ( l - 1 ) } ) \mathbf q _ { k } \mathbf q _ { j } ^ { \top }$$

where f ( j ) is defined in 5.

It has been proved that the layer outputs of prefix LM have the same dynamics of multi-step gradient descent on a linear regression problem Ding et al. [2023b], which means the iterative weights w ( ) l converges to the stationary point w ∗ in a linear rate.

For causal LM, for each j , the iterative weights w ( ) l j converge to the stationary point w ∗ j , but w ∗ j may not converge to the the optimal solution w ∗ Ding et al. [2023b].

For j = m n -1 , · · · , m n in ISM, the iterative weights w ( ) l j converge to the stationary point w ∗ similar to prefix LLM. Thus, for m &gt; m n n -1 , ISM also has the same multi-step gradient descent.

## 3.3 ISM vs. Causal LLM

As we proofed in Section 3.2, ISM converges to the stationary point w ∗ in most situations ( m &gt; n m n -1 ) and causal LM may not converge to w ∗ . It indicates that in multi-turn dialogue learning, ISM will perform better than causal LM when the length of query from the last dialogue is longer than 1.

To apply our ISM to causal LLMs, we modify the causal mask as shown in Figure 1 a) into ISM as shown in Figure 1 c) to apply bidirectional attention on queries in dialogue history.

## 3.4 ISM vs. Prefix LLM

Compared with prefix LLM, the standout benefit of our ISM is its ability to reuse the KV cache across dialogue rounds, effectively reducing the generation latency.

KV Cache is a commonly used acceleration technique when LLMs perform generation. However, it's not reusable between dialogue rounds in prefix LLM. Given dialogue history h = [ P Q , 1 , A 1 , · · · , Q n -1 , A n -1 ] and current query Q n to generate A n . Assuming we have dialogue history's KV cache C h , for causal LLM and our ISM, we only need to compute the current query's KV cache C q and concatenate them to get the whole sequence's cache [ C h ; C q ] , so that we

Table 1: GPT-4 pairwise comparison results between two types of models and our corresponding ISM version in KdConv and NaturalConv.

| Base Model Type   | Causal     | Causal           | Causal   | Prefix     | Prefix           | Prefix   |
|-------------------|------------|------------------|----------|------------|------------------|----------|
| Models            | Qwen1.5-7b | Qwen1.5-7b (ISM) | Tie      | ChatGLM-6b | ChatGLM-6b (ISM) | Tie      |
| Win Rate          | 31.87%     | 33.59%           | 34.54%   | 36.02%     | 38.63%           | 25.35%   |

can generate A n with linear complexity. However, for prefix LLM, due to the bidirectional attention on prefix sequences as shown in Figure 1 b), the KV cache of dialogue history C h needs to be recomputed when computing KV cache of current query Q n . Therefore, the KV cache of dialogue history C h isn't reusable in prefix LLM.

To apply our ISM on prefix LLM, we choose ChatGLM as the base model. It designs a special 2D positional encoding method as shown in Figure 2 a). When facing multi-turn dialogue data, it applies bidirectional attention on prefix tokens (dialogue history) while generating answers. In contrast, our ISM applies alternate bidirectional and unidirectional attention on history queries and answers. Therefore, we follow GLM to apply the 2D position encoding for previous query and answer spans in Figure 2 b). Position 1 and 2 are utilized to encode the inter- and intra-span positions, respectively.

## 4 Experiments

## 4.1 Datasets

Daily Dialogue. Daily DialogueLi et al. [2017] is a human-written high-quality multi-turn conversation dataset. The dialogues in the dataset reflect daily communication way and cover various topics about daily life. It contains 13,118 dialogues, the average turns of each dialogue is 7.9.

KdConv. KdConv Zhou et al. [2020] is a Chinese knowledge-driven conversation dataset. It grounds the topics in multi-turn conversations to knowledge graphs. It contains 4.5K conversations from three domains (film, music, and travel), and 86K utterances with an average turn number of 19.0.

NaturalConv. NaturalConv Wang et al. [2021] is a Chinese multi-turn topic-driven conversation dataset, which allows the participants to chat anything they want as long as any element from the topic is mentioned. It contains 19.9K conversations from six domains, and 400K utterances with an average turn number of 20.1.

## 4.2 Implementation Details

For Daily Dialogue, we choose Llama2-7b as the base model and apply our ISM in it as described in Section 3.3. We call the modified model Llama2-7b (ISM). We finetune them on the train split with the same settings. Specifically, we utilize 4 NVIDIA A100 GPUs to full-finetune Llama2-7b and Llama2-7b (ISM) on Daily Dialogue's train split for 5 epochs, respectively. We train models with AdamW Loshchilov and Hutter [2018], the learning rate is set as 3e-5, global batch size is 64.

For KdConv and NaturalConv, we choose Qwen1.5-7b as the base model. As these two datasets are both in Chinese and topic-driven. We mix their train split to finetune Qwen1.5-7b and Qwen1.5-7b (ISM) and evaluate on the mixture of their test split. The training settings are the same as Llama2-7b's. In addition, to evaluate our ISM on prefix LLM, we apply ISM in ChatGLM as illustrated in Section 3.4 and utilize KdConv and NaturalConv to train ChatGLM-6b and ChatGLM-6b (ISM).

For evaluation, we follow previous works Zheng et al. [2024] to apply powerful GPT-4 as a stand-in for human judge. We design appropriate instructions to ask GPT-4 to compare the answers between the base model and our corresponding ISM version and to decide which answer is better or tie. Considering the position bias Zheng et al. [2024], we swap the position of answers and compute the average result. Specific prompts can be seen in Appendix A.2. In addition, we randomly select 200 samples for human evaluation to compare the consistency between GPT-4 and human judgement.

## 4.3 Quantitative Analysis

We report the GPT-4 pairwise comparison results in Table 1. For causal model, we compare Qwen 1.5-7b and Qwen 1.5-7b (ISM) on the mixture test split of KdConv and NaturalConv. We can find

Table 2: GPT-4 pairwise comparison results in Daily Dialogue.

| Base Model Type Models   | Causal    | Causal         | Causal   |
|--------------------------|-----------|----------------|----------|
|                          | Llama2-7b | Llama2-7b(ISM) | Tie      |
| Win Rate                 | 26.89%    | 33.53%         | 39.58%   |

Table 3: Pairwise comparison results between AntGLM-10b and AntGLM-10b (ISM). The Short Split contains samples with 1 and 2 rounds. The Long Split contains samples with 3 rounds or more.

| Base Model Type   | Prefix      | Prefix           | Prefix      | Prefix     | Prefix           | Prefix   |
|-------------------|-------------|------------------|-------------|------------|------------------|----------|
| Splits            | Short Split | Short Split      | Short Split | Long       | Split            | Long     |
| Models            | AntGLM-10B  | AntGLM-10B (ISM) | Tie         | AntGLM-10B | AntGLM-10B (ISM) | Tie      |
| Win Rate          | 19.00%      | 19.20%           | 61.80%      | 13.70%     | 22.37%           | 63.93%   |

that the win rate of Qwen 1.5-7b (ISM) exceeds Qwen 1.5-7b by 1.72%. For prefix model, we compare ChatGLM-6b and ChatGLM-6b (ISM). Considering the inefficient training process of prefix LLM in multi-turn dialogue dataset, we expand the training data for ChatGLM. Specifically, given a multi-turn dialogue sample, we expand it into five samples manually. For instance, given a sample with 15 turns of dialogue, we will expand it into samples with 3, 6, 9, 12, and 15 turns. We can find the win rate of ChatGLM-6b (ISM) outperforms ChatGLM-6b by 2.61%. It demonstrates that our ISM can improve both causal LLM and prefix LLM's generation quality in multi-turn dialogues.

Similarly, we compare Llama2-7b and Llama2-7b (ISM) on Daily Dialogue, the results are reported in Table 2. We observe that Llama2-7b (ISM) demonstrates substantial performance enhancements compared to the base model, achieves 5.64% increasement in win rate.

Further, we apply ISM in our private model AntGLM-10BLi et al. [2023] to evaluate the performance. AntGLM has the same architecture as GLM, belongs to prefix LLM. We evaluate AntGLM and AntGLM (ISM) in our private AntEval dataset. AntEval dataset is collected from real-world conversations about insurance and medical. It contains 1219 samples in total. Including 944 samples with one turn and 56 samples with two turns, we call these as Short Split. In addition, other 219 samples contain conversation over 2 rounds, with an average turn number of 4.6, we call these as Long Split. We evaluate AntGLM and AntGLM (ISM) in these two splits, respectively, the results are shown in Table 3. We can observe that AntGLM (ISM) achieves 8.67% win rate over AntGLM in the Long Split, while the improvement is only 0.2% in the Short Split. It illustrates that our approach exhibits a pronounced advantage in longer dialogue samples.

## 4.3.1 Human Evaluation

Designing instructions and applying GPT-4 to evaluate LLM's performance has been widely adopted, but it's still a subjective evaluation method. Therefore, we randomly select 100 samples from Daily Dialogue and KdConv&amp;NaturalConv separately for human evaluation to compare the judgement consistency between human and GPT-4. As shown in Figure 3, we observe a high level of consistency between human and GPT-4 evaluation, 81% and 78% in Figure 3 a) and b), respectively. Furthermore, it is notable that the majority of inconsistencies are centered around determining whether there is a tie or not. It demonstrates that the evaluation results from GPT-4 are relatively reliable.

Figure 3: Consistency comparison between human evaluation and GPT-4 evaluation.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00539v1-figure-1811814c-af86-4612-849c-fec899e6aac8.jpg)

Figure 4: Time-to-first-token (TTFT) latency comparison between AntGLM and AntGLM (ISM). AntGLM (ISM) reduces TTFT significantly by reusing the KV cache across dialogue rounds.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00539v1-figure-b24d87e3-6890-45d4-bc5f-ef5d53a669a0.jpg)

## ####Based on the above examples, determine the following results####

Dialogue history:

User 1:

Are you busy tomorrow morning ?

User 2:

I'm free . What's up ?

User 1:

Someone has to pick up the boss at the airport .

Chatbot 1:

You mean I have to go to the airport?

Chatbot 2:

You mean it's not you?

## GPT-4 Judgement:

Explanation:

In  this  scenario,  User  2  responds  positively  to  being  free  tomorrow  morning, indicating a willingness to help. Chatbot 1's response is more proactive and assumes a  role  of  responsibility  by  asking  if  they  have  to  go  to  the  airport.  This  shows initiative and readiness to assist without waiting for further instructions. Chatbot 2's response,  while  also  showing  readiness,  lacks  the  proactive  nature  of  Chatbot  1's response and instead seeks confirmation if they are the one being asked to go.

Judgement:

1

## ####Based on the above examples, determine the following results####

Dialogue history:

User 1:

This  position  demands  a  higher  writing  ability  ,  so  please  say  something  about your writing ability .

Chatbot 1:

I have been writing for many years, and I have always been interested in creative writing.

Chatbot 2:

I  have  been  writing  for  3  years,  and  I  have  won  the  first  prize  in  many  writing contests at college and university level.

## GPT-4 Judgement:

Explanation:

Bot 2's response is better. It directly addresses the question about writing ability and provides specific evidence to support its claim, demonstrating both experience and achievement in writing contests. Bot 1's response, while indicating an interest in creative writing, lacks the specificity and  evidence  provided  by  Bot  2. Therefore,  Bot  2's  response  better  meets  the  criteria  of  being  useful,  accurate, reasonable, contextually relevant, and fluent in the given context.

Judgement:

2

Figure 5: Two cases are shown, each one contains the dialogue history, generated responses from Llama2-7b (Chatbot 1), Llama2-7b(ISM) (Chatbot 2), and GPT-4's judgement.

## 4.4 Latency Analysis

As described in Section 1, prefix LLM suffers from high generation latency in multi-turn dialogues due to the inability to reuse the KV cache across dialogue rounds. In this section, we conduct experiments to compare the time-to-first-token (TTFT) latency between AntGLM-10B and AntGLM10B (ISM). Specifically, we construct a simulated dialogue sample consisting of 20 turns, with each turn having a sequence length of 200. The query length is 40, and the answer length is 160. We deploy them on 2 NVIDIA A10 GPUs and repeatedly call the inference API 10 times to calculate the average TTFT. As shown in Figure 4, it's evident that the TTFT latency of AntGLM-10B surges quickly as the input sequence length expands. Conversely, the TTFT of AntGLM-10B (ISM) increases much slower due to the reuse of the KV cache. Specific analysis can be seen in Appendix A.1

## 4.5 Case Study

In Figure 5, we showcase two cases including dialogue sample, generated responses and GPT-4 judgement results. We can find that GPT-4 follows our instructions well to make judgements. In the first case, GPT-4 not only considers the dialogue history, but also takes account of User 2's "willingness" and "responsibility". Cases with longer dialogue history can be seen in Appendix A.3.

## 4.6 Limitation

Current ISM is applied to existing causal or prefix base models for finetuning. The data utilized for finetuning is relatively small compared with the data used during their pretraining stage. The performance of our ISM in the pretraining stage needs to be explored.

## 5 Conclusion and Future Work

In this paper, we propose a novel attention masking scheme called Intermittent Semi-working Mask (ISM) to achieve the high generation quality and low generation latency in multi-turn dialogue. Specifically, we apply alternate bidirectional and unidirectional attention on queries and answers in the dialogue history. In this way, our ISM is able to maintain the advantages of both prefix LLM and causal LLM simultaneously. Extensive experiments illustrate that our ISM achieves significant performance. In the future, we will try to apply our ISM to the pretraining stage of LLM and delve deeper into exploring its advantages.

## References

- J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.
- R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403 , 2023.
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.
- W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/ .
- A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1-113, 2023.
- J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , pages 4171-4186. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1423. URL https://doi.org/10.18653/v1/n19-1423 .
- N. Ding, Y. Chen, B. Xu, Y. Qin, Z. Zheng, S. Hu, Z. Liu, M. Sun, and B. Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233 , 2023a.
- N. Ding, T. Levinboim, J. Wu, S. Goodman, and R. Soricut. Causallm is not optimal for in-context learning. In The Twelfth International Conference on Learning Representations , 2023b.
- L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, and H.-W. Hon. Unified language model pre-training for natural language understanding and generation. In Proceedings of the 33rd International Conference on Neural Information Processing Systems , pages 13063-13075, 2019.
- Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 320-335, 2022.
- J. Fu, S.-K. Ng, Z. Jiang, and P. Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166 , 2023a.
- Y. Fu, H. Peng, T. Khot, and M. Lapata. Improving language model negotiation with self-play and in-context learning from ai feedback. arXiv preprint arXiv:2305.10142 , 2023b.
- D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020.
- Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, Y. Fu, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems , 36, 2024.
- J. D. M.-W. C. Kenton and L. K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT , pages 4171-4186, 2019.
- M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, 2020.

- Q. Li, X. Yang, H. Wang, Q. Wang, L. Liu, J. Wang, Y. Zhang, M. Chu, S. Hu, Y. Chen, et al. From beginner to expert: Modeling medical knowledge into general llms. arXiv preprint arXiv:2312.01040 , 2023.
- Y. Li, H. Su, X. Shen, W. Li, Z. Cao, and S. Niu. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957 , 2017.
- C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74-81, 2004.
- Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.
- I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.
- L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.
- K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311-318, 2002.
- A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018.
- C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1-67, 2020.
- Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, J. Wei, X. Wang, H. W. Chung, D. Bahri, T. Schuster, S. Zheng, et al. Ul2: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations , 2022.
- H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023a.
- H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023b.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.
- J. Von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning , pages 35151-35174. PMLR, 2023.
- H. Wang, R. Wang, F. Mi, Y. Deng, Z. Wang, B. Liang, R. Xu, and K.-F. Wong. Cue-CoT: Chain-of-thought prompting for responding to in-depth dialogue questions with LLMs. In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 12047-12064, Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.806. URL https://aclanthology.org/ 2023.findings-emnlp.806 .
- X. Wang, C. Li, J. Zhao, and D. Yu. Naturalconv: A chinese dialogue dataset towards multi-turn topic-driven conversation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 14006-14014, 2021.
- C. Xu, D. Guo, N. Duan, and J. McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.01196 , 2023.

- R. Zhang, S. Frei, and P. L. Bartlett. Trained transformers learn linear models in-context. arXiv preprint arXiv:2306.09927 , 2023.
- L. Zheng, W.-L. Chiang, Y. Sheng, T. Li, S. Zhuang, Z. Wu, Y. Zhuang, Z. Li, Z. Lin, E. Xing, et al. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. arXiv preprint arXiv:2309.11998 , 2023.
- L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36, 2024.
- H. Zhou, C. Zheng, K. Huang, M. Huang, and X. Zhu. KdConv: A Chinese multi-domain dialogue dataset towards multi-turn knowledge-driven conversation. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7098-7108, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.635. URL https://aclanthology.org/2020.acl-main.635 .

## A Appendix

## A.1 Inference Acceleration Estimation on ISM

ISM inference can be accelerated by reusing the KV Cache across turns. We analyze the computational complexity of transformers before and after acceleration.

Given the following variable definitions for a Transformer model:

- · b = batch size
- · s = input sequence length
- · n s = new input sequence length
- · h = hidden dimension
- · h n = number of attention heads
- · h h = dimension per attention head, where h h = h h n

## A.1.1 Before Acceleration

The FLOPs for the self-attention layer ( F self-attention ) can be estimated as follows:

- · Computing queries, keys, values:

$$F _ { \mathfrak { q } \mathbf v } = 3 \cdot 2 \cdot b \cdot s \cdot h ^ { 2 } = 6 \cdot b \cdot h ^ { 2 } \cdot s$$

- · Computing attention scores:

$$F _ { \text{as} } = 2 \cdot 2 \cdot b \cdot h _ { n } \cdot s \cdot h _ { h } \cdot s = 4 \cdot b \cdot h \cdot s ^ { 2 }$$

- · Computing weighted sum:

$$F _ { \text{ws} } = 2 \cdot b \cdot h ^ { 2 } \cdot s$$

- · The total FLOPs for self-attention is therefore:

$$F _ { \text{self-attention} } = F _ { \text{qkv} } + F _ { \text{as} } + F _ { \text{ws} } = 4 \cdot b \cdot h \cdot s ^ { 2 } + 8 \cdot b \cdot h ^ { 2 } \cdot s \quad \quad ( 1 8 )$$

The FLOPs for the feed-forward network layer ( F ffn ) is:

$$F _ { \text{if} } = 1 6 \cdot b \cdot h \cdot s ^ { 2 }$$

Thus, the total FLOPs for transformers ( F transformers ) combines both the self-attention and feed-forward FLOPs:

$$F _ { \text{transformers} } = F _ { \text{self-attention} } + F _ { \text{ffn} } = 4 \cdot b \cdot h \cdot s ^ { 2 } + 2 4 \cdot b \cdot h ^ { 2 } \cdot s$$

Overall, the computational complexity of the Transformer is quadratically related to the input sequence length

## A.1.2 After Acceleration

By reusing the KV cache from the previous turn of dialogue, we can trim the input sequence to the new uncached segment. In this way, it is equivalent to replacing one s(input sequence length) in the aforementioned computation process with n\_s (new input sequence length), thus the total computational complexity after inference acceleration is:

$$F _ { \text{acelarated-transformers} } = 4 \cdot b \cdot h \cdot s * n _ { s } + 2 4 \cdot b \cdot h ^ { 2 } \cdot n _ { s } \quad \quad ( 2 1 )$$

When n s is much smaller than s and h , n s can be neglected. Therefore, the total computational FLOPs is approximately 4 · b · h · s +24 · b · h 2 , where the relationship between computational cost and input sequence length becomes a linear polynomial. Thus, the computational cost is optimized from a quadratic polynomial of input sequence length to a linear polynomial.

Given an multi-turn dialogue between two user (User 1 and User 2) and two responses generated from two AI Chatbots (Chatbot 1 and Chatbot 2). Please act as an impartial judge to evaluate the quality of two responses and judge which one is a better response for User2, the criteria are as follow:

- 1. Assess the current response with reference to the dialogue history and the current question.
- 2. The model's response should be useful, accurate, reasonable, contextually relevant, and fluent.
- 3. If the user's question in the current round does not have a clear intent, the model's response should exhibit a degree of creativity or be able to guide the conversation forward.
- 4. Ensure that the order of responses does not influence your decision.
- 5. Do not judge the quality of the responses solely based on their length.
- 6. To be as objective as possible.

Compare the responses of both bots and provide a brief explanation. After providing your explanation, output your final judgment in the following format strictly: if Bot 1's response is better, output 1; if Bot 2's response is better, output 2; if both are about equal or incorrect, output 0. Ensure that you only output 0, 1, or 2.

## ####Case 1####

Dialogue history:

User 1:

Oh , so Dalian is hotter than New York now .

User 2:

Not hot but warm . Very comfortable .

User 1:

Are you staying at the Hilton ?

Chatbot 1:

No, I'm staying at the Friendship Hotel. Chatbot 2:

No, I'm at the New York. Evaluation process:

## ####Case 2####

Dialogue history:

User 1:

Room service , is there anything I can do for you ? User 2:

Yes , I am afraid there are not enough outlets .

User 1:

Well , we have 6 for each room .

Chatbot 1:

6 outlets? I need 3.

Chatbot 2:

But it's still not enough for me. Evaluation process:

## Explanation:

The  second  response  is  more  reasonable.  From  the dialogue history, we can know the customer think the outlets are not enough. So when the server says it has 6  for  each  room,  Chatbot1's  answer  "I  need  3"  is contradictory to the previous reply.

Judgement:

## Explanation:

The  first  response  is  better  because  it  is  more  clear and directly answers the question. In addition, according to the history answer we can speculate that the person is more likely in Dalian rather New York. Judgement:

1

2

####Based on the above examples, determine the following results####

......

Figure 6: Prompt for GPT-4. Instruction contains the task description and evaluation criteria. We provide 2 cases and corresponding human-written annotations for few-shot learning.

## A.2 Prompt

We design instruction prompt carefully as shown in Figure 6 to leverage GPT-4 in assessing the quality of responses from the models to be compared. The whole prompt can be divided into three parts, the first part describe the task and the specific evaluation criteria, the second part contains two cases and corresponding human written evaluation result for few-shot learning, the third part contains sample to be judged.

## A.3 Cases

We showcase two cases with longer dialogue history and corresponding GPT-4 judgements, shown in Figure 7 and 8. From Figure 7 we can observe that facing dialogues with longer history, our ISM is able to attend history context better and generate more reasonable response.

## ####Based on the above examples, determine the following results####

Dialogue history:

User 1:

Good morning , Angela , how have you been lately ?

User 2:

Morning , Michael . I ' Ve been very busy lately . One of our other vendors is going out of business and I ' Ve been searching for a suitable replacement .

User 1:

Well , rest assured that you can count on us to be here for the long run , sitdown.Coffee ?

User 2:

No , thanks . I ' Ve been trying to cut down on the caffeine .

User 1:

Haha , I could never do that . I ' d be a zombie if I didn ' t have my morning coffee fix . Let ' s get down to business then .

User 2:

Yes . I ' Ve come to talk with you about ordering the eight megapixel cameras for our new methone . The demand for phone cameras is growing , and Pear has been falling behind in the market .

## User 1:

That ' s great ! I ' m glad to hear that Pear has finally jumped on the bandwagon . Right now our contract is for the five megapixel cameras . Is Pear still interested in having those ? Chatbot 1:

Yes, but we ' re also interested in having a 10 megapixel option.

Chatbot 2:

No, we ' Ve decided to upgrade to eight megapixels. We were hoping that by making the cameras better, we could also make the phones better.

Evaluation process:

## GPT-4 Judgement:

Explanation:

Both  Chatbot  1  and  Chatbot  2  provide  contextually  relevant  responses,  showing  an  understanding  that  Pear  is looking to improve their product. Chatbot 1's response introduces a new specification (10 megapixels) that was not previously mentioned by User 2. While this indicates a potential interest in further upgrades, which could be seen as a  display  of  creativity  or  forward-thinking,  it  does  not  directly  answer  User  1's  question  regarding  the  current contract. Instead, it introduces ambiguity as to whether Pear is no longer interested in the five-megapixel cameras or if they are considering multiple options.

Chatbot 2's response directly acknowledges the shift from five to eight-megapixel cameras, demonstrating a clear intent to upgrade based on market demands. This response directly addresses User 1's question about the contract for the five-megapixel cameras, making it clear that Pear's interest has shifted to the higher quality cameras.

For the purpose of a direct, useful, and accurate response to the question posed by User 1, Chatbot 2's answer is more aligned with the dialogue's progression and User 2's previously stated intentions.

Judgement:

2

Figure 7: Case with longer dialogue history, includs the dialogue history, generated responses from Llama2-7b (Chatbot 1) and Llama2-7b(ISM) (Chatbot 2), and GPT-4's judgement. Llama2-7b (ISM) wins.

## ####Based on the above examples, determine the following results####

Dialogue history:

User 1:

Have you got any experience in advertising ?

User 2:

Yes , I have been working in the Public Relations Section of a company in the past two years . I plan the advertising campaign and cooperate the work of artists and typographers . Sometimes I have to do the work of a specialist when there ' s something urgent .

User 1:

Do you have any practical experience as a secretary ?

User 2:

Yes , sir . I acted as a secretary in an insurance company .

User 1:

Would you tell me the essential qualities a secretary should maintain ?

User 2:

Well , to begin with , I would say she needs to be diligent , and the second point is that she has to do a lot of things on her own initiative . Finally , she can make notes in shorthand and types and has the skills in report writing , summary writing , keep minutes at meetings , and so on . Most important of all is that she seems to have a better memory than average .

User 1:

Have you had any experience with computer ?

User 2:

Yes , I studied in a computer training program , and can process data through the computer .

User 1:

That's fine . What about operating the fax and duplicator ?

User 2:

I can handle them without any trouble

User 1:

What have you learned from jobs you have held ?

Chatbot1:

I have learned to cooperate with my colleagues and to be responsible for my work.

Chatbot2:

I have learned to be patient, professional and gained a lot of practical experience in work.

Evaluation process:

## GPT-4 Judgement:

Explanation:

Both Chatbot 1 and Chatbot 2 provide responses that are relevant to the question and exhibit qualities desirable in a job candidate. Chatbot 1 emphasizes cooperation and responsibility, while Chatbot 2 highlights patience, professionalism, and  practical  experience.  Both  responses  are  reasonable  and  could  be  considered  positive  attributes  in  a  potential employee.

Judgement:

0

Figure 8: Case with longer dialogue history, includs the dialogue history, generated responses from Llama2-7b (Chatbot 1) and Llama2-7b(ISM) (Chatbot 2), and GPT-4's judgement. The two are tied.
Mingcong Lu Jiangcai Zhu Wang Hao Zheng Li Shusheng Zhang Kailai Shao Chao Chen Nan Li Feng Wang Xin Lu
Intermittent Semi-working Mask: A New Masking Paradigm for LLMs
Multi-turn dialogues are a key interaction method between humans and Large
Language Models (LLMs), as conversations extend over multiple rounds, keeping
LLMs' high generation quality and low latency is a challenge. Mainstream LLMs
can be grouped into two categories based on masking strategy: causal LLM and
prefix LLM. Several works have demonstrated that prefix LLMs tend to outperform
causal ones in scenarios that heavily depend on historical context such as
multi-turn dialogues or in-context learning, thanks to their bidirectional
attention on prefix sequences. However, prefix LLMs have an inherent
inefficient training problem in multi-turn dialogue datasets. In addition, the
attention mechanism of prefix LLM makes it unable to reuse Key-Value Cache (KV
Cache) across dialogue rounds to reduce generation latency. In this paper, we
propose a novel masking scheme called Intermittent Semi-working Mask (ISM) to
address these problems. Specifically, we apply alternate bidirectional and
unidirectional attention on queries and answers in the dialogue history. In
this way, ISM is able to maintain the high quality of prefix LLM and low
generation latency of causal LLM, simultaneously. Extensive experiments
illustrate that our ISM achieves significant performance.