QAConv: Question Answering on Informative Conversations

This paper introduces QAConv, a new question answering (QA) dataset that uses conversations as a knowledge source. We focus on informative conversations, including business emails, panel discussions, and work channels. Unlike open-domain and task-oriented dialogues, these conversations are usually long, complex, asynchronous, and involve strong domain knowledge. In total, we collect 34,608 QA pairs from 10,259 selected conversations with both human-written and machine-generated questions. We use a question generator and a dialogue summarizer as auxiliary tools to collect and recommend questions. The dataset has two testing scenarios: chunk mode and full mode, depending on whether the grounded partial conversation is provided or retrieved. Experimental results show that state-of-the-art pretrained QA systems have limited zero-shot performance and tend to predict our questions as unanswerable. Our dataset provides a new training and evaluation testbed to facilitate QA on conversations research.

Introduction
Having conversations is one of the most common ways to share knowledge and exchange information. Recently, many communication tools and platforms are heavily used with the increasing volume of remote working, and how to effectively retrieve information and answer questions based on past conversations becomes more and more important. In this paper, we focus on QA on conversations such as business emails (e.g., Gmail), panel discussions (e.g., Zoom), and work channels (e.g., Slack). Different from daily chit-chat (Li et al., 2017) and task-oriented dialogues (Budzianowski et al., 2018), these conversations are usually long, complex, asynchronous, multi-party, and involve strong domain knowledge. We refer to them as informative conversations and an example is shown in Figure 1.
However, QA research mainly focuses on document understanding (e.g., Wikipedia) not dialogue understanding, and dialogues have significant differences with documents in terms of data format and wording style, and important information is scattered in multiple speakers and turns (Wolf et al., 2019b;Wu et al., 2020). Moreover, existing work related to QA and conversational AI focuses on conversational QA (Reddy et al., 2019;Choi et al., 2018) instead of QA on conversations. Conversational QA has sequential dialogue-like QA pairs that are grounded on a short document paragraph, but what we are more interested in is to have QA pairs grounded on conversations, treating past dialogues as a knowledge source.
QA on conversation has several unique challenges: 1) information is distributed across multiple speakers and scattered among dialogue turns; 2) Harder coreference resolution problem of speakers and entities, and 3) missing supervision as no training data in such format is available. The most related work to ours is the FriendsQA dataset (Yang and Choi, 2019) and the Molweni dataset (Li et al., 2020). However, the former is built on chit-chat transcripts of TV shows with only one thousand dialogues, and the latter has short conversations in a specific domain (i.e., Ubuntu). The dataset comparison is shown in Table 1.
Therefore, we introduce QAConv dataset, sampling 10,259 conversations from email, panel, and channel data. The longest dialogue sample in our data has 19,917 words (or 32 speakers), coming from a long panel discussion. We segment long conversations into shorter conversational chunks to collect human-written (HW) QA pairs or to modify machine-generated (MG) QA pairs from Amazon Mechanical Turk (AMT). We train a multi-hop question generator and a dialogue summarizer to generate QA pairs. We use QA models to identify uncertain samples and conduct an additional human verification stage. The data collection flow is shown in Figure 1. In total, we collect 34,608 QA pairs.
We construct two testing scenarios: 1) In the chunk mode, a conversational chunk is provided to answer questions, similar to the SQuAD dataset (Rajpurkar et al., 2016); 2) In the full mode, a conversational-retrieval stage is required before answering questions, similar to the open-domain QA dataset (Chen and Yih, 2020). We explore several state-of-the-art QA models such as the span extraction RoBERTa-Large model (Liu et al., 2019) trained on SQuAD 2.0 dataset, and the generative UnifiedQA model (Khashabi et al., 2020) trained on eight different QA datasets and showed its generalization ability to 12 unseen QA corpora. We investigate the statistic-based BM25 (Robertson et al., 1994) retriever and the neural-based dense passage retriever (Karpukhin et al., 2020) trained on Wikipedia (DPR-wiki). We show zero-shot and finetuning performances in both modes and conduct improvement study and error analysis.
The main contributions of our paper are threefold: 1) QAConv provides a new testbed for QA on informative conversations including emails, panel discussions, and work channels. We show the po-tential of treating long conversations as a knowledge source, and point out a performance gap between QA on documents and QA on conversations; 2) We incorporate question generation (QG) model into the QA data collection, and we show the effectiveness of such approach in human evaluation. 3) We introduce chunk mode and full mode settings for QA on conversations, and our training data enables existing QA models to perform better on dialogue understanding.

QAConv Dataset
Our dataset is collected in four stages: 1) selecting and segmenting informative conversations, 2) generating question candidates by QG models, 3) crowdsourcing question-answer pairs on those conversations/questions, and 4) conducting quality verification and data splits.

Selection and Segmentation
Full data statistics are shown in Table 2. First, we use the British Columbia conversation corpora (BC3) (Ulrich et al., 2008) and the Enron Corpus (Klimt and Yang, 2004) to represent business email use cases. The BC3 is a subset of the World Wide Web Consortium's (W3C) sites that are less technical. We sample threaded Enron emails   One of the main challenges in our dataset collection is the length of input conversations and thus resulting in very inefficient for crowd workers to work on. For example, on average there are 13,143 words per dialogue in the Court dataset, and there is no clear boundary annotation in a long conversation of a Slack channel. Therefore, we segment long dialogues into short chunks by a turn-based buffer to assure that the maximum number of tokens in each chunk is lower than a fixed threshold, i.e., 512. For the Slack channels, we use the disentanglement script from (Chatterjee et al., 2020) to split channel messages into separated conversa-tional threads, then we either segment long threads or combine short threads to obtain the final conversational chunks.

Question Generation
Synthetic dataset construction has been shown to improve robustness (Gupta et al., 2021) and improve the complexity of test sets (Feng et al., 2021). We leverage a question generator and a dialogue summarizer to generate and recommend some questions to workers. We train a T5-Base (Raffel et al., 2019) model on HotpotQA (Yang et al., 2018), which is a QA dataset featuring natural and multihop questions, to generate questions for our conversational chunks. By the second hypothesis, we first train a BART (Lewis et al., 2020) summarizer on News (Narayan et al., 2018) and dialogue summarization corpora (Gliwa et al., 2019) and run QG models on top of the generated summaries.
We filter out generated questions that a QA model can predict the same answers we used in our QG model, which we hypothesize that these questions could be easy questions that we would like to avoid. Note that our QG model has grounded answers since it is trained to generate questions by giving a text context and an extracted entity. We hypothesize that these questions are trivial questions in which answers can be easily found, and thus not interesting for our dataset. Examples of   our generated multi-hop questions are shown in the Appendix (Table 18).

Crowdsourcing QA Pairs
We use two strategies to collect QA pairs, human writer and machine generator. We first ask crowd workers to read partial conversations, and then we randomly assign two settings: 1) writing QA pairs themselves or 2) selecting one recommended machine-generated question to answer. We apply several on-the-fly constraints to control the quality of the collected QA pairs: 1) questions should have more than 6 words with a question mark in the end; 2) questions and answers cannot contain firstperson and second-person pronouns (e.g., I, you, etc.); 3) answers have to be less than 20 words , and 4) all words have to appear in source conversations. We randomly select four MG questions from our question pool and ask crowd workers to answer one of them, without providing any potential answers. They are allowed to modify questions if necessary. To collect unanswerable questions, we ask crowd workers to write questions with at least three entities mentioned in the given conversations but they are not answerable. We pay crowd workers roughly $8-10 per hour, and the average time to read and write one QA pair is approximately 4 minutes.

Quality Verification and Data Splits
We design a filter mechanism based on different potential answers: human writer's answers, answer from existing QA models, and QG answers. If all the answers have a pairwise fuzzy matching ratio (FZ-R) scores 2 lower than 75%, we then run another crowdsourcing round and ask crowd workers to select one of the following options: A) the QA pair looks good, B) the question is not answerable, C) the question has a wrong answer, and D) the question has a right answer but I prefer another answer. We run this step on around 40% samples which are uncertain. We filter the questions of the (C) option and add answers of the (D) option into the ground truth. In questions marked with option (B), we combine them with the unanswerable questions that we have collected. In addition, we include 1% random questions (questions that are sampled from other conversations) to the same batch of data collection as a qualification test. We filter crowd workers' results if they fail to indicate such a question as an option (B). Finally, we split the data into 27,287 training samples, 3,660 validation samples, and 3,661 testing samples. There are 4.7%, 5.1%, 4.8% unanswerable questions in train, validation, and test split, respectively.

QA Analysis
In this section, we analyze our collected questions and answers. We first investigate question type distribution and we compare human-written questions and machine-generated questions. We then analyze answers by an existing named-entity recognition (NER) model and a constituent parser.

Question Analysis
Question Type. We show the question type tree map in Figure 2 and the detailed comparison with other datasets in Table 3. In QAConv, the top 5 question types are what-question (29%), which-question (27%), how-question (12%), whoquestion (10%), and when-question (6%). Comparing to SQuAD 2.0 (49% what-question), our dataset have a more balanced question distribution. The question distribution of unanswerable questions is different from the overall distribution. The top 5 unanswerable question types are what-question (45%), why-question (15%), howquestion (12%), which-question (10%), and whenquestion (8%). Human Writer v.s. Machine Generator. As shown in Table 4, there are 41.7% questions which are machine-generated questions. Since we still give crowd workers the freedom to modify questions if necessary, we cannot guarantee these questions are unchanged. We find that 33.56% of our recommended questions have not been changed (100% fuzzy matching score) and 19.92% of them are slightly modified (81%-99% fuzzy matching score). To dive into the characteristics and differences of these two question sources, we further conduct the human evaluation by sampling 200 conversation chunks randomly. We select chunks that have QG questions unchanged (i.e., sampling from the 33.56% QG questions). We ask three annotators to first write an answer to the given question and conversation, then label fluency (how fluent  and grammatically correct the question is, from 0 to 2), complexity (how hard to find an answer, from 0 to 2), and confidence (whether they are confident with their answer, 0 or 1). More details of each evaluation dimension (Section A.4) and performance difference (Table 12) are shown in the Appendix. The results in Table 4 indicate that QG questions are longer, more fluent, more complex, and crowd workers are less confident that they are providing the right answers. This observation further confirmed our hypothesis that the question generation strategy is effective to collect harder QA examples.

Answer Analysis
Following Rajpurkar et al. (2016), we used Part-Of-Speech (POS) (Kitaev and Klein, 2018) and Spacy NER taggers to study answers diversity. Firstly, we use the NER tagger to assign an entity type to the answers. However, since our answers are not necessary to be an entity, those answers without entity tags are then pass to the POS tagger, to extract the corresponding phrases tag. In Table 5, we can see that Noun phrases make up 30.4% of the data; followed by People, Organization, Dates, other numeric, and Countries; and the remaining are made up of clauses and other types. Full category distribution is shown in the Appendix ( Figure 3). Note that there are around 1% of answers in our dataset are coming from multiple source text spans (examples are shown in Appendix Table 17).

Chunk Mode and Full Mode
The main difference between the two modes is whether the conversational chunk we used to collect QA pairs is provided or not. In the chunk mode, our task is more like a traditional machine reading comprehension task that answers can be found (or cannot be found) in a short paragraph, usually less than 500 words. In the full mode, on the other hand, we usually need an information retrieval stage before the QA stage. For example, in the Natural Question dataset (Kwiatkowski et al., 2019), they split Wikipedia into millions of passages and retrieve the most relevant one to answer.  We define our full mode task with the following assumptions: 1) for the email and panel data, we assume to know which dialogue a question is corresponding to, that is, we only search chunks within the dialogue instead of all the possible conversations. This is simpler and more reasonable because each conversation is independent; 2) for slack data, we assume that we only know which channel a question belongs to but not the corresponding thread, so the retrieval part has to be done in the whole channel. Although chunk mode may be a better way to evaluate the ability of machine reading comprehension, the full mode is more practical as it is close to our setup in the real world.

State-of-the-art Baselines
There are two categories of question answering models: span-based extractive models which predict answers' start and end positions, and free-form text generation models which directly generate answers token by token. All the state-of-the-art models are based on large-scale language models, which are first pretrained on the general text and then finetuned on other QA tasks. We evaluate all of them on both zero-shot and finetuned settings (further finetuned on the QAConv training set), and both chunk mode and full mode with retrievers. In addition, we run these models on the Molweni (Li et al., 2020) dataset for comparison and find out our baselines outperform the best-reported model, DADgraph (Li et al., 2021a) model, which used expensive discourse annotation on graph neural network. We show the Molweni results in the Appendix (Table 11).

Span-based Models
We use several models finetuned on the SQuAD 2.0 dataset as span extractive baselines. We use uploaded models from huggingface (Wolf et al., 2019a) library. DistilBERT (Sanh et al., 2019) is a knowledge-distilled version with 40% size reduction from the BERT model, and it is widely used in mobile devices. The BERT-Base and RoBERTa-Base (Liu et al., 2019) models are evaluated as the most commonly used in the research community. We also run the BERT-Large and RoBERTa-Large models as stronger baselines. We use the wholeword masking version of BERT-Large instead of the token masking one from the original paper since it performs better.

Free-form Models
We run several versions of UnifiedQA models (Khashabi et al., 2020) as strong generative QA baselines. UnifiedQA is based on T5 model (Raffel et al., 2019), a language model that has been pretrained on 750GB C4 text corpus. UnifiedQA further finetuned T5 models on eight existing QA corpora spanning four diverse formats, including extractive, abstractive, multiple-choice, and yes/no questions. It has achieved state-of-the-art results on 10 factoid and commonsense QA datasets. We finetune UnifiedQA on our datasets with T5-Base, T5-Large size, and T5-3B. We report T5-11B size for the zero-shot performance.

Retrieval Models
Two retrieval baselines are investigated in this paper: BM25 and DPR-wiki (Karpukhin et al., 2020). The BM25 retriever is a bag-of-words retrieval function weighted by term frequency and inverse document frequency. The DPR-wiki model is a BERT-based dense retriever model trained for opendomain QA tasks, learning to retrieve the most relevant Wikipedia passage.

Computational Details
We train most of our experiments on 2 V100 NVIDIA GPUs with a batch size that maximizes their memory usage, except T5-3B we train on four A100 NVIDIA GPUs with batch size 1 with several parallel tricks, such as fp16, sharded_ddp and deepseep library. We train 10 epochs for all T5 models and 5 epochs for all BERT-based models. We release hyper-parameter setting and trained models to help reproduce baseline results.

Evaluation Metrics
We follow the standard evaluation metrics in the QA community: exact match (EM) and F1 scores. The EM score is a strict score that predicted answers have to be the same as the ground truth    answers. The F1 score is calculated by tokens overlapping between predicted answers and ground truth answers. In addition, we also report the FZ-R scores, which used the Levenshtein distance to calculate the differences between sequences. We follow Rajpurkar et al. (2016) to normalize the answers in several ways: remove stop-words, remove punctuation, and lowercase each character. We add one step with the num2words and word2number libraries to avoid prediction difference such as "2" and "two".

Chunk Mode
As the chunk mode results on the test set shown in Table 6, UnifiedQA T5 models, in general, outperform BERT/RoBERTa models in the zero-shot setting, and the performance increases as the size of the model increases. This observation matches the recent trend that large-scale pretrained language model finetuned on aggregated datasets of a specific downstream task (e.g., QA tasks (Khashabi et al., 2020) or dialogue task (Wu et al., 2020)) can show state-of-the-art performance by knowledge transfer. Due to the space limit, all the development set results are shown in the Appendix.
We observe a big improvement from all the baselines after finetuning on our training set, suggesting the effectiveness of our data to improve dialogue understanding. Those span-based models, meanwhile, achieve similar performance to Uni-fiedQA T5 models with smaller model sizes. BERT-Base model has the largest improvement gain by 22.6 EM score after finetuning. We find that the UnifiedQA T5 model with 11B parameters cannot achieve performance as good as the 3B model, we guess that the released checkpoint has not been optimized well by Khashabi et al. (2020). In addition, we estimate human performance by asking crowd workers to answer around 10% QA pairs in test set. We collect two answers for each question and select one that has a higher FZ-R score. We observe an EM score at around 80% and an F1 score at 90%, which still shows a considerable gap with existing  models.

Full Mode
The retriever results are shown in Table 8, in which we find that BM25 outperforms DPR-wiki by a large margin in our dataset on the recall@k measure, where we report k = 1, 3, 5, 10. The two possible reasons are that 1) the difference in data distribution between Wikipedia and conversation is large and DPR is not able to properly transfer to unseen documents, and 2) questions in QAConv are more specific to those mentioned entities, which makes the BM25 method more reliable. We show the full mode results in Table 7 using BM25 (DPRwiki results in the Appendix Table 16). We use the top one retrieved conversational chunk as input to feed the trained QA models. As a result, the performance of UnifiedQA (T5-3B) drops by 18.2% EM score in the zero-shot setting, and the finetuned results of RoBERTa-Large drop by 22.2% EM score as well, suggesting a serious error propagation issue in the full mode that requires further investigation in the future work.

Error Analysis
We further check the results difference between answerable and unanswerable questions in Table 9. The UnifiedQA T5 models outperform span-based models among the answerable questions, however, they are not able to answer any unanswerable questions and keep predicting some "answers". More interestingly, we observe that those span-based models perform poorly on answerable questions, as they can achieve a high recall but a low F1 score on unanswerable questions with a binary setting (predict answerable or unanswerable). This implies that existing span-based models tend to predict our task as unanswerable, revealing their weakness of dialogue understanding ability.
Then we check what kinds of QA samples in the test set are improved the most while finetuning on our training data using RoBERTa-Large. We find that 75% of such samples are incorrectly predicted to be unanswerable, which is consistent with the results in Table 9. We also analyze the error prediction after finetuning. We find that 35.5% are what-question errors, 18.2% are which-question errors, 12.1% are how-question errors, and 10.3% are who-question errors.
In addition, we sample 100 QA pairs from the errors which have an FZ-R score lower than 50% and manually check and categorize these predicted answers. We find out that 20% of such examples are somehow reasonable and may be able to count as correct answers (e.g., UCLA v.s. University of California, Jay Sonneburg v.s. Jay), 31% are predicted wrong answers but with correct entity type (e.g., Eurasia v.s. China, Susan Flynn v.s. Sara Shackleton), 38% are wrong answers with different entity types (e.g., prison v.s. drug test, Thanksgiving v.s., fourth quarter), and 11% are classified as unanswerable questions wrongly. This finding reveals the weakness of current evaluation metrics that they cannot measure semantic distances between two different answers.

Related Work
QA datasets can be categorized into four groups. The first one is cloze-style QA where a model has to fill in the blanks. For example, the Children's Book Test (Hill et al., 2015)  In general, our task is also related to conversations as a knowledge source. The dialogue state tracking task in task-oriented dialogue systems can be viewed as one specific branch of this goal as well, where tracking slots and values can be reframed as a QA task (McCann et al., 2018;Li et al., 2021b). Moreover, extracting user attributes from open-domain conversations (Wu et al., 2019), getting to know the user through conversations, can be marked as one of the potential applications. The very recently proposed query-based meeting summarization dataset, QMSum (Zhong et al., 2021), can be viewed as one application of treating conversations as databases and conduct an abstractive question answering task.

Conclusion
QAConv is a new dataset that conducts QA on informative conversations such as emails, panels, and channels. We show the unique challenges of our tasks in both chunk mode with oracle partial conversations and full mode with a retrieval stage. We find that state-of-the-art QA models have limited dialogue understanding and tend to predict our answerable QA pairs as unanswerable. We provide a new testbed for QA on conversation tasks to facilitate future research.

Ethical Considerations
The QAConv benchmark proposed in this work could be helpful in creation of more powerful conversation retrieval and QA on conversations. However, QAConv benchmark only covers a few domains as background conversations. Furthermore, even with our best efforts to ensure high quality and accuracy, the dataset might still contain incorrect labels and biases in some instances, which could be the inherent mistakes from the original dialogue datasets. This could pose a risk if models that are evaluated or built using this benchmark are used in domains not covered by the dataset or if they leverage evidence from unreliable or biased dialogues. Thus, the proposed benchmark should not be treated as a universal tool for all domains and scenarios. We have used only the publicly available transcripts data and adhere to their guideline, for example, the Media data is for research-purpose only and cannot be used for commercial purpose. As conversations may have biased views, for example, specific political opinions from speakers, the transcripts and QA pairs will likely contain them. The content of the transcripts and summaries only reflect the views of the speakers, not the authors' point-of-views. We would like to remind our dataset users that there could have potential bias, toxicity, and subjective opinions in the selected conversations which may impact model training. Please view the content and data usage with discretion.

A.1 Dataset documentation and intended uses
We follow datasheets for datasets guideline to document the followings.

A.1.1 Motivation
• For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled?
-QAConv is created to test understanding of informative conversations such as business emails, panel discussions, and work channels. It is designed for QA on informative conversations to fill the gap of common Wikipedia-based QA tasks.
• Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
-Salesforce AI Research team and HKUST CAiRE team work together to create this dataset.
• Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.
-Salesforce AI research team funded the creation of the dataset.

A.1.2 Composition
• What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.
-QAConv has conversations (text) among speakers (people) and a set of corresponding QA pairs (text).
• How many instances are there in total (of each type, if appropriate)?
-QAConv has 34,608 QA pairs and 10,259 conversations. Each conversation has 568.8 words in average and the longest one has 19,917 words.
• Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).
-The conversations in QAConv are randomly sampled from several conversational datasets, including BC3, Enron, Court, Media, and Slack, and the number of samples is decided based on related work and the budget.
• What data does each instance consist of? "Raw" data (e.g., unprocessed text or images) or features? In either case, please provide a description.
-Each sample has raw text of conversations, speaker names, and QA pairs.
• Is there a label or target associated with each instance? If so, please provide a description.
-Each answerable sample has at least one possible answer in a list format.
• Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.
-We do not include the crowd worker information due to the potential privacy issue.
• Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicit. -N/A • Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.
-We provide official training, development, and testing splits.
• Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.
-There could have some potential noise of question or answer annotation.
• Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions] (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.
• Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctorpatient confidentiality, data that includes the content of individuals' nonpublic communications)? If so, please provide a description.
-No, all the samples in QAConv is public available.
• Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why. -No • Does the dataset relate to people? If not, you may skip the remaining questions in this section. -Yes • Does the dataset identify any subpopulations (e.g., by age, gender)? If so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset.
-QAConv contains different speakers with their names. Some samples have their role information, e.g., petitioner.
• Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how.
-Yes, because some of the conversations are coming from public forums, therefore, people may be able to find the original speaker if they find the original media source.
• Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial or ethnic origins, sexual. orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description. -N/A.

A.1.3 Collection Process
• How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-ofspeech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.
-The QA data is collected by Amazon Mechanical Turk. The data is directly observable.
• What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated? If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?
-The QA data is collected by Amazon Mechanical Turk, we design a user interface with instructions on the top and then given partial conversation as context.
• Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?
-Crowdworkers. We paid them roughly $8-10 per hour, calculated by the average time to read and wriite one QA pair is approximately 4 minutes.
• Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.
-The data was collected during Feb 2021 to March 2021.
• Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation.
-We have conduct an internal ethical review process by Salesforce ethical AI team, https://einstein.ai/ethics.
• Does the dataset relate to people? If not, you may skip the remainder of the questions in this section. -Yes.
• Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?
-We obtain the data through AMT website.
• Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the notification itself.
-Yes, the turkers know the data collect procedure. Screenshots are shown Figure 4, Figure 5, Figure 6 in the Appendix.
• Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.
• If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate).
• Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation. -N/A

A.1.4 Preprocessing/cleaning/labeling
• Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the. remainder of the questions in this section.
-We conduct data cleaning such as removing code snippets before asking the crowd workers to provide corresponding QA pairs. Thus, no additional cleaning or preprocessing is done for the released dataset, only the reading scripts used to change the format for model reading are used.
• Was the "raw" data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the "raw" data.
-Yes, in the same link.
• Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point.

A.1.5 Uses
• Has the dataset been used for any tasks already? If so, please provide a description.
-It is proposed to use for QA on conversations task.
• Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point.
-It is a new dataset. We run existing state-ofthe-art models and release the code.
• What (other) tasks could the dataset be used for?
-Many conversational AI related tasks can be applied or transferred, for examples, conversational retrieval and conversational machine reading.
• Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms?
-Different ways to disentangle conversations could impact the overall performance. In our current setting, we use and release the buffer-based chunking mechanism.
• Are there tasks for which the dataset should not be used? If so, please provide a description.
-Conversations from Media corpus should not be used for commercial usage.

A.1.6 Distribution
• Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description. -No.
• How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)?
• When will the dataset be distributed? -Released.
• Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.
• Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions. -No.
• Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.
-Media dataset is restricted their conversations to be research-only usage.
-Salesforce AI Research team. Chien-Sheng (Jason) Wu is the corresponding author.
• How can the owner/curator/manager of the dataset be contacted (e.g., email address)?
-Create an open issue on our Github repository or contact the authors.
• Is there an erratum? If so, please provide a link or other access point. -No.
• Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)?
-No. If we plan to update in the future, we will indicate the information on our Github repository.
• If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced. -No.
• Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to users.
-Yes. If we plan to update the data, we will keep the original version available and then release the follow-up version, for example, QAConv-2.0 • If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to other users? If so, please provide a description.
-Yes, they can submit a Github pull request or contact us privately.

A.2 Test Data Additional Verification
After random split, we run an additional verification step on the dev and test set. If the new collected answer is very similar with the original answer (FZR score > 90), we keep the original answer. If the new answer is similar within a margin (90 > FZR score > 75), we keep both answers. If the new answer is very different from the original answer (75 > FZR score), we will run one more verification step to get the 3rd answers. We pick the most similar two answers as the gold answers if their FZR score is > 75, otherwise, we manually looked into those controversial QA pairs and made the final judgement.
• Enron: Creative Commons Attribution 3.0 United States license.
• Court: This material is based upon work supported in part by the National Science Foundation under grant IIS-0910664. Any opinions, findings, and conclusions or recommendations expressed above are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
• Media: Only the publicly available transcripts data from the media sources are included.
• Slack: Numerous public Slack chat channels (https://slack.com/) have recently become available that are focused on specific software engineering-related discussion topics.
A.4 Human evaluation description of human-written and machine-generated questions.
Rate [Fluency of the question]: • (A) The question is fluent and has good grammar. I can understand clearly.
• (B) The question is somewhat fluent with some minor grammar errors. But it does not influence my reading.
• (C) The question is not fluent and has serious grammar error. I can hardly understand it.
Rate [Complexity of the question]: • (A) The answer to the question is hard to find. I have to read the whole conversation backand-forth more than one time.
• (B) The answer to the question is not that hard to find. I can find the answer by reading several sentences once.
• (C) The answer to the question is easy to find. I can find the answer by only reading only one sentence.
Rate [Confidence of the answer]: • (A) I am confident that my answer is correct.
• (B) I am not confident that my answer is correct.        Steve Duffy: ..., but I don't know if Enron would even consider this. Studdert might have\nthe best feel for this. Separately, the defendant group will get back to us\non any offer they might be willing to make to settle just the Montana case,\nbut it appears that their real interest would be in a \"global\" deal. Any\ncomments? SWD Michael Burke: Steve, Stan and I have discussed this and we agree that Mike Moran should\ntake the lead and explore all aspects of an Enron Global deal. I know that\nyou will assist Mike in this endeavor. thanks, mike MR. FREEDMAN (RESPONDENT): . . . They both deserve the death penalty. They -they were -the prosecutors were aware that the -the death penalty is what stirs the pot here, and so they were urging somebody to be the shooter to get the death penalty. If this wasn't a death penalty case, I don't think they -it would have mattered who killed who. And so they were urging -JUSTICE KENNEDY: Well, I think there's quite a difference in -in case A where you say our position is that Stumpf was the shooter, pure and simple. That's it. In case B, they say we think Stumpf was the shooter. We're not 100 percent sure, but he should get the death penalty. The alternative is before the sentencer and the sentencer can make that determination.

Question
Which person was mentioned as the shooter in case A and B?


extfieldsofstudy:
Computer Science
Engineering
Mathematics

text:
Spectral Efficiency Analysis of Cell-free Distributed Massive MIMO Systems with Imperfect Covariance Matrix

In this paper, the impacts of imperfect channel covariance matrix on the spectral efficiency (SE) of cell-free distributed massive multiple-input multiple-output (MIMO) systems are analyzed. We propose to estimate the channel covariance matrix by alternately using the assigned pilots and their phase-shifted pilots in different coherent blocks, which improves the accuracy of channel estimation with imperfect covariance matrix and reduces pilot overhead. Under this scheme, the closed-form expressions of SE with maximum ratio combination (MRC) and zero-forcing (ZF) receivers are derived, which enables us to select key parameters for better system performance. Simulation results verify the effectiveness of the proposed channel estimation method and the accuracy of the derived closed-form expressions. When more coherent blocks are used to estimate the covariance matrix, we can get better system performance. Moreover, some insightful conclusions are arrived at from the SE comparisons between different receiving schemes (ZF and MRC) and different pilot allocation schemes (orthogonal pilot and pilot reuse).

I. INTRODUCTION
D UE to the increasing requirements for high system throughput, low latency, ultra-reliability and near-instant connection, more innovative technologies are needed to pave the way for the development of B5G and 6G technologies [1]. A large number of access points (APs) distributed in an area serve all users in the same time-frequency resource [2], [3]. Cell-free distributed massive MIMO can get all the benefits from massive MIMO (which facilitates propagation and channel strengthening when multiple antennas are used at the APs [4]) and network MIMO (which increases macrodiversity gain). Therefore, it has a very high spectral efficiency, energy efficiency and coverage. In addition, the cell-free structure can solve the problem that user performance and user location are related in the cellular network. Compared with the centralized system, the cell-free distributed massive MIMO system has advantages such as channel diversity, no switching, higher coverage and no need deploying cells in specific areas [5]. Therefore, in recent years, cell-free distributed massive MIMO has aroused widespread research interest [6]- [10].
Obtaining channel state information (CSI) is essential for taking advantage of the cell-free distributed massive MIMO system. There have been some studies on channel estimation methods. In most of these studies, the minimum mean square error (MMSE) channel estimation method is extended to the cell-free distributed MIMO system, or make certain improvements on this basis [6], [11]- [14]. There may be other channel estimation methods, such as least-squares (LS) channel estimation, but this results in very poor system performance [15]. However, these estimations are performed under the premise that the covariance matrix is perfect. Actually, the covariance matrix is usually imperfect, and the imperfect covariance matrix will have an impact on system performance. Because covariance matrix information is important for resource allocation and to suppress pilot contamination, imperfect covariance matrix will cause serious pilot contamination and affect the system performance [16].
Generally speaking, there are six typical methods to obtain the covariance matrix: 1) Use a specific phase for learning the covariance matrix where every user in the network uses a unique orthogonal pilot, from which a sample correlation matrix can be formed, possibly by using regularization to get robustness [17]; 2) Eliminate the interference by sending pilots from other interfering users to obtain the covariance matrix of the target user [16]; 3) Add additional pilot sequences for covariance estimation in the coherent blocks to obtain accurate channel estimation [18]; 4) Instead of reserving a specific phase for learning the covariance matrix, the pilot assignment is changed between different coherence blocks [19]; 5) Use the MMSE estimator of the structure model as the blueprint of the neural network structure, and a similarly effective (but suboptimal) estimator was obtained [20]; 6) Phase-rotate the pilot sequence in each coherence block to approximately decorrelate the channel observations at the APs [21]. However, these methods are all covariance matrix acquisition methods in a centralized cellular scenario, and there is no relevant research in a cell-free distributed massive MIMO scenario.
To meet the needs of super-dense user scenarios, the total number of antennas in cell-free massive MIMO systems need to be increased greatly. With the increase of the total number of antennas in the system, the spatial multiplexing gain of the system will be greatly improved, and the number of users who can serve in the same frequency will also be greatly increased. However, the increase of system capacity is established when the CSI is known. If the orthogonal pilot is used to estimate the CSI, the pilot overhead will increase linearly with the increasing of the number of antennas. If a completely orthogonal pilot is used, the pilot overhead will be very large, which ultimately restricts the improvement of the overall transmission efficiency. For uplink wireless transmission, although the pilot overhead is only proportional to the number of users and independent of the number of antennas, it is still too overhead if the completely orthogonal pilot is adopted considering the large number of users the cellfree distributed massive MIMO needs to support. Therefore, it is necessary to adopt a pilot multiplexing scheme to improve the utilization of pilot resources, which will result in pilot contamination [22]- [25]. Pilot contamination makes the estimation of the covariance matrix more complicated, because the users using the same pilot sequences will cause interference, so the channel estimates from which the covariance matrix estimates are obtained are themselves contaminated. Naively utilizing the contaminated channel estimates in a sample covariance estimator will result in the target user covariance matrix estimate containing the covariance matrices of the interference users, which in turn leads to inaccurate estimation of the covariance matrix and ultimately affects the overall transmission efficiency.
In this paper, we propose a new covariance matrix estimation method and derive the closed-form expressions for the uplink achievable rates in a cell-free distributed massive MIMO system with both MRC and ZF receiver under pilot contamination. The main contributions of this paper are summarized as follows: 1) We propose to estimate the channel covariance matrix by using alternate assigned pilots and their phase-shifted pilots in different coherent blocks, which obtains accurate channel estimation with low complexity and without extra pilots. 2) Given the estimated channel covariance matrix, we derived accurate closed-form expressions of the uplink achievable rates with both MRC and ZF receivers which enables the spectral efficiency analysis of cellfree distributed massive MIMO systems with imperfect covariance matrix. 3) Numerical simulations are performed to corroborate our theoretical analysis, and insightful conclusions are drawn from the comparison between system performance with perfect and imperfect covariance matrix using different receivers with different pilot allocation and the analyze of the relationship between the number of coherent blocks used to calculate the covariance estimation and the SE.
The remainder of the paper is organized as follows. In Section II, we describe the system model including the system configuration, channel model, and channel estimation with pilot contamination in the case of perfect or imperfect covariance matrix. Section III contains pilot structure design and covariance matrix estimation. The spectral efficiency is analyzed in section IV. Representative numerical results are given in Section V before we conclude the paper in Section VI.
Notation: Boldface letters stand for matrices (upper case) or vectors (lower case). The transpose and conjugate transpose are denoted by (·) T and (·) H respectively. I MN stands for the M × N identity matrix, and CN (µ, σ 2 ) denotes the circularly symmetric complex Gaussian distribution with mean µ and variance σ 2 , while W(N, R) denotes Wishart matrix with N degrees of freedom and R is the covariance matrix that corresponds to underlying Gaussian random vectors.

II. SYSTEM MODEL
This paper considers a cell-free distributed massive MIMO system. There are M APs equipped with N antennas and K single-antenna users. Assuming that the system adopts a time division duplex (TDD) mode, the BS uses the reciprocity of the uplink and downlink channels to perform downlink multiuser precoding based on the uplink channel estimation results.

A. Channel Model
Considering a block fading channel, and the channel vector from user k to all APs can be modeled as [26] with where λ km ∆ = d −ζ k,m s k,m represents the large-scale fading between user k and AP m. d k,m is the distance from user k to AP m, ζ is the path loss exponent, s k,m is a log-normal shadow fading variable. In addition, h k ∈ C MN ×1 models small-scale fast fading, and each elements follows CN (0, 1).
The uplink received signal model is: where y represents information received from all users, ρ is the sending power, x k is the sending information, n ∼ N c 0, σ 2 is the noise and σ 2 is the noise power. We assume that g k and n are independent of each other.

B. Uplink Channel Estimation
As widely recognized in most of articles, we first assume that the large-scale fading Λ k is known to the BS, and the small-scale fading h k needs to be estimated at the BS. Then, based on the actual situation that the covariance matrix is unknown, so the large-scale fading Λ k is also unknown to the BS, and both the large-scale fading Λ k and the small-scale fading h k need to be estimated at the BS.

1) Case I: Perfect covariance matrix
It is assumed that there are P orthogonal pilot sequences of length τ to be used for uplink channel estimation, which is a constant independent of K, and the pilot signal is allocated to K users when accessing the network. When P < K, there will be multiple users using the same pilot frequency, resulting in pilot contamination, which leads to the deterioration of the accuracy of channel estimation. Let U p ⊂ {1, . . . , K} denote the subset of users using the pilot p, all users using the pth pilot transmit the pilot signal X p on the same timefrequency resource, and satisfy X p X H p = I P . After correlating the received training signal with the conjugate of the pilot sequence, the BS estimates the channel based on the following observations Since g k satisfies N c (0, Λ k ), for k ∈ U p , the minimum mean square error (MMSE) estimate of channel g k should bê with where Σ p is the covariance matrix of the received signal. Therefore, for k ∈ U p , the channel estimation expression with perfect covariance matrix iŝ 2) Case II: Imperfect covariance matrix In practice, the covariance matrix is usually imperfect, and the aforementioned channel estimation under the perfect covariance matrix does not meet actual requirements. At this point, both Λ k and Σ p are unknown quantities, so the channel estimation expression is transformed intô Hence, if we want to obtain the accurate channel estimation with imperfect covariance matrix ,we must accurately estimate the covariance matrix. At the same time, it can also be seen in both (6) and (8) that the covariance matrix cannot be calculated directly by simply using the contaminated channel, and the covariance matrix needs to be estimated separately.
Assuming that the channel is a block fading model, the channel remains unchanged within a certain coherence bandwidth B c and a certain coherence time T c , that is, the channel remains unchanged within τ c = B c T c symbols. The covariance matrix is constant in the transmission bandwidth. Compared with the rapid change of the small-scale fading, it changes slowly in time, so it can be reasonably assumed that it remains unchanged within τ s = B s T s /B c T c = B s T s /τ c coherent blocks. Therefore, channel estimation is required for each coherent block, but the covariance matrix only needs to be estimated once in τ s coherent block. Therefore, according to this property, we estimate the covariance matrix and channel by using a special pilot structure. In the following section, we propose a method of using different pilot in adjacent coherent blocks. Compared with the common method of adding extra pilots for covariance matrix estimation, this scheme can greatly reduce the pilot overhead without losing the estimation accuracy.

III. PROPOSED PILOT STRUCTURE AND METHOD FOR ESTIMATING COVARIANCE MATRIX
In this section, we first describe the pilot structure design. Then based on the structure, we estimate the sample covariance and individual covariance respectively.

A. Pilot Structure Design
Considering the block fading model, multiple coherent blocks can be used to estimate the covariance matrix. Since each coherent block sends pilots for channel estimation, in order not to add additional pilot, we consider using the channel estimated pilots for simultaneous covariance matrix estimation. However, due to pilot contamination, in order to obtain interference-free observations for each user, we should change the pilot structure. This pilot structure needs to be able to eliminate interference for accurate covariance matrix estimation without affecting channel estimation, so we can change the pilot by introducing random phase shift.
In addition, in order to have a low complexity system, we should use a relatively simple pilot structure. So we consider a use of alternate pilots to estimate the covariance matrix, that is, each user alternately sends allocated pilot X p and phaseshifted pilot Φ k in adjacent coherent blocks. Φ k is the pilot after the pilot X p undergoes a random phase shift where n is the index of coherent blocks, θ k,2n is a random phase shift, which is irrelevant to the channel vector and noise and satisfies E e jθ k,2n = 0. So this random phase-shiftedt pilot Φ k is irrelevant to the channel and noise. The detailed structure of the pilot is depicted in Fig. 1.
Assuming that different users using the same pilot has different random phase shift θ k,2n . By observing the received signals of the alternate pilot, using the phase shift term irrelevance to remove interference from other users and noise, an accurate covariance matrix estimation can be obtained. In practical applications, θ k,2n can be obtained by a pseudorandom sequence generator, and it is assumed that these phase shifts are known at the central processing unit (CPU).

B. Sample Covariance Matrix Estimation
Based on the pilot structure, the sample correlation matrix is used to approximate Σ p . We select the coherent blocks that send pilot X p , that is, the interval coherent blocks are used to estimate Σ p . Assuming that the pilots of N Σ ≤ τ s /2 coherent blocks are received, the sample covariance matrix can be obtained aŝ where y p [2n − 1] represents the received signal of the (2n − 1)th coherent block. When N Σ is larger, the estimatedΣ p will be more accurate.
In fact, it is not necessary to select coherent blocks sending pilot X p , as long as the interval coherence blocks are selected to estimate the sample correlation matrix, the same result will be obtained.

C. Individual Covariance Matrix Estimation
In order to estimate individual covariance matrixΛ k , adjacent coherent blocks are needed to remove the correlation. Based on the pilot sequence Φ k , the pilot signal received by the APs from the users is: According to the received signals of the alternate pilots, for k ∈ U p , the observed values of the received pilot signals of the two adjacent coherent blocks (i.e. coherent block pairs) are respectivelŷ and whereĥ (1) k [2n − 1] is obtained by (4). For k ∈ U p , the covariance matrix of the channel estimated according to the observations of the adjacent alternate pilot signals is The equation is established because it is assumed that g k and n p are independent of each other, and because the random phase θ k,2n is introduced, the two noises are not correlated and interference terms from other users using the same pilot can also be eliminated. Therefore, the estimate of Λ k can be obtained aŝ However, for a finite N Λ , the estimateΛ k in (15) is not necessarily Hermitian symmetric. Therefore, this matrix can be regularized by approximating it with a positive semidefinite matrix. We approximateΛ k with the positive semidefinite matrix closest to the Frobenius norm, i.e.,Λ PSD k UD + U H , where D + is a diagonal matrix that contains only the positive eigenvalues of the symmetric part ofΛ k , i.e., D + Λ k +Λ H k /2 , and U contains the corresponding eigenvectors. Therefore,Λ k is written in the form ofΛ PSD k , which is written aŝ where N Λ represents the number of coherent block pairs required to estimate Λ k . When N Λ → ∞, the estimated covariance matrix converges to the real covariance matrix, i.e., Λ k → Λ k . In summary, we substitute the estimatedΛ k andΣ p into (8), and we can get the channel estimation when the covariance matrix is imperfect whereŴ is the deterministic matrix for the estimators. Remark 1: We select alternate pilots with different phase shifts for individual covariance matrix estimation, and then use interval phase-shift-free pilots for sample covariance estimation. In this scheme, we can obtain accurate covariance matrix estimation and accurate channel estimation without adding additional pilots in the case of low system complexity.

IV. UPLINK PERFORMANCE WITH IMPERFECT CHANNEL COVARIANCE INFORMATION
In this section, we analyze the spectral efficiency of the system based on the above channel estimation method. To derive the closed-form expressions of the achievable rates, we first give the ideal lower boundary of the user achievable rates and then use random matrix theory to perform expectation transformation.
The signal model (3) received by the APs can be rewritten as [27] where w k represents the receiving vector, which can be defined as whereẑ k is the kth column ofĜ Ĝ HĜ −1 ,Ĝ = [ĝ 1 ,ĝ 2 , · · · ,ĝ K ], andĝ k is the estimation of g k .
The lower boundary of the user achievable rate can be expressed as [18] where P/τ c represents the symbols occupied by the pilot. According to (19), the signal-to-noise ratio (SINR) γ k is defined as

A. MRC
When MRC receiver is used, the receiver vector is equal to the channel estimate. Substituting (8) into (22) and based on Lemma 1 shown below, we can get the closed-form expression of uplink achievable rates.
Lemma 1: For a complex Wishart matrix W satisfies W(n, I) and n > m, it has the property [28] For n > m + 1 where A ∈ C m×m is a random matrix. Theorem 1: When MRC receiver is used, the closed-form approximation of SINR is given by In addition, when the total number of antennas M N → ∞, the limit rate of each user given in (27) at the top of next page. It can be seen that when the MRC receiver is used, even if the AP is equipped with a large number of antennas, the SINR of the receiver still tends to be a constant related to the covariance matrix, and its performance still has a bottleneck. It can also be seen that the SINR of the receiver can be improved by reasonably allocating users using the same pilot.

B. ZF
After ZF detection, the received signal can be written as whereG = [g 1 ,g 2 , · · · ,g K ],g k represents the error of channel estimationg Therefore, the SINR γ k can be expressed as whereΓ is defined as the interference item Lemma 2: For k ∈ U p , the channels between the users using the same pilot and the APs have the following relationship where, for i ∈ U p has Based on the above expressions, when M N → ∞, (30) can be transformed into Next, by approximating the estimates, the theoretical expression of SINR of the ZF receiver can be obtained.
Theorem 2: When ZF receiver is used, the closed-form approximation of SINR is given by where andΓ is (42) at the top of the page, whereΣ p is a Wishart matrix, and it satisfies W (N Σ , I MN ). Proof: Please refer to Appendix B. It can be seen that unlike MRC, using ZF receiver, when the number of antennas of each APs N → ∞, the SINR increases linearly as (39). At the same time, it can be known that the characteristic of matrix Ξ p is an important factor that restricts system performance, and it is closely related to pilot contamination. When Ξ p is rank-deficient, the performance of the ZF receiver will also be seriously degraded.
In summary, the closed-form expressions of SINR in both MRC and ZF receivers are related to the number of coherent blocks N Σ and N Λ to calculate the covariance matrix. Better estimation of the covariance matrix can lead to higher SINR and thus improve the system performance.

V. SIMULATION RESULTS
In this section, we fit the closed-form expressions with the simulation value, and compare them with the case where the covariance matrix is perfect. With the numerical result, we analyze the accuracy of channel estimation with imperfect covariance matrix.
We consider a cell-free distributed massive MIMO system, and there are M = 5 APs with N = 50 antennas randomly distributed in the area. We use the channel model in (1), where the path loss exponent ζ is 3.7, the reference distance is 1, and the variance of shadow fading satisfies exponential normal distribution. Set all APs to be uniformly distributed in a circle with a diameter of 1 km, and all K = 5 users are randomly distributed in the area. We set the number of coherent blocks with constant channel covariance matrix τ s = 20000, the number of symbols in a coherent block τ c = 200 , and the number of pilot symbols τ = 10. For comparison, we consider the following three aspects.  (2), and we perform the MMSE estimation of the channel to get the sum of user rates. • The simulated value of channel estimation with imperfect covariance matrix(Simulated): We directly use the covariance matrix estimation results of (10) and (16) for channel estimation, and get the sum of user rates to judge system performance. • The theoretical value of channel estimation with imperfect covariance matrix(Theoretical): We derive the Theorem 1 and the Theorem 2, and obtain the closed expression of the SINR to get the sum of user rates.
Based on the above three aspects, we simulate MRC and ZF respectively. Fig. 2 and Fig. 3 respectively show the sum of user rates when N Σ = 1000 and N Σ = 3000 with completely orthogonal pilots. It can be seen from Fig. 2 and Fig. 3 that the simulated value fits well with the theoretical value. Therefore, the correctness of our derived closed-form expressions is proved. When using MRC and ZF, the sum of user rates with perfect covariance are close to the situation when the covariance is imperfect, and MRC is closer than ZF. We can even see that when using MRC receivers, our covariance estimation  Fig. 4. The sum of user rates when N Σ = 1000 with P=8 pilots scheme can achieve 95% or higher spectral efficiency with a perfect covariance matrix. This proves that the covariance matrix estimation method we used can bring good system performance. But when using ZF, the sum of user rates are higher than using MRC. This is because the ZF receiver can eliminate inter-user interference. From the curve of the sum of user rates with perfect covariance matrix, it can be seen that the sum of user rates increase as the number of pilots N Λ used to calculate the individual covariance estimate increases. This result is due to the fact that the covariance estimation is not accurate when the inserted N Λ is small, and the inaccurate channel estimation results in a lower sum of user rates. As N Λ continues to increase, the accuracy of the channel estimation improves, and the sum of user rates increase accordingly. In addition, comparing Fig. 2 and Fig. 3, it can be seen that the number coherent blocks N Σ used to estimate the covariance matrix Σ p of the received signal will actually affect the system performance, and a more accurate estimation of Σ p will have better system performance. The increase of N Σ does not change the trend of the curve, because the increase of N Σ is only a better estimate of Σ p under the existing structure, and does not change the relationship between sum of user rates and N Λ . At this time, the sum of user rates of the imperfect covariance matrix are closer to the perfect situation. This is at the cost of calculation. In actual situations, a good CPU to increase N Σ to get better system performance is required. Fig. 4 shows the sum of user rates when N Σ = 1000 with P = 8 pilots. Because of the large number of users in a dense distributed MIMO system, the situation of each user using completely orthogonal pilots cannot be satisfied. So it is necessary to consider the situation of pilot contamination, that is, multiple users use the same pilot. It can be seen that pilot contamination has a certain impact on the performance of MRC and ZF. Pilot multiplexing can improve system resource utilization, and a good pilot allocation method can achieve a trade-off between resource utilization and system performance.

VI. CONCLUSION
This paper proposes a method to estimate the covariance matrix by alternating pilot in adjacent coherent blocks. This method improves the channel estimation method without extra pilot overhead to achieve better system performance when the covariance matrix is imperfect. We derived the closedform expression of uplink SE with MRC and ZF receivers and verified the closed-form expressions. We also analyze the influence of the covariance matrix calculation with different number of coherent blocks on the system performance. The simulation results show that the channel estimation algorithm with imperfect channel covariance information can achieve high estimation accuracy. At the same time, a better estimation of the covariance matrix of the received signal leads to better system performance. In addition, for different receivers, pilot contamination can decrease system performance.

APPENDIX A PROOF OF THEOREM 1
Based on the random matrix theory (see ( [28])), we provide the following proof which is divided into two steps. First, if the combining vector w k =Ŵ k y p is used, we need to convert the expression toŴ k . For E w H k g k , where (a) is obtained because the noise n p and the channel g k are independent of each other, and x H y = tr yx H for any vectors x, y. By using the same trace rule as above and identifying Σ p , for E w H k w k , we have Finally, for E w H k g i 2 ,


extfieldsofstudy:
Computer Science
Mathematics

text:
An Experimental Study of Latency for IEEE 802.11be Multi-link Operation

Will Multi-Link Operation (MLO) be able to improve the latency of Wi-Fi networks? MLO is one of the most disruptive MAC-layer techniques included in the IEEE 802.11be amendment. It allows a device to use multiple radios simultaneously and in a coordinated way, providing a new framework to improve the WLAN throughput and latency. In this paper, we investigate the potential latency benefits of MLO by using a large dataset containing 5 GHz spectrum occupancy measurements. Experimental results show that when the channels are symmetrically occupied, MLO can improve latency by one order of magnitude. In contrast, in asymmetrically occupied channels, MLO can sometimes be detrimental and increase latency. To address this case, we introduce Opportunistic Simultaneous Transmit and Receive (STR+) channel access and study its benefits.

I. INTRODUCTION
The importance of wireless connectivity in a globalized society is unquestionable, and forced lockdowns reminded us how dependable Wi-Fi is. We resorted to Wi-Fi to be in touch with our loved ones, to make online purchases, and to get work done and keep the economy afloat. In a post-pandemic world, Wi-Fi technologies will be vital for accessing fair and remotefriendly education, medical care, and business opportunities in the unlicensed spectrum. There will be nearly 628 million public Wi-Fi hotspots by 2023 [1], one out of ten equipped with Wi-Fi 6 based on the IEEE 802.11ax amendment [2].
As the popularity of Wi-Fi grows, so does the demand for augmented data rates, higher reliability, and lower latency, driving the development of a new Wi-Fi 7 generation based on the IEEE 802.11be Extremely High Throughput (EHT) specification [3]- [7]. Despite its name, Wi-Fi 7 will be chasing much more than peak throughput. Indeed, the 802.11be Task Group acknowledges the need for lower delays to enable delay-sensitive networking use cases, including augmented and virtual reality, cloud computing, and cross-factory floor communications in next-generation enterprises [8]- [12].
In a quest for lower delays, one of the most disruptive features being proposed for 802.11be is Multi-Link Operation (MLO) [13]- [15]. In MLO, devices can make simultaneous use of different channels or bands, potentially allowing delay-sensitive traffic to be transmitted through multiple links to ensure M. Carrascosa and B. Bellalta were supported by WINDMAL PGC2018-099959-B-I00 (MCIU/AEI/FEDER,UE) and Cisco.
G. Geraci was supported by MINECO's Project RTI2018-101040 and by a "Ramón y Cajal" Fellowship from the Spanish State Research Agency. its timely reception. With its standardization process being consolidated, and prompted by the increasing interest from the research community [16]- [19], a fundamental question arises as to whether and to what extent MLO can reduce Wi-Fi latency in real-world scenarios.
In this paper, capitalizing on over-the-air measurements of spectrum occupancy for the entire 5 GHz band recently collected [20], [21] and freely available in open source 1 , we experimentally investigate the latency 2 performance of 802.11be MLO. Atop these traces, which include scenarios with high access point (AP) density and crowded environments and span multiple hours, we develop an emulation tool that fuses a Wi-Fi MLO state machine with the high-resolution spectrum measurements. Besides legacy Wi-Fi Single-Link Operation (SLO), we study the two MLO channel access modes currently under consideration by the IEEE 802.11be Task Group [3]: (i) MLO-STR, where two radio interfaces are operated independently, and (ii) MLO-NSTR, where one interface acts as primary and the other as secondary. Our main contributions can be summarized as follows: • We show that when using two links with statistically symmetrical occupancy, MLO reduces average and 95th percentile latency by up to 78% with respect to SLO by availing of a second radio interface. • In contrast, we surprisingly discover that when using two links with asymmetrical occupancy, MLO-STR can sometimes worsen the latency performance with respect to SLO. In the worst case, we observe an increase of up to 112% in terms of 95th percentile latency. • To overcome the aforementioned issue, we propose an alternative implementation of STR, denoted MLO-STR+. By running in parallel as many backoff instances as interfaces, MLO-STR+ allocates each packet to the interface whose backoff expires first. This way, STR+ guarantees same delay as or lower than SLO, with reductions of up to 70% in the best observed case.
II. MULTI-RADIO MULTI-LINK OPERATION IEEE 802.11be considers two main channel access methods to support Multi-link Operation: Simultaneous Transmit and Receive (MLO-STR), and Non-simultaneous Transmit and Receive (MLO-NSTR) [3]. We introduce them in the following, from the perspective of an AP equipped with two radio interfaces and thus able to operate on two different channels simultaneously: • MLO-STR: The two radio interfaces operate independently and asynchronously, and a packet waiting for transmission is allocated to a radio interface as soon as the latter becomes available. If both radio interfaces are available, the packet is randomly allocated to either. Once an interface is allocated a packet, it starts channel contention by initializing a backoff instance. • MLO-NSTR: One interface acts as primary, and the other as secondary. When there are packets waiting for transmission, the primary interface undergoes contention to access the channel through a backoff counter. Once the backoff counter reaches zero, packets are sent through both interfaces if the secondary one has been idle for at least a PIFS interval. Otherwise, only the primary interface is used to transmit.
Besides the MLO modes, IEEE 802.11be also considers the conventional Single-link Operation, where an AP is equipped with only one radio interface. In the case of MLO-STR, arriving packets are allocated to whichever interface becomes available first. This results in a significant delay reduction for packets #1, #2 and #4. In the case of MLO-NSTR, the secondary channel's dependence on the primary sometimes prevents efficiently using the two radio interfaces. As a result, and unlike MLO-STR, the delay for packets #1 and #4 cannot be reduced with respect to SLO.

III. EXPERIMENTAL SETUP
In this work, we consider a target WLAN Basic Service Set (BSS) consisting of one AP and one station (STA), both equipped with two Wi-Fi interfaces each operating in the 5 GHz band on channels 36 and 100, respectively denoted primary and secondary. On these channels, the target BSS observes the environment activity, i.e., the transmissions generated by Orthogonal Basic Service Sets (OBSS). The BSS and OBSS under consideration are illustrated in Figure 2 in blue and red, respectively. For this setup, we consider the three modes of operation described in Section II, namely: (i) SLO, where only the primary channel interface is available; (ii) MLO-STR, where both interfaces are available and work independently; and (iii) MLO-NSTR, where both interfaces are available but usage of the secondary channel is conditioned on the primary also being unoccupied.
For the above scenario, we consider downlink traffic, i.e., from the AP to the STA. We assume packet arrivals to follow a Poisson process, and transmitted packets to have a constant size of L = 12000 bits. Table I summarizes the main parameters used in the Wi-Fi state machine.

A. WACA Dataset
In order to evaluate the latency of 802.11be MLO in a realworld setting, we employ the WACA dataset, containing overthe-air measurements of the 5 GHz band occupancy that we have recently collected and made publicly available. This dataset was obtained by conducting extensive measurement campaigns on different days and in multiple locations, including a sold-out football stadium (F. C. Barcelona's Camp Nou).
In this paper, we employ the football stadium measurements since they range from completely idle to fully occupied channels. In the dataset, spectrum samples consist of 1 s of consecutive, 10 µs receive signal strength indicator (RSSI) measurements. We refer the reader to [20], [21] for further details on the dataset. Compared to [20], [21], in this work we have implemented a wholly new Wi-Fi state machine, capable of (i) fully characterizing the temporal dynamics of the system under finite traffic loads, i.e., non-full buffer conditions, and (ii) supporting multiple Wi-Fi interfaces and packet buffers.

B. Trace-based Simulations Methodology
In order to study the effect of channel occupancy on latency, we partition the available traces in our dataset for both primary and secondary channels into different average channel occupancy regimes: {10%, 20%, . . . , 90%}. Then, we run each simulation as follows: 1) We select the occupancy regime of interest for the primary and secondary channels, e.g., 10% and 40%, respectively; 2) We combine uniformly at random one spectrum sample each for the primary and secondary channels; 3) For each spectrum sample pair and given a particular traffic load of interest, we compute the packet arrival times at the AP; 4) We execute the Wi-Fi state machine for SLO, MLO-STR, and MLO-NSTR over each spectrum sample, considering for each channel access mode the same packet arrival times. We consider the same hinder interaction model as in [21]. 5) We store the individual delay experienced by each packet over all spectrum samples.
For a fair comparison between SLO and MLO, we guarantee that all results are obtained in non-saturation conditions and thus we discard any simulations where less than 95% of all the transmitted packets are received.

IV. DELAY PERFORMANCE
This section investigates the delay performance of both SLO and MLO modes for different combinations of channel occupancies and traffic loads.
To evaluate the gains of MLO in terms of delay, we consider the same traffic load for both SLO and MLO modes. In addition, the primary channel considered in the MLO mode is the same channel used in SLO. Therefore, we could expect that adding another channel in MLO mode, regardless of its occupancy, should yield lower delays.

A. Symmetrically Occupied Channels
Here we study the case of symmetric channel occupancies in which both MLO interfaces have channels with similar occupancy levels. In particular, we study the delay performance with pairs of channels in the ranges of 10%, 40%, and 70% occupancy. In those cases, the average full-buffer throughput under SLO is 37, 22, and 6.8 Mbps, respectively. For these three scenarios (symmetric low, medium, and high occupancy), we feed the Wi-Fi state machine with Poisson traffic and vary the intensity as a fraction of this SLO average full-buffer throughput, namely 0.2, 0.4, 0.6, 0.8. Figure 3 shows the average and 95th percentile delay for all channel access modes and the different channel occupancies. First, we observe that when both channels have 10% occupancy (Figure 3a), the three schemes have strikingly different scaling with increasing traffic load as MLO delay does not increase at the same rate as SLO delay. For example, at 20% traffic load, STR and NSTR offer a modest decrease in average delay compared to Single Link Operation of 17% and 9% respectively. In contrast, when the traffic load is 80%, STR and NSTR reduce the average delay by 69% and 62%. This scaling is even more pronounced analyzing the 95th percentile of delay, in which MLO achieves up to a 78% delay reduction. Thus, for both average and 95th percentile delay, the benefits of MLO are increasingly pronounced under higher traffic load as in this case, there are often multiple packets in the buffer such that both interfaces can be used. Moreover, with a relatively low channel occupancy of 10%, both channels are often available.
Next, we consider the case that both channels have symmetrical medium (40%) occupancy ( Figure 3b). Here, while SLO's average delay increases only modestly with traffic (i.e., from 2 to 18 ms), the 95th percentile delay increases much more rapidly, exceeding 100 ms. In contrast, STR can yield a staggering order of magnitude reduction in 95th percentile delay   compared to SLO. The reason is that STR avails usage of two channels and can access either or both of them. STR, therefore, realizes delay benefits compared to SLO unless both channels are occupied.
Unfortunately, unlike STR, the benefits of NSTR over SLO are limited and mostly confined to how the average delay scales with traffic load. Indeed, NSTR can only gain access to the secondary channel if the primary channel is also idle, implying that the average delay is guaranteed to be lower than the average delay under SLO. However, the 95th percentile delays are triggered by long periods of occupancy of the primary channel, thus making any availability of the secondary channel during this time irrelevant. As a result, the 95th percentile delay under NSTR rapidly grows as the normalized traffic load increases.
Lastly, when both channels have high (70%) occupancy (Figure 3c), STR again has the most favorable 95th percentile delay scaling with traffic load, providing substantial reductions as compared to both SLO and NSTR. Nonetheless, at such high channel occupancies, even STR has difficulty finding transmission opportunities on either channel, so both mean and 95th percentile delays are increasing. Additionally, NSTR provides negligible benefits in both average and 95th percentile delay compared to SLO.
Findings: When both channel occupancies are symmetrically medium to high load, NSTR fails to provide significant 95th percentile delay benefits compared to SLO. The key reason is that NSTR is only able to realize a benefit compared to SLO if both channels are simultaneously unoccupied, an increasingly unlikely occurrence in this scenario. Fortunately, STR yields significant 95th percentile latency benefits (compared to both SLO and NSTR) even in the challenging regime of increasing occupancies and traffic. This is because STR can utilize either available channel, and reduce the packets waiting time even if it cannot simultaneously utilize both available channels.

B. Asymmetrically Occupied Channels
Here, we employ the same normalized traffic loads from the previous section, but change the channel occupancy of our interfaces so that they lie in different ranges. Between the two channels, we always assume the primary to be the less occupied one. Note that the opposite case favors both MLO modes in this comparison: SLO would always incur a high delay, and both MLO modes would take advantage of a more idle secondary channel. Figure 4a depicts the case of a low (10%) primary and medium (40%) secondary channel occupancy. As expected, NSTR offers deterministically lower delays than SLO, with the highest benefits occurring under higher traffic loads. However, STR surprisingly incurs a higher average and 95th percentile delay than SLO for the lowest traffic load of 0.2. Indeed, STR starts contention by initializing the backoff counter as soon as a channel is detected to be idle. Unfortunately, such channel may be occupied before the backoff timer expires, thus pausing the backoff counter. If the backoff is paused too often (or for long intervals), the packet could incur even higher delays than it would have if the other channel-initially busy-had been selected.
In Figure 4b, this effect is exacerbated due to the even higher occupancy of the secondary channel, as selecting an idle secondary channel incurs the risk of the latter being occupied before the backoff counter expires. When this occurs, the 95th percentile delay can be twice as high as that with SLO, albeit still confined to below 10 ms. However, STR average and 95th percentile delays grow at a lower rate than those of SLO as the traffic load increases. Indeed, STR can still take advantage of a secondary channel (even when highly occupied) to reduce congestion and curb the latency when it is caused not only by the channel occupancy patterns but also by the amount of traffic.
Finally, Figure 4c considers the more symmetrical case of primary and secondary channel occupancy of 40% and 70%, respectively. Similar to the prior case of Figures 3b and 3c, STR scales well with the increasing traffic load, keeping the average delay below that of SLO and the decreasing the 95th percentile by up to a half. Compared to Figure 4b, the primary channel occupancy has grown, leading to a faster increase in the SLO delay vs. traffic load. However, STR is capable of leveraging both links and thus achieves lower delays. Findings: Channel occupancy is a crucial factor to account for when selecting a secondary channel in MLO mode. For STR, specifically, using a secondary channel that is much busier than the primary can lead to even higher delays than using SLO. This is owed to packets being suboptimally assigned to an interface before carrying out the backoff, with the latter likely to be interrupted on the busier channel. This effect is exacerbated when the difference between channel occupancies increases.

C. Opportunistic MLO-STR
We have shown that MLO-STR can lead to even higher delays than using SLO in the case of channels with different occupancies. To overcome this issue, we now propose an alternative implementation which we denote as Opportunistic MLO-STR (MLO-STR+): • MLO-STR+: When both interfaces are idle, one backoff instance is started on each. Packet allocation is deferred until either backoff counter expires, and the first waiting packet is allocated to the interface whose counter expires first. This approach differs from MLO-STR, where a packet is assigned to a channel as soon as the latter is idle, without waiting for its backoff counter to expire.
The main advantage of the proposed STR+ lies in the fact that, if one channel becomes occupied during the backoff, a transmission opportunity may be found on the other, avoiding unnecessarily delaying the waiting packet. In practice, implementing MLO-STR+ only requires a minor firmware update on the current Wi-Fi state machine: the ability to control when an interface can initiate, pause, and complete the backoff countdown without actually being allocated a packet. Figure 5 shows the average and 95th percentile delay for the same cases studied in Figure 4. We still take SLO as the baseline and compare MLO-STR and MLO-STR+ modes.
In Figure 5a, STR+ consistently outperforms STR and SLO in both average and 95th percentile delay, since packets are transmitted either at the same time as in SLO, or faster via the secondary interface.
In Figure 5b, when the secondary channel has a 70% occupancy, we encounter the worst scenario for STR. In this case, STR selects the secondary channel when it undergoes a short idle periods. However, since the latter are typically followed by longer intervals of occupancy, the backoff counter often remains frozen, leading to 95th percentile delays more than twice as high as those with SLO. This shortcoming is avoided altogether by the proposed STR+, assigning a packet to either interface only after ensuring that the corresponding backoff counter has expired.
Finally, Figure 5c depicts the case of 40% and 70% occupancy on the primary and secondary channels, respectively. As the former has increased, the SLO delay grows rapidly. STR already outperforms SLO in average and 95th percentile delay, and STR+ slightly reduces these values further.
Findings: Multiple channels can be better exploited by running several backoff counters in parallel and allocating a waiting packet to whichever interface whose counter expires first. This proposed approach, which we denote MLO-STR+, gains more frequent transmission opportunities in the face of bursty spectrum occupancy, and guarantees a delay reduction with respect to SLO.

V. CONCLUSIONS
In this paper, we provided an experimental study of latency for IEEE 802.11be MLO. Using the WACA dataset, which contains real-world channel occupancy measurements in the 5 GHz spectrum, we cast light upon the latency performance of two MLO channel access modes, namely (i) MLO-STR, where two radio interfaces are operated independently, and (ii) MLO-NSTR, where one interface acts as primary and the other as secondary. We showed that when both channels are on average equally occupied, both MLO modes can reduce the 95th percentile latency by nearly one order of magnitude as they avail of a second radio interface. In contrast, in asymmetrically occupied channels, we surprisingly found the use of MLO-STR to be detrimental and cause even higher latency values than SLO. To overcome this issue, we proposed an alternative implementation, denoted MLO-STR+. By running in parallel as many backoff instances as interfaces, MLO-STR+ allocates each packet to the interface whose backoff expires first, thus achieving a significantly lower latency.