summaries: The paper proposes a simple attention-based model for conversational and multi-hop QA tasks. The model use BERT-like pre-trained LM ETC separately encodes questions and paragraph (i.e., a collection of sentences). Besides the encodings on sentence-level, the final context encodings also contain extra paragraph embeddings, which are a weighted sum of sentences’ encodings using a simple dot product attention. For the QA interaction, the models use a hard attention mechanism to select an entry representing either a sentence or a paragraph.  The experiments on two extractive QA datasets HYBRIDQA and QASPER show the model performs worse than the MATE model on HYBRIDQA but marginally better than other baselines on QASPER. On multi-hop QA and conversational QA tasks, the model performs marginally better than baselines on *expanded dataset*, but authors do not provide results on original datasets.

summaries: This paper introduces DocHopper, a new model for complex question answering over long documents (e.g., multi-hop QA over multiple paragraphs, conversational QA, reasoning over scientific documents). DocHopper is based on ETC (Ainslie et al., 2020) and DocHopper extends the existing hierarchical attentions from ETC with a new approach to update query representations in latent space. Their model does not jointly encode a question and context and does not require re-encoding of queries as in prior work, which leads to their effectiveness at inference time. They evaluate DocHopper on four different datasets: ShARC, QASPER, HotpotQA, and HybridQA. The proposed method achieves strong performance on those datasets, reducing the computational cost at the inference time.

summaries: The paper proposes an iterative approach for multi-hop question answering. At high-level the proposed model breaks a question into multiple sub-questions and then adds information relevant to each sub-question to the query vector for the next step retrieval. At each iteration an ETC encoder is used to encode the document and a sub-question; the vector corresponding to sub-question is then updated iteratively by contextualizing it over sentences and paragraphs in the document and is used to extract the final answer using a subsequent BERT reader. Evaluation results show improvements on 3 of 4 datasets.

summaries: This paper provides a novel MRC model (DocHopper) for multi-hop QA over long structured documents. In multi-hop QA, the evidence necessary to answer a user's question is spread across different parts of the long document. Previous approaches find the evidence by iteratively updating the user's query. The problems in these previous approaches are 1) computational efficiency and 2) ineffective modeling strategy for figuring out the relations of the evidence. DocHopper resolves these problems with a hierarchical attention mechanism. Hierarchical attention mechanism provides two types of embedding vectors for a single paragraph: 1) local sentence vectors and 2) global context vector of the paragraph. DocHopper computes the similarities between the query vectors and these sentence/paragraph vectors and selects the proper evidence. Since the sentence/paragraph vectors can be pre-computed, the only inference time required for this method is the time for question embedding, and this brings drastic improvement in the computational efficiency. This paper uses four types of datasets for evaluation: 1) conversational QA (ShARC), 2) TableQA (HybridQA), 3) QA on academic paper (QASPER), and 4) multi-hop factual QA (HotpotQA), and shows the QA performance and computational efficiency of their model.

abstractText: We propose a new model, DOCHOPPER, that iteratively attends to different parts of long, heirarchically structured documents to answer complex questions. Similar to multi-hop question-answering (QA) systems, at each step, DOCHOPPER uses a query q to attend to information from a document, combines this “retrieved” information with q to produce the next query. However, in contrast to most previous multi-hop QA systems, DOCHOPPER is able to “retrieve” either short passages or long sections of the document, thus emulating a multi-step process of “navigating” through a long document to answer a question. To enable this novel behavior, DOCHOPPER does not combine document information with q by concatenating text to the text of q, but by combining a compact neural representation of q with a compact neural representation of a hierarchical part of the document, which can potentially be quite large. We experiment with DOCHOPPER on four different QA tasks that require reading long and complex documents to answer multi-hop questions, and show that DOCHOPPER achieves state-of-the-art results on three of the datasets. Additionally, DOCHOPPER is efficient at inference time, being 3–10 times faster than the baselines.1

authors:
Joshua Ainslie
Santiago Ontanon
Chris Alberti
Vaclav Cvicek
Zachary Fisher
Philip Pham
Anirudh Ravula
Sumit Sanghai
Qifan Wang
Li Yang

title: Etc: Encoding long and structured inputs in transformers

venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),

authors:
Iz Beltagy
Matthew E Peters
Arman Cohan

title: Longformer: The long-document transformer

venue: arXiv preprint arXiv:2004.05150,

authors:
Ming-Wei Chang
Kristina Toutanova
Kenton Lee
Jacob Devlin

title: Language model pre-training for hierarchical document representations

venue: arXiv preprint arXiv:1901.09128,

authors:
Wenhu Chen
Hanwen Zha
Zhiyu Chen
Wenhan Xiong
Hong Wang
William Wang

title: Hybridqa: A dataset of multi-hop question answering over tabular and textual data

venue: Findings of EMNLP 2020,

authors:
Pradeep Dasigi
Kyle Lo
Iz Beltagy
Arman Cohan
Noah A Smith
Matt Gardner

title: A dataset of information-seeking questions and answers anchored in research papers

venue: arXiv preprint arXiv:2105.03011,

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova

title: BERT: Pre-training of deep bidirectional transformers for language understanding

authors:
Bhuwan Dhingra
Manzil Zaheer
Vidhisha Balachandran
Graham Neubig
Ruslan Salakhutdinov
William W Cohen

title: Differentiable reasoning over a virtual knowledge base

venue: arXiv preprint arXiv:2002.10640,

authors:
Julian Martin Eisenschlos
Maharshi Gor
Thomas Müller
William W. Cohen

title: Mate: Multi-view attention for table transformer efficiency, 2021

authors:
Yuwei Fang
Siqi Sun
Zhe Gan
Rohit Pillai
Shuohang Wang
Jingjing Liu

title: Hierarchical graph network for multi-hop question answering

authors:
Yifan Gao
Chien-Sheng Wu
Jingjing Li
Shafiq Joty
Steven CH Hoi
Caiming Xiong
Irwin King
Michael R Lyu

title: Discern: Discourse-aware entailment reasoning network for conversational machine reading

authors:
Alexios Gidiotis
Grigorios Tsoumakas

title: A divide-and-conquer approach to the summarization of long documents

venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,

authors:
Kelvin Guu
Kenton Lee
Zora Tung
Panupong Pasupat
Ming-Wei Chang

title: Realm: Retrievalaugmented language model pre-training

venue: arXiv preprint arXiv:2002.08909,

authors:
Vladimir Karpukhin
Barlas Oğuz
Sewon Min
Ledell Wu
Sergey Edunov
Danqi Chen
Wen-tau Yih

title: Dense passage retrieval for open-domain question answering

venue: arXiv preprint arXiv:2004.04906,

authors:
Omri Koshorek
Adir Cohen
Noam Mor
Michael Rotman
Jonathan Berant

title: Text segmentation as a supervised learning task

venue: arXiv preprint arXiv:1803.09337,

authors:
Patrick Lewis
Ethan Perez
Aleksandra Piktus
Fabio Petroni
Vladimir Karpukhin
Naman Goyal
Heinrich Küttler
Mike Lewis
Wen tau Yih
Tim Rocktäschel
Sebastian Riedel
Douwe Kiela

title: Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

authors:
Shaobo Li
Xiaoguang Li
Lifeng Shang
Xin Jiang
Qun Liu
Chengjie Sun
Zhenzhou Ji
Bingquan Liu

title: Hopretriever: Retrieve hops over wikipedia to answer complex questions, 2020

authors:
Sewon Min
Danqi Chen
Luke Zettlemoyer
Hannaneh Hajishirzi

title: Knowledge guided text retrieval and reading for open domain question answering, 2020

authors:
Aaron van den Oord
Yazhe Li
Oriol Vinyals

title: Representation learning with contrastive predictive coding

venue: arXiv preprint arXiv:1807.03748,

authors:
Peng Qi
Haejun Lee
Oghenetegiri ”TG” Sido
Christopher D. Manning

title: Retrieve, read, rerank, then iterate: Answering open-domain questions of varying reasoning steps from text, 2021

authors:
Lin Qiu
Yunxuan Xiao
Yanru Qu
Hao Zhou
Lei Li
Weinan Zhang
Yong Yu

title: Dynamically fused graph network for multi-hop reasoning

venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,

authors:
Sebastian Ruder
Parsa Ghaffari
John G Breslin

title: A hierarchical model of reviews for aspectbased sentiment analysis

venue: arXiv preprint arXiv:1609.02745,

authors:
Marzieh Saeidi
Max Bartolo
Patrick Lewis
Sameer Singh
Tim Rocktäschel
Mike Sheldon
Guillaume Bouchard
Sebastian Riedel

title: Interpretation of natural language rules in conversational machine reading

venue: arXiv preprint arXiv:1809.01494,

authors:
Haitian Sun
Bhuwan Dhingra
Manzil Zaheer
Kathryn Mazaitis
Ruslan Salakhutdinov
William W Cohen

title: Open domain question answering using early fusion of knowledge bases and text

venue: arXiv preprint arXiv:1809.00782,

authors:
Haitian Sun
Tania Bedrax-Weiss
William W. Cohen

title: Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text, 2019

authors:
Haitian Sun
Pat Verga
Bhuwan Dhingra
Ruslan Salakhutdinov
William W Cohen

title: Reasoning over virtual knowledge bases with open predicate relations

venue: arXiv preprint arXiv:2102.07043,

authors:
A. Talmor
J. Berant

title: The web as a knowledge-base for answering complex questions

venue: In North American Association for Computational Linguistics (NAACL),

authors:
Pat Verga
Haitian Sun
Livio Baldini Soares
William W Cohen

title: Facts as experts: Adaptable and interpretable neural memory over symbolic knowledge

venue: arXiv preprint arXiv:2007.00849,

authors:
Wei Wang
Ming Yan
Chen Wu

title: Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering

authors:
Johannes Welbl
Pontus Stenetorp
Sebastian Riedel

title: Constructing datasets for multi-hop reading comprehension across documents

venue: Transactions of the Association for Computational Linguistics,

authors:
Wen Xiao
Giuseppe Carenini

title: Extractive summarization of long documents by combining global and local context

venue: arXiv preprint arXiv:1909.08089,

authors:
Wenhan Xiong
Xiang Lorraine Li
Srini Iyer
Jingfei Du
Patrick Lewis
William Yang Wang
Yashar Mehdad
Wen tau Yih
Sebastian Riedel
Douwe Kiela
Barlas Oğuz

title: Answering complex open-domain questions with multi-hop dense retrieval, 2021

authors:
Zhilin Yang
Peng Qi
Saizheng Zhang
Yoshua Bengio
William W. Cohen
Ruslan Salakhutdinov
Christopher D. Manning

title: HotpotQA: A dataset for diverse, explainable multi-hop question answering

venue: In Conference on Empirical Methods in Natural Language Processing (EMNLP),

authors:
Zichao Yang
Diyi Yang
Chris Dyer
Xiaodong He
Alex Smola
Eduard Hovy

title: Hierarchical attention networks for document classification

venue: In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies,

authors:
Xingxing Zhang
Furu Wei
Ming Zhou

title: Hibert: Document level pre-training of hierarchical bidirectional transformers for document summarization, 2019

authors:
Chen Zhao
Chenyan Xiong
Jordan Boyd-Graber
Hal Daumé III

title: au2. Multi-step reasoning over unstructured text with beam dense retrieval, 2021

heading: 1 INTRODUCTION

text:
In this work we focus on the problem of answering complex questions over long structured documents. A long document typically contains coherent information on a certain topic, and the contents are grouped by sections or other structures. To answer complex, multi-hop questions over long documents often requires navigating through different parts of the documents to find different pieces of information relevant to a question. This navigation, in turn, requires understanding high-level information about the structure of the document. For example, to answer the question “What modules in DOCHOPPER will be finetuned in all the experiments?”, one might first turn to the section title “Model” to identify the different modules in DOCHOPPER, and then read the “Experiments” with these modules in mind, potentially further attending to specific subsections (such as the ones titled “Implementation Details”). As in academic papers (Dasigi et al., 2021), similar tasks are common for questions about government policies (Saeidi et al., 2018) or legal documents. This type of QA tests not only the ability to understand short passages of text, but also the ability to understand the goals of the question and the structure of documents in a domain.
A common approach of solving multi-hop questions is to iteratively find evidence for one hop, and then use that evidence to update the query used in the next hop of the QA process. The update can be performed by either explicitly predicting the intermediate answers (Talmor & Berant, 2018; Sun et al., 2019) or directly appending previous evidences to the questions (Zhao et al., 2021; Qi et al., 2021; Li et al., 2020; Xiong et al., 2021). While appending retrieved evidence to a query works well on many factual QA tasks, where it is possible to answer questions with evidences that are short pieces of text, this approach is expensive if one wishes to retrieve larger pieces of text as evidences (e.g., the “experiments” section of a paper). Another disadvantage is that appending together many small fragments of text intuitively fails to capture the relationships between them, and the structure of the document from which they were extracted.
To capture high-level structural information in a document as well as detailed information from short passages, hierarchical attention mechanisms have been proposed, which learn neural representations
1We will open-source our code and data.
at different levels that are then mixed to make final predictions for simple questions (Wang et al., 2018; Chang et al., 2019). Hierarchical attention has also been adopted in pretrained language models, e.g. ETC (Ainslie et al., 2020), which introduced a global-local attention mechanism where embeddings of special global tokens are used to encode high-level information. Our DOCHOPPER system incorporates ETC as a document encoder. However, while ETC has previously performed well on multi-hop QA tasks like HotpotQA and WikiHop (Yang et al., 2018; Welbl et al., 2018) which require combining information from a small number of short passages, it has not been previously evaluated on tasks of the sort considered here. Our experiments show that DOCHOPPER outperforms past approaches to using ETC for multi-hop questions.
DOCHOPPER extends the existing hierarchical attention methods with a novel approach to updating queries in a multi-hop setting. DOCHOPPER iteratively attends to different parts of the document, either at fine-grained level or at higher level. This process can be viewed as either retrieving a short passage, or navigating to a part of a document. In each iteration, the query vector is updated in the embedding space rather than by re-encoding a sequence of concatenated tokens. This updating step is end-to-end differentiable and efficient. In our experiments, we also show it is effective on four different benchmarks involving complex queries over long structured documents.
In particular, we evaluate DOCHOPPER on four different tasks: conversational QA for discourse entailment reasoning, using the ShARC (Saeidi et al., 2018) benchmark2; factual QA with table and text, using HybridQA (Chen et al., 2020); information seeking QA on academic papers, using QASPER (Dasigi et al., 2021); and multi-hop factual QA, using HotpotQA (Yang et al., 2018). Since the outputs of the four tasks are different, additional layers or simple downstream models are appended to DOCHOPPER to get the final answers. DOCHOPPER achieves state-of-the-art results on three of datasets, outperforming the baseline models by 3%-5%. Additionally, DOCHOPPER runs 3– 10 faster than the baseline models, because the neural representations of documents is pre-computed, which significantly reduces computation cost at inference time.

heading: 2 RELATED WORKS

text:
Graph-based models have been widely used for answering multi-hop questions in factual QA (Min et al., 2020; Sun et al., 2018; 2019; Qiu et al., 2019; Fang et al., 2019). However, most of the graphbased models are grounded to entities, i.e., evidences (from knowledge bases or text corpus) are connected by entities in the graph. The graph construction step also heavily relies on many discrete features such as hyperlinks or entities predicted with external entity linkers. It’s not clear how to apply these models to more general tasks if context is not entity-centric, such as questions about academic papers or government documents. Similar problems also exist in memory-augmented language models that achieved the state-of-the-art on many factual QA tasks (Guu et al., 2020; Lewis et al., 2021; Verga et al., 2020; Dhingra et al., 2020; Sun et al., 2021).
Alternatively, one can adopt the “retrieve and read” pipeline to answer multi-hop questions over long documents. Recent works proposed to extend the dense retrieval methods (Karpukhin et al., 2020) to multi-hop questions (Zhao et al., 2021; Qi et al., 2021; Li et al., 2020). However, such models retrieve one small piece of evidence at a time, lacking the ability of navigating between different parts of the documents to find relevant information at both higher and lower levels of the documentstructure hierarchy. Another disadvantage of these iterative models is that they are not end-to-end differentiable. Updating the questions for the next hop requires re-encoding the concatenated tokens from the questions and previously retrieved evidences. It also makes the model inefficient because re-encoding tokens with large Transformer models is very expensive.
Besides question answering tasks, hierarchical attention has been successfully used in tasks such as document classification (Yang et al., 2016; Chang et al., 2019), summarization (Gidiotis & Tsoumakas, 2020; Xiao & Carenini, 2019; Zhang et al., 2019), sentiment analysis (Ruder et al., 2016), text segmentation (Koshorek et al., 2018), etc. It is worth mentioning that ETC (Ainslie et al., 2020) was also used on a key-phrase extraction task on web pages using the structured DOM tree. However, none of these models can be easily adapted to answering complex questions over long documents.
2We modify the original dataset and replace the oracle snippet with the entire web page as input. Please see section §4.1 for more details.

heading: 3 MODEL

text: In this section, we first introduce how to compute neural representations for questions and context at different levels. Then, we present the iterative process that attends to information in the document and updates the query vector. Additional layers or simple downstream models required for different downstream tasks will be explained in the next section (§4).

heading: 3.1 BACKGROUND

text:
Input A long document usually contains multiple levels of hierarchy, e.g. sections, sub-sections, paragraphs, sentences, etc. For simplicity, we only consider two levels of hierarchy in this paper: paragraph-level and sentence-level. A sentence is the lowest granularity that can be retrieved, while a paragraph is an abstraction of a collection of sentences, which can be used to represent sections or other levels in the hierarchy, depending on the application. Formally, let d = {p0, . . . , p|d|} ∈ D be a document in the corpus D that contains a sequence of paragraphs pi, and let a paragraph pi = {si0, . . . , si|pi|} contain a sequence of sentences s i 0. A sentence s i j will be encoded into a fixed length vector sij ∈ Rd. Pretrained ETC We use ETC (Ainslie et al., 2020) as our query and context encoder as it is pretrained to produce both token-level and sentence-level embeddings. ETC is pretrained as a Mask Language Model (MLM). Different from BERT (Devlin et al., 2019), it employs an additional global-local attention mechanism. ETC assigns to each sentence a special global token that only attends to local tokens in the sentence, and its embedding is trained to summarize the information of local tokens in the sentence. A global token also attends the global tokens of other sentences in the input. ETC additionally adopts Contrastive Predictive Coding (CPC) (Oord et al., 2018) to train the embedding of global tokens to make them aware of other sentences in the context. We use the embeddings of global tokens in ETC as encodings for sentences.
Specifically, ETC takes a list of sentences pi = {si0, . . . , si|pi|} as input, and returns a list of vectors si0, . . . , si|pi|, where each s i j ∈ Rd represents the embedding of a sentence sij .
si0, . . . , s i |pi| = ETC({s i 0, . . . , s i |pi|}) ∈ R
|pi|×d

heading: 3.2 QUERY EMBEDDINGS

text:
We discuss the strategy of computing query vectors for two types of input, conversational QA and multi-hop QA. We use ETC (Ainslie et al., 2020) as our query encoder for the reasons discussed above. Other pretrained language models for similar purposes should work as well.
Conversational QA A conversational question contains multiple rounds of interaction between the machine and human. As the conversation goes on, the topic of the conversation will shift, but questions asked in the next turn are related to the history of the conversation. We consider this multi-turn process as a multi-hop QA task. In our specific task, the conversation starts with an initial question q0. The model will answer the question if there’s enough information. Otherwise, the model will ask a followup question f1 and expect the user’s answer a1. The model will keep asking followup questions for a few more iterations. The task is to predict the answer if there is enough information after reading the full conversation history, or mark the question as not answerable.
We denote ai as the answer to the followup question fi received from the users at the i’th iteration. We consider the full conversation history as the question pq = {q0, (f1, a1), . . . , (fn, an)} with the initial question denoted q0 and the i-th followup question-answer pair denoted (fi, ai). The answer ai to the followup question fi is concatenated to the end of the questions and represented as a single sentence. We call pq a question paragraph as it contains a sequence of question-answer pairs. We use ETC to compute the query embeddings qi, one for each sentence in the query paragraph pq . The query embeddings qi will be used to perform iterative attention over the document.
q0, . . . ,qn = ETCq({q0, (f1, a1), . . . , (fn, an)}) (1)
Multi-hop QA A multi-hop question, e.g. “Which gulf is north of the Somalian city with 550,000 residents”, requires the model to first find the answer of the first “hop” of the question, and then perform a second round to find the final answer. Assume for now there are exactly two hops. Different from conversational QA, questions in multi-hop QA do not have a clear split between the two hops of questions, making it impossible to explicitly split a question into two sub-questions. Instead, we add a dummy question qnull to the question paragraph qp = {q0, qnull}. The global-to-local attention mask of ETC is modified to allow the global token of the dummy question to attend to tokens in the question q0. With this modification, the query embeddings q0 and q1 for q0 and qnull can to attend any part of the question, but each can also attend to different parts of the question. One could append additional dummy questions to the question paragraph pq if the number of hops is larger. Thus we let
q0,q1 = ETCq({q0, qnull}) (2)
Note that we assume the true number of hops is known in multi-hop QA tasks, which is two or four for the experiments in this paper. However, one can train models to decide when to stop if questions have various numbers of hops. We leave this as a topic for future work.

heading: 3.3 NEURAL REPRESENTATIONS OF PARAGRAPHS

text:
We compute embeddings sij for the sentence sij with the context encoder. We use ETC as our context encoder, but similar pretrained language models will also work. Recall that a paragraph pi = {si0, . . . , si|pi|} contains a sequence of sentences s i j . The sentence embeddings sij in the paragraph pi are simply computed by applying ETC on the paragraph input.
si0, . . . , s i |pi| = ETCc({s i 0, . . . , s i |pi|})
The paragraph embeddings are directly derived from sentence embeddings sij and dependent on the queries qt, the embedding of the t’th hop of the question. The paragraph embedding pi is a weighted sum of sentence embeddings sij in the paragraph pi, where αj is the attention weights of the query vector qt to the sentence embedding sij . The paragraph embeddings pi are thus dependent on the query, but do not require jointly encoding tokens from queries and context, as in many BERT-style reading comprehension models. Computing paragraph embeddings with Eq.3 is hence very efficient.
pi = ∑ j αj sij , αj = softmax(q T t s i j) (3)

heading: 3.4 ITERATIVE ATTENTION

text:
We put the sentence embeddings and paragraph embeddings of document d into a combined context embedding table, so the model has the flexibility to decide which sentence or paragraph to attend to. Different update rules will be applied according to whether sentences or paragraphs are attended to.
To construct the embedding table, we iterate through all paragraphs in a document and apply the ETC encoder on each paragraph to compute the paragraph and sentences embeddings. Sentence and paragraph embeddings from all paragraphs are then merged together to form the combined embedding table. We denote the combined embedding table for document d as Cd = {p0, s00, . . . , s0|p0|,p1, s 1 0, . . . , s1|p1|, . . . }. Let cm be the embedding of the m’th entry from Cd; we emphasize that cm can represent either a sentence or a paragraph embedding.
Attention Step At each iteration, DOCHOPPER computes the inner product scores between the query vector qt and embeddings cm in Cd, and returns the entry ĉ with the largest score, which is usually referred as hard attention. (As we will see ĉ is not directly used for computation, but it is helpful in explaining the attention method).
ĉ = argmaxcm(q T t cm)
Mixing Step DOCHOPPER then mixes the embedding of the attended entry ĉ with the query vector qt to find the missing information. Since the combined embedding table Cd contains both sentence and paragraph embeddings, the selected entry ĉ can represent either a sentence or a paragraph. The two cases are separately considered. If ĉ is a sentence, i.e. ĉ = sij , DOCHOPPER computes the mixed embeddings as
q̃t = W T q [qt; s i j ] (4)
where [qt; sij ] is the concatenation of two vectors qt and sij . Assume ĉ is a paragraph, i.e., ĉ = pi, DOCHOPPER first looks up sentences in pi, i.e. the list {si0, . . . , si|pi|}. The following process is then used to compute the mixed embedding q̃t: (1) DOCHOPPER computes the attention weights of the query vector qt to the embeddings of associated sentences {si0, . . . , si|pi|} that measures the relevance scores between the query vector and the sentences. This attention weight is the same as the weight αj in Eq.3 that is used to compute the paragraph embeddings, so we reuse αj in this equation. In the implementation, we also re-use the value of αj if it has been computed for the query-dependent paragraph embeddings. (2) The query vector qt is compared with every sentences in paragraph to extract missing information. qt is broadcast to the sentence embedding sij with the weight αj , i.e. the query vector is more important if more relevant. The comparison is performed as a linear projection of the concatenated query and sentence embeddings, qt and sij .
kj = WTq [αj qt; s i j ] (5)
Then (3) the concatenated vectors kj are summed with the weight βj , where βj is the attention weight of a learned vector v to the concatenated vector kj . The learned vector v coordinates the importance of sentences from the attended paragraph after comparing them with the query vector and decides what information to pass to the next step of retrieval.
q̃t = ∑ j βj kj , βj = softmax(vT kj) (6)
It is not hard to see that computing the mixed embedding in Eq. 6 for the case that a paragraph is retrieved is essentially the same as in Eq. 4 if the retrieved paragraph pi only contains one sentence, i.e. αj = 1 and βj = 1 if |pi| = 1; hence the same logic can be used regardless of whether ĉ is a sentence or a paragraph.
Update Step The mixed embedding q̃t is then used to update the query vector qt+1 for the next step. Intuitively, q̃t is the residual from the previous step. Adding the residual embedding encourages the model to attend to information that is not fully satisfied from previous steps.
qt+1 ← qt+1 + q̃t (7)
Loss Function Attention is supervised if (distantly) supervised labels are available in the dataset. qTt cm is the inner product score between the query vector qt and a context embedding cm. Icm is an indicator function that equals to 1 iff the label of cm is positive.
lt = cross entropy(softmax(qTt cm), Icm)
The loss function is computed at the final step, and possibly at intermediate steps if labels are available. Supervision labels are sometimes distantly constructed. For example, in the extractive QA task, a positive candidate is the sentence or paragraph that contains the answer span (see §4).

heading: 3.5 RUNTIME EFFICIENCY

text: DOCHOPPER is very efficient at runtime thanks to the precomputed context embeddings (§3.3) at inference time. Different from previous reading comprehension models that jointly encode questions and context (Beltagy et al., 2020; Ainslie et al., 2020), DOCHOPPER encode question embeddings and context embeddings independently. At inference time, DOCHOPPER lets questions directly attend to the precomputed context embeddings, significantly reducing the computation cost compared to cross-attention models that jointly encode questions and context.

heading: 4 EXPERIMENTS

text: We evaluate DOCHOPPER on four different datasets: ShARC (Saeidi et al., 2018), HybridQA (Chen et al., 2020), QASPER (Dasigi et al., 2021), and HotpotQA (Yang et al., 2018).3 Since the downstream tasks of the four datasets are different, we apply an additional layer for ShARC, and a Transformer-based extractive QA model for HybridQA, QASPER and HotpotQA to extract the final answers. We will discuss the setup for each case separately.

heading: 4.1 EXTRACTIVE QA: HYBRIDQA AND QASPER

text:
Dataset HybridQA is a dataset that requires jointly using information from tables and hyperlinked text from cells to find the answers. Please see the Appendix for more information. This dataset requires the model to first locate the correct row in the table, and then find the answer from cells (or their hyperlinked text). To apply DOCHOPPER on the HybridQA dataset, we first convert a table with hyperlinked text into a long document. Each row in the table is considered a paragraph by concatenating the column header, cell text, and hyperlinked text if any. The average length of the documents is 9345.5 tokens.
Dataset QASPER (Dasigi et al., 2021) is a QA dataset constructed from NLP papers. The dataset contains a mixture of extractive, abstractive, and yes/no questions. We experiment with the subset of extractive questions (51.8% of the datasets) in this paper. Some questions in the dataset are answerable with a single-hop of retrieval. However, as described in the original paper, 55.5% of the questions have multi-paragraph evidence, and thus aggregating multiple pieces of information should improve the accuracy. Answers in the QASPER dataset have an average of 14.4 tokens. We treat each subsection as a paragraph and prepend the section title and subsection title to the beginning of the subsection.
Implementation Details Question embeddings q0,q1 are initialized from Eq. 2 and updated as in Eq. 7. For the two datasets, we perform 2-hop attention: the first hop is trained to attend to a paragraph, and the second hop to attend to a sentence. Note that we do not require that sentences attended to in the second hop must come from the previously attended paragraphs. The attention scores of paragraphs and sentences are linearly combined to find the best sentence that contains the answer, which will then be read by a BERT-based reader to extract the final answer. The final attention score of a sentence sij ∈ pi is
score(sij) = q T 1 s i j + λ1 · qT0 pi + λ2 · sparse(q0, pi) (8)
3We modify the original ShARC and HotpotQA datasets to evaluate them in the long-document setting. Please see the dataset descriptions for more details.
where qT1 sij and qT0 pi are the attention scores at sentence and paragraph levels, and sparse(q0, pi) is the similarity score using sparse features.4 λ1 and λ2 are hyper-parameters tuned on dev data. We set λ1 = 1.5 and λ2 = 3.0 for HybridQA, and λ1 = 0.5 and λ2 = 0.0 for QASPER.
Baselines We compare DOCHOPPER with the previous state-of-the-art models, MATE (Eisenschlos et al., 2021) and LED (Dasigi et al., 2021), and several other competitive baselines to show the efficacy of DOCHOPPER. MATE (Eisenschlos et al., 2021) is a pretrained Transformer model with sparse attention between cells that is specifically designed for tabular data. HYBRIDER (Chen et al., 2020) is a pipeline system that (1) links cells, (2) reranks linked cells, (3) hops from one cell to another, and (4) reads text to find answers. All four stages are trained separately. LED is an encoderdecoder model that builds on Longformer (Beltagy et al., 2020). We also experiment with directly reading the documents with a Transformer-based reader ETC (Ainslie et al., 2020): though it can’t fit the entire document into its input, it is still one of the best models for reading long sequences (up to 4096 tokens). To handle longer documents, we adopt the sequential reading strategy: the model reads the document paragraph by paragraph, and picks the most confident prediction as the answer. We also report the numbers of a “retrieve and read” pipeline with a BM25 retriever and a finetuned ETC reader. The numbers are shown in Table 1. Runtime is measured as examples per second with a batch size of 1.
Results and Analysis On QASPER, DOCHOPPER outperforms the previous state-of-the-art models by 3-5%, and runs more than 10 times faster. DOCHOPPER performs worse than MATE (Eisenschlos et al., 2021) on HybridQA. This is due to three reasons. (1) MATE is specifically designed and pretrained to understand tabular data, while DOCHOPPER is applied to general documents. (2) To convert tables to DOCHOPPER’s input, DOCHOPPER serializes the tables by rows and thus loses information from other table structures, e.g. cells and columns, that are commonly used to understand tabular data. In an ablated experiment with the assumption that sentences are grouped by cells, we let DOCHOPPER return cells that contain the selected sentences and pass the cells (instead of single sentences) to the underlying extractive QA model (labeled w/ cell). This improves the performance by 5.4 points. (3) MATE restricts the length of text in cells to 5 sentences and enforces it by retrieving top-k sentences from the hyperlinked text. It also restricts the total length of tables to 2048 tokens. DOCHOPPER does not put any restrictions on the length of cell or the length of tables, and can thus be easily applied to more general tasks.
We additionally performed more ablated experiments with DOCHOPPER. The query update (see the row w/o query update) in Eq. 7 is also important, causing 3.5% and 1.8% difference in performance on both datasets. We also ablated the model by using one step of attention to select the most relevant sentence from the document (single-hop), and note again that performance drops noticeably. Adding one more step of attention, while only attending to sentences (sentence-only in the table), leads to
4Similar to the baseline model (Chen et al., 2020), we use paragraph-level sparse features to improve accuracy. In HybridQA, sparse(q0, pi) computes the length of longest common substrings in the question q0 and the paragraph pi. Sparse features are only used at the end of retrieval, not at any intermediate steps.
HotpotQA-Long Answer Support
IRRR (direct) 53.6 / 64.8 44.7 / 73.2 IRRR (rerank) 62.1 / 75.6 54.5 / 81.3
DOCHOPPER (direct) 57.4 / 69.5 45.1 / 73.8 DOCHOPPER (rerank) 66.5 / 79.7 61.4 / 85.9
ETC (distractor) – / 81.7 – / 89.4 HGN (distractor) – / 83.4 – / 89.2
Table 2: EM/F1 performance on answer and supporting evidence on HotpotQA (dev set) with full Wikipedia pages as context. HGN Fang et al. (2019) and ETC Fang et al. (2019) are evaluated on the distractor setting.
ShARC-Long Easy Strict runtime
ETC 61.1 – 11.2/s Retrieval + DISCERN 65.2 50.7 42.0/s Sequential (DISCERN) 63.7 54.2 8.8/s
DOCHOPPER 72.3 60.2 164.3/s (w/o query update) 72.4 59.9 – (sentence only) 62.2 43.2 – (single-hop) 68.0 52.7 –
Table 3: Classification accuracy on ShARC-Long dataset. The Easy setting only checks the predicted labels, while the Strict setting additionally checks if all required evidences are retrieved. DISCERN is run with their open-sourced codes.
some improvement, but is still worse than attending at both paragraph and sentence levels. The performance of sentence selection (with ablated numbers) is presented in the Appendix.

heading: 4.2 HOTPOTQA-LONG

text:
Dataset HotpotQA requires multi-hop reasoning to answer two types of questions, bridge and comparison. The original dataset provides the first paragraphs of 10 Wikipedia pages as its context. To test the model’s ability to extract information from a longer context, we use first 2048 tokens of the 10 Wikipedia pages, and concatenate the 10 pages into a single document. This increases the average length of context from 897 to 9970 tokens. We call this variant of the dataset HotpotQA-Long.
Implementation Details The experiment setup for HotpotQA-Long is similar to HybridQA and QASPER. Attention for this dataset is supervised, using the information about which evidences support each answer provided in the dataset. Here, we refer to a Wikipedia page (with a maximum of 2048 tokens) as a paragraph. The iterative attention is performed similarly to §4.1, but repeated for 4 hops: paragraph, sentence, paragraph, and sentence. The first two hops are supervised by the first supporting evidence, and the last two hops are supervised by the second supporting evidence. The final score for each evidence is computed as in Eq. 8 with λ1 = 0.7 for the first evidence and λ1 = 0.0 for the second, and λ2 = 0 for both. As suggested by Xiong et al. (2021), reranking can improve the overall performance, so we take the top 4 sentences for each step and rerank the 16 combinations.5
Baselines We compare DOCHOPPER with IRRR (Xiong et al., 2021) which is an multi-hop retrieval model that iteratively retrieves a small piece of evidence and re-encodes query vectors at each step by concatenating tokens from the retrieved passages at the previous step.6 We compare to IRRR both with or without the reranking step. Since questions in HotpotQA require combining evidences from multiple paragraphs, it’s not clear how to run the sequential reading baseline on HotpotQALong, but as a reference, we show the performance of ETC and HGN in the distractor setting (with the original context, which is 10x smaller) in Table 2.
Results DOCHOPPER outperforms the baseline models by 4-6 points on the accuracy of both answer and supporting evidence. More surprisingly, the performance of DOCHOPPER in the long document setting is only ∼4 points lower than the state-of-the-art on the distractor setting, even though the context is more than 10 times longer. Please see the Appendix for ablated experiments and analysis.

heading: 4.3 CONVERSATIONAL QA: SHARC-LONG

text:
Dataset ShARC Saeidi et al. (2018) is a conversational QA dataset for discourse entailment reasoning. Each example consists of a document that describes a government policy and a conversation history about the document between the machine and the user. The conversation starts with an initial questions asked by the user and follows by a sequence of clarification questions and answers collected from the interaction between the machine and the user. The model takes the conversation
5Please refer to Xiong et al. (2021) for more details on the reranking step. 6IRRR is the best model (ranked #9 on leaderboard) on HotpotQA (fullwiki) with open-sourced codes.
history as input and predicts the answer to the initial question. An answer has one of the four labels: “Yes”, “No”, “Irrelevant”, or “Inquire”. “Irrelevant” means the question is not related to the provided context and “Inquire” means there’s not enough information to answer the question. Similar to HotpotQA-Long, we expand the original context to test the long document setting. The expanded context contains 737.1 tokens on average, 13.5 times longer than the original context. Please see the Appendix for more details and examples.
Implementation Details The query embeddings q0, . . . ,qn are initialized from Eq. 1 where n is the number of followup questions in the conversation. The query vector qt at the step t will be updated with the residuals following the update rule in Eq.7. The iterative attention process is distantly supervised at intermediate steps. See the Appendix for information on distant supervision.
We add a simple classification layer on DOCHOPPER to make the final prediction. As input to this layer, we reuse the concatenated vector kj in Eq. 5. Let k (t) j be the concatenated vector of the sentence attended at the t’th retrieval step, either a sentence directly attended or associated from a attended paragraph. Concatenated embeddings at all attention steps {k(0)0 , . . . ,k (0) ∗ , . . . ,k (n) 0 , . . . ,k (n) ∗ } are linearly combined into one vector k̃ that will be used to make the final prediction, using weights γ(t)j computed across the attended sentences from all steps t. m ∈ R4 holds the logits of the 4 classes that is used to compute the softmax cross entropy with the one-hot encoding of the positive class labels.
k̃ = ∑ t ∑ j γ (t) j k (t) j , γ (t) j = softmax(u T k(t)j ), m = W T c k̃ ∈ R4
Baselines We compare our model with ETC and a “retrieve and read” baseline. In using ETC, we concatenate the full context and the conversation into a single input, and prepend a global [CLS] token. The embedding of the global [CLS] token will be used to make the final prediction. For the “retrieve and read” pipeline, we adopt the previous state-of-the-art model DISCERN as the reader, and pair it with a learned retriever.7 We also run a sequential reading baseline with DISCERN where documents are chunked every 128 tokens with a stride of 32 tokens and then read with DISCERN, predicting the class with the highest probability among all chunked inputs.8Runtime is measured as examples per second.
Results The evaluation is performed in two settings: Easy and Strict. The Easy setting only evaluates the accuracy of classification. In the Strict setting, we additionally require that all evidences (provided in the original dataset) are retrieved. We report the (micro) accuracy as the evaluation metric. DOCHOPPER outperforms all baseline models by more than 7 points in both easy and strict settings, while being more than 3 times faster than all baseline models. We performed the previous ablated experiments and observed similar changes in performance. Please see the Appendix for more numbers on evidence selection.

heading: 5 CONCLUSION

text:
We consider on the problem of answering complex questions over long structured documents. Like multi-hop open QA tasks, this problem requires not only conventional “machine reading” abilities, but the ability to retrieve relevant information and refine queries based on retrieved information. Additionally, it requires the ability to navigate through a document, by understanding the relationship between sections of the document and parts of the question. In our framework, navigation is modeling similarly to retrieval in multi-hop models: the model attends to a document section, and uses a compact neural encoding of the section to update the query. Unlike most prior multihop QA models, however, queries are updated in embedding space, rather than by appending to a discrete representation of question text. This approach is end-to-end differentiable and very fast. Experiments also demonstrate that this use of hierarchical attention can significantly improve the performance on QA tasks: in fact, the DOCHOPPER model achieves the start-of-the-art results on four challenging QA datasets, outperforming the baseline models by 3–5%, while also being 3–10 times faster.
7The retriever is a ablated DOCHOPPER model that attends at the paragraph level only. 8The model is often extremely confident on the “Irrelevant” class because most chunked inputs are obviously irrelevant. We tuned a hyper-parameter = 0.99 so the model predicts “Irrelevant” if the probabilities of all chunked inputs are larger than .

heading: 6 ETHICS STATEMENT

text:
Pretrained language models (LM) are effective in many tasks but are criticized due to their high training cost and environmental concerns. Recently, more attention has been brought into development efficient models. The DOCHOPPER model proposed in this paper follows research in this direction. DOCHOPPER exploits the existing language models released by other research institutes. Without any additional pretraining, DOCHOPPER achieved competitive or state-of-the-art results in several challenging QA datasets. All experiments in this paper were conducted with a single GPU.
Besides being efficient in training, DOCHOPPER is also extremely efficient during inference. Experiments show that DOCHOPPER runs 3-10 times faster compared to existing QA models and thus could be deployed using limited computation resources.

heading: 7 REPRODUCIBILITY STATEMENT

text: Experiments in this paper are conducted on publicly available datasets. Codes and processed data will be open-sourced upon the acceptance of this paper.

heading: A APPENDIX

text:
A.1 DATASET DETAILS
HybridQA (Chen et al., 2020) is a dataset that requires jointly using information from tables and text hyperlinked from table cells to find the answers of multi-hop questions. A row in the table describes attributes of an instance, for example, a person or an event. Attributes are organized by columns. For example, the table of Medalist of Sweden in 1932,9 contains a row “[Medal:] Gold; [Name:] Rudolf Svensson; [Sport:] Wrestling (Greco-Roman); [Event:] Men’s Heavyweight”. Text in the square brackets are the headers of the table. The medal winner “Rudolf Svensson” and the event “Wrestling (Greco-Roman)” are hyperlinked to the first paragraph of their Wikipedia pages. A question asks “What was the nickname of the gold medal winner in the men ’s heavyweight grecoroman wrestling event of the 1932 Summer Olympics?” requires the model to first locate the correct row in the table, and find the answer from other cells in the row or their hyperlinked text.
To apply our model on the HybridQA dataset, we first convert a table with hyperlinked text into a long document. Each row in the table is considered a paragraph by concatenating the column header, cell text, and hyperlinked text if any. The column name and cell text are each treated as one sentence. Hyperlinked text is also split into sentences. In the example above, the row becomes “Medal. Gold. Name. Rudolf Svensson. Johan Rudolf Svensson (27 March 1899 – 4 December 1978) was a Swedish wrestler. He competed ...”. The average length of the documents is 9345.5.
QASPER (Dasigi et al., 2021) is a QA dataset constructed from NLP papers. They hired graduate students to read the papers and ask questions. A different group of students are hired to answer the questions. For example, a question asks “What are the baseline models used in this paper?”. The answers are {“BERT”, “RoBERTa”}. The dataset contains a mixture of extractive, abstractive, and yes/no questions. We focus on the subset of extractive questions (51.8% of the datasets) in this paper. Some questions in the dataset are answerable with a single-hop. However, as suggested in the original paper, 55.5% of the questions have multi-paragraph evidence, and thus aggregating multiple pieces of information should improve the accuracy. Answers in the QASPER dataset are longer, with an average of 14.4 tokens. We treat each subsection as a paragraph and prepend the section title and subsection title to the beginning of the subsection.
ShARC Saeidi et al. (2018) is a conversational QA dataset for discourse entailment reasoning. Questions in ShARC are about government policy crawled from government websites. Users engage with a machine to check if they qualify for some benefits. A question in the dataset starts with a
9https://en.wikipedia.org/wiki/Sweden_at_the_1932_Summer_Olympics
initial question, e.g. “Can I get standard deduction for my federal tax return?”, with a user scenario, e.g. “I lived in the US for 5 years with a student visa”, and a few followup questions and answers through the interaction between the machine and users, e.g. “Bot: Are you a resident alien for tax purpose? User: No”. The model reviews the conversation and predicts one of the three labels: “Yes”, “No”, or “Irrelevant”. If the model think there’s not enough information to make the prediction, it should predict a fourth label “Inquire”.
Besides the conversation, each example in the ShARC dataset provides a snippet that the conversation is originated from. A snippet is a short paragraph that the conversation is created from, e.g. “Certain taxpayers aren’t entitled to the standard deduction: (1) A married individual filing as married... (2) An individual ...”. Since the snippets are usually short, with an average of 54.7 tokens, previous models, e.g. DISCERN Gao et al. (2020), concatenate the snippet and the conversation, and jointly encode them with Transformer-based models, e.g. BERT or RoBERTa. Here we consider instead a more challeging long-document setting, in which the snippet is not known, and the model must also locate the snippet from the document. We crawl the web pages with the provided URL. The pages contain 737.1 tokens on average, 13.5 times longer than the original snippets, and the longest page contains 3927 tokens. We name this new variant ShARC-Long.
A.2 IMPLEMENTATION DETAILS FOR SHARC-LONG
Changes to Context Representations Instead of computing the paragraph embeddings as a weighted sum of sentence embeddings, we directly obtain the paragraph embeddings from ETC output for this dataset. Recall that a paragraph pi = {si0, . . . , si|pi|} contains a sequence of sentences sij . We prepend a dummy sentence snull to the beginning of the paragraph, and again, we modify the global-to-local attention mask to allow the global token of the dummy sentence to attend to all tokens in the paragraph pi. Let pi ∈ Rd be the embedding of paragraph pi. The embeddings for a paragraph and its contained sentences are:
pi, s i 0, . . . , s i |pi| = ETC({snull, s i 0, . . . , s i |pi|})
Distant Supervision The iterative attention process is distantly supervised with supervision at intermediate steps. At each step, the model is trained to attend to both the correct paragraph and the correct sentences if they exists. Since the embedding table Cd consists of both paragraph and sentence embeddings, we only need to compute the attention scores once at each step, but consider both the correct paragraph and the correct sentence as positive. The positive paragraph is one of the paragraphs from the crawled web page with the highest BLEU score.10 We notice that some web pages at the provided URLs have been changed significantly, so the snippets provided in the datasets may not exist any more, hence we discard the associated data if the highest BLEU scores of the paragraphs is less than 0.7. We follow the heuristics used by baseline models (Gao et al., 2020) to get positive sentence candidates by finding the sentence with the minimum edit distance.
A.3 ADDITIONAL RESULTS
We report the performance of eventually selecting the correct evidences in Table 4, 5, and 6.
Comments on HotpotQA-Long We also observe that ablated experiment on evidence selection (w/o query update) is only 7.8 points lower than the full model. To understand the underlying
10We drop the brevity penalty term in BLEU score.
reason, we train the model to perform a one-step attention only for supporting facts of the second hop (for bridge questions). The accuracy is 71.7, only 3.6 points lower than the accuracy of the full multi-hop process. This is likely due to the high surface form overlap between the questions and their context.


summaries: The paper proposes a label refinement approach. Starting from an initial set of labels, k-means clustering is used to refine the labels. The approach is described in the context of dubbing/voice casting. The method tries to obtain voice characteristics which can be further used for dubbing/voice-casting. Experiments are shown on video game datasets MassEffect and Skyrim and they primarily investigate how different parameters of k-means (# of clusters) and distance measure affect the label refinement.

summaries: The paper tackles a voice similarity system task for voice casting problem. The authors trained voice embedding network using voice character label and cluster the embedding features and used these clusters to train final voice embedding network. The introduction is well-written, however, the proposed method is known or marginal improvements from the existing technical skill in machine learning community (pseudo labeling with embedding feature clustering).

summaries: This paper addresses the task of finding similar-sound voices with application to voice-dubbing (e.g. finding an actor to record dialog in English, translated from original French dialog, such that the English speaker sounds similar to the original French speaker). The paper proposes a method called "label refining". This method is based on p-vectors, a prior representation found to model similarity between characters. The method uses 1) non-expert "initial labels" to train a p-vector system 2) use k-means to cluster the resulting p-vectors and 3) use the groups learned by k-means to re-group the labels (the "refined labels"). To evaluate the method, English-French pairs of voice data from the video games Mass-Effect 3 and Skyrim are used. Performance is measured using clustering measures v-measure and purity-K, as well as accuracy on the test set (which it seems the ground-truth labels are mapping to the correct dubbing speaker). The method achieves a higher accuracy of 0.70 over a state-of-the-art system that achieves 0.69. A second experiment is performed using Skyrim data as a subsidiary corpus used to cluster, with the goal of "bringing out vocal characteristics" from the initial labels, which shifts the optimal K towards 2, which the paper claims is because the new representations start to model gender.

summaries: The paper presents a "label-refining" technique, which helps users pick voices to provide a better user experience in dubbed video games. The idea is that a voice talent's voice in the target language should match the character's voice characteristics in the source language. The method seems to work by attaching labels to data-driven clusters, and refining these using a second corpus, on which the labels' value for discrimination is measured.

abstractText: A characteristic is a distinctive trait shared by a group of observations which may be used to identify them. In the context of voice casting for audiovisual productions, characteristic extraction has an important role since it can help explaining the decisions of a voice recommendation system, or give modalities to the user with the aim to express a voice search request. Unfortunately, the lack of standard taxonomy to describe comedian voices prevents the implementation of an annotation protocol. To address this problem, we propose a new semi-supervised learning method entitled Label Refining that consists in extracting refined labels (e.g. vocal characteristics) from known initial labels (e.g. character played in a recording). Our proposed method first suggests using a representation extractor based on the initial labels, then computing refined labels using a clustering algorithm to finally train a refined representation extractor. The method is validated by applying Label Refining on recordings from the MassEffect 3 video game. Experiments show that, using a subsidiary corpus, it is possible to bring out interesting voice characteristics without any a priori knowledge.

authors:
J. Ajmera
H. Bourlard
I. Lapidot
I. McCowan

title: Unknown-multiple speaker clustering using hmm

venue: In INTERSPEECH,

authors:
Yoshua Bengio
Aaron Courville
Pascal Vincent

title: Representation learning: A review and new perspectives

title: Representation learning for underdefined tasks

venue: Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications,

authors:
J.S. Chung
A. Nagrani
A. Zisserman

title: Voxceleb2: Deep speaker recognition

venue: In Interspeech,

authors:
Filip Karlo Došilović
Mario Brčić
Nikica Hlupić

title: Explainable artificial intelligence: A survey

venue: 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO),

authors:
Adrien Gresse
Mickael Rouvier
Richard Dufour
Vincent Labatut
Jean-Francois Bonastre

title: Acoustic pairing of original and dubbed voices in the context of video game localization

authors:
Adrien Gresse
Mathias Quillot
Richard Dufour
Vincent Labatut
Jean-François Bonastre

title: Similarity metric based on siamese neural networks for voice casting

venue: In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),

authors:
Adrien Gresse
Mathias Quillot
Richard Dufour
Jean-Francçois Bonastre

title: Learning Voice Representation Using Knowledge Distillation For Automatic Voice Casting

authors:
Ping Hu
Dongqi Cai
Shandong Wang
Anbang Yao
Yurong Chen

title: Learning supervised scoring ensemble for emotion recognition in the wild

venue: In Proceedings of the 19th ACM International Conference on Multimodal Interaction,

title: Convergence problems of mahalanobis distance-based k-means clustering

venue: IEEE International Conference on the Science of Electrical Engineering in Israel (ICSEE),

authors:
Aansh Malik
Ha Nguyen

title: Exploring automated voice casting for content localization using deep learning

venue: SMPTE Motion Imaging Journal,

authors:
Nicolas Obin
Axel Roebel

title: Similarity search of acted voices for automatic voice casting

authors:
Nicolas Obin
Axel Roebel
Grégoire Bachman

title: On automatic voice casting for expressive speech: Speaker recognition vs. speech classification

authors:
Raghuveer Peri
Monisankha Pal
Arindam Jati
Krishna Somandepalli
Shrikanth Narayanan

title: Robust speaker recognition using unsupervised adversarial invariance

authors:
Mathias Quillot
Richard Dufour
Jean-François Bonastre

title: Assessing speaker-independent character information for acted voices

venue: In Alexey Karpov and Rodmonga Potapova (eds.), Speech and Computer,

authors:
Mathias Quillot
Jarod Duret
Richard Dufour
Mickael Rouvier
Jean-François Bonastre

title: Influence of speaker pre-training on character voice representation

authors:
Andrew Rosenberg
Julia Hirschberg

title: V-measure: A conditional entropy-based external cluster evaluation measure

venue: In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL),

title: An evaluation of forensic similarity hashes

venue: Digital Investigation,

heading: 1 INTRODUCTION

text:
Localization of a multimedia work, such as a video game or a motion picture, consists in making changes and adaptations to the original work so that international distributors can market the resulting product in a target country. For this purpose, companies need to translate their works into a target language. Various options are available to them to give substance to their translation. Subtitling is the simplest and cheapest option, but not the most practical one for spectators: a large audience is more comfortable with hearing speech, in general in their mother tongue, rather than reading subtitles while hearing speech in another language. Dubbing, which is more expensive and takes longer to set up, better retains the audience’s immersion by replacing the original voices with dubbing comedian voices. It involves a voice selection process, which means choosing among several candidate voices in a target language. This selection is referred to as voice casting.
With the emergence of streaming platforms (e.g. Netflix, Disney+, Amazon Prime Video) and the rapid development of the video game industry, the number of works to be localized is increasing dramatically. As a result, more and more actors are available on the voice market. Vocal Casting is an artisanal process that requires a lot of precision and is carried out by human operators. As it is, it cannot be applied to a very large number of actors. Operators very often rely on their memory, which is limited to a small number of actors with whom they have worked and in whom they have confidence. However, this operation can degrade the immersion of the audience since the most oftused voices become easily recognizable. They then become references and create cognitive biases in the public’s perception of the character. Automatic actor search and recommendation tools, based on voice processing, would help operators find new actors who would then enrich the vocal diversity of the works to the delight of the audience.
In this article, we wish to address some of the complexity of voice casting and we start with the voices of professional actors playing characters for the gaming industry. The articles Gresse et al. (2020; 2019; 2017); Malik & Nguyen (2021); Obin & Roebel (2016); Obin et al. (2014); Quillot et al. (2021b;a) cited in this paper are the only ones dealing with the speech-processing-based automation
of Voice Casting in the literature to our knowledge. We rely on Gresse et al. (2020), a recent work which defines the p-vector, a neural-network-based representation of the voice dedicated to the character aspects in acted voices. The application context in Gresse et al. (2020) (like in related previous works Gresse et al. (2017; 2019)) is voice casting for audio dubbing. The final objective of this work is to assess how close an actor – acting in a target language – is to the voice of a given character, speaking in a source language. This can be referred to as character-based voice similarity. In contrast to Obin et al. (2014); Obin & Roebel (2016), which proposed a voice similarity system for the acted voice based on data labeled with speech classes (age, gender, emotions ...) by an expert, the p-vector approach does not use any expert labels. This representation is learnt from audio associated with the played character label, without any additional information. Also, the authors of Quillot et al. (2021a) recently posed the following hypothesis: when a character is played, the actor adds to the voice some information that characterizes the character and is independent of the speaker identity. They then introduced SICI (Speaker-Independent Character Information) and showed that there is speaker-independent character information in the acted voice and that it is present in p-vectors pretrained on a speaker task Quillot et al. (2021b).
Although these approaches deal with final works (video games), another one consists in working directly on the decision data from Artistic Directors. Recently, researchers from Warner Bros. evaluated their models on this kind of data. Unfortunately, these data are sensitive and their acquisition is not trivial since it requires to work on the critical voice casting process. This is why we decided to position ourselves in a task quite different from Malik & Nguyen (2021), since we do not use Artistic Director decisions.
Voice similarity systems alone are not enough for Voice Casting operators. They do not trust the computed recommendations and need a more complete interface to interact with the system. Many works propose explainability approaches Došilović et al. (2018) but these are too complicated to be understood by a user who is not expert in computer science, and more particularly in machine learning. We strongly believe that the introduction of explanatory labels will help the operators understand the decisions by giving, for example, the list of speech characteristics shared by two recordings/actors. Such labels could also be used to express queries in voice search systems. Unfortunately, voice casting is an underdefined task Bonastre (2019). The operators themselves cannot formally define their decision criteria. There is also no standard taxonomy in cinema to describe voices, making it impossible to create an annotation protocol. The question is then: how can we extract features from voices when we have no a priori knowledge of them? Different representation learning approaches exist in the literature, but none of them corresponds to our expectations. 1) Feature extraction surveyed in Bengio et al. (2014) consists in representing an observation by a vector of reals but not in the form of binary vectors that we call characteristics in this paper. 2) Supervised methods like Hu et al. (2017) require labels that we do not have access to in order to extract characteristics. 3) Unsupervised methods like Peri et al. (2020) propose generic representations from which we could not extract characteristics dedicated to the domain of acted voice and character identity.
Hence, to address the lack of method to extract characteristics without ground truth, this article proposes a totally new approach entitled Label Refining. It stats with training a representation extractor using non-expert initial labels. A clustering algorithm is then employed with the purpose to emphasizing characteristics shared by groups of observations which we term refined labels. A last step consists in extracting a new representation from refined labels and comparing it with the initial representation by evaluating it on a downstream task. Unlike generic representations that use no label, this method has the advantage of guiding the extracted representations by using the non-expert initial labels.
To validate this method, we apply it to dubbing and more specifically to the extraction of characteristics from actors’ voices. We therefore p-vector features from characters played in a video game. We apply a clustering algorithm, k-means, in the p-vector space to extract refined labels. In order to evaluate the quality of the refined labels, we train a neural network to extract refined p-vectors. We then subject them to a voice character similarity task to compare them with the initial p-vectors and thus measure the loss or gain of dedicated character information. In this experiment, we test the following hypothesis: during the clustering step, using the recordings of the training corpus only highlights the initial labels, the characters. During this step, a second corpus (close to the domain) is needed to highlight the vocal features extracted by the p-vectors. Since distance plays an essen-
tial role in the k-means algorithm, we also explore the use of cosine and Mahalanobis distances to compare with Euclidean one.
Section 2 presents Label Refining. We define the term “refined label” and present the method. The experimental protocol to apply Label Refining is described on Section 3. In Section 4, we present experiments and discuss about the obtained results. For reproducibility reasons, scripts and models are available on GitHub1. Finally, we conclude by presenting some takeaways and possible directions for future work in Section 5.

heading: 2 LABEL REFINING

text: Before experimenting, we first introduce the Label Refining method. It consists in extracting refined labels from initial ones. In this section, we give a definition of refined labels and we detail the method of extraction and evaluation.

heading: 2.1 THE DEFINITION OF REFINED LABELS

text:
In this work, we assume that labels are decomposable into various lower-level sublabels. In our case, we wish to use the knowledge provided by the played character labels associated to the recordings with the aim to infer the characteristics such that the emotion played (e.g. agressive, happy), a voice color (e.g. white, red) or a timbre (e.g. husky, enlightened).
Equation 1 formalizes the definition of what a refined label is. Let R the set of all the refined labels and a(l)i ∈ {r1, r2...r|R|} the ith refined label associated to the initial label l. Let us also specify that the number n of refined labels is different depending on l. Thus, from the knowledge given by initial labels in L we wish to find the refined labels a(l)i of each l.
l  a (l) 1 a (l) 2
... a (l) n
(1)
Let us note the functions f : e 7→ l and g : e 7→ r where o is an observation (e.g. vocal recording). The equation 2 gives fictives examples. It shows the expected images of the functions f and g according to two initial labels l1 and l2. In our model of the problem, we assume that a refined label r may appear in observations associated to different initial labels. Here, this is the case with r1 which appears both in an observation associated to l1 and an observation associated to l2.
l1
{ f : o
(l1) 1 7→ l1 f : o (l1) 2 7→ l1 , l1
{ g : o
(l1) 1 7→ r1 g : o (l1) 2 7→ r2 , l2
{ f : o
(l2) 1 7→ l2 f : o (l2) 2 7→ l2 , l2
{ g : o
(l2) 1 7→ r1 g : o (l2) 2 7→ r3
(2)

heading: 2.2 OUR PROPOSITION TO REFINE LABELS

text:
Figure 1 presents the refined label extraction method that we propose in three steps. 1 consists in training a system to extract a representation (i.e. vector, hash Roussev (2011)) of the observation o ∈ O using the initial labels l ∈ L (e.g. label of the character played in a recording). 2 consists in using a clustering algorithm in order to associate new labels r ∈ R to the observations O. 3 consists finally in training an new system to extract refined labels by using labels r.
In order to evaluate the relevance of labels r, we use two methods. The first method, taking place at step 2 , consists in applying different clustering quality measures such as v-measure Rosenberg & Hirschberg (2007) or the purity Ajmera et al. (2002). Applied to the step 1 and step 3 , the second method consists in extracting a representation of the observations O and evaluating it on
1To keep the identity of the authors secret for the reviewers, the GitHub URL will be added if the paper is accepted.
a downstream task. The downstream task can be diverse: speaker recognition, speech recognition or, in our case, character voice similarity. Thanks to the second method, it is possible to highlight the gain or loss of information by comparing the results obtained at steps 1 and 3. In this work, we postulate that the refined labels are pertinent if they do not lead to a loss of information when extracting refined representation.
Depending on the clustering algorithm, it is likely to have to try different values of k. In the following experiments, we use k-means and test different values of k. In this context, we apply the method for all the k values and compare the obtained results. We postulate that Label Refining is validated if the number of refined labels k differs from the number of characters and if the character information is at least preserved in the refined representation.

heading: 3 EXPERIMENTAL PROTOCOL

text: In the previous section, we introduced the generic method for extracting refined labels. We describe in this Section the protocol we follow to validate the method. We apply Label Refining on acted voice recordings in order to extract voice characteristics dedicated to the character played. A resume of the protocol is presented on the figure 2. Each step is detailed on the next subsections.

heading: 3.1 MAIN (MASSEFFECT) AND SECOND (SKYRIM) CORPORA

text:
We use MassEffect data as the main corpora to train feature extractor and evaluate our system on downstream task. We also explore the use of a second dataset from the video game Skyrim during
the second step (i.e. clustering algorithm) with the aim to emphasize voice characteristics models by previous trained p-vectors. This subsection present both MassEffect and Skyrim datasets.
3.1.1 MASSEFFECT, THE DATASET USED TO TRAIN AND EVALUATE p-VECTORS
The main corpus is composed of voice recordings coming from the Mass-Effect 3 role-playing game. Contrary to movies, voices of video games present some particularities explained in Gresse et al. (2019) (radio effects are present in the original voices of our corpus). They are also easier to collect since they are separated from the ambient sounds even in their final form, contrary to the dubbed film archives. Originally released in English, the game has been translated and revoiced in several other languages. In our experiments, we use the English and French versions of the audio sequences, representing about 7.5 hours of speech in each language. Segments (or recordings) are 3.5 seconds long on average. A character is then defined by a unique French-English couple of two distinct speakers. To avoid speaker identity biases, we consider only a small subset where we are certain that none of the actors play more than one character. A single audio segment corresponds to a unique speaking slot from an actor in a particular language. We then apply a filter that keeps only recordings longer than 1 second. Finally, we only keep the 16 characters for which we have the largest number of recordings, as Gresse et al. (2019) did.
In order to measure the ability of our systems to generalize, we propose a breaking down protocol. We only keep 12 characters, and their 24 corresponding speakers, for the training phase and we use the remaining 4 characters for the testing phase. We break the corpus down into training, validation and test sets. The train and val sets are composed of different recordings, but coming from the same 12 characters. To build the train, val and test subsets, we randomly select for each character 144, 36 and 180 recordings respectively, while balancing the number of French and English recordings. We then have a total of 1, 728 (train), 432 (val) and 720 (test) recordings. We name the three sets Sn.

heading: 3.1.2 SKYRIM, A SUBSIDIARY DATASET FOR EMPHASIZING VOICE CHARACTERISTICS

text: Skyrim is also a known video game. We use its recordings as a second corpus in addition to the one of MassEffect previously presented. This dataset can be viewed as a validation set. We propose to use this corpus to emphasize the characteristics of voices. Indeed, the validation set of MassEffect suffers from a lack of variability since it contains the same characters as the training set. Unfortunately, we do not have available MassEffect data with unknown characters to add to the validation set since they are used on the test. Thus, we assume that using Skyrim as a second validation set will help to bring more variability and highlight voice characteristics extracted from the p-vectors trained on MassEffect. This subset is composed by 18, 250 recordings of different 32 characters. As on MassEffect, each recording played in english is associate to a french one and vice-versa but we don’t balance the number of recording by character.

heading: 3.2 EXTRACTING VOICE REPRESENTATION FROM CHARACTER LABELS

text: This part of the protocol corresponds to the step 1 described in the Figure 2. In order to extract the p-vectors, we follow scrupulously the best architecture described in Quillot et al. (2021b). Training this architecture need to follow two steps: 1) training a speaker identification system following a x-vector architecture on Voxceleb 2 corpora (Chung et al. (2018)) 2) replacing the output softmax of this system with p-vector layers, and training it to recognise the characters of the MassEffect dataset.

heading: 3.2.1 REFINING LABELS WITH K-MEANS ALGORITHM

text: This part of the protocol corresponds to the step 2 described in the Figure 2. In this work, we wish to validate our method described in 2.2. For this purpose, we focus on the k-means algorithm. Unlike others, this algorithm is easier to comprehend. We use the implementation of the toolkit sklearn with the default parameters, namely: k − means + + as initializer of the barycentres in order to converge more quickly to the solution. The max number of iterations is 300 of non convergence. The algorithm is runned 10 times, then we select the clustering model with the best intertia.

heading: 3.3 EXTRACTING REFINED VOICE REPRESENTATION BY USING CLUSTERING GROUPS

text: This part of the protocol corresponds to the step 3 described in the Figure 2. Once the clustering algorithm is complete, we build for each k-value a new neural network using the same neural architecture as used for the p-vector. Instead of predicting the initial labels, this new network is trained to predict the refined labels r ∈ R using the clustering algorithm. After training, we can extract from its last layer a representation of the provided record that we name p-refined-vectors. The used architecture is exactly the same as for the step 1 since this neural network is also firstly pre-trained on a speaker identification task by using Voxceleb 2. Only the last layer of the p-vector network is changed to correspond to the number k of groups (of the refined labels r) instead of the number of character labels l.

heading: 3.4 EVALUATION

text: In this subsection, we present the two ways to evalute the quality of the refining. The further one is computing clustering metrics. The second one is evaluating the representation extracted using the labels on a downstream task. We give all the details in this subsection.

heading: 3.4.1 COMPUTING CLUSTERING METRICS

text:
To evaluate a clustering model, we use the following measures: entropy, v-measure Rosenberg & Hirschberg (2007), class purity, group purity, purity-K Ajmera et al. (2002). These measures require to know for each entity in the database the group to which the algorithm has associated it as well as its original label. Since their formal definition are defined in the cited papers, we only present below their meaning and behaviour.
These measures are calculated from two different perspectives. The entropy, homogeneity and group purity measures check whether each group is associated with entities belonging to the same class. They provide a group view. The completeness and class purity measures check if each class is associated to entities belonging to the same group. They provide a class view.
To analyze the behavior of these measures, let C a clustering model learnt on the observations X with parameter k, the a priori number of clusters. Let f the function which measure the quality of the model C (e.g. entropy) on X . The table 1 shows the behaviour of the measures by computing for each of them the limit of f i k → +∞ et k → 0. These measures all have their meaning and are generally misinterpreted. This is particularly the case for the entropy applied to clustering algorithms. The latter is a measure similar to that of homogeneity that has been inverted. Although it helps to compare two systems, it has a major drawback. It is bounded between 0 and +∞. Interpreting the measure of a single model is therefore difficult unlike homogeneity which is bounded between 0 and 1. The latter thus gives a rate of homogeneity that entropy cannot give. Without going into details here, the group purity is a measure very similar to the homogeneity.
The measures of completeness and class purity, in addition to being very similar, are the antithesis of the previous measures. They are no longer measures of group homogeneity but measures of class homogeneity. The closer the value is to 1, the more the class in question is observed in only one group. The closer the value is to 0, the more the class in question is observed in a large number of groups at similar quantities.
Hybrid measures are weighted averages of class and group measures. Their limitations in k → +∞ and k → 0 are not clear as they depend on the balance of power between homogeneity (group purity) and completeness (class purity).
Demonstrating the limits of each function is quite simple. For homogeneity and group purity, it is maximal when the elements of each group belong to the same class. When we reachk = |X|, each element is then assigned to a cluster and each cluster is assigned to an element. There is then only one class for each group. The homogeneity is thus maximal, the group purity too. In the opposite case, when k = 1, all observations belong to the same group. The group is then maximally heterogeneous and the values of homogeneity and group purity then tend towards 0. For completeness and class purity, the reasoning is similar and then becomes trivial.

heading: 3.4.2 EVALUATING VOICE REPRESENTATION

text: As described in Quillot et al. (2021b), we evaluate p-vectors on a character voice similarity task using a character-similarity measure based on a Siamese network. We generate Target trials composed of pairs of recordings belonging to the same character and Non-target trials made up of recordings belonging to two different characters. To avoid any bias, the number of targets and non-targets is balanced, as well as the number of pairs between two actors. So, we generate 124, 416 (train), 7, 776 (val) and 64, 800 (test) trials using Sn dataset (MassEffect). Skyrim is not used during this evaluation step. The threshold is set a posteriori at the Equal Error Rate (EER) point. System performance is then expressed in terms of accuracy on the test.

heading: 4 EXPERIMENTS AND RESULTS

text:
In the experiments we present in the following sections, we decided to test different values of k in order to limit the number of calculations to be performed. Several values seemed obvious to test such as 12, 24, 4 and 8 which are the number of characters and speakers respectively in the train and test. We also assume that the number of speech features we are looking for is higher than the number of characters our system has to learn to model. We therefore also tested the values 18, 32, 48 and 64 which are arbitrary values but greater than 12. On the other hand, we have also tested k = 2 and k = 6 as values less than 4 and 12 as a counter-hypothesis.
With the aim to compare results obtained on the refined p-vectors, we first evaluate original pvectors. These p-vectors are trained by using Sn. They are evaluated by training 10 siamese neural networks and selecting the best on the validation. The best model gives us 0.69 of accuracy on the test. This corresponds to the actual state-of-art performance presented in Quillot et al. (2021b).

heading: 4.1 REFINING CHARACTER LABELS BY USING MASSEFFECT

text:
In Table 2, we plot the observed metrics v-measure and purity-K on our clustering models computed on the training dataset of Sn (MassEffect). We then train 10 siamese neural networks and select the best on the validation. Its accuracy score on the validation and the test are plotted on the last two lines. For the three metrics presented in the table, the higher the value, the better the system.
On the validation set, the metrics (v-measure and purity-K) highlight the 12 characters contained in it and known to the system. Indeed, we naturally have higher scores for k = 12 with 0.91 for the v-measure and 0.90 for the purity-K.
On the test, we observe a somewhat different phenomenon. Remember that the test is composed of 4 characters. If the points – which are projections of the recordings into p-vector space – were perfectly discriminated by the neural classification system, they would then be organized into groups of records belonging to the same character. We would therefore have stronger measure scores for k = 4. In our case, we can observe this phenomenon with a loss due to the difficulty for the system to separate the records of characters that are unknown to it. Indeed, the highest values are at k = 6 for the v-measure with 0.58 and at k = 2 for the purity with 0.70. These groups are finally the closest to 4 and show a loss of quality of the representations on unknown data. However, they do not show, a priori, the existence of features that could interest us.
Finally, we are interested in the performances obtained on the Siamese networks that we have trained. Recall that refined p-vectors are provided as input to these Siamese networks which are trained on a character verification task – character voice similarity. These vectors are extracted from a neural network trained to recognize the clustering group (refined label) corresponding the given recording (observation). Without surprise, we observe a best accuracy for k = 12. This corresponds to the number of characters. Contrary to the previous metrics, we do not observe a normal distribution in the results. Indeed, when k is moving away from 12, the obtained accuracy is not necessarily lower. For instance, at k = 64 the obtained accuracy is not significantly lower than the one obtained at k = 12. Even more interesting, the accuracy obtained on the test subset is about 0.70 that is higher than the state-of-the-art system, 0.69.

heading: 4.2 REFINING CHARACTER LABELS BY USING SKYRIM AS A SUBSIDIARY CORPUS

text:
In a second experiment, we wish to verify if it is possible to use a subsidiary corpus to extract voice characteristics. For this purpose, we propose this time to apply the clustering algorithm on the pvectors representations extracted from the Skyrim corpus for step 2. The p-vectors have learned to highlight the played character but it is impossible to bring out the vocal characteristics it extracts to perform this classification. We hope that the use of a secondary corpus such as Skyrim will create enough variability to bring out vocal characteristics from the initial labels. For step 1, the p-vector extractor is exactly the same as the previous experiment. It is trained on MassEffect with Sn and the Skyrim data is completely unknown to it. For step 2, we extract p-vectors using Skyrim data and perform on them the k-means algorithm. For step 3, we finally associate a cluster to each recording of MassEffect and train refined p-vectors as realized in the previous experiment.
Table 3 presents the obtained results. We note× as accuracy result when labels are missing when we associate them to MassEffect data with the aim to train the step 3 neural network. We can observe that this time the v-measure and the purity-K maximum scores move away from the number of characters that we originally had – 32 for Skyrim. So we know that the semantics of these clusters differ from the semantics of the character labels. In this example, purity-K highlights k = 2. In the k = 2 case, the two groupings primarily separate men and women. Each group is therefore the gender characteristic of the voice. For the v-measure, the score correlates with the number k of groups. The best score is therefore for k = 64, but it might have gone up by increasing k. Unlike k = 2, it is difficult to understand the semantics of each group and to associate words that make sense to a human operator. Note that these measures have no known bias unlike the previous experiment since here the number of characters in the Skyrim data is 32 and the k = 2 and k = 64 values are significantly off.

heading: 4.3 ASSESSING THE IMPACT OF THE DISTANCE SELECTED FOR K-MEANS

text:
As we know, distance in the k-means algorithm plays an essential role. For this reason, we propose to realise the same experiment as presented in the previous section (4.2) but using cosine distance instead of Euclidian one. We also use the Mahalanobis distance k-means presented in Lapidot (2018) – without constraints to gain computing time.
The table 4 presents results obtained by using the cosine and the Mahalanobis distances. We can observe that the Mahalanobis distance offers superior results on v-measure and purity-K compared to
the Euclidean distance and the cosine distance. Moreover, while the cosine distance tends to draw the same conclusions as the Euclidean distance through these measurements, the Mahalnobis distance differs in the number of clusters it highlights with the purity-K measure. Indeed, the two other distances put forward k = 2 while this one puts forward k = 32. The training of Siamese networks seems to agree with the choice k = 32 for Mahalnobis. However, this accuracy is significantly lower than the one obtained with cosine distance, 0.64 < 0.67. Mahalanobis does not seem to fit the data so as to conserve character information.

heading: 5 CONCLUSION

text:
This article introduces Label Refining, a semi-supervised method that consists in extracting refined labels from initial ones. It meets the needs of characteristics extraction, especially vocal, without a priori knowledge. To validate this method, we applied it to dubbing and more specifically to the extraction of characteristics from actors’ voices. A first experiment showed that using the same data during all the steps of Label Refining will only highlight the initial labels. A second experiment showed that using a subsidiary corpus during the clustering step permits to highlight refined labels. It might also be noted that, in one case, our method brought to light gender characteristics. In the other case, giving labels intelligible meaning required further analysis. A last experiment showed that the Euclidean distance seems to fit the data better than cosine and Mahalanobis distances since it preserves the character information better.
With these experiments, we can conclude that Label Refining permits to extract refined labels that deviate from the original ones when using a subsidiary corpus. These labels conserve or bring the wanted information, namely character information in our case. While it is not straightforward to give meaning to these characteristics without a perceptive study, we propose in future works to build explainability methods based on Label Refining, especially for voice similarity systems.

title: LABEL REFINING: A SEMI-SUPERVISED METHOD TO EXTRACT VOICE CHARACTERISTICS WITHOUT GROUND TRUTH