text:
\section{Credits}

This document has been adapted by Yulan He
from the instructions for earlier ACL and NAACL proceedings, including those for
ACL 2020 by Steven Bethard, Ryan Cotterrell and Rui Yan,
ACL 2019 by Douwe Kiela and Ivan Vuli\'{c},
NAACL 2019 by Stephanie Lukin and Alla Roskovskaya,
ACL 2018 by Shay Cohen, Kevin Gimpel, and Wei Lu,
NAACL 2018 by Margaret Michell and Stephanie Lukin,
2017/2018 (NA)ACL bibtex suggestions from Jason Eisner,
ACL 2017 by Dan Gildea and Min-Yen Kan,
NAACL 2017 by Margaret Mitchell,
ACL 2012 by Maggie Li and Michael White,
ACL 2010 by Jing-Shing Chang and Philipp Koehn,
ACL 2008 by Johanna D. Moore, Simone Teufel, James Allan, and Sadaoki Furui,
ACL 2005 by Hwee Tou Ng and Kemal Oflazer,
ACL 2002 by Eugene Charniak and Dekang Lin,
and earlier ACL and EACL formats written by several people, including
John Chen, Henry S. Thompson and Donald Walker.
Additional elements were taken from the formatting instructions of the \emph{International Joint Conference on Artificial Intelligence} and the \emph{Conference on Computer Vision and Pattern Recognition}.

\section{Introduction}

The following instructions are directed to authors of papers submitted to EMNLP 2020 or accepted for publication in its proceedings.
All authors are required to adhere to these specifications.
Authors are required to provide a Portable Document Format (PDF) version of their papers.
\textbf{The proceedings are designed for printing on A4 paper.}

\section{Electronically-available resources}

ACL provides this description and accompanying style files at
\begin{quote}
\url{https://2020.emnlp.org/downloads/emnlp2020-templates.zip}
\end{quote}
We strongly recommend the use of these style files, which have been appropriately tailored for the EMNLP 2020 proceedings.

\paragraph{\LaTeX-specific details:}
The templates include the \LaTeX2e{} source (\texttt{\small emnlp2020.tex}),
the \LaTeX2e{} style file used to format it (\texttt{\small emnlp2020.sty}),
an ACL bibliography style (\texttt{\small acl\_natbib.bst}),
an example bibliography (\texttt{\small emnlp2020.bib}),
and the bibliography for the ACL Anthology (\texttt{\small anthology.bib}).

\section{Length of Submission}
\label{sec:length}

The conference accepts submissions of long papers and short papers.
Long papers may consist of up to eight (8) pages of content plus unlimited pages for references.
Upon acceptance, final versions of long papers will be given one additional page -- up to nine (9) pages of content plus unlimited pages for references -- so that reviewers' comments can be taken into account.
Short papers may consist of up to four (4) pages of content, plus unlimited pages for references.
Upon acceptance, short papers will be given five (5) pages in the proceedings and unlimited pages for references.
For both long and short papers, all illustrations and tables that are part of the main text must be accommodated within these page limits, observing the formatting instructions given in the present document.
Papers that do not conform to the specified length and formatting requirements are subject to be rejected without review.

The conference encourages the submission of additional material that is relevant to the reviewers but not an integral part of the paper.
There are two such types of material: appendices, which can be read, and non-readable supplementary materials, often data or code.
Additional material must be submitted as separate files, and must adhere to the same anonymity guidelines as the main paper.
The paper must be self-contained: it is optional for reviewers to look at the supplementary material.
Papers should not refer, for further detail, to documents, code or data resources that are not available to the reviewers.
Refer to Appendices~\ref{sec:appendix} and \ref{sec:supplemental} for further information.

Workshop chairs may have different rules for allowed length and whether supplemental material is welcome.
As always, the respective call for papers is the authoritative source.

\section{Anonymity}
As reviewing will be double-blind, papers submitted for review should not include any author information (such as names or affiliations). Furthermore, self-references that reveal the author's identity, \emph{e.g.},
\begin{quote}
We previously showed \citep{Gusfield:97} \ldots
\end{quote}
should be avoided. Instead, use citations such as
\begin{quote}
\citet{Gusfield:97} previously showed\ldots
\end{quote}
Please do not use anonymous citations and do not include acknowledgements.
\textbf{Papers that do not conform to these requirements may be rejected without review.}

Any preliminary non-archival versions of submitted papers should be listed in the submission form but not in the review version of the paper.
Reviewers are generally aware that authors may present preliminary versions of their work in other venues, but will not be provided the list of previous presentations from the submission form.

Once a paper has been accepted to the conference, the camera-ready version of the paper should include the author's names and affiliations, and is allowed to use self-references.

\paragraph{\LaTeX-specific details:}
For an anonymized submission, ensure that {\small\verb|\aclfinalcopy|} at the top of this document is commented out, and that you have filled in the paper ID number (assigned during the submission process on softconf) where {\small\verb|***|} appears in the {\small\verb|\def\aclpaperid{***}|} definition at the top of this document.
For a camera-ready submission, ensure that {\small\verb|\aclfinalcopy|} at the top of this document is not commented out.

\section{Multiple Submission Policy}
Papers that have been or will be submitted to other meetings or publications must indicate this at submission time in the START submission form, and must be withdrawn from the other venues if accepted by EMNLP 2020. Authors of papers accepted for presentation at EMNLP 2020 must notify the program chairs by the camera-ready deadline as to whether the paper will be presented. We will not accept for publication or presentation the papers that overlap significantly in content or results with papers that will be (or have been) published elsewhere.

Authors submitting more than one paper to EMNLP 2020 must ensure that submissions do not overlap significantly ($>$25\%) with each other in content or results.

\section{Formatting Instructions}

Manuscripts must be in two-column format.
Exceptions to the two-column format include the title, authors' names and complete addresses, which must be centered at the top of the first page, and any full-width figures or tables (see the guidelines in Section~\ref{ssec:title-authors}).
\textbf{Type single-spaced.}
Start all pages directly under the top margin.
The manuscript should be printed single-sided and its length should not exceed the maximum page limit described in Section~\ref{sec:length}.
Pages should be numbered in the version submitted for review, but \textbf{pages should not be numbered in the camera-ready version}.

\paragraph{\LaTeX-specific details:}
The style files will generate page numbers when {\small\verb|\aclfinalcopy|} is commented out, and remove them otherwise.

\subsection{File Format}
\label{sect:pdf}

For the production of the electronic manuscript you must use Adobe's Portable Document Format (PDF).
Please make sure that your PDF file includes all the necessary fonts (especially tree diagrams, symbols, and fonts with Asian characters).
When you print or create the PDF file, there is usually an option in your printer setup to include none, all or just non-standard fonts.
Please make sure that you select the option of including ALL the fonts.
\textbf{Before sending it, test your PDF by printing it from a computer different from the one where it was created.}
Moreover, some word processors may generate very large PDF files, where each page is rendered as an image.
Such images may reproduce poorly.
In this case, try alternative ways to obtain the PDF.
One way on some systems is to install a driver for a postscript printer, send your document to the printer specifying ``Output to a file'', then convert the file to PDF.

It is of utmost importance to specify the \textbf{A4 format} (21 cm x 29.7 cm) when formatting the paper.
Print-outs of the PDF file on A4 paper should be identical to the hardcopy version.
If you cannot meet the above requirements about the production of your electronic submission, please contact the publication chairs as soon as possible.

\paragraph{\LaTeX-specific details:}
PDF files are usually produced from \LaTeX{} using the \texttt{\small pdflatex} command.
If your version of \LaTeX{} produces Postscript files, \texttt{\small ps2pdf} or \texttt{\small dvipdf} can convert these to PDF.
To ensure A4 format in \LaTeX, use the command {\small\verb|\special{papersize=210mm,297mm}|}
in the \LaTeX{} preamble (below the {\small\verb|\usepackage|} commands) and use \texttt{\small dvipdf} and/or \texttt{\small pdflatex}; or specify \texttt{\small -t a4} when working with \texttt{\small dvips}.

\subsection{Layout}
\label{ssec:layout}

Format manuscripts two columns to a page, in the manner these
instructions are formatted.
The exact dimensions for a page on A4 paper are:

\begin{itemize}
\item Left and right margins: 2.5 cm
\item Top margin: 2.5 cm
\item Bottom margin: 2.5 cm
\item Column width: 7.7 cm
\item Column height: 24.7 cm
\item Gap between columns: 0.6 cm
\end{itemize}

\noindent Papers should not be submitted on any other paper size.
If you cannot meet the above requirements about the production of your electronic submission, please contact the publication chairs above as soon as possible.

\subsection{Fonts}

For reasons of uniformity, Adobe's \textbf{Times Roman} font should be used.
If Times Roman is unavailable, you may use Times New Roman or \textbf{Computer Modern Roman}.

Table~\ref{font-table} specifies what font sizes and styles must be used for each type of text in the manuscript.

\begin{table}
\centering
\begin{tabular}{lrl}
\hline \textbf{Type of Text} & \textbf{Font Size} & \textbf{Style} \\ \hline
paper title & 15 pt & bold \\
author names & 12 pt & bold \\
author affiliation & 12 pt & \\
the word ``Abstract'' & 12 pt & bold \\
section titles & 12 pt & bold \\
subsection titles & 11 pt & bold \\
document text & 11 pt  &\\
captions & 10 pt & \\
abstract text & 10 pt & \\
bibliography & 10 pt & \\
footnotes & 9 pt & \\
\hline
\end{tabular}
\caption{\label{font-table} Font guide. }
\end{table}

\paragraph{\LaTeX-specific details:}
To use Times Roman in \LaTeX2e{}, put the following in the preamble:
\begin{quote}
\small
\begin{verbatim}
\usepackage{times}
\usepackage{latexsym}
\end{verbatim}
\end{quote}

\subsection{Ruler}
A printed ruler (line numbers in the left and right margins of the article) should be presented in the version submitted for review, so that reviewers may comment on particular lines in the paper without circumlocution.
The presence or absence of the ruler should not change the appearance of any other content on the page.
The camera ready copy should not contain a ruler.

\paragraph{Reviewers:}
note that the ruler measurements may not align well with lines in the paper -- this turns out to be very difficult to do well when the paper contains many figures and equations, and, when done, looks ugly.
In most cases one would expect that the approximate location will be adequate, although you can also use fractional references (\emph{e.g.}, this line ends at mark $295.5$).

\paragraph{\LaTeX-specific details:}
The style files will generate the ruler when {\small\verb|\aclfinalcopy|} is commented out, and remove it otherwise.

\subsection{Title and Authors}
\label{ssec:title-authors}

Center the title, author's name(s) and affiliation(s) across both columns.
Do not use footnotes for affiliations.
Place the title centered at the top of the first page, in a 15-point bold font.
Long titles should be typed on two lines without a blank line intervening.
Put the title 2.5 cm from the top of the page, followed by a blank line, then the author's names(s), and the affiliation on the following line.
Do not use only initials for given names (middle initials are allowed).
Do not format surnames in all capitals (\emph{e.g.}, use ``Mitchell'' not ``MITCHELL'').
Do not format title and section headings in all capitals except for proper names (such as ``BLEU'') that are
conventionally in all capitals.
The affiliation should contain the author's complete address, and if possible, an electronic mail address.

The title, author names and addresses should be completely identical to those entered to the electronical paper submission website in order to maintain the consistency of author information among all publications of the conference.
If they are different, the publication chairs may resolve the difference without consulting with you; so it is in your own interest to double-check that the information is consistent.

Start the body of the first page 7.5 cm from the top of the page.
\textbf{Even in the anonymous version of the paper, you should maintain space for names and addresses so that they will fit in the final (accepted) version.}

\subsection{Abstract}
Use two-column format when you begin the abstract.
Type the abstract at the beginning of the first column.
The width of the abstract text should be smaller than the
width of the columns for the text in the body of the paper by 0.6 cm on each side.
Center the word \textbf{Abstract} in a 12 point bold font above the body of the abstract.
The abstract should be a concise summary of the general thesis and conclusions of the paper.
It should be no longer than 200 words.
The abstract text should be in 10 point font.

\subsection{Text}
Begin typing the main body of the text immediately after the abstract, observing the two-column format as shown in the present document.

Indent 0.4 cm when starting a new paragraph.

\subsection{Sections}

Format section and subsection headings in the style shown on the present document.
Use numbered sections (Arabic numerals) to facilitate cross references.
Number subsections with the section number and the subsection number separated by a dot, in Arabic numerals.

\subsection{Footnotes}
Put footnotes at the bottom of the page and use 9 point font.
They may be numbered or referred to by asterisks or other symbols.\footnote{This is how a footnote should appear.}
Footnotes should be separated from the text by a line.\footnote{Note the line separating the footnotes from the text.}

\subsection{Graphics}

Place figures, tables, and photographs in the paper near where they are first discussed, rather than at the end, if possible.
Wide illustrations may run across both columns.
Color is allowed, but adhere to Section~\ref{ssec:accessibility}'s guidelines on accessibility.

\paragraph{Captions:}
Provide a caption for every illustration; number each one sequentially in the form:
``Figure 1. Caption of the Figure.''
``Table 1. Caption of the Table.''
Type the captions of the figures and tables below the body, using 10 point text.
Captions should be placed below illustrations.
Captions that are one line are centered (see Table~\ref{font-table}).
Captions longer than one line are left-aligned (see Table~\ref{tab:accents}).

\begin{table}
\centering
\begin{tabular}{lc}
\hline
\textbf{Command} & \textbf{Output}\\
\hline
\verb|{\"a}| & {\"a} \\
\verb|{\^e}| & {\^e} \\
\verb|{\`i}| & {\`i} \\
\verb|{\.I}| & {\.I} \\
\verb|{\o}| & {\o} \\
\verb|{\'u}| & {\'u}  \\
\verb|{\aa}| & {\aa}  \\\hline
\end{tabular}
\begin{tabular}{lc}
\hline
\textbf{Command} & \textbf{Output}\\
\hline
\verb|{\c c}| & {\c c} \\
\verb|{\u g}| & {\u g} \\
\verb|{\l}| & {\l} \\
\verb|{\~n}| & {\~n} \\
\verb|{\H o}| & {\H o} \\
\verb|{\v r}| & {\v r} \\
\verb|{\ss}| & {\ss} \\
\hline
\end{tabular}
\caption{Example commands for accented characters, to be used in, \emph{e.g.}, \BibTeX\ names.}\label{tab:accents}
\end{table}

\paragraph{\LaTeX-specific details:}
The style files are compatible with the caption and subcaption packages; do not add optional arguments.
\textbf{Do not override the default caption sizes.}

\subsection{Hyperlinks}
Within-document and external hyperlinks are indicated with Dark Blue text, Color Hex \#000099.

\subsection{Citations}
Citations within the text appear in parentheses as~\citep{Gusfield:97} or, if the author's name appears in the text itself, as \citet{Gusfield:97}.
Append lowercase letters to the year in cases of ambiguities.
Treat double authors as in~\citep{Aho:72}, but write as in~\citep{Chandra:81} when more than two authors are involved. Collapse multiple citations as in~\citep{Gusfield:97,Aho:72}.

Refrain from using full citations as sentence constituents.
Instead of
\begin{quote}
  ``\citep{Gusfield:97} showed that ...''
\end{quote}
write
\begin{quote}
``\citet{Gusfield:97} showed that ...''
\end{quote}

\begin{table*}
\centering
\begin{tabular}{lll}
\hline
\textbf{Output} & \textbf{natbib command} & \textbf{Old ACL-style command}\\
\hline
\citep{Gusfield:97} & \small\verb|\citep| & \small\verb|\cite| \\
\citealp{Gusfield:97} & \small\verb|\citealp| & no equivalent \\
\citet{Gusfield:97} & \small\verb|\citet| & \small\verb|\newcite| \\
\citeyearpar{Gusfield:97} & \small\verb|\citeyearpar| & \small\verb|\shortcite| \\
\hline
\end{tabular}
\caption{\label{citation-guide}
Citation commands supported by the style file.
The style is based on the natbib package and supports all natbib citation commands.
It also supports commands defined in previous ACL style files for compatibility.
}
\end{table*}

\paragraph{\LaTeX-specific details:}
Table~\ref{citation-guide} shows the syntax supported by the style files.
We encourage you to use the natbib styles.
You can use the command {\small\verb|\citet|} (cite in text) to get ``author (year)'' citations as in \citet{Gusfield:97}.
You can use the command {\small\verb|\citep|} (cite in parentheses) to get ``(author, year)'' citations as in \citep{Gusfield:97}.
You can use the command {\small\verb|\citealp|} (alternative cite without  parentheses) to get ``author year'' citations (which is useful for  using citations within parentheses, as in \citealp{Gusfield:97}).

\subsection{References}
Gather the full set of references together under the heading \textbf{References}; place the section before any Appendices.
Arrange the references alphabetically by first author, rather than by order of occurrence in the text.

Provide as complete a citation as possible, using a consistent format, such as the one for \emph{Computational Linguistics\/} or the one in the  \emph{Publication Manual of the American
Psychological Association\/}~\citep{APA:83}.
Use full names for authors, not just initials.

Submissions should accurately reference prior and related work, including code and data.
If a piece of prior work appeared in multiple venues, the version that appeared in a refereed, archival venue should be referenced.
If multiple versions of a piece of prior work exist, the one used by the authors should be referenced.
Authors should not rely on automated citation indices to provide accurate references for prior and related work.

The following text cites various types of articles so that the references section of the present document will include them.
\begin{itemize}
\item Example article in journal: \citep{Ando2005}.
\item Example article in proceedings, with location: \citep{borschinger-johnson-2011-particle}.
\item Example article in proceedings, without location: \citep{andrew2007scalable}.
\item Example arxiv paper: \citep{rasooli-tetrault-2015}.
\end{itemize}

\paragraph{\LaTeX-specific details:}
The \LaTeX{} and Bib\TeX{} style files provided roughly follow the American Psychological Association format.
If your own bib file is named \texttt{\small emnlp2020.bib}, then placing the following before any appendices in your \LaTeX{}  file will generate the references section for you:
\begin{quote}\small
\verb|\bibliographystyle{acl_natbib}|\\
\verb|

\section{Appendix A: Details of Experiments}

\subsection{Details of Datasets}
Consider the multi-party conversations, we adopt a publicly available Ubuntu dataset from DSTC-8 track of ``NOESIS II: Predicting Responses"\cite{kim2019eighth}. This Ubuntu dataset consists of four tasks and we use datasets from three of them. Task 1 is a single-topic multi-party dialogue setting which is considered for our topic prediction, and each test sample of the test set consists of 1 positive response and 4 negative responses randomly selected from 99 negatives. Task 2 is a long Ubuntu chat log with multi-party conversations on-going simultaneously which is ideal for our response selection evaluation, and each test sample in the test set contains 1 positive response and 99 negative responses. Task 4 is formed by multi-party conversations with link annotations, where each instance has 1500 utterances with around 500 annotation links. The numbers of conversations in task 4 datasets are: 17,751 in the training set, 785 in the dev set, and 964 in the test set. The corresponding numbers of links are: 69,395 in the training set, 2,607 in the dev set, and 5,187 in the test set. Table 1 in the main paper gives the statistics of this dataset used in our experiments.

\subsection{Evaluation Metrics}
Recall@N is used for topic prediction and response selection, which counts how often the correct answer is within the top N specified by the model. The values of N can be 1, 5, and 10. In addition, the Mean Reciprocal Rank (MRR) is also considered which is a widely used metric from the ranking literature.

\if 0
\begin{table}[ht]
\centering
\small
\setlength{\tabcolsep}{3pt}
\scalebox{1}
{\begin{tabular}{c|ccc}
\toprule
{\bf Tasks@Track2} & {\bf Train set} & {\bf Val set} & {\bf Test set}  \\
 \midrule
 Task 1& 225,367 & 4,827 & 5,529 \\
 Task 2& 112,262 & 9,565 & 9,027 \\
 Task 4 &153&10&10 \\
\bottomrule
\end{tabular}}
\caption{DSTC-8 Ubuntu Dataset Statistics}
\label{table:dstc8-data}
\vspace{-0.11in}
\end{table}
\fi

\begin{table}[ht]
\centering
\small
\setlength{\tabcolsep}{3pt}
\scalebox{1}
{\begin{tabular}{c|ccc}
\toprule
{\bf Tasks@Track2} & {\bf Turns (Avg)} & {\bf Speakers (Avg)}  \\
 \midrule
 \hline
 Task 1 Train & 6.0 & 2.4 \\
 Task 1 Val & 6.1 & 2.4 \\
 Task 1 Test & 6.1 & 2.4 \\
 \hline
 Task 2 Train & 54.7 & 19.6  \\
 Task 2 Val & 54.3 & 18.5  \\
 Task 2 Test & 54.6 & 18.2  \\
 \hline
 Task 4 Train & 3.9 & 1.5  \\
 Task 4 Val & 3.6 & 1.8  \\
 Task 4 Test & 3.0 & 1.5  \\
\bottomrule

\end{tabular}}
\caption{DSTC-8 Ubuntu Dataset Statistics: Average Turns and Average Speakers per conversation}
\label{table:dstc8-data}
\vspace{-0.11in}
\end{table}

\subsection{Implementation Details}
\label{BERT topic pairs}
We adopt the vanilla pretrained uncased $\text{BERT}_{\text{base}}$ (which has 12 layers with a hidden size of 768, provided by hugging face library) \footnote{\url{https://github.com/huggingface/transformers}} to initialise Topic-BERT with uncased $\text{BERT}_{\text{base}}$ configuration. In addition, according to section of Topic-BERT Pretraining, we obtained 13561221 train sentence pairs, 576414 validation sentence pairs and 435680 test sentence pairs from Task 1 corresponding dataset. Inspired by \cite{gu2020pre,BERT-enhance}, we posttrained our input-encoder (uncased $\text{BERT}_{\text{base}}$) 10 epochs more with the training topic sentence pairs and the batch size is set to 96, the MLM training objective was on positive training pairs only. Then 2 more epochs with validation topic sentence pairs was used for fine-tuning.

We used cross-entropy loss and Adam optimizer with a learning rate of 2e-5 and a linear decay schedule with warm-up 0.1. Regarding some hyper-parameters, the maximum sequence length of the concatenation of a context-response pair was set to 192 (96 for response), and the ratio of dropout is 0.1 to prevent over-fitting. Then our trained Topic-BERT will be used to encode input sentences pairs (10 utterance pairs for each batch to limit the context) to obtain the topic vectors for the end 3 tasks. 4 Tesla V100-SXM2 GPUs was used to train Topic-BERT, the training time for each epoch is around 30.2 hours and the validation time is around 45.2 minutes to complete one epoch for posttraining.

\section{Appendix B: Details of Baseline Models}
\begin{table}[t!]
\centering
\setlength{\tabcolsep}{3pt}
\scalebox{0.85}
{\begin{tabular}{l|cccc}
\toprule
{\bf Model} & {\bf Recall@1} & {\bf Recall@5} & {\bf Recall@10} & {\bf MRR}  \\
\midrule
\hline

Switch-BERT & 0.506 & 0.755 & 0.834 & 0.621\\
BERT-ESIM & 0.596 & 0.847 & 0.904 & 0.707 \\
XLNet & 0.637 & 0.856 & 0.914 & 0.735 \\
Adapt-BERT & 0.706 & 0.916 & 0.957 & 0.799\\\hline

\end{tabular}}
\vspace{-0.5em}
\caption{Released response selection results on DSTC-8 Ubuntu task 2. The Switch-BERT result is obtained from \cite{gu2020pre}, BERT-ESIM result is obtained from \cite{BERT-ESIM}, Adapt-BERT result is obtained from \cite{BERT-enhance}, XLNet result is obtained from \cite{BERT-SM}}
\label{table:response selection base}
\vspace{-0.5em}
\end{table}

\subsection{Baselines for Response Selection}
\label{BRS}
\paragraph{Baseline Models.} We compare the proposed Topic-BERT approach with several existing and state-of-the-art approaches for response selection:
\begin{itemize}[leftmargin=*,itemsep=0.1pt]
\item\textbf{BERT.} The vanilla pretrained uncased $\text{BERT}_{\text{base}}$ \footnote{\url{https://github.com/huggingface/transformers}} was the base model. We follow \cite{BERT-enhance} to posttrain $\text{BERT}_{\text{base}}$ on task 1 dataset 10 more  epochs with NSP and MLM objectives. The maximum sequence length was set to 512 for the document-level input with a concatenation of the whole context and response to capture the most present tokens. Adam optimizer with a learning rate of 2e-5 and a linear decay schedule with warm-up 0.1. Batch size is 16.

In fine-tuning, 10 more epoch was used for response selection on task 2 dataset. Adam optimizer with a learning rate of 3e-5 and a linear decay schedule with warm-up 0.06 was selected. For response selection, the candidate pool may not contain the correct response, so we need to choose a threshold. When the probability of positive labels was smaller than the threshold, we predicted that candidate pool did not contain the correct response. The threshold was selected from the range [0.7, 0.75, .., 0.95]\cite{gu2020pre}, based on the validation set, it was set to 0.90 finally. We used the validation set to set the stop condition to select the best model for testing.

\item\textbf{ToD-BERT.} ToD-BERT \cite{wu2020tod} is a pre-trained language model with 9 task-oriented dialogue datasets, it surpasses BERT in downstream response selection task. We compare our topic pretrained BERT with the ToD-BERT. We initialized BERT model with ToD-BERT, and posttrain it 10 more epochs on task 1 dataset with NSP and MLM objectives. A limit of maximum sequence is set to 512 by taking a document-level context history and response as input sequence. Adam optimizer with a learning rate of 1e-5 and a linear decay schedule with warm-up 0.1.

There are 10 more epochs for fine-tuning on task 2 response selection as well. Adam optimizer with a learning rate of 2e-5 and a linear decay schedule without warm-up was selected. Based on the validation set, the none reply threshold was set to 0.95.

\item\textbf{Switch-BERT.} \citet{gu2020pre} introduced switch embedding to indicate a switch of turn when the conversation is proceeding. External data (Ubuntu FQA) will be utilized for BERT pretraining. We list down their reported result as baseline in \cref{table:response selection base}.

\item\textbf{BERT-ESIM.} Model ensembling of ESIM and BERT with gradient boosting classifier which rank as the second team in DSTC-8 response selection \cite{BERT-ESIM}. ESIM is the state-of-the-art model in DSTC-7 by applying cross attention to get the inter-sentence matching. Then ESIM and BERT will be ensembled with a gradient boost classifier. We list down their reported result as baseline in \cref{table:response selection base}.

\item\textbf{Adapt-BERT.} BERT model with task-related pretraining and context modeling through hard and soft context modeling which rank as the top-1 team in DSTC-8 response selection challenge \cite{BERT-enhance}. The hard context retrieval and data augmentation make the response selection performance more accurate. We list down their reported result as baseline in \cref{table:response selection base}.
\end{itemize}

\subsection{Baselines for Topic Prediction}
\label{BERT TP}
\begin{itemize}[leftmargin=*,itemsep=0.1pt]
\item\textbf{BERT.} \cref{BRS} posttrained BERT will be fine-tuned with topic sentence pairs which are generated from task 1 for 10 epochs and the batch size is set to 96. We used cross-entropy loss and Adam optimizer with a learning rate of 2e-5 and a linear decay schedule with warm-up 0.1. Regarding some hyper-parameters, the maximum sequence length of the concatenation of a context-response pair was set to 192 (96 for response), and the ratio of dropout is 0.1 to prevent over-fitting.
\item\textbf{ToD-BERT.} The posttrained ToD-BERT in \cref{BRS} will be fine-tuned with 10 more epochs on task 1 topic sentence pairs with a batch size of 96 and up to 10 utterance counts per batch, the maximum sequence length is 192 (96 for response), an Adam optimizer with a learning rate of 1e-5 and a linear decay schedule with warm-up 0.1 were used in this experiment.
\end{itemize}

\subsection{Baselines for Topic Disentanglement}

\begin{itemize}[leftmargin=*,itemsep=0.1pt]
\item\textbf{BERT and ToD-BERT.}
We use our fine-tuned \textbf{BERT} and \textbf{ToD-BERT} model mentioned in \cref{BERT TP} as our baselines by taking the context utterance-response pairs from a dialogue as inputs, following \citet{gu2020pre} the initial learning rate is 2e-5 with maximum sequence length of 100, a single-layer BiLSTM with hidden size of 384 (BiLSTM output hidden size will be 768 to match with BERT output hidden size) is employed on top of the outputs of BERT [cls] outputs to capture the semantics across different messages and a single-layer feedforward NN with 768×4 hidden units will be the classifier to calculate the link-relevance score. The classifier has 3072 hidden units with 0.1 dropout to prevent overfitting. An Adam optimizer with a learning rate of 2.5e-5 and a linear decay schedule with warm-up 0.05 were used in this experiment.

\item\textbf{Feed-Forward.} We use the feed-forward model from \citet{acl19disentangle} as the baseline model. In DSTC-8, \citet{acl19disentangle} provided a fully trained model. \footnote{\url{https://github.com/dstc8-track2/NOESIS-II/tree/master/subtask4}} The provided model is trained using a two-layer neural network. The input is 77 hand engineered features combined with word average embeddings from pretrained Glove embeddings. We list down their reported result as baseline in main paper Table 4 to compare with.

\item\textbf{Masked Hierarchical (MH) BERT.} It is a two stage BERT to model the conversation structure where the low level BERT is to capture the utterance-level contextual representation between utterances. And the high level BERT is to modeling the conversation structure with an ancestor masking approach to avoid irrelevant connections \cite{zhu2019did}. We list down their reported result as baseline in main paper Table 4 to compare with.
\end{itemize}

\subsection{Baselines for Experiment IV}
The Topic-BERT with standalone response selection setting is fine-tuned with Ubuntu Corpus V1 with following hyperparameter setting, Adam optimizer with initial learning rate 5e-5, the maximum sequence length of the concatenation of a context-response pair was set to 192 (96 for response),batch size is 96 and 10 utterance pairs per batch, 10 epochs in total.

Below are the most commnon baseline models for Ubuntu Corpus V1 with a brief introduction. We list down their reported result as baseline in main paper Table 5 to compare with \cite{whang2019domain}.

\begin{itemize}[leftmargin=*,itemsep=0.1pt]
\item\textbf{DL2R.} It is a deep neural network to capture the relations cross sentence pairs. Query will be reformulated along with context utterance to enrich the contextual information for response selection \cite{yan2016learning}.

\item\textbf{Multi View.} It is a combination of word sequence model and utterance sequence model, the word-view and utterance-view will be used through a hierarchical RNN for response matching \cite{zhou2016multi}.

\item\textbf{SMN.} Response will be matched with each contextual utterances at multiple levels of granularity, then SMN will accumulate these matching information to select response \cite{SMN}.

\item\textbf{AK-DE-biGRU.} It is attention based dual encoder with external data to incorporate domain knowledge to improve response selection \cite{chaudhuri2018improving}

\item\textbf{DUA.} \citet{zhang2018modeling} proposed utterance aggregation approach with attention matching for response selection.

\item\textbf{DAM.} It is a transformer based model to utilize utterances self-attention and context-to-response cross attention to leverage the hidden representation at multi-grained level \cite{DAM}.

\item\textbf{IMN \& ESIM} Both of them enrich sentence representation with inter-sentence matching to solve response selection problem \cite{gu2019interactive,ESIM2,ESIM1}.
\end{itemize}
\begin{table*}[ht!]
\centering
\setlength{\tabcolsep}{3pt}
\scalebox{0.8}
{\begin{tabular}{c|c}
\toprule
\textbf{Speaker}& \textbf{Utterance}  \\
 \midrule
 None & None \\
 None & None \\
 participant\_23 & <participant_0> participant_1, ctrl+alt+del? \\
 participant\_25 & i would just like to know, how can i found out the support webcams for dapper? \\
 participant\_23 & in sessions what do i have to add to the start of my command to run the command as root when i boot up ? \\
 participant\_25 & *supported \\
 participant\_23 & \textbf{to start lampp} \\
 participant\_23 & \textbf{opt/lampp/lampp start} \\
 participant\_25 & is there a command in shell, or a section in the ubuntu website to find out all the supported webcams for dapper drake? \\
 participant\_23 & ok ty participant\_20\\
\midrule
ToD-BERT & participant\_25 , permission on /home/luigi/wwwebserver http://pastebin.ubuntu.com/75435/ \\
Topic-BERT & \textbf{ill put it in rc.local}\\
Ground-truth & ill put it in rc.local\\
 \bottomrule
\end{tabular}}
\caption{Top-1 response selection performance comparison}
\label{table:example1}
\end{table*}

\begin{table*}[ht!]
\centering
\setlength{\tabcolsep}{3pt}
\scalebox{0.8}
{\begin{tabular}{c|c}
\toprule
\textbf{Speaker}& \textbf{Utterance}  \\
 \midrule
 \hline
 participant\_0 & participant\_1, ctrl+alt+del? \\
 None & None \\
 participant\_1 & \bf participant\_0: no response, this is when i logged in \\
participant\_2 & \bf is medibuntu down? \\
 participant\_3 & I'm unable to browse a samba share on my own computer using nautilus. If I try to access a windows  \\
           & share I only get a empty window. Any hints/ideas?\\
 participant\_4 & hi, when I downlad files from the repositories, where in Ubuntu Linux they will be saved? \\
 participant\_0 & participant\_2, yes \\
 participant\_5 & participant\_4: depends on what you want. \\
 participant\_2 & \br participant\_0: is there a mirror somewhere? \\
 participant\_0 & participant\_1, power switch it is then\\

\midrule
\hline
ToD-BERT & participant\_5:  I want to save some of them to a usbstick after they where saved on my harddisk \\
Topic-BERT & \bf participant\_0: seems like even DNS is down\\
Ground-truth & participant\_0: seems like even DNS is down\\
\bottomrule
\end{tabular}}
\caption{Top-1 response selection performance comparison}
\label{table:example2}
\end{table*}
\section{Appendix C: Qualitative Evaluation}

\if 0
\begin{table}[h]
\vspace{-0.5em}
\centering
\setlength{\tabcolsep}{3pt}
\scalebox{0.85}
{\begin{tabular}{l|c|cccc}
\toprule
& &\multicolumn{4}{|c}{\textbf{Precision@N-gram}} \\
{\bf Model} & {\bf BLEU4} & {\bf N = 1 } & {\bf N = 2} & {\bf N = 3} & {\bf N = 4} \\
\midrule
\hline
ToD-BERT & 0.671 & 7.568 & 1.894 & 0.218 & 0.065\\
Topic-BERT & 0.75 & 7.876 & 2.032 & 0.250 & 0.078\\\hline
\end{tabular}}
\vspace{-0.5em}
\caption{BLEU4 and N-gram precision are calculated using SacreBLEU \footnote{\url{https://github.com/mjpost/sacrebleu}} on incorrectly selected responses.}
\label{table:BLEU score}
\vspace{-0.5em}
\end{table}
\fi

\Cref{table:example1}, \Cref{table:example2} shows our qualitative comparison. Our topic pretraining strategy could retrieve response based on topic relevance, those informative non-topic relevant utterances will not influence our response selection. During the evaluation, we find that several response candidates such as "Yes",  "Haha", "np" may misleading the response selection. Those terms doesn't have intrinsic/specific meaning in topic, they could be over attended as the dominant factor for response selection. This also address our interest in future potential research area such as topic noise decoupling.

\subsection{Dataset}
Considering our setting, a public available Ubuntu dataset in DSTC-8\cite{kim2019eighth} is suitable for our problem setting. Ubuntu Task 1 dataset is a single-topic multi-party dialogue setting which is considered for our topic prediction and the test set contains 1 positive response and 4 randomly selected negative responses out of 99 for each testing instance. Ubuntu Task 2 dataset is a long Ubuntu chat log with multi-party conversations on-going simultaneously which is used for our response selection and the test set contains 1 positive response and 99 negative responses for each testing instance. Ubuntu Task 4 dataset is formed by multi-party conversations with link annotations, where each instance has 1500 utterances with around 500 annotation links. In addition, Ubuntu V1 \cite{lowe2015ubuntu} with 1M train set, 500K validation and 500K was

\subsection{Experimental Setup}
\label{BERT topic pairs}
We adopt the vanilla pretrained uncased BERT-Base (which has 12 layers with a hidden size of 768, provided by hugging face library) \footnote{\url{https://github.com/huggingface/transformers}} as our input-encoder, we initialised our BERT weights with uncased BERT-Base configuration. In addition, according to the methodology we discussed in section \ref{Self-supervised BERT Pretraining}, we obtained 13561221 train sentence pairs, 576414 validation sentence pairs Task 1 corresponding dataset. Inspired by \cite{gu2020pre,BERT-enhance}, we post trained our input-encoder (uncased BERT-Base) 10 epochs more with augmented train and validation topic sentence pairs and the batch size is set to 96 with maximum 10 utterance pairs. Then 2 more epochs with test topic sentence pairs was used for finetuning.  We used cross-entropy loss and Adam optimizer with a learning rate of 2e-5 and a linear decay schedule with warm-up 0.1. Regarding some hyper-parameters, the maximum sequence length of the concatenation of a context-response pair was set to 192 (96 for response), and the ratio of dropout is 0.1 to prevent over-fitting. The evaluation metric is the average of accuracy, precision and recall. Then, linear feed-forward layer layer with a hidden size of 768$\times$2 to reduce it from vector to scalar for downstream tasks (weights are not shared).

\begin{table}[t!]
\centering
\setlength{\tabcolsep}{3pt}
\scalebox{1}
{\begin{tabular}{c|ccc}
\toprule
Task & Train set & Validation set & Test set  \\
 \midrule
 Task 1& 225,367 & 4,827 & 5,529 \\
 Task 2& 112,262 & 9,565 & 9,027 \\
 Task 4 &153&10&10 \\
\bottomrule
\end{tabular}}
\caption{DSTC-8 Ubuntu Dataset Statistics}
\label{table:data}
\end{table}
For response selection, the candidate pool may not contain the correct response, so we need to choose a threshold. When the probability of positive labels was smaller than the threshold, we predicted that candidate pool did not contain the correct response. The threshold was selected from the range [0.7, 0.75, .., 0.95]\cite{gu2020pre}, based on the validation set, it was set to 0.90 finally. We used the validation set to set the stop condition to select the best model for testing.

\subsection{Evaluation Metrics}
Track 2 considered a range of metrics
when comparing models. Following \cite{lowe2015ubuntu}, the
Recall@N is used for topic prediction and response selection, which counts how
often the correct answer is within the top N specified by the model. The values of N can be 1, 5, and 10. In addition, the Mean Reciprocal Rank (MRR) is also considered which is a widely used metric from the ranking literature. In the final comparison, the mean of Recall@10 and MRR is used.

For disentanglement, exact match F-1 calculated using the number of perfectly matching conversations, excluding conversations with only one message. For topic prediction, we simply use precision, recall and F-1 metrics

\subsection{Baseline Models}
\subsubsection{Response Selection}
\label{RS}
We compare the proposed Topic-BERT approach with several existing and state-of-the-art approaches for response selection:
\begin{itemize}
\item\textbf{BERT.} The vanilla pretrained uncased BERT-Base\footnote{\url{https://github.com/huggingface/transformers}} was the base model. We follow \cite{gu2020pre} to posttrain BERT-Base on task 1 dataset with 10 more epochs for response selection and topic prediction with batch size of 16. The initial learning rate was set to 2e-5. The maximum sequence length of the concatenation of a context-response pair was set to 512. In finetuning, 2 more epoch was used for response selection on task 2 dataset. Adam optimizer with a learning rate of 2e-5 and a linear decay schedule with warm-up 0.06 was selected.

\item\textbf{ToD-BERT.} ToD-BERT \cite{wu2020tod} is a pre-trained language model with 9 task-oriented dialogue datasets, it surpasses BERT in downstream response selection task. We compare our topic pre-trained BERT with the ToD-BERT in topic prediction, response selection and disentanglement.

\item\textbf{BERT-ESIM.} Model ensembling of ESIM and BERT with gradient boosting classifier which rank as the second team in DSTC-8 response selection \cite{BERT-ESIM}.

\item\textbf{Adapt-BERT.} BERT model with task-related pretraining and context modeling through hard and soft context modeling which rank as the top-1 team in DSTC-8 response selection challenge \cite{BERT-enhance}.
\end{itemize}

\subsubsection{Topic Prediction}
\label{BERT TP}
\begin{itemize}
\item\textbf{BERT.} We use our post trained BERT which is discussed in \ref{BERT topic pairs} as our BERT baseline for topic prediction.
\item\textbf{ToD-BERT.} We initialise BERT model with ToD-BERT and finetune it with our obtained topic sentences pairs \ref{Self-supervised BERT Pretraining} as our ToD-BERT baseline for topic prediction.
\end{itemize}
\subsubsection{Disentanglement}
We use our finetuned \textbf{BERT} and \textbf{ToD-BERT} model mentioned in \ref{BERT TP} as our baselines by taking the context utterance-response pairs from a dialogue as inputs, in addition,a single-layer BiLSTM with hidden size of 384 (BiLSTM output hidden size will be 768 to match with BERT output hidden size) is employed on top of the outputs of BERT [cls] outputs to capture the semantics across different messages which are denoted as and a single layer feedforward neural network with 768 $\times$ 4 hidden units will be the classifier.
\begin{itemize}
\item\textbf{Feed-Forward.} We use the feed-forward model from \cite{acl19disentangle} as the baseline model. In DSTC-8, \citet{acl19disentangle} provided a fully trained model.\footnote{\url{https://github.com/dstc8-track2/NOESIS-II/tree/master/subtask4}} The provided model is trained using a two-layer neural network. The input is 77 hand engineered features combined with word average embeddings from pre-trained Glove embeddings.
\item\textbf{Masked Hierarchical (MH) BERT.} It is a two stage BERT to model the conversation structure where the low level BERT is to capture the utterance-level contextual representation between utterances. And the high level BERT is to modeling the conversation structure with an ancestor masking approach to avoid irrelevant connections \cite{zhu2019did}.
\end{itemize}

\subsection{Experiment Result}
Table~\ref{table:response selection} presents the evaluation results of our methods on the response selection task. Table~\ref{table:topic prediction} shows the evaluation results of our methods on the topic prediction task.
And Table~\ref{table:disentanglement} presents the evaluation results of our methods on the disentanglement task.

From Table~\ref{table:response selection}, we can see that our context filtering, self-supervised topic training and topic attention contribute positively to our model, the metric of Recall@1 improved from 0.298 (BERT) to 0.696 (Topic BERT with standalone response selection), which shows pre-training BERT with task related data can improve BERT model in response selection task.Furthermore, the performance continues to increase from 0.696 to 0.710 when we jointly train response selection and topic prediction, it shows an effective utilization of topic information in selecting response to outperform Adapt-BERT which is the current state-of-the-art performance in DSTC-8 response selection task \cite{kim2019eighth}. Then we replace topic prediction with disentanglement, the performance further improved from 0.710 to 0.720, which shows response selection can utilize topic tracing by sharing the link relationship of utterances. Finally, our multi-task model with topic prediction, disentanglement and response selection achieve a better result comparing to the baselines.

Meanwhile, From Table~\ref{table:response selection},\ref{table:topic prediction},\ref{table:disentanglement}, we can see that our pre-training with topic sentence pairs data is more effective than BERT pre-training with dialogue data. Our post train strategy leverage utterance-utterance topic relationship which makes sentence level encoding simpler and effective.

\begin{table}[t!]
\centering
\setlength{\tabcolsep}{3pt}
\scalebox{0.85}
{\begin{tabular}{l|cccc}
\toprule
Model & Recall@1 & Recall@5 & Recall@10 & MRR  \\
\midrule
\hline
BERT & 0.298 & 0.513 & 0.589 & 0.377\\
ToD-BERT & 0.588 & 0.823 & 0.885 & 0.691 \\
BERT-ESIM & 0.596 & 0.847 & 0.904 & 0.707 \\
Adapt-BERT & 0.706 & 0.916 & 0.957 & 0.799\\\hline
Topic-BERT & {\bf 0.726} & {\bf 0.930} & {\bf 0.970} & {\bf 0.807} \\
\textminus TP & 0.720 & 0.927 & 0.964 & 0.803  \\
\textminus D & 0.710 & 0.924 & 0.960 & 0.800 \\
\textminus TP \textminus D & 0.696 & 0.910 & 0.950 & 0.790 \\\hline

\end{tabular}}
\caption{Response selection result with our pre-trained Topic BERT. ``RS" indicates our model architecture with response selection task, ``TP" indicates our model architecture with topic prediction task and ``D" indicates our model architecture with topic disentanglement task. }
\label{table:response selection}
\end{table}

\begin{table}[t!]
\centering
\setlength{\tabcolsep}{3pt}
\scalebox{1}
{\begin{tabular}{l|ccc}
\toprule
Model &Precision & Recall & F-1   \\
\midrule
\hline
BERT & 0.523 & 0.482 & 0.502 \\
ToD-BERT & 0.626 & 0.563 & 0.593\\\hline
Topic-BERT & {\bf 0.890} & {\bf 0.847} & {\bf 0.868} \\
\; w/o FT & 0.848 & 0.781 & 0.813 \\
\textminus D & 0.891 & 0.845 & 0.867 \\
\textminus RS &  0.889 & 0.840 & 0.864 \\
\textminus D \textminus RS & 0.866 & 0.793 & 0.828  \\
\bottomrule
\end{tabular}}
\caption{Topic prediction result with our pre-trained Topic BERT. ``RS" indicates our model architecture with Response Selection task, ``TP" indicates our model architecture with Topic Prediction task and ``D" indicates our model architecture with topic Disentanglement task.}
\label{table:topic prediction}
\end{table}

\if 0
\begin{table}[t!]
\centering
\setlength{\tabcolsep}{3pt}
\scalebox{1}
{\begin{tabular}{l|ccc}
\toprule
Model& \multicolumn{3}{|c}{\textbf{Result} } \\
&Precision & Recall & F-1   \\
\midrule
\hline
TP& 0.866 & 0.793 & 0.828  \\
+ RS & 0.891 & 0.845 & 0.867 \\
+ D& 0.889 & 0.840 & 0.864  \\
\midrule
RS + TP + D &0.890 & 0.847 & 0.868 \\
\bottomrule
\hline
Our Topic BERT & 0.848 & 0.781 & 0.813 \\
BERT & 0.523 & 0.482 & 0.502 \\
\end{tabular}}
\caption{Topic prediction result with our pre-trained Topic BERT. ``RS" indicates our model architecture with Response Selection task, ``TP" indicates our model architecture with Topic Prediction task and ``D" indicates our model architecture with topic Disentanglement task.}
\label{table:topic prediction}
\end{table}

\begin{table}[t!]
\centering
\setlength{\tabcolsep}{3pt}
\scalebox{1}
{\begin{tabular}{l|ccc}
\toprule
Model& \multicolumn{3}{|c}{\textbf{Result} } \\
&Precision & Recall & F-1   \\
\midrule
\hline
D& 0.689 & 0.678 & 0.683  \\
+ TP & 0.705 & 0.692 & 0.698 \\
+ RS & 0.749 & 0.727 & 0.737  \\
\midrule
RS + TP + D &0.754 & 0.725 & 0.739\\
\bottomrule
\hline
Feed-Forward & 0.748 & 0.718 & 0.733 \\
\end{tabular}}
\caption{Topic disentanglement result with our pre-trained Topic BERT. ``RS" indicates our model architecture with response selection task, ``TP" indicates our model architecture with topic prediction task and ``D" indicates our model architecture with topic disentanglement task.}
\label{table:disentanglement}
\end{table}
\fi

\begin{table}[t!]
\centering
\setlength{\tabcolsep}{3pt}
\scalebox{1}
{\begin{tabular}{l|ccc}
\toprule
Model & Precision & Recall & F-1   \\
\midrule
\hline
BERT & 0.431 & 0.417 & 0.424  \\
MH BERT & 0.539 & 0.517 & 0.528\\
ToD-BERT & 0.612 & 0.603 & 0.607\\
Feed-Forward & 0.748 & 0.718 & 0.733 \\\hline
Topic-BERT & {\bf 0.754} & {\bf 0.725} & {\bf 0.739}\\
\textminus TP & 0.749 & 0.727 & 0.737  \\
\textminus RS & 0.705 & 0.692 & 0.698 \\
\textminus TP \textminus RS& 0.689 & 0.678 & 0.683  \\
\bottomrule
\end{tabular}}
\caption{Topic disentanglement result with our pre-trained Topic BERT. RS indicates our model architecture with response selection task, TP indicates our model architecture with topic prediction task and D indicates our model architecture with topic disentanglement task.}
\label{table:disentanglement}
\end{table}

\if 0
\begin{table*}[t!]
\centering
\setlength{\tabcolsep}{3pt}
\scalebox{0.8}
{\begin{tabular}{c|c}
\toprule
\textbf{Speaker}& \textbf{Utterance}  \\
 \midrule
 None & None \\
 None & None \\
 participant\_23 & [n=olid@ACC9C91A.ipt.aol.com] has joined #ubuntu \\
 participant\_25 & i would just like to know, how can i found out the support webcams for dapper? \\
 participant\_23 & in sessions what do i have to add to the start of my command to run the command as root when i boot up ? \\
 participant\_25 & *supported \\
 participant\_23 & \textbf{to start lampp} \\
 participant\_23 & \textbf{opt/lampp/lampp start} \\
 participant\_25 & is there a command in shell, or a section in the ubuntu website to find out all the supported webcams for dapper drake? \\
 participant\_23 & ok ty participant\_20\\
\midrule
ToD-BERT & participant\_25 , permission on /home/luigi/wwwebserver http://pastebin.ubuntu.com/75435/ \\
Topic-BERT & \textbf{ill put it in rc.local}\\
Ground-truth & ill put it in rc.local\\
 \bottomrule
\end{tabular}}
\caption{Top-1 response selection performance comparison}
\label{table:example}
\end{table*}
\fi

Similarly, the result in Table~\ref{table:topic prediction} also shows our topic attention enhance the topic matching of two utterances in the metric of F-1 by 1.5\%. Response selection and disentanglement shows an equivalent effect on topic prediction in multi-task setting which shows contextual and link relationship can help topic prediction.

In addition, from the result shown in Table~\ref{table:disentanglement}, we can see that the multi-task setting enrich the link relationship together with topic prediction and response selection, the highly related tasks make our model outperform the baseline feed forward network with hand-crafted features. Meanwhile, due to the self-links in disentanglement not existing in topic prediction and response selection. It beats the SOTA baseline marginally.

\if 0
Table~\ref{table:example} shows our qualitative comparison. Our topic pre-training strategy could retrieve response based on topic relevance, those informative non-topic relevant utterances will not influence our response selection.
\fi

\subsection{Experimental Setup}
\label{BERT topic pairs}
We adopt the vanilla pretrained uncased $\text{BERT}_{\text{base}}$ (which has 12 layers with a hidden size of 768, provided by hugging face library) \footnote{\url{https://github.com/huggingface/transformers}} as our input-encoder, we initialised our BERT weights with uncased $\text{BERT}_{\text{base}}$ configuration. In addition, according to (\cref{Self-supervised Topic-BERT Pretraining}), we obtained 13561221 train sentence pairs, 576414 validation sentence pairs and 435680 test sentence pairs from Task 1 corresponding dataset. Inspired by \cite{gu2020pre,BERT-enhance}, we post-trained our input-encoder (uncased $\text{BERT}_{\text{base}}$) 10 epochs more with the training topic sentence pairs and the batch size is set to 96 with maximum 10 utterance pairs for every batch. Then 2 more epochs with validation topic sentence pairs was used for finetuning.  We used cross-entropy loss and Adam optimizer with a learning rate of 2e-5 and a linear decay schedule with warm-up 0.1. Regarding some hyper-parameters, the maximum sequence length of the concatenation of a context-response pair was set to 192 (96 for response), and the ratio of dropout is 0.1 to prevent over-fitting. Then our trained Topic-BERT will be used to encode input sentences pairs (10 utterance pairs for each batch) to obtain the topic vectors for the end 3 tasks.
\fi

\paragraph{Evaluation Metrics}
DSTC-8 Track 2 considered a range of metrics for comparing models. We follow their evaluation metrics. Recall@N is used for topic prediction and response selection, which counts how often the correct answer is within the top N specified by the model. The values of N can be 1, 5, and 10. In addition, the Mean Reciprocal Rank (MRR) is also considered which is a widely used metric from the ranking literature.

For disentanglement, we use precision, recall and F-scores (\emph{w.r.t.}\ \ true-class) for link-level  predictions. Similarly, for topic prediction, we use precision, recall and F-scores for correctly classifying the actual response to be in the same topic.

\subsection{Experiment I: Response Selection}
\label{RS}
\paragraph{Baseline Models.} We compare the proposed Topic-BERT approach with several existing and state-of-the-art approaches for response selection:
\begin{itemize}[leftmargin=*,itemsep=0.1pt]
\item\textbf{BERT.} We adopt the vanilla pretrained uncased $\text{BERT}_{\text{base}}$\footnote{\url{https://github.com/huggingface/transformers}} as the base model, and follow \cite{gu2020pre} to post-train $\text{BERT}_{\text{base}}$ for 10 epochs on DSTC-Task 1 (response selection in a single-topic dialog). We take the whole context with the response as one input sequence. We then finetune it on Task 2's response selection for 10 more epochs. More details can be found in Appendix.

\item\textbf{ToD-BERT.} This is a domain-specific pretrained BERT from \citet{wu2020tod}, which is pretrained on a combination of 9 Task-oriented Dialogue datasets and surpasses BERT in several downstream response selection tasks.
\item\textbf{BERT-ESIM.} This model ensembles both ESIM \cite{ESIM} and BERT with gradient boosting classifier, and ranks the second best in DSTC-8 response selection \cite{BERT-ESIM}.
\item\textbf{Adapt-BERT.} This is based on BERT model with task-related pretraining and context modeling through hard and soft context modeling, and ranks as top-1 in the DSTC-8 response selection challenge \cite{BERT-enhance}.
\end{itemize}

\paragraph{Results.} From \Cref{table:response selection}, we can see that our Topic-BERT model outperforms the baselines by a large margin. By examining our model in detail, we found that our context filtering, self-supervised topic training and topic attention contribute positively to our model, boosting the metric of Recall@1 from 0.287 ($\text{BERT}_\text{base}$) to 0.696 (Topic-BERT with standalone response selection task). This shows our topic pretraining with task related data improves BERT for response selection task.

Furthermore, the performance continues to increase from 0.696 to 0.710, when we jointly train response selection and topic prediction (2nd last row), validating an effective utilization of topic information in selecting response.
Then we replace topic prediction with disentanglement, which further improves from 0.710 to 0.720, showing response selection can utilize topic tracing by sharing the connection of utterances. Finally, our Topic-BERT with the multi-task learning achieves the best result (0.726) and significantly outperform the prior state-of-the-art Adapt-BERT in DSTC-8 response selection task \cite{kim2019eighth}.

We further compute BLEU4 SacreBLEU \cite{post2018call} for the incorrectly selected responses by Topic-BERT and ToD-BERT. From \Cref{table:BLEU score}, we see that responses retrieved by Topic-BERT are more relevant even if they are not the top one.

\begin{table}[t!]
\centering
\setlength{\tabcolsep}{3pt}
\scalebox{0.80}
{

\begin{tabular}{l|cccc}
\toprule
{\bf Model} & {\bf Recall@1} & {\bf Recall@5} & {\bf Recall@10} & {\bf MRR}  \\
\midrule
\midrule
$\text{BERT}_{\text{base}}$ & 0.287 & 0.503 & 0.572 & 0.351\\
BERT{\scriptsize +post-train} & 0.532 & 0.797 & 0.840 & 0.677\\
ToD-BERT & 0.588 & 0.823 & 0.885 & 0.691 \\
\midrule
Adapt-BERT & 0.706 & 0.916 & 0.957 & 0.799\\
\midrule
Topic-BERT & {\bf 0.726} & {\bf 0.930} & {\bf 0.970} & {\bf 0.807} \\
~~~\textminus TP & 0.720 & 0.927 & 0.964 & 0.803  \\
~~~\textminus D & 0.710 & 0.924 & 0.960 & 0.800 \\
~~~\textminus TP \textminus D & 0.696 & 0.910 & 0.950 & 0.790 \\

\bottomrule
\end{tabular}}

\caption{Response selection results on DSTC-8 Ubuntu.
``-TP" means our model excluding topic prediction loss and ``-D" means excluding topic disentanglement loss. Adapt-BERT results are obtained from \cite{BERT-enhance}, other DSTC-8 released baselines are in Appendix.}
\label{table:response selection}
\end{table}

\begin{table}[t]
\centering
\setlength{\tabcolsep}{3pt}
\scalebox{0.8}
{\begin{tabular}{l|c|cccc}
\toprule
& &\multicolumn{4}{|c}{\textbf{Precision@N-gram}} \\
{\bf Model} & {\bf BLEU4} & {\bf N = 1 } & {\bf N = 2} & {\bf N = 3} & {\bf N = 4} \\
\midrule
ToD-BERT & 0.67 & 7.568 & 1.894 & 0.218 & 0.065\\
Topic-BERT & 0.75 & 7.876 & 2.032 & 0.250 & 0.078\\\hline
\end{tabular}}
\caption{BLEU4 and N-gram precision are calculated using SacreBLEU on incorrectly selected responses.}
\label{table:BLEU score}
\end{table}

\subsection{Experiment II: Topic Prediction}
\label{BERT TP}
This experiment aims to examine how significant our Topic-BERT can improve over the baselines on the topic prediction task, which is important for both response selection and topic disentanglement.
\paragraph{Baseline Models.}
\begin{itemize}[leftmargin=*,itemsep=0.1pt]
\item\textbf{BERT.} {We use our post-trained $\text{BERT}_{\text{base}}$ from \cref{RS} and fine-tune it on Task 1 topic sentence pairs as our BERT baseline for topic prediction.}
\item\textbf{ToD-BERT.} We adopt our post-trained ToD-BERT and fine-tune it with our obtained topic sentences pairs
as the ToD-BERT baseline.
\end{itemize}

\paragraph{Results.} \Cref{table:topic prediction} gives the topic prediction results on DSTC-8 {task-1}. From the results, we can see that our Topic-BERT outperforms the baselines BERT and ToD-BERT significantly in the topic prediction task. Compared with our pretrained Topic-BERT without fine-tuning (last row), the proposed topic attention further enhances the topic matching of two utterances by improving the F-score by 1.5\% (from 0.813 to 0.828). Joint training with response selection or disentanglement tasks show similar effect on topic prediction tasks, and the contextual topic information sharing by Topic-BERT multi-task model add a marginal improvement in topic prediction. Compared with vanilla BERT, ToD-BERT \cite{wu2020tod} makes substantial improvement for the topic prediction task, but not as significant as ours. This further confirms the importance and efficacy of our learning scheme. {Meanwhile, if we compare our pretrained Topic-BERT without fine-tuning (last row) with the BERT model that does not use STP (first row), the significant improvement gives us an impression on how much our Topic-BERT benefits from the STP loss.}
\begin{table}[t!]
\centering
\scalebox{0.8}
{\begin{tabular}{l|ccc}
\toprule
{\bf Model} & \quad{\bf Precision} & {\bf Recall} & {\bf F-Score} \\
\midrule
\hline
BERT & 0.523 & 0.482 & 0.502 \\
ToD-BERT & 0.626 & 0.563 & 0.593\\
\midrule
Topic-BERT &  0.890 & {\bf 0.847} & {\bf 0.868} \\
~~~\textminus D & {\bf0.891} & 0.845 & 0.867 \\
~~~\textminus RS &  0.889 & 0.840 & 0.864 \\
~~~\textminus D \textminus RS & 0.866 & 0.793 & 0.828  \\
\; w/o FT & 0.848 & 0.781 & 0.813 \\
\bottomrule
\end{tabular}}
\caption{Topic prediction results on DSTC-8 Ubuntu.
``w/o FT" means our Topic-BERT without fine-tuning, ``-RS" means our model excluding the Response Selection loss, ``-D" means excluding Disentanglement loss.}
\label{table:topic prediction}
\end{table}

\subsection{Experiment III: Disentanglement}
This experiment aims to examine how well can Topic-BERT tackle the topic disentanglement task.

\begin{table}[t]
\centering
\setlength{\tabcolsep}{3pt}
\scalebox{0.8}
{\begin{tabular}{l|ccc}
\toprule
{\bf Model} & \; {\bf Precision} & \;\;\quad{\bf Recall}\quad\;\;\quad & {\bf F-Score}   \\
\midrule
\hline
BERT & 0.431 & 0.417 & 0.424  \\
MH BERT & 0.539 & 0.517 & 0.528\\
ToD-BERT & 0.612 & 0.603 & 0.607\\
Feed-Forward & 0.748 & 0.718 & 0.733 \\
\midrule
Topic-BERT & {\bf 0.754} & {\bf 0.725} & {\bf 0.739}\\
~~~\textminus TP & 0.749 & 0.727 & 0.737  \\
~~~\textminus RS & 0.705 & 0.692 & 0.698 \\
~~~\textminus TP \textminus RS& 0.689 & 0.678 & 0.683  \\
\bottomrule
\end{tabular}}
\caption{Disentanglement results on DSTC-8 Ubuntu.
``-RS" means our model excluding Response Selection loss, and ``-TP" means excluding Topic Prediction loss.
}
\label{table:disentanglement}
\end{table}

\paragraph{Baseline Models.}
\begin{itemize}[leftmargin=*,itemsep=0.1pt]

\item\textbf{BERT} \& \textbf{ToD-BERT}
We use our fine-tuned {BERT} and {ToD-BERT} models in \cref{BERT TP} as our baselines by taking the history of utterances ($u_1, \ldots,  u_{n-1}, u_n$) and pair each with the current utterance $u_n$ itself from a dialogue as input. Following \cite{gu2020pre}, A single-layer BiLSTM is applied to extract the cross message semantics of \texttt{[CLS]} outputs. Then we take the differences and element-wise products (Eq. \ref{eq:esim}) between the history and current utterance. Finally, a feedforward layer is used for link prediction.

\item\textbf{Feed-Forward.} This is the baseline model\footnote{\url{https://github.com/dstc8-track2/NOESIS-II/tree/master/subtask4}} from DSTC-8 task organizers that has the best result for task 4 \cite{acl19disentangle}, which is trained by employing a two-layer feed-forward neural network on a set of {\it 77 hand engineered features} combined with word average embeddings from pretrained Glove embeddings.
\item\textbf{Masked Hierarchical (MH) BERT.} This is a two-stage BERT proposed by \citet{zhu2019did} to model the conversation structure, in which the low-level BERT is to capture the utterance-level contextual representation between utterances, and the high-level BERT is to model the conversation structure with an ancestor masking approach to avoid irrelevant connections.
\end{itemize}
\paragraph{Results.}
From the results in \Cref{table:disentanglement}, we can see that
our Topic-BERT achieves the best result and outperforms all the BERT based baselines significantly. This shows our multi-task learning can enrich the link relationship for improving disentanglement together with topic prediction and response selection. The improvement of Topic-BERT over the baseline model using feed-forward network and hand-crafted features is relatively less, but our approach is able to avoid manual feature engineering. {Many of these features are dataset/domain specific and they do not generalize across datasets/domains.}

\subsection{Experiment IV: Evaluation on New Task}
\begin{table}[t!]
\centering
\setlength{\tabcolsep}{3pt}
\scalebox{0.7}
{\begin{tabular}{l|ccc}
\toprule
{\bf Model} & \quad\textbf{Recall}{10}@1\quad & \quad\textbf{Recall}{10}@2\quad  & \quad\textbf{Recall}{10}@5\quad\\
\midrule
\hline
DL2R & 0.626 & 0.783 & 0.944 \\
Multi View & 0.662 & 0.801 & 0.951\\
$\text{SMN}_{\text{dynamic}}$ & 0.726 & 0.847 & 0.961 \\
AK-DE-biGRU & 0.747& 0.868 & 0.972 \\
DUA & 0.752 & 0.868 & 0.962 \\
DAM & 0.767 & 0.874 & 0.969 \\
IMN & 0.777 & 0.888 & 0.974 \\
ESIM & 0.796 & 0.894 & 0.975\\
$\text{MRFN}_{\text{FLS}}$ & 0.786 & 0.886 & 0.976\\
\midrule
\midrule
$\text{BERT}_{\text{base}}$ & 0.817 & 0.904 & 0.977 \\
BERT-DPT & 0.851 & 0.924 & 0.984 \\
Topic-BERT
& {\bf0.861} & {\bf0.933} & {\bf0.985}\\
\bottomrule
\end{tabular}}
\caption{Response selection results on Ubuntu Corpus v1.
All other results are from \cite{whang2019domain}
}
\label{table:ubuntu v1response selection}
\end{table}
Finally, we examine our Topic-BERT's transferability on a new task based on another Ubuntu Corpus v1 dataset by comparing with various state-of-the-art response selection methods in \Cref{table:ubuntu v1response selection}. Ubuntu Corpus V1 contains 1M train set, 500K validation and 500K test set \cite{lowe2015ubuntu}.

\paragraph{Baseline Models.} Here we mainly introduce the state-of-the-art baseline: BERT-DPT \cite{whang2019domain}, which fine-tunes BERT by optimizing the domain post-training (DPT) loss comprising both NSP and MLM objectives for response selection.
Details of other baselines can be found in Appendix.
\paragraph{Results.} Our Topic-BERT with standalone response selection task fine-tuned on Ubuntu Corpus v1 outperforms the state-of-the-art BERT-DPT,
improved by about $1\%$ for $\text{Recall}_{10}$@1.
This result shows that the learned topic relevance in Topic-BERT can be potentially transferable to a novel task, the topic information influences the response selection positively, and our utterance-level topic tracking is effective for response selection.

\subsection{Subtasks}

\subsection{Topic-BERT Pretraining}
\label{Self-supervised Topic-BERT Pretraining}

One crucial advantage of our topic-based task formulation is that it allows us to pretrain BERT directly on a very relevant task in a self-supervised way, without requiring any human annotation. In other words, our goal is to pretrain BERT such that it can be used to encode relevant topic information for our task(s). For this, we assume that a single-threaded conversation between {two or more  participants} covers a single topic and the utterance pairs in that thread can be used to pretrain our Topic-BERT with relevant self-supervised objectives.

To collect such single-threaded conversational data in an opportunistic way, we can simply adopt the heuristics (unsupervised) used by \citet{lowe2015ubuntu} to collect the popular Ubuntu Dialogue Corpus from multi-threaded chatlogs. Alternatively, we can extract two-party conversations from other sources as done in previous work  \cite{henderson2019convert,wu2020tod}. In our experiments, we use the data from DSTC-8 task 1 \cite{kim2019eighth}, which was automatically collected from Ubuntu chat logs.
This dataset contains detached speaker-visible conversations between two or more participants from the Ubuntu IRC channel.

To pretrain Topic-BERT, we first initialise it with the pretrained uncased BERT$_{base}$ \cite{BERT}. We treat the training setting similar to our \emph{topic predection} task in \Cref{sec:taskform}. Formally, the pretraining dataset is ${\mathcal{D}}_{pr} = \{(u_{i},r_i^{+},r_{i,j}^{-})\}_{i=1}^{|{\mathcal{D}}_{pr}|}$, where each utterance pair from the same conversation (including the true response) constitutes a positive pair $(u_{i},r_i^{+})$, and for each such positive pair we randomly sample 4 negative responses ($r_{i,j}^{-}$) from the 100 candidate pool to balance the positive and negative ratio. We (re)train Topic-BERT on ${\mathcal{D}}_{pr}$ with two self-supervised objectives as follows.

\paragraph{Masked Language Modeling (MLM)} We follow the same MLM training of the original BERT \cite{BERT} by masking 15\% of the input tokens at random, and replacing the masked word with \texttt{[MASK]} token at 80\% of the time, with a random word at 10\% of the time, and with the original word at 10\% of the time. The MLM objective is only applied to the positive samples.

\begin{comment}
\begin{itemize}[leftmargin=*,itemsep=0.1pt]
  \item \textbf{Inter-context positive instances:} \\$(u_{m},u_{n})$  $\forall$ $u_{m},u_{n}$ $\in$ $c_{i}$ if $m < n$.
  \item \textbf{Cross context-response positive instances:} $(u_{m},r_i^+)$ $\forall$ $u_{m}$ $\in$ $c_{i}$.
  \item \textbf{Cross context-response negative instances:}  $(u_{m},r_i^-)$ $\forall$ $u_{m}$ $\in$ $c_{i}$, $\forall$ $r_i^-$ $\in$ $\{r_{neg}\}$.
\end{itemize}
\end{comment}

\paragraph{Same Topic Prediction (STP)}  Each training pair ($(u_{i},r_i^{+})$ or $(u_{i},r_{i,j}^{-})$) is fed into the Topic-BERT as (\texttt{[CLS]}, $[u_1]$, \texttt{[SEP]}, $[u_2]$, \texttt{[SEP]}). Similar to the original BERT's Next Sentence Prediction (NSP) task, the position embedding, segment embedding and token embedding are added together to get input layer token representations. The token representations are then passed through multiple transformer \cite{transformer} encoder layers, where each layer is comprised of a self-attention and a feed-forward sublayer. Different from the original BERT, Topic-BERT uses the \texttt{[CLS]} representation to predict whether the training instance is a positive (same topic) pair or a negative (different topic) pair. Thus, the \texttt{[CLS]} representation encodes topic relationship between the two utterances and will be used as the topic-aware contextual embedding to determine whether the two utterances are matched in topic.

\subsection{Topic-BERT Multi-Task Framework}

As shown in \Cref{fig:model}(b), the encoded representations from our Topic-BERT are passed through a topic attention layer (\cref{sec:topicattn}) to get the corresponding topic vectors, which are then used for the end tasks.

\subsubsection{Topic Attention Layer} \label{sec:topicattn}

We apply an attention layer to enhance topic information in the encoded vector. We use the Topic-BERT's \texttt{[CLS]} representation $T_{\textsc{cls}}$ as query to attend to the remaining $K$ tokens $\{T_j\}_{j=1}^K$:
\begin{eqnarray}
\small
\label{eq:Topic Attention1}
\centering
    e_{j} &=& {\bm{v}}_{a}^{T} \tanh ({\bm{W}}_{a} T_{\textsc{cls}} + {\bm{U}}_{a}T_{j}); \\
     a_{j} &=& \frac{\exp({e_{j}})}{\sum_{j=1}^{K}\exp{(e_{i}})} \\
T_{\text{topic}} &=& \sum_{j=1}^{K} a_{j}T_{j}
\end{eqnarray}

where ${\bm{v}}_a$, ${\bm{W}}_a$ and ${\bm{U}}_a$ are trainable parameters. The concatenation of $T_{\text{topic}}$ and $T_{\textsc{cls}}$ constitutes the final topic vector, {\em i.e.,}\ \ ${\bm{t}} = [T_{\textsc{cls}};T_{\text{topic}}]$. We repeat this encoding process for the $n$ utterances in the context $c_i = \{u_{1},u_{2}, \ldots, u_{n}\}$ by pairing each with the candidate response $r_{i,j}$ to get $n$ different topic vectors ${\bm{T}}_j = \{{\bm{t}}_1, \ldots, {\bm{t}}_n \}$. ${\bm{T}}_j$ represents $r_{i,j}$'s topic relevance to the  context utterances, which will be fed to the task-specific layers.

\subsubsection{Topic Prediction} \label{sec:toppred}

Topic prediction is done for each utterance-response pair $(u_k, r_{i, j})$ for all $u_k \in c_i$ to decide whether $u_k$ and $r_{i, j}$ should be in the same topic (\cref{sec:taskform}). The Topic-BERT encoded topic vector corresponding to the $(u_k, r_{i, j})$ pair is ${\bm{t}}_k \in {\bm{T}}_j$. We define the binary topic classification model as:

\begin{equation}
\centering
    g_{{\theta_{t}}}(u_k, r_{i,j}) = \sigma( {\bm{w}}_{p}^T {\bm{t}}_k)
\end{equation}
where ${\bm{w}}_p$ is the task-specific parameter. We use a binary cross entropy loss computed as:
\begin{equation}
\begin{split}
\label{eq:topic prediction loss}
    {\mathcal{L}}_{\text{topic}} = - y \log (g_{{\theta_{t}}}) - (1-y) \log (1-g_{{\theta_{t}}})
\end{split}
\end{equation}
where $y \in \{0,1\}$ is the ground truth indicating same or different topic. Note that topic prediction is an auxiliary task intended to help our main task of response selection, as we describe next.

\subsubsection{Response Selection}

In response selection, our goal is to measure relevance of a candidate response $r_{i,j}$ with respect to the context $c_i$. For this, we first apply the same hard context retrieval method proposed by \citet{BERT-enhance} to filter out irrelevant utterances and to reduce the context size. Then, we put each context utterance paired with the response $r_{i,j}$ as the input to Topic-BERT to compute the corresponding topic vectors ${\bm{T}}_j$ through the topic attention layer.

We pass the topic vectors ${\bm{T}}_j \in \mathbb{R}^{n\times d}$ through a scaled dot-product self-attention layer \cite{transformer} to learn all-pair topic relevance at the utterance level. Formally,
\begin{equation}
\label{eq:Self-Attention}
\centering
     {\bm{T}}_j^{'} = \mathcal{S}\big(\frac{ ({\bm{T}}_j {\bm{W}}_q) ({\bm{T}}_j{\bm{W}}_k)^{\top}}{\sqrt{d}}\big) ({\bm{T}}_j{\bm{W}}_v)
\end{equation}
where $\{{\bm{W}}_q,{\bm{W}}_k,{\bm{W}}_v\} \in \mathbb{R}^{n\times d}$ are the query, key and value parameters, respectively, and $d$ denotes the hidden dimension of $768$.

We add a max-pooling layer to select the most important information followed by a linear layer and a $\mathcal{S}$ to compute the relevance score of the response $r_{i,j}$ with the context $c_i$. Formally,
\begin{equation}
\centering
    \hspace{-0.3em}f_{{\theta_{r}}}(u_k, r_{i,j}) = \mathcal{S} ({\bm{W}}_r  (\text{maxpool}({\bm{T}}_j^{'})))
\end{equation}
where ${\bm{W}}_r$ is the task-specific parameter. We use the standard cross entropy loss defined as:
\begin{equation}
\label{eq:response selection loss}
    {\mathcal{L}}_{\text{rs}} = - \sum_{i,j} \mathbbm{1} (y_{i,j}) \log (f_{\theta_{r}})

\end{equation}
where $\mathbbm{1} (y_{i,j})$ is the one-hot encoding of the ground truth label.

\subsubsection{Topic Disentanglement}

For topic disentanglement (\cref{sec:taskform}), our goal is to find the ``reply-to'' links between the utterances (including the candidate response) to track which utterance is replying to which previous utterance.

For training on topic disentanglement, we simulate a sliding window over the entire (entangled) conversation. Each window constitutes a context $c_i = \{u_{1},u_{2}, \ldots, u_{n}\}$ and the model is trained to find the parent of $u_n$ in $c_i$, in other words, we try to find the reply-to link $(u_n, u_{n_{p}})$ for $1 \le {n_{p}} \le n$.

For the input to our Topic-BERT (\Cref{fig:model}b), we treat $u_n$ as the response, thus allowing also response-response $(u_n, u_n)$ interactions through Topic-BERT's encoding layers to facilitate \emph{self-link} predictions (the fact that $u_n$ can point to itself).

In the task-specific layer for disentanglement, we take the self-attended topic vectors ${\bm{T}}_j^{'} = \{{\bm{t}}_1^{'}, \ldots, {\bm{t}}_n^{'}\}$ as input, and separate it into two parts: context topic vectors encapsulated in ${\bm{T}}_c^{'} = \{{\bm{t}}_1^{'}, \ldots, {\bm{t}}_{n-1}^{'}\}$ $\in \mathbb{R}^{(n-1) \times d}$ and the response topic vector ${\bm{t}}_n^{'}$ $\in \mathbb{R}^{d}$. In order to model high-order interactions between the response and context utterances, we compute the differences and element-wise products between them \cite{ESIM2}. We duplicate the response message ${\bm{t}}_n^{'}$ to obtain ${\bm{T}}_r^{'} \in \mathbb{R}^{(n-1) \times d}$ and concatenate them as:
\begin{equation}
    {\bm{T}}^{''} = [{\bm{T}}_r^{'},{\bm{T}}_c^{'},{\bm{T}}_r^{'} \odot {\bm{T}}_c^{'}, {\bm{T}}_r^{'} - {\bm{T}}_c^{'}] \label{eq:esim}
\end{equation}

Then, we compute the {\it reply-to} distribution as: $h_{\theta_{d}}(u_n, c_i) = \mathcal{S} ({\bm{T}}{''} {\bm{w}}_d) \in \mathbb{R}^{n \times 1}$, and optimize with the following cross-entropy loss:
\begin{equation}
\label{eq:topic disentanglement loss}
    {\mathcal{L}}_{\text{dis}} = -\sum_{j=1}^n \mathbbm{1} (y_{j}) \log (h_{\theta_{d}})
\end{equation}

\begin{comment}
And the score was computed to represent the confidence of link between $(u_k,r_{i,j})$. The pointing distribution from $r_{i,j}$ to the leading utterance is calculated as:
\begin{equation}
\hspace{-0.3em}
\label{disentangle probability pointing}
    h_{\theta_{d}} (u_n, u_{j}) = \frac{\exp{g_{\theta_{dis}}(u_k,r_{i,j})}}{\sum_{u_m\in [c_i;r_{i,j}]}g_{\theta_{dis}}(u_m,r_{i,j})}
\end{equation}
The topic disentanglement loss $L_{dis}$ is defined
\end{comment}

\noindent For inference, we compute~$\argmax_j h_{\theta_{d}}(u_n, c_i)$.

\subsubsection{Multi-task Learning}

We jointly train the three tasks: response selection, topic prediction and topic disentanglement, which share the same topic attention weights to benefit each other. Response selection should benefit from dynamic topic prediction and disentanglement. Similarly, topic prediction and disentanglement should benefit from the response prediction.
The overall loss is a combination of the three task losses from Equations \ref{eq:topic prediction loss}, \ref{eq:response selection loss}, and \ref{eq:topic disentanglement loss}:
\begin{equation}
    {\mathcal{L}} = \alpha {\mathcal{L}}_{\text{rs}} + \beta {\mathcal{L}}_{\text{topic}} + \gamma {\mathcal{L}}_{\text{dis}}
\end{equation}
where $\alpha$, $\beta$, and $\gamma$ are parameters which are chosen from $[0,0.1,0.2,...,1]$ by optimizing our model response selection accuracy on dev dataset.

\subsection{Response Selection}

A dual encoder framework was proposed to match the context and response \cite{lowe2015ubuntu}, and the long short-term memory (LSTM) was utilized to learn the long and short term dependencies among tokens. Beyond tokens, the sentence view matching was introduced by applying a hierarchical recurrent neural network to model sentence level relationships \cite{zhou-etal-2016-multi}. However, context utterances and response are encoded separately without interaction; thus the semantics extracted from context are not based on the response. Recent approaches such as Sequential Matching Netowrk (SMN) \cite{SMN} leverage the contextual information by matching each contextual utterance with response and the multi channel Convolutional Neural Network (CNN) was proposed to generate multiple levels of granularity of matched segment.

These hierarchy-based methods use LSTM to encode the text, which is not cost effective to capture multi-grained segment representation \cite{lowe2015ubuntu,zhou-etal-2016-multi,SMN}. A particular work on sequence-based method stand out in DSTC-7; Enhanced Sequential Inference Model (ESIM) \cite{ESIM} achieves the state-of-the-art performance in DSTC-7 by taking advantage of inter-sentence matching \cite{ESIM1,ESIM2}. It converts multi-turn dialogue setting to natural language inference setting. In addition, transformer-based approach Deep Attention Matching (DAM) solve response selection problem by attention mechanism \cite{DAM}. It utilizes utterance self-attention and context-to-response cross attention to leverage the hidden representation at multi-grained level. Similar to DAM, Multi-hop Selector Network (MSN) was proposed by \citet{yuan2019multi} to fuse and select relevant context utterances and match it with the response utterance. In addition, \citet{tao2019one} model the relationship between a context utterance and the response in multiple levels.

Compared to LSTM-based approaches, methods based on transformers \cite{transformer}  present a promising performance in both accuracy and efficiency \cite{BERT-SM}. \citet{BERT} proposed BERT, a  transformer-based large-scale pretrained language model, which achieves state-of-the-art performance in different NLP tasks. BERT is also a good match to response selection problem as shown by \citet{vig2019comparison}. Our Topic-BERT is initialised with BERT$_{base}$ and posttrained with topic related sentence pairs.

\subsection{Hard Context Retrieval}

The side effect of multi-speaker multi-turn context is crucial; a lot of noise will be introduced in the context utterances. The speaker and addressee information are essential to decide the structure of conversation, thus can also benefit conversational response selection \cite{zhang2017addressee,le2019speaking,hu2019gsn}.
A hard context retrieval method was proposed by \citet{BERT-enhance} to minimize the context size, while keeping only the utterances whose speaker is the same as the response candidates or referred by the response candidates.
However, it cannot guarantee clean context with a single topic of conversation. Indeed, topic tracking is necessary along with hard context retrieval.

\subsection{Conversation Disentanglement}

Traditional statistical learning  based approaches and linguistic features have shown to be effective for conversation disentanglement \cite{mayfield2012hierarchical, du2016discovering}. Recent methods demonstrate that  neural networks could be applied to have a better linguistic representation of the utterances to retrieve relevant conversation. Hand crafted features and pretrained word embeddings are utilized to predict the link-to relationship between utterances \cite{acl19disentangle}. Recently, BERT has been adapted in disentangling task to capture the semantics across utterances \cite{gu2020pre}. Also, a masked transformer has been applied to learn the graphical representation {of utterances based on the reply-to links \cite{zhu2019did}}. \citet{Tao-emnlp-20} apply a pointer network for online disentanglement of conversations.

\section*{Appendix}
\input{Content/appendix}

\section{Introduction}
\input{Content/introduction-shafiq}

\section{Related Work}
\input{Content/related_works_revise}

\section{Task Formulation}
\label{sec:taskform}
\input{Content/task_definition}

\section{Our Topic-BERT Framework}
\input{Content/methods_revise}
\section{Experiments}
\input{Content/experiment_steven}
\section{Conclusion}
\input{Content/conclusion-revise}

\section{Introduction}
\input{Content/introduction_revise}
\section{Task Definition}
\input{Content/task_definition}
\section{Our Proposed Method}
\input{Content/methods_revise}
\section{Experiment}
\input{Content/experiment_revise}
\section{Conclusion}
\input{Content/conclusion}

redpajama_set_name: RedPajamaArXiv