\section{Introduction}

\begin{figure}[h!]
    \centering

    \includegraphics[width=0.83\linewidth]{figs/protocols_diagram_version18.png}
    \caption{The three protocols compared in this study. Each crowdworker is part of only one protocol.
    \label{fig:protocol_diagram}
\end{figure}

Many popular datasets for training and evaluating natural language understanding (NLU) models consist of examples written by non-expert annotators.
While it is convenient and relatively inexpensive to gather large datasets from non-expert crowdworkers, the resulting datasets often suffer from systematic gaps and artifacts.
Through \emph{post hoc} analysis, experts have identified many such problems and found that augmenting datasets with targeted examples can mitigate these issues \cite{yanaka-etal-2019-help,min-etal-2020-syntactic}.
Though non-expert created data is often flawed, it is easier to scale up compared to expert annotations, and so it is widely used in the creation of large training datasets.
With this in mind, we investigate how to leverage expert linguistic knowledge during annotation by having linguists dynamically identify artifacts, biases, and gaps in the data, then communicate with non-expert annotators to instruct them towards annotations that address issues as they arise.

We focus on natural language inference \citep[NLI;][i.a.]{dagan2006pascal}, a task where the goal is to predict the label (\textsc{entailment}, \textsc{contradiction}, \textsc{neutral}) that defines the logical relationship of a hypothesis to a premise (e.g., for the premise \textit{Jenny loves all animals} the hypothesis \textit{Jenny loves cats} is an \textsc{entailment}, and \textit{Jenny hates dogs}, a \textsc{contradiction}).
We choose NLI because it is among the best-studied NLU tasks, with demonstrated value, but also multiple well-documented data quality issues that arise in crowdsourced data collection, many of which can be traced to a given \textit{heuristic}.
Because these heuristic-based issues are prevalent, we focus on NLI with the aim that our methodology can inform data collection for new tasks in which there are fewer known heuristics.

Previous attempts to develop more effective NLU data collection protocols have been limited in their ability to assess the efficacy of their interventions, as they often lack direct comparisons between different collection methods.
We test three levels of expert involvement: (i) a baseline group with no hands-on expert involvement (`Baseline'), (ii) a group that followed linguistically-motivated constraints that experts developed to target heuristic-based weaknesses in the data (`linguist-in-the-loop' (LitL)), and (iii) a group that extended the LitL protocol to add direct interaction with the experts, including individual-level discussion about the task on the chat platform Slack (`LitL Chat').
These three protocols are shown in Figure~\ref{fig:protocol_diagram}, and a task example with a constraint from LitL and LitL Chat is shown in Figure \ref{fig:sample_hit}

\begin{figure}[t]
\centering
\begin{minipage}{0.92\linewidth}
\newcommand{0.5}{0.5}
\newcommand{0.9}{0.9}
\tiny

\textbf{\scriptsize Text:} They inhabit the near-boiling water of geysers in Yellowstone, and the even hotter water in volcanic vents on the ocean floor. \newline

$\mathord{\scalerel*{\Box}{gX}}$ The definitely correct sentence {\color{red}does not reuse nouns, verbs, adjectives, or adverbs from the text} (\$0.10) \vadjust{\vspace{0.5 pt}}\newline
$\bullet$ Your definitely correct statement must not contain any of the following words: {\color{red}there, can, may, might, some, people} \vadjust{\vspace{0.9 pt}}\newline
\textbf{\scriptsize Definitely correct statement:}\vadjust{\vspace{0.5 pt}}\newline
\framebox[\linewidth]{}\vadjust{\vspace{3pt}}\newline

$\mathord{\scalerel*{\Box}{gX}}$ The maybe correct sentence {\color{red}does not reuse nouns, verbs, adjectives, or adverbs from the text} (\$0.05) \vadjust{\vspace{0.5 pt}}\newline
$\bullet$ Your maybe correct statement must not contain any of the following words: {\color{red}often, several, many, most, some, other, will} \vadjust{\vspace{0.9 pt}}\newline
\textbf{\scriptsize Maybe correct statement:}\vadjust{\vspace{0.5 pt}}\newline
\framebox[\linewidth]{}\vadjust{\vspace{3pt}}\newline

$\mathord{\scalerel*{\Box}{gX}}$ The definitely incorrect sentence {\color{red}does not reuse nouns, verbs, adjectives, or adverbs from the text} (\$0.05) \vadjust{\vspace{0.5 pt}}\newline
$\bullet$ Your definitely incorrect statement must not contain any of the following words: {\color{red}any, never, no, nothing, not/n't, only, always, all} \vadjust{\vspace{0.9 pt}}\newline
\textbf{\scriptsize Definitely incorrect statement:}\vadjust{\vspace{0.5 pt}}\newline
\framebox[\linewidth]{}

\end{minipage}

\caption{Round 5 HIT with the optional \textit{No Overlap} constraint shown.}
\label{fig:sample_hit}
\end{figure}

We observe that expert involvement (LitL and LitL Chat) during data collection reduces the impact of some learnable heuristics and results in a more challenging final dataset with model performances that are 5 points lower on validated data compared to Baseline.
Examples in all protocols qualitatively appear equally free of noise (incorrect labels, typos, etc.), and lexical diversity increases in later rounds for the protocols with linguist intervention.\footnote{Appendix \ref{app:examples} contains a sample of validated examples.}
However, we find no evidence of better model accuracy on adversarial examples or out-of-domain datasets.
Further, we do not find any benefit to providing a chatroom for crowdworkers to interact directly with linguists.

\section{Related Work}

\paragraph{NLI Data Collection Methods}
Large-scale human-elicited datasets include the Stanford Natural Language Inference Corpus \citep[SNLI;][]{bowman2015large}, the Multi-genre Natural Language Inference Corpus \citep[MNLI;][]{williams2020mnli}, the Chinese OCNLI corpus \citep[][]{hu2020ocnli}, and Adversarial NLI \citep[ANLI;][]{nie2020anli}.
All four datasets use non-expert annotators to write hypotheses and annotate inference labels from pre-defined short texts, though only OCNLI and ANLI add interventions to increase data diversity.
OCNLI
hires students specializing in linguistics and language studies to construct hypotheses and adds data collection rounds with instructions for avoiding known sources of bias.
ANLI uses a human-and-model-in-the-loop procedure to elicit examples that are progressively more difficult for their model, resulting in a dataset with a large human--model performance gap,
though identifying the cause for model failure is left up to the discretion of the worker.

Efforts to improve on sentence writing tasks for NLI have yielded mostly negative results in head-to-head protocol comparisons.
In an experimental comparison on different NLI crowdsourcing protocols, \citet{vania2020asking} find that automatically selecting premise-hypothesis pairs
for crowdworker annotation does not yield a better dataset compared to a baseline sentence writing protocol.
\citet{bowman2020new} compare interventions aimed at improving NLI writing, using protocol variants that constrain the worker's task, but they do not see improvements in transfer learning results
compared to a baseline protocol.

\paragraph{NLI Dataset Bias}
Several studies have identified systematic biases in NLI datasets that the models trained on them subsequently learn (often robustly).
Hypothesis-only bias, where a model correctly labels the relationship of a premise-hypothesis pair based only on the hypothesis, is a well-documented issue \citep[i.a.]{poliak-etal-2018-hypothesis,gururangan2018artifactsNLI}.
Lexical overlap between the premise and hypothesis is another source of bias \cite{mccoy-etal-2019-right,naik2018stress}, as greater overlap between a premise and hypothesis is associated with a greater likelihood the label is \textsc{entailment}.
Sources of bias can also be due to gaps in the training data, and \citet{sinha2020unnatural} point to the lack of syntactic understanding in NLI models as one such example, noting that models often ignore syntactic information entirely.
The well-studied biases in NLI make the task a good test case for protocols designed to assess biases as data is collected.

\paragraph{Methods for Filling the Gaps in Datasets}
To collect challenging examples for NLU tasks, researchers have explored altering labeled data to create targeted or adversarial examples.
\citet{kaushik2020learning} have crowdworkers make minimal edits to annotations to align with a revised label.
\citet{gardner2020evaluating} create contrast sets for evaluation by having experts alter already-annotated examples such that the resulting label changes.
\citet{wei-zou-2019-eda} use simple automatic data manipulations to augment datasets for several text classification tasks, resulting in more robust models.
More linguistically sophisticated manipulations have been used to augment MNLI to improve monotonicity reasoning \cite{yanaka-etal-2019-help} and mitigate lexical overlap heuristic \cite{min-etal-2020-syntactic}.
These methods are applied after crowdsourced data collection is complete, so it is not clear if the gaps they identify in a final dataset would have been avoidable if addressed during the data collection process.

Most similar to our approach, OCNLI's instructions nudge annotators towards writing examples that address \textit{known} sources of bias.
They find that encouraging annotators to follow constraints such as avoiding negation in a \textsc{contradiction} label results in a harder dataset.
We expand on this work by introducing a wider range of constraints and assessing their effects throughout data collection.
Our approach is also similar to \citeauthor{vidgen2020learning}'s (\citeyear{vidgen2020learning}) human-generated hate-speech dataset.
They introduce \textit{pivots} during data collection in which they instruct annotators about how to write to fool their model.
We expand on their method by
qualitatively assessing the annotations to identify issues specific to our data as it is collected.

\paragraph{Expert Interaction with Crowdworkers}
\citet{DBLP:conf/www/TangYH19} report that direct communication among crowdworkers leads to improved task performance on image labeling, optical character recognition, and audio transcription.
This suggests that collecting higher quality data is possible when workers have real-time group interaction.
Other studies have reported that interaction among crowdworkers is an effective tool for limiting some forms of bias and increasing accuracy \citep{drapeau2016microtalk, schaekermann2018resolvable}.
\citet{roit2020controlled}, in a different strategy, gives annotators detailed feedback during training, then selects only a small number of those workers for the larger annotation task.
This strategy frontloads the work of the experts and relies on the selected workers to perform the task consistently.

Despite the potential benefits of real-time interaction between crowdworkers and experts, there has not yet been a direct comparison of protocols that differ based on this variable.
To our knowledge, this study is both the first to test the effect of this interaction and the first head-to-head experimental assessment of human-in-the-loop dataset collection methods, allowing us to make conclusions about the causal effects of the different interventions compared to a baseline.

\section{Dataset Collection}
\label{subsec:datacollec}

\paragraph{Task Description}
Our task is modeled on MNLI's data collection procedure.
We present workers with a text, for which they write statements they consider definitely correct, maybe correct, and definitely incorrect.
Following each round of sentence writing, crowdworkers validate 500 examples from their protocol.
We collect four validations for each example that we validate and use these labels plus the original one to assign a gold label based on majority vote.
Examples for which no gold label can be assigned are removed from the data.
We use the validated data to evaluate our models and the unvalidated data for training.
Workers with a validation rate below 70\% or whose validation responses fail to match the gold label at least 70\% of the time are subject to disqualification.
Throughout the study, we disqualified three workers from Baseline, three from LitL, and two from LitL Chat.

\paragraph{Pay Structure}
To retain crowdworkers for all five rounds, we increase the base pay of \$1/HIT by \$0.05 each round and pay a \$20.00 bonus after the last round.
To ensure we collect sufficient examples from each worker, we award a bonus worth 10\% of their base pay for reaching milestones of 10, 50, and 100 HITs each round.
To encourage workers to write high-quality examples, we pay a \$5.00 bonus each round to workers with over 25 HITs and at least a 95\% validation rate.
We estimate that, with bonuses, a worker who completes 70 HITs with a high validation rate will earn \$81 in Round 1 ($\sim$\$16/hr), with that rate rising to \$95 in Round 5 ($\sim$\$19/hr).
Workers in LitL and LitL Chat earn additional bonuses for completing challenge options (\$0.05-\$0.10), and workers in LitL Chat earn bonuses for participation in the chatroom (\$1.50 for any engagement, \$10.00 for active engagement).

\subsection{Crowdworker recruitment}
We use a pre-test to recruit workers via Amazon Mechanical Turk (MTurk).
The pre-test is open to workers in the United States with approval rates at or above 98\% and more than 1000 HITs approved.
The pre-test is a sentence-writing task where workers see a premise and write
hypotheses under each of the three NLI labels.
To assess if workers can follow more complicated instructions, they are also asked to write one entailed sentence that uses a conjunction and one neutral sentence that does not re-use any words from the text.

We collect responses from 155 crowdworkers, of whom 145 indicate interest in completing future, similar HITs.
From those 145, we read their responses and exclude 24 for failing to adequately complete the task (many due to responses that do not follow instructions).
The remaining 121 crowdworkers are retained and split between the three experimental protocols in a pseudo-random way such that (i) the three workers who asked not to participate in a chat forum are placed in the Baseline or LitL protocol,\footnote{Though a potential design confound, this was necessary and had minimal effect. \textit{Requiring} workers to sign up for a third party service violates Amazon's terms of service, so we allow participants to opt out. Only three participants opted out of the chat (two of whom dropped out after Round 1), and many workers placed in a non-chat protocol had indicated a willingness to participate in the chat.} and (ii) groups are matched based on a 4-point rating scale of their qualitative performance on the pre-test.
We ultimately had 37 annotators participate in data collection in Baseline, 30 in LitL, and 32 in LitL Chat.

\subsection{Annotation Details}
\label{sec:annot_details}

Crowdworkers annotate data in five rounds, with each round lasting one week.
Between rounds, we conduct several planned diagnostics on our datasets to monitor the impact of our intervention and inform annotator feedback for the following round.

\paragraph{Annotation stage}
Annotators construct hypotheses based on premises taken from the \textsc{Slate} subset of MNLI. \textsc{Slate} hosts popular culture articles from the archives of Slate Magazine.
After Round~1, we exclude premises that are shorter than six tokens based on feedback from annotators that many of the very short premises are incomplete, nonsensical, or confusing to write hypotheses for.

\paragraph{Diagnostic stage}
After each round, we fine-tune RoBERTa \citep{liu2019roberta} models using data collected up to that round.
We then evaluate the models on diagnostic examples from GLUE \citep{wang2019GLUE} and HANS \citep{mccoy-etal-2019-right}.
The GLUE examples target different aspects of linguistic reasoning including lexical semantics, predicate-argument structure, logic, and world knowledge.
HANS tests for three shallow heuristics, including lexical overlap between a premise and hypothesis.
We also train and evaluate RoBERTa models using hypothesis-only inputs to assess hypothesis-only biases in the data \citep{gururangan2018artifactsNLI}.
Finally, we assess the distribution of hypothesis lengths and the pointwise mutual information (PMI) between each word in the vocabulary and label.
Hypothesis length does not appear to differ by protocol or label, so it never informs our constraints.

We use these diagnostics as well as qualitative reviews of the data to devise linguistically-motivated guidelines for the following round, allowing us to adapt feedback for crowdworkers in a structured way as the data is collected.
This process is conducted by five of the authors who have graduate training in English syntax and semantics.

\begin{table*}[ht]
    \centering
    \resizebox{\textwidth}{!}
    \rowcolors{2}{gray!25}{white}
    \begin{tabular}{p{0.125\textwidth}p{0.42\textwidth}p{0.42\textwidth}rrr}
    \toprule
    \bf Constraint&\bf Premise&\bf Hypothesis&\bf Label&\multicolumn{2}{c}{\bf Attempt rate} \\
    & & & & LitL & Chat\\\midrule
Hypernym or hyponym & Does anyone know what happened to \textbf{chaos}? & Whatever happened to the \textbf{lack of order} is certainly a mystery. & E & 22.8 & 23.7\\
Banned word in diff. label & Inflation is supposed to be a deadly poison, not a useful medicine. & \textbf{All} people believe inflation is supposed to be a useful medicine & C & 43.7 & 27.7\\
Temporal reasoning & John Kasich dropped his presidential bid. & They said that \textbf{earlier}, John Kasich had dropped his presidential bid. & E & 34.1 & 10.0\\
Synonym or antonym & 2) This particular instance of it \textbf{stinks}. & This instance is perceived to be \textbf{a good thing}. & C & 39.5 & 24.5\\
All overlap & News argues that most of America's 93 million volunteers aren't doing much good. & News argues that volunteers aren't doing much good. & E & 21.8 & 30.4\\
Register change & First, the horsemen brought out a teaser horse. & Teaser horses are commonly thought to be both entertaining and tragic. & N &25.3 & 15.0\\
No overlap & and she doesn't floss while driving. & The woman has an automated car. & N &29.2 & 22.3\\
Relative clause & Sun Ra's spaceships did not come, as it were, out of nowhere. & The spaceships \textbf{that belong to Sun Ra} came out of nowhere & C & 35.0 & 24.3 \\
Reverse argument order & After an \textbf{inquiry} regarding \textbf{Bob Dole}'s ... & It is illegal for \textbf{Bob Dole} to receive \textbf{inquiries}. & N &36.7 & 29.4 \\
Grammar change & The Bush campaign \textbf{has} a sweet monopoly on that. & The Obama campaign \textbf{had} a sweet monopoly on that. & C & 22.6 & 13.4 \\
Sub-part & He was crying like his mother had just walloped \textbf{him}. & He cried a lot, as though he were walloped on \textbf{his behind}. & E & 23.2 & 19.1 \\
Background knowledge & In both \textbf{Britain and America}, the term covers nearly everybody. & The term generally applied to \textbf{countries in two opposite sides of the world}. & E & 32.9 & 15.9 \\\bottomrule
    \end{tabular}
    }
    \caption{Sentence pairs displaying each challenge option. Where applicable, relevant contrasts are bolded. Examples are randomly drawn from data that passed validation on the constraint with the restriction that both sentences be fewer than 80 characters ($\sim$ 32\% of the data).
    The last column shows the percentage of the challenges attempted.}
    \label{tab:guideline_examples}
\end{table*}

\subsection{Constraints}
\paragraph{Banned Words}
After Round 1, crowdworkers in LitL and LitL Chat are instructed not to use certain words when writing sentences for each label.
In each round, 5-7 new banned words are identified based on the PMI for each label.
We use PMI to identify words to ban under each label, as words with high label PMI are a major contributor to hypothesis-only bias.
This constraint is mandatory in all HITs.
Figure \ref{fig:sample_hit} shows examples of the banned words during Round 5.

\paragraph{Challenge Options}
We use constraints, framed as \textit{challenge options} to the worker, to target heuristics that we identify in the data during the diagnostic state.
By explicitly telling workers to avoid these heuristics, we aim to lower their contribution to any bias in the final dataset.
We determine constraints through qualitative assessment of the data, taking into consideration syntactic diversity, lexical choice, and semantic or world-knowledge-based reasoning patterns.
For example, after noticing that the majority of hypotheses relied only on the stated information from the premise in Round 1, we encouraged workers in Round 2 to focus on ``background knowledge'' (example in Table \ref{tab:examples}) that they know to be true, but isn't explicitly stated, such as the knowledge that Britain and America are countries on opposite sides of the world.

After Round 1, each HIT in LitL and LitL Chat lists one constraint.
This task is optional for the workers, as some constraints are incompatible with some examples.
The 12 challenge options are defined in Appendix \ref{challenge_options},
with examples in Table \ref{tab:guideline_examples}.

\subsection{Protocols}
\label{subsec:protocols}

\paragraph{Baseline Protocol}
Our Baseline protocol follows the basic task description in the beginning of \S \ref{subsec:datacollec} and does not include any direct expert involvement.

\paragraph{Linguist-in-the-Loop (LitL) Protocol}
LitL extends the Baseline protocol with constraints (described in \S \ref{sec:annot_details}).
As the constraints make the task more difficult, we award bonuses to workers who indicate that they attempted a challenge option.
The bonus is \$0.05-\$0.10 per example, determined by the linguists' assessment of the difficulty.
For example, the \textit{No Overlap} constraint is more difficult in entailment examples than neutral, so a \textit{No Overlap} entailment example receives a higher bonus.

During the validation round, examples with challenge options are validated for whether they adhere to the given constraint, regardless of whether the annotator indicated that they had attempted the challenge.
For any worker whose validation rate on the bonus challenges is below 50\%, we contact them to explain the source of their errors.

\paragraph{LitL Chat Protocol}
We provide direct communication with expert linguists on Slack.
We encourage workers to ask task-specific questions for anything they find challenging or confusing.
Most questions seek to clarify if a certain strategy `counts' as adhering to a constraint.
Feedback given via email in the LitL protocol is instead given via direct message in Slack, unless the worker initiates contact over email.
Additionally, at the beginning of Rounds 3--5, we identify creative examples written in a previous round and post them to Slack for inspiration, with a brief comment.

\subsection{Annotator Performance}

\paragraph{Inter-Annotator Agreement}
Baseline shows the highest inter-annotator agreement with $\kappa$ = 0.71, while LitL and LitL Chat have 0.64 and 0.63, respectively.
All three protocols meet the standard threshold for ``substantial agreement.''
Validation rates are 93.7\% for Baseline, 89.76\% for LitL, and 91.36\% for LitL Chat.
LitL and LitL Chat may have slightly lower validation rates than Baseline because the constraints lead to challenging examples, making the validator's task more difficult.

\paragraph{Proportion of Constraints Attempted}
The attempt rate of bonus challenges differs substantially between constraints (Table \ref{tab:guideline_examples}).
Overall, more abstract categories (e.g., background knowledge) are attempted less often than more concrete constraints.
There are also differences by protocol, as LitL had a higher attempt rate than LitL Chat.
One reason for this difference may be that workers in LitL Chat were more selective about identifying good examples on which to apply the constraints.
Supporting this possibility, we find that LitL Chat had higher constraint validation rates than LitL in Rounds 4 and 5, indicating that workers in LitL Chat adhered to the constraints more accurately after practice.

\paragraph{Use of Slack}
The total number of active workers in Slack fell from 23 in Round 1 to just 16 by Round 4.\footnote{Round 5 was even lower, but spanned the US Thanksgiving holiday, which likely artificially lowered participation.}
The total number of messages sent also fell with each round, going from about 215 posts and replies in Round 1 to 162 in Round 4.
It may be that workers rely on the chat less as they become more familiar with the task.

\section{Experiments}
For each round and protocol, we collect 3.5k examples and use the 500 validated examples (\S \ref{subsec:protocols}) as validation data and the remaining 3k for training.\footnote{Data and code: \href{https://github.com/Alicia-Parrish/ling_in_loop/tree/master}{github.com/Alicia-Parrish/ling\_in\_loop}} We then fine-tune a RoBERTa$_{\rm{Lg}}$ model on all data accumulated up to Round $n
. For example, the Round 2 model is trained on examples from Rounds 1 and 2 with training and validation sizes of 6k and 1k, respectively. We also fine-tune a RoBERTa$_{\rm{Lg}}$ model previously trained on MNLI (RoBERTa$_{\rm{Lg+MNLI}}$) and find similar results (details in Appendix \ref{app:mnli_trained_results}). After each round, we evaluate our models on the diagnostics described in \S \ref{sec:annot_details}.

After the final round of data collection, we evaluate models trained on our data on MNLI-mismatched \citep{williams2020mnli} and ANLI \citep{nie2020anli}.
The MNLI corpus includes two evaluation sets,  MNLI-matched and MNLI-mismatched, with examples sourced from different genres.
We evaluate on MNLI-mismatched, as we source our premise sentences from an MNLI-matched genre.
Evaluating on held-out sets allows us to test if our interventions lead to increased model accuracy on datasets generated through different protocols or from different sources while ensuring that we do not overly tune our feedback to these benchmarks.

We estimate average accuracy and confidence intervals by fine-tuning 10 additional models with a sample of 90\% of the collected training data. We use the best hyperparameters for each protocol and round from our hyperparameter search described below. In sampling the data, we first sort the data by annotator and successively remove 10\% of examples, allowing us to study variation among annotators while controlling for training set size.

\paragraph{Implementation}
To fine-tune our models, we perform a grid search over learning rate $ \in \{5e-6, 1e-5, 2e-5, 3e-5\}$ and batch size $ \in \{16, 32\} $ and use the hyperparameters yielding the best in-domain validation accuracy. We train for 20 epochs, since each round of data collection yields 3k training examples, and longer training has been shown to help smaller training sets \citep{Zhang20Revisiting}. Our code
is based on \texttt{jiant} \citep{phang2020jiant}, which uses PyTorch \citep{pytorch2019} and Transformers \citep{wolf-etal-2020-transformers}.

\section{Results}

\begin{figure}[h]
    \centering
    \includegraphics[width=0.5\textwidth]{figs/roberta-large/combined_indomain.pdf}
    \caption{Performance of RoBERTa$_{\rm{Lg}}$ fine-tuned on data collected through different protocols on in-domain validation data trained with either the normal sentence pairs (top) or hypothesis-only (bottom) input. Higher hypothesis-only accuracy indicates more bias. For each round, we include training and validation data \textit{accumulated} up to Round $n$. Dashed black line marks average majority class baseline across protocols.}
    \label{fig:indomain_val}
\end{figure}

\begin{figure*}
    \centering
    \includegraphics[width=\textwidth]{figs/roberta-large/combined_itereval.pdf}
    \caption{Performance of RoBERTa$_{\rm{Lg}}$ fine-tuned on data collected through different protocols on the GLUE diagnostic set (top) and HANS non-entailment examples (bottom).}
    \label{fig:iter_val}
\end{figure*}

\begin{table}[]
    \centering
    \small
    \begin{tabular}{l r r r}
        \toprule
        \bf Round & \bf Baseline & \bf LitL &\bf LitL Chat \\
        \midrule
        R1 & 84$_{\text{(1.2)}}$ & 82$_{\text{(1.1)}}$ & 86$_{\text{(1.1)}}$ \\
        R2 & 85$_{\text{(0.6)}}$ & 81$_{\text{(1.0)}}$ & 85$_{\text{(0.9)}}$ \\
        R3 & 86$_{\text{(0.7)}}$ & 82$_{\text{(0.8)}}$ & 84$_{\text{(0.9)}}$ \\
        R4 & 86$_{\text{(0.8)}}$ & 83$_{\text{(0.7)}}$ & 84$_{\text{(1.0)}}$ \\
        R5 & 87$_{\text{(0.5)}}$ & 82$_{\text{(0.6)}}$ & 84$_{\text{(0.9)}}$ \\
        \bottomrule
    \end{tabular}
    \caption{Average performance of RoBERTa$_{\rm{Lg}}$ fine-tuned on MNLI over 10 random restarts on validated examples accumulated up to Round $n$. Values in parentheses indicate standard deviation of performance.}
    \label{tab:mnli-only_indomain}
\end{table}

\paragraph{Evaluation Difficulty}
We test whether data collected with expert intervention leads to a more challenging test set by comparing  in-domain performance for each protocol for RoBERTa$_{\rm{Lg}}$,
using training and validated evaluation data accumulated up to Round $n$ (Figure \ref{fig:indomain_val}).
This allows us to study the characteristics of an iteratively collected corpus using $n$ rounds of each protocol.
We see that LitL and LitL Chat performance falls below Baseline after the introduction of linguistically-informed constraints in Round 2.
Table \ref{tab:mnli-only_indomain} shows a similar trend -- performance from RoBERTa$_{\rm{Lg}}$ fine-tuned \textbf{only on MNLI} on the same validation sets
decreases or remains relatively low for LitL and LitL Chat, while performance on Baseline increases as more data is collected.
As we evaluate on validated examples, it is unlikely that this lower performance is due to noise in the data. Rather, these findings indicate we are able to create more challenging evaluation data using the LitL and LitL Chat interventions.

\paragraph{Hypothesis-Only Bias}
We test whether the data collected with linguist intervention leads to a reduction in hypothesis-only bias by comparing accuracy for each protocol for RoBERTa$_{\rm{Lg}}$ trained on hypothesis-only input, where lower accuracy suggests less bias in the data (Figure \ref{fig:indomain_val}).
Both LitL and LitL Chat result in lower hypothesis-only bias than Baseline, and this gap widens in later rounds.
To assess whether this widening from Round~1 to~5 is statistically reliable, we conduct a two-way ANOVA of round by protocol, which yields a significant interaction (\textit{p} = 0.049), indicating that while hypothesis-only performance increases for all protocols with more training examples, this increase in bias is significantly reduced in LitL and LitL Chat compared to Baseline.
The lower bias in LitL and LitL Chat may be due to the lower average word-label PMI, which increases over rounds for Baseline while consistently falling in both LitL and LitL Chat.\footnote{A two-way ANOVA again reveals a significant interaction of protocol by round (\textit{p} = 0.022) on word-label PMI values.}
However, for all protocols, accuracies are still above chance performance, leaving room to further reduce hypothesis-only bias.

\paragraph{Diagnostic Sets}
We use diagnostic tests to evaluate whether fine-tuning on data collected with linguist involvement leads to a model that has higher performance on challenge test sets.
Figure \ref{fig:iter_val} shows model performance on the GLUE diagnostic set and HANS non-entailment examples.
A two-way ANOVA of round by protocol does not reveal any significant interactions or main effects for GLUE.
For HANS, we see higher accuracy from LitL and LitL Chat for Lexical Overlap and Subsequence examples in Rounds 4 and 5 after introducing \textit{No} and \textit{All Overlap} constraints, though the interaction is only significant with RoBERTa$_{\rm{Lg+MNLI}}$ (\textit{p} = 0.0021 and \textit{p} = 0.0017 for Lexical Overlap and Subsequence, respectively).
Performance on HANS entailment examples are in line with \citet{mccoy-etal-2019-right} with median accuracies of 90\% or higher (Appendix \ref{app:hans_entailment}).

To investigate whether rates of lexical overlap differ by protocol, we assess classification accuracy for a linear model trained only on the example's overlap rate, defined as the proportion of words in the hypothesis that are also in the premise. We observe that the potential bias introduced from overlap rate is strongest in the Baseline protocol, which performs 9.52 points above majority class guessing, while LitL and LitL Chat perform 8.06 and 6.88 points above majority class guessing, respectively.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.5\textwidth]{figs/roberta-large/combined_val.pdf}
    \caption{Performance of RoBERTa$_{\rm{Lg}}$ fine-tuned on data collected through different protocols on MNLI-mismatched (top) and ANLI (bottom).}
    \label{fig:combined_val}
\end{figure}

\paragraph{Held-Out Evaluation Sets}
We next test whether models fine-tuned on data collected with linguist involvement show better out-of-domain performance.
Figure \ref{fig:combined_val} shows that there is little difference in ANLI and MNLI-mismatched performance among all protocols.
We perform a more granular analysis on ANLI examples using the tags from \citet{williams2020anlizing} and again find no clear effect of protocol (details in Appendix \ref{app:anli_by_tag}).
Even though our interventions reduce hypothesis-only bias and improve model performance on HANS non-entailment examples, we have no evidence that these benefits transfer to out-of-domain examples or examples from adversarial protocols.

\section{Considerations in Choosing a Protocol}

In broad terms, we observe a benefit from dynamically updating annotator guidelines to address gaps and biases observed during data collection.
This procedure increased the average cost per example by 4.1\% over an average base cost of \$0.367.
In an exit survey,
over 50\% of annotators in LitL and LitL Chat indicated that they would have completed more optional challenge examples if the pay had been higher.
We offered \$0.05 to \$0.10 per example, but given the somewhat low rate at which annotators chose to attempt the challenges (28.6\% and 21.2\% of examples for LitL and LitL Chat, respectively), we find it likely that increasing the amount offered per example would have increased participation, potentially also increasing the benefits observed in model performance.
We recommend that future work using challenge options offer bonuses worth at least 15\% of the base pay.

\paragraph{Cost of Linguist Involvement}
The iterative analyses and updates to the guidelines used for LitL and LitL Chat protocols took 10--12 hours of expert time per week, compared to a base time cost of about one hour per week for just monitoring task completion.
The use of Slack nearly doubled the expert time needed, adding an \textit{additional} 8--10 hours each week just for LitL Chat over LitL, even after taking into account the slight reduction in time spent replying to email questions that shifted to Slack.
If we value linguist time at \$40/hr, then this raises the final price per example to \$0.378 in Baseline, with LitL 31.2\% higher, and LitL Chat 58.5\% higher than baseline.
Despite the extra time spent with annotators, we did not observe any measurable benefit to this more hands-on intervention.

\paragraph{Qualitative Considerations}
Though many annotators in LitL Chat expressed that they enjoyed the extra communication and help, annotators from LitL and LitL Chat rated the task as `more enjoyable' than typical MTurk tasks at nearly identical rates (85.2\% and 87.5\% respectively, compared to 67.7\% in Baseline).
Ratings of the difficulty of writing and validation tasks were also nearly identical between the LitL and LitL Chat protocols.

\section{Conclusion}
We conclude that for some tasks, it is beneficial to integrate expert analysis of data \textit{during} data collection, so that the expert can dynamically update guidelines and constraints based on existing gaps and biases in the dataset.
Through a controlled experiment, we have shown that
this expert involvement leads to higher accuracy in measures related to the dataset gaps and biases that were identified and targete
, and the iterative procedure allowed us to identify new areas of weakness at each round.
As we did not observe any increases in out-of-domain accuracy, we conclude that
there is no more general benefit from our interventions beyond the targeted areas of weakness.
Future work could extend this protocol to identify additional interventions that would lead to datasets with better generalizability.
Finally, we find no evidence to support the claim reported by some studies that one-on-one interactions between experts and crowdworkers is beneficial in more challenging tasks.

\bibliographystyle{acl_natbib}