text:
\section{Introduction}

Natural Language Processing (NLP) research has advanced considerably in recent years, developing reliable predictors for argumentation \cite{mirkin2018listening,lippi2016argument,Aharoni:14,Biran:11,Carlile:18,Habernal:14,Park:18,Stab:14,Stab:17}, specificity \cite{Li:15,Li:16,Louis:12,Gao:19}, and collaboration \cite{richey2016sri}. The majority of corpora informing these developments are made up of written texts (e.g., documents, asynchronous online discussions) extracted from the web and often written by individuals. Additionally, annotations in these corpora typically focus on one linguistic dimension at a time (e.g., specificity \textit{or} dimensions of argumentation). As such, few published corpora both (1) focus on synchronous multi-party argumentation and (2) include multiple simultaneous annotations.

To address the lack of multi-party synchronous argumentation corpora that include multiple simultaneous annotations, we are releasing the Discussion Tracker corpus, which includes transcriptions of 29 multi-party synchronous argument-based dialogues collected in high school English classes and annotated for three simultaneous but distinct discourse dimensions (argumentation, specificity and collaboration). In addition to providing descriptive statistics on the corpus, we provide code and benchmark performance figures for predicting each of the three annotated dimensions, illustrating the challenging nature of this corpus.  We then illustrate the benefits the corpus has already afforded NLP algorithms for improving argument mining with multi-task learning and discuss other potential uses of the corpus for further NLP research.

\section{Related Work}

The development of argument mining tasks has been heavily informed by corpora of written texts extracted mostly from the web (e.g., online newspapers, Wikipedia, blog posts, user comments) \cite{al2018modeling,Brian:11,Aharoni:14,Habernal:14,Park:18,Swanson:15,Cabrio:14,Boltuzic:14}, from student essays \cite{Stab:14,Stab:17,Carlile:18}, from legal documents (parliamentary records and court reports) \cite{Mochales:11,Ashley:13}, or from speeches \cite{mirkin2018listening,lippi2016argument}. Similarly, NLP developments in specificity have typically drawn on newspaper articles \cite{Louis:11,Li:15,Li:16} or online content \cite{Gao:19,Ko:19}. Although work in Computer Supported Collaborative Learning (CSCL)  \cite{Weinberger:06,Fischer:13,Noroozi:13,Scheuer:14,Dillenbourg:08} has used mainly multi-party online data, it has typically drawn from asynchronous written discourse as opposed to synchronous dialogue.

Many corpora of written text data that have been annotated for similar dimensions as the Discussion Tracker corpus have focused on single and independent tasks like argument components (e.g. claims, premises) \cite{Brian:11,Aharoni:14,Habernal:14,Mochales:11}, relations between pairs of arguments (e.g. support, attack) \cite{Park:18}, or specificity \cite{Li:15,Li:16,Louis:12}. Some corpora include multi-level annotations that allow for performing multiple tasks (e.g. argument components and relations) \cite{al2018modeling,Stab:14,Stab:17,Ashley:13,Carlile:18}. However, multi-level annotations like these have often been heavily dependent on one another. For example, although Carlile et al. analyzed two simultaneous annotations (argument components and specificity), their definition of specificity was contingent on argument component type, defining specificity of claims differently from specificity of premises. In this way, their multi-level annotations were tightly coupled and could only be analyzed in conjunction with one another. In cases like Al Khatib et al. \shortcite{al2018modeling}, which provides distinct multi-level annotations for asynchronous on-line argumentation, they do not include specificity at all. By providing annotations for multi-level and distinct annotations for collaboration, argumentation, and specificity in synchronous dialogues in natural learning environments, we believe the Discussion Tracker corpus is capable of offering the NLP community new opportunities for research.

Of the extant multi-party synchronous data, the most similar corpus to ours  is Richey et al.'s SRI corpus \cite{richey2016sri}. Their corpus was developed for analysis of the ways student talk patterns correlated with collaborative learning. Equipped with multi-speaker, small group audio recordings of middle school students discussing mathematical solutions, the SRI corpus provides multi-level annotations of collaboration indicators and collaboration quality. However, although the SRI corpus includes multi-party synchronous argumentation data, some major differences stand out from the Discussion Tracker corpus. First, the scope of the the Discussion Tracker corpus extends beyond collaborative dimensions to include dimensions of argumentation and specificity as well, whereas the SRI corpus focuses only on collaboration. Second, whereas the group size of the multi-party dialogues in the SRI corpus is three students, the group sizes in the Discussion Tracker corpus average around 15 students per discussion. Third, in addition to providing the gender of speakers, as the SRI corpus does, the Discussion Tracker corpus also includes identification of the racial background of speakers. Fourth, Discussion Tracker will be released as written transcriptions with corresponding annotations whereas the SRI data is released as audio files with time-stamped annotations. The differences between these corpora are not to suggest one is better than the other, but simply that different formats afford different avenues for analysis.

\begin{table*}[!htb]
\centering
\begin{tabular}{|l|l|r|l|r|l|r|r|r|r|r|}
\hline
Tchr & M/F & \begin{tabular}[c]{@{}l@{}}Tchr\\  Exp\end{tabular} & Loc. & Grade & Course & \begin{tabular}[c]{@{}l@{}}Class\\ size\end{tabular} & Male & Fem. & \begin{tabular}[c]{@{}l@{}}Race\\ White\end{tabular} & \begin{tabular}[c]{@{}l@{}}Race\\ Non-white\end{tabular} \\ \hline
1 & F & 12 & suburb & 10 & regular & 29 & 20 & 9 & 28 & 1 \\ \hline
2 & M & 18 & suburb & 11 & honors & 11 & 5 & 6 & 7 & 4 \\ \hline
3 & F & 6 & suburb & 9 & honors & 16 & 11 & 5 & 8 & 8 \\ \hline
4 & M & 12 & suburb & 12 & AP & 13 & 4 & 9 & 6 & 7 \\ \hline
5 & F & 15 & urban & 9 & regular & 16 & 11 & 5 & 13 & 3 \\ \hline
6 & F & 14 & urban & 11 & AP & 18 & 11 & 7 & 14 & 4 \\ \hline
7 & F & 30 & urban & 10 & regular & 25 & 15 & 10 & 21 & 4 \\ \hline
8 & F & 30 & rural & 12 & AP & 13 & 3 & 10 & 12 & 1 \\ \hline
9 & F & 20 & rural & 10 & regular & 15 & 11 & 4 & 15 & 0 \\ \hline
10 & M & 20 & rural & 9 & honors & 25 & 16 & 9 & 25 & 0 \\ \hline
\end{tabular}
\caption{Characteristics of 10 classroom environments. Columns from left to right: teacher, teacher's gender (male or female), years of teaching experience, school location, grade level, course type, class size, number of male students, female students, white students, non-white students.}
\label{tab:Classrooms}
\end{table*}

\section{Data Collection}
\begin{table}[!htb]
\begin{tabular}{|l|l|r|r|r|}
\hline
Disc.  & Text & Stu. & Min. & \#Trns \\ \hline \hline
1a & \begin{tabular}[c]{@{}l@{}}Death of \\ Ivan Illych\end{tabular} & 27 & 40 & 208 \\ \hline
1b & Night & 28 & 41 & 134 \\ \hline
1c & The Name & 24 & 42 & 216 \\ \hline \hline
2a & \begin{tabular}[c]{@{}l@{}}Lgnd of \\ Slpy Hllw\end{tabular} & 11 & 32 & 49 \\ \hline
2b & \begin{tabular}[c]{@{}l@{}}The Mnstr's \\ Black Veil\end{tabular} & 9 & 19 & 43 \\ \hline
2c & \begin{tabular}[c]{@{}l@{}}Dickinson \\ Poems\end{tabular} & 10 & 33 & 110 \\ \hline \hline
3a & \begin{tabular}[c]{@{}l@{}}Lord of \\ the Flies\end{tabular} & 16 & 40 & 99 \\ \hline
3b & \begin{tabular}[c]{@{}l@{}}To Kill \\ A Mbird\end{tabular} & 12 & 35 & 81 \\ \hline
3c & \begin{tabular}[c]{@{}l@{}}To Kill \\ A Mbird\end{tabular} & 14 & 41 & 77 \\ \hline \hline
4a & \begin{tabular}[c]{@{}l@{}}Heart of \\ Darkness\end{tabular} & 13 & 44 & 109 \\ \hline
4b & \begin{tabular}[c]{@{}l@{}}Scarlett Ltr\end{tabular} & 13 & 45 & 63 \\ \hline
4c & \begin{tabular}[c]{@{}l@{}}A Mdsmmr \\ Night's  Drm\end{tabular} & 11 & 39 & 119 \\ \hline \hline
5a & \begin{tabular}[c]{@{}l@{}}To Kill \\ A Mbird\end{tabular} & 15 & 29 & 106 \\ \hline
5b & \begin{tabular}[c]{@{}l@{}}Smthg Wckd \\ This Wy Cms\end{tabular} & 16 & 35 & 105 \\ \hline
5c & \begin{tabular}[c]{@{}l@{}}The Little \\ Prince\end{tabular} & 15 & 38 & 111 \\ \hline \hline
6a & \begin{tabular}[c]{@{}l@{}}The Immortal \\ Lf of H. Lcks\end{tabular} & 6 & 35 & 124 \\ \hline
6b & The Crucible & 6 & 38 & 79 \\ \hline
6c & Into the Wild & 7 & 33 & 293 \\ \hline \hline
7a & \begin{tabular}[c]{@{}l@{}}Of Mice \\and Men\end{tabular} & 25 & 25 & 192 \\ \hline
7b & \begin{tabular}[c]{@{}l@{}} Fahr. \\ 451\end{tabular} & 11 & 34 & 332 \\ \hline
7c & MLK Jr. & 13 & 38 & 141 \\ \hline \hline
8b & \begin{tabular}[c]{@{}l@{}}The Yellow \\ Wllppr\end{tabular} & 11 & 39 & 127 \\ \hline
8c & Antigone & 13 & 28 & 248 \\ \hline \hline
9a & \begin{tabular}[c]{@{}l@{}}Salem Witch \\ Trials\end{tabular} & 15 & 35 & 275 \\ \hline
9b & The Crucible & 14 & 25 & 264 \\ \hline
9c & \begin{tabular}[c]{@{}l@{}}The Prks of \\ Bng Wllflwer\end{tabular} & 15 & 38 & 267 \\ \hline \hline
10a & \begin{tabular}[c]{@{}l@{}}Bleachers
\end{tabular} & 23 & 38 & 165 \\ \hline
10b & JFK Speech & 25 & 33 & 148 \\ \hline
10c & \begin{tabular}[c]{@{}l@{}}To Kill \\ A Mbird\end{tabular} & 25 & 33 & 188 \\ \hline
\end{tabular}
\caption{Overview of discussions by teacher. ``Disc'' = Discussion, numbers correspond to teacher in Table \ref{tab:Classrooms}; ``Text''=the titles of the texts under discussion; ``Stu.''= amount of students present in discussion; ``Min.'' = length in minutes; ``\#trns''= number of turns per discussion}
\label{tab:discussions}
\end{table}
The Discussion Tracker corpus is based on audio-recorded multi-party spoken discussions in 10 different high school English teachers' classrooms across three different school districts (suburban, urban, and rural) (see Table \ref{tab:Classrooms}). Between October 2018 - March 2019 we recorded a total of three literature discussions per classroom. Omitting one discussion that was off-topic, the corpus we are releasing contains 29 transcriptions of high school literature discussions based on 985 minutes of audio (see Table \ref{tab:discussions}).

Across the ten classrooms, the mean number of discussion participants was 15 students (SD 6), ranging from 6 to 29.
In accordance with educational research examining racial and gender inequities in instructional practice \cite{kelly2008race,sherry2014indirect}, we collected metadata for race and gender demographics. Based on notes taken during data collection, we estimated that on average student discussants were 50\% male and 50\% female (SD 0.18), 77\% white (SD 0.22) and 23\% nonwhite (SD 0.22). Of the nonwhite students most appeared to be Indian (58\%), 18\% appeared Black, 15\% East Asian, and the remaining 9\% appeared Latinx or other.

In order to maintain the authenticity of the instructional environment, teachers were free to facilitate their discussions according to their pedagogical expertise, so long as they arranged students to face the microphone (which was placed in the center of the classroom) and attempted to ensure that students sat in the same seats for each discussion. Speaker demographics varied slightly across discussions within the same classroom due to student absences and discussion styles (e.g., holding a small group discussion in which only a portion of the students were expected to speak). Descriptions of the classrooms (Table \ref{tab:Classrooms}) reflect the maximum number of possible discussants when all students were present. Any variation between these classroom descriptions and the descriptions of the discussions within each classroom (Table \ref{tab:discussions})  can be explained either by student absences or discussion style.

In most discussions, the entire class participated, although five discussions were set up as small groups and thus limited to a subset of the students present (see 6a, 6b, 6c, 7b, 7c in Table \ref{tab:discussions}). Discussions were recorded by a member of the research team using a Zoom H6 six-track portable audiorecorder placed in the center of the classroom with student discussants arranged in circles around the device. In addition to creating a map that linked numerical IDs to the locations of each discussant, a researcher also kept handwritten notes to identify speakers. Transcriptions of the audio data were outsourced to a professional service (Rev.com) and were reviewed by research assistants for accuracy. In a test of four transcripts, an average of 4\% words per transcript were incorrectly transcribed and required revision. In addition to aligning speaker IDs to transcribed talk, research assistants also transferred data to an excel document formatted specifically for annotation.

\section{Data Annotation}
The Discussion Tracker corpus includes annotations for three dimensions of student talk that researchers in classroom discourse have associated with positive educational outcomes \cite{howe2019teacher,applebee2003,chisholm2011,soter2008,sohmer2009guided,juzwik2013,nystrand1991} (see Table \ref{fig:transcript}); similar dimensions have also been annotated by NLP researchers for other types of data. Using a classroom discussion annotation scheme optimized for NLP development \cite{lugini2018annotating}, we annotated student talk for {\it argument moves} (claims, evidence and explanations) and {\it specificity} (low, medium and high). In addition, we developed an annotation scheme for {\it collaboration} that  synthesized findings in both classroom discourse research \cite{engle2002guiding,keefer2000} as well as the computer-supported collaborative learning (CSCL) literature \cite{samei,zhangcscl}.

Prior to annotation, speakers were identified using the handwritten notes taken during data collection. Cases in which speaker IDs were difficult to determine were labeled either as `St?' if the speaker was likely a student or `?' if it was unclear whether the speaker was a student or teacher. Student talk was segmented into both turns at talk and argument discourse units. Talk at the turn level was annotated for collaboration, and talk at the argument discourse unit was annotated for argument move values (claim, evidence, explanation) and specificity level (low, medium, high). The coding instructions for all  annotated dimensions are briefly reviewed below. More details can be found in Lugini et al. \shortcite{lugini2018annotating}, and a sample coded transcript with all annotation manuals can be found in the link provided in section \ref{sec:public}

Similar to argument mining systems like Nguyen \shortcite{nguyen2018argument}, our pipeline for annotating collaborative argumentation involved several steps. (1) While examining only student talk, we flagged turns  that contained no substantive argumentation and were thereby deemed \textit{non-argumentative}. Turns that included both non-argumentative and argumentative phrases were considered argumentative. In addition to talk that was inaudible or off-topic (``I have to go to the bathroom''), non-argumentative talk also included meta-discourse talk (``okay you can take a turn,''), discussion prompts (``Describe the imagery in the poem'') and brief agreements (``yeah'').
(2) Argumentative turns  were annotated for one of four collaboration dimensions, \textit{New Ideas}, \textit{Agreements}, \textit{Extensions}, and \textit{Probes/Challenges}. (3) Turns containing collaborative argumentation were further segmented into argument discourse units, which were, (4) labeled for argument move type (\textit{claims}, \textit{evidence}, or \textit{explanations}), and (5) annotated for specificity.

\subsection{Collaboration}

Each argumentative turn  was annotated for one of four possible collaborative relationships with prior turns at talk. (1) \textit{New Ideas}: turns that did not reference ideas in prior turns at talk. (2) \textit{Agreements}: turns that repeated verbatim or almost verbatim the idea in a prior turn. (3) \textit{Extensions}: turns that built on prior ideas, either the speaker's own or another student's. (4) \textit{Probes/Challenges}:  turns that questioned or disagreed with a prior idea. Also included in collaboration annotations was a reference to the prior turn with which the current turn was in a collaborative relationship (turn reference number). After coding approximately one-third of the transcripts, analyses revealed that  30\% of turns had a collaborative relationship with one of the previous two turns, and 95\% had a collaborative relationship with turns within the previous four turns. Thus a limit for turn references was set at no more than four annotated previous turns unless the speaker's reference to an earlier turn was explicit (e.g., a speaker said, ``Going back to John's comment about authority" when John had commented 10 turns previously).

\subsection{Segmentation}
Prior to annotating for argumentation and specificity, argumentative turns at talk were segmented further into argument discourse units (ADUs). Similar to Ghosh et al. \shortcite{ghosh2014analyzing}, who segmented ADUs into either ``stance'' vs ``rationale,'' annotators were instructed to divide turns at talk into interpretive vs. factual/ information-based segments of talk. For example, as seen in Table \ref{fig:transcript}, Speaker 1's turn at talk was first segmented when they offer examples ``throughout history'' of their claim. The turn was segmented a second time when the speaker offered an interpretation of how the examples related to their claim. Annotators were not expected to get so fine-grained as Stab and Gurevych \shortcite{stab2017parsing}, whose ADU segmentation accounted for sub-claims and multiple premises. Thus, annotators were instructed not to segment turns into multiple claims or multiple units of evidence.

\subsection{Argumentation}

As in Lugini et al.~\shortcite{lugini2018annotating}, annotations for argumentation were derived from classical models of argument structure \cite{toulmin1958}, and were simplified to include three labels: \textit{claim} (an arguable statement that presents an interpretation of a text or topic), \textit{evidence} (facts or information to support a claim) and \textit{explanation} (reasoning or justification for why the given evidence supports the claim).

\subsection{Specificity}
Our annotation scheme for specificity differs from Carlile et al. \shortcite{Carlile:18} in that our  labels were not contingent on argumentation, but rather stood independent of argumentation labels. Like Lugini et al. \shortcite{lugini2018annotating}, we defined specificity as the existence of particularity, detail, content-language (use of disciplinary terminology like ``symbolism'' or ``irony''), and/or a chain of reasons. Argument units that included two or more of these four characteristics above were annotated as {\it high specificity}; argument units containing one of the characteristics were annotated as {\it medium specificity}; and argument units containing none of the above characteristics were annotated as {\it low specificity}.
\begin{table*}[]
\begin{tabular}{|c|c|p{2.5in}|c|c|c|c|}
\hline
\textbf{Turn}      & \textbf{Speaker}       & \textbf{Talk}                                                                                & \textbf{\begin{tabular}[c]{@{}c@{}}Collab-\\ oration\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Reference\\ Turn\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Argument-\\ ation\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Spec-\\ ificity\end{tabular}} \\ \hline
\multirow{3}{*}{1} & \multirow{3}{*}{St 1}  & My interpretation of it is that, without a middle ground, you are left with two very extreme points. Whether or not the middle ground directly centered, we have a range. We have a spectrum.[...]                       & \multirow{3}{*}{New}                                               & \multirow{3}{*}{}  & Claim                                                              & Medium                                                           \\ \cline{3-3} \cline{6-7}
                   &                        & Throughout history, whether you go back to ancient Europe, and you look at tyrannies and dictatorships, not even ancient Europe. If you go back to the Holocaust and what Hitler was doing over in Germany [...] if you go back to Communism, as well [...]                                                                                &                                                                    &                    & Evidence                                                           & Medium                                                           \\ \cline{3-3} \cline{6-7}
                   &                        & Those are two extremes, and neither of them ended well, and just anarchy there. There is no order there, there is no civilized kind of society to base anything around. I think the middle ground is necessary just to create some kind of spectrum that we can go off of.                                                                                &                                                                    &                    & Explanation                                                        & Medium                                                           \\ \hline
2                  & St 9                   & I acknowledge your point, but there wasn't nobody going against anything until this happened, until this event occurred.                                                                                & Challenge                                                          & 1                  & Claim                                                              & Low                                                              \\ \hline
3                  & St 1                   & Does that make the way they were living right, thought?                                                                                & Challenge                                                          & 2                  & Claim                                                              & Low                                                              \\ \hline
4                  & St 9                   & If they were happy, I believe they were perfectly fine.                                                                                & New                                                                &                    & Claim                                                              & Low                                                              \\ \hline
5                  & St 17                  & My assessment of the topic at hand is, there needs to be a balance between state rights and user rights. [xx] slide, and to what extent was it off balance.                                                                                & Extension                                                          & 1                  & Claim                                                              & Medium                                                           \\ \hline
6                  & St 14                  & I concur with both St 19 and 17's statements. I also think that if we have society in which people are afraid to go against the core, then the rights of them are restricted. They're afraid that if they step outside the lines, then it won't end good for them, so everybody's afraid [...] they won't be accepted.                                                                                & Extension                                                          & 5                  & Claim                                                              & Medium                                                           \\ \hline
\multirow{2}{*}{7} & \multirow{2}{*}{St 18} & I concur with St 14.                                                                                & \multirow{2}{*}{Extension}                                         & \multirow{2}{*}{6} & Claim                                                              & Low                                                              \\ \cline{3-3} \cline{6-7}
                   &                        & Because back in the day when we have the Civil War going on, people were on different sides. People were afraid to come and say, ``oh, I'm in between,'' because then they would be afraid that they'd be treated just like African Americans. As St.,9 said, it got to the point where they were on two different sides and they couldn't decide on something, so they said, ``hey, let's fight this out, and whoever wins basically decides what action happens.'' &                                                                    &                    & Evidence                                                           & High                                                             \\ \hline
\end{tabular}
\caption{Sample transcript from discussion 6b on the play ``The Crucible''.}
\label{fig:transcript}
\end{table*}

\subsection{Reliability Analyses}
Reliability analyses of segmentation and annotations yielded high inter-rater agreement. We calculated reliability measures for each of the following categories:
1) selecting collaborative argumentative turns at talk, 2) annotations for collaboration labels, 3) segmentation of turns into argument discourse units, 4) annotations for each argument move label and 5) specificity labels.

Recall that annotators were instructed to consider turns at talk as \textit{non-argumentative} if they did not include argumentation.
Agreement between annotators for argumentative turns versus not was based on a sample with highly skewed class distributions yielding low Kappa (0) but high raw percentage of agreement (92\%).

Basing our argumentative turns segmentation metric on Habernal and Gurevych \shortcite{Habernal:17}, we computed agreement on 54 transcripts from a classroom corpus we collected before Discussion Tracker by comparing the segmentation of two trained annotators. The average alpha was 0.96, min was 0.72, max was 1, and standard deviation was 0.048. All data in the Discussion Tracker corpus was segmented by the same trained annotator.

A portion of the transcripts were double-annotated and yielded substantial agreement for collaboration (Cohen's Kappa, 0.74) and specificity (Quadratic Weighted Kappa, .70), and near-perfect agreement for argumentation (Cohen's Kappa, .89). Because our specificity annotation was based on an ordered scale (low, medium, high), we employed an inter-rater agreement measure (Quadratic weighted Kappa) that could account for degrees of disagreement. Because argument move annotations were not based on an ordered scale we simply used Cohen's Kappa in which disagreements were not specially weighted.

After achieving satisfactory agreement in double-coding, the remaining transcripts were single-annotated for collaboration, argumentation, and specificity.
Differences between annotators were resolved through deliberation  to construct gold standard annotations for public release.

\subsection{Corpus Statistics}

Of Discussion Tracker's 3261 student turns, 2128 were considered \textit{argumentative} and were annotated for collaborative argumentation. As seen in Table \ref{tab:stats}, the large majority of annotated turns were labeled as either \textit{New Ideas} (37.69\%) or \textit{Extensions} (47.70\%), with \textit{Challenges/Probes} and \textit{Agreements} making up only a narrow portion of the corpus.

\begin{table}[!t]
\begin{tabular}{|l|l|r|r|}
\hline
 & Annotation & Total Count & Percentage \\ \hline
\textbf{\begin{tabular}[c]{@{}c@{}}Collab-\\ oration\end{tabular}} & New & 802 & 37.69\% \\ \cline{2-4}
 & Agree & 37 & 1.74\% \\ \cline{2-4}
 & Extensions & 1015 & 47.70\% \\ \cline{2-4}
 & Chall/ Probes & 274 & 12.88\% \\ \cline{2-4}
 & \textbf{Total} & \textbf{2128} & \textbf{100.00\%} \\ \hline
\textbf{\begin{tabular}[c]{@{}l@{}}Argum-\\ entation\end{tabular}} & Claims & 2047 & 65.30\% \\ \cline{2-4}
 & Evidence & 762 & 24.31\% \\ \cline{2-4}
 & Explanations & 326 & 10.40\% \\ \cline{2-4}
 & \textbf{Total} & \textbf{3135} & \textbf{100.00\%} \\ \hline
\textbf{Specificity} & Low & 1189 & 37.93\% \\ \cline{2-4}
 & Medium & 1071 & 34.16\% \\ \cline{2-4}
 & High & 875 & 27.91\% \\ \cline{2-4}
 & \textbf{Total} & \textbf{3135} & \textbf{100.00\%} \\ \hline
\end{tabular}
\caption{Descriptive statistics of corpus annotations.}
\label{tab:stats}
\end{table}

The argumentative turns at talk were further segmented into 3135 argumentative discourse units to be annotated for argument move type and specificity. The corpus is made up of mostly claims (65.30\%),  less than half of which were supported with evidence (24.31\%), and still less were elaborated with explanations (10.40\%). Specificity of argument moves was more evenly distributed with 37.93\% annotated as low, 34.16\% as medium, and 27.91\% high.

\section{Public Release}
\label{sec:public}
The Discussion Tracker Corpus will be freely available for research purposes, with the release coordinated with the publication of this paper. The release will include 29 separate .xlsx documents segmented for both turns at talk and argument discourse units. Each document will contain the discussion in full, including teacher talk and non-annotated student talk for context. Additionally, transcripts will contain unique ID numbers (e.g., T127.1.Heartdark) for each turn at talk indicating the de-identified teacher (e.g., T127),
the discussion (e.g., .1-- referring to the first discussion from that teacher's classroom), the text (e.g., ``Heartdark'' for Joseph Conrad's \textit{Heart of Darkness} and the turn number in the discussion (the final number in the ID). Directly to the right of each turn at talk will be columns containing collaboration annotations and their corresponding turn reference numbers. Talk segmented at the ADU level will include annotations for each argument move type and specificity.

The corpus is available at \url{https://discussiontracker.cs.pitt.edu}. Included in the corpus are 29 transcripts with complete annotations, a metadata file containing information for the speaker demographics and grade levels, and the coding manuals for creating the annotations.

The code for all classification experiments from Section~\ref{sec:case} is also available via the Discussion Tracker website to provide a performance benchmark for future  research that uses this corpus.

\section{Case Studies Using the Corpus}
\label{sec:case}
Our corpus provides NLP researchers  the opportunity of several uses.
First, each of the three annotated dimensions of collaborative argumentation can be used individually to train a classifier for automated prediction.
Second, we believe that one of the most interesting characteristics of the Discussion Tracker corpus is the fact that it provides annotations for multiple dimensions of collaborative argumentation simultaneously.
It is possible, then, to analyze if and how these dimensions are related. If that is the case, it may be possible to develop more robust and accurate models for automated classification of such dimensions.
Carlile et al. \shortcite{Carlile:18} annotated argumentative discourse units for argument component and specificity (among other things) and were able to make use of these two aspects simultaneously to predict argument persuasiveness in written essays.
Similarly, we showed in our previous study that specificity can be used to improve the performance of argument component classifiers \cite{Lugini:18b}.
We found that models trained through multi-task learning where the primary task consists of argument component classification and the secondary task consists of specificity classification almost always outperform models that only perform argument component classification.
However, the corpus used in our previous study is not publicly available and therefore our previous results are not reproducible by other members of the research community.

To provide reproducible performance baselines to facilitate future classifier evaluations using the Discussion Tracker corpus, we present our experiments in learning models for individual classification tasks and then jointly learning multiple classifiers.
The performance of each model was evaluated using the same ten-fold cross-validation: each fold consists of 26 transcripts as training set and 3 as test set (except for one fold where 27 transcripts are used for training and 2 for testing). We report accuracy, Cohen Kappa (quadratic-weighted for specificity and unweighted for the remaining tasks) and macro f-score as evaluation metrics.
The particular folds used for the cross-validation experiments presented in this paper will  be made  available in the corpus release in the form of a json file containing a list of all training and test transcripts for each fold.

\subsection{Learning through Individual Annotations}
\subsubsection{Argumentation}
\label{sec:case_study_argumentation}
As a baseline for evaluating the performance of argument component classification on the Discussion Tracker corpus, we tested our previously proposed model \cite{Lugini:18b}, which showed significantly higher performance on a previously examined set of classroom discussions compared to argument mining models developed for other types of corpora.
It consists of a hybrid model which combines embeddings generated through a neural network with a set of handcrafted features.

The handcrafted features consist primarily of two sets: Speciteller feature set, derived from prior work on specificity \cite{Li:15}, and online dialogue feature set, derived from prior work on argument mining in online dialogues \cite{Swanson:15}.
The Speciteller feature set contains the following features, extracted independently for each argument move: average of 100-dimensional word vectors \cite{Turian:10} for words in the argument move, number of connectives, number of words, number of symbols, number of numbers, number of capital letters, number of stopwords normalized by argument move length, average characters per word, number of subjective and polar words (extracted using the MPQA \cite{Wilson:09} and the General Inquirer \cite{Stone:63} dictionaries), average word familiarity (extracted using MRC Psycholinguistic Database \cite{Wilson:88}), and inverse document frequency statistics (maximum, minimum).
The online dialogue feature sets includes: number of pronouns, number of occurrences of words of different lengths, descriptive word-level statistics, term frequency - inverse document frequency of unigrams and bigrams (with frequency greater than 5), and part of speech tag features (unigrams, bigrams and trigrams).\footnote{One feature set from our prior work is not included in this study, namely wLDA \cite{Nguyen:16}, since it greatly increased model complexity leading to overfit.}

The neural network model is composed of a series of 3 convolutional/max-pooling layers, in which the convolutional layer consists of 16 filters of size 7.
Each word in an argument move is first processed through an embedding module which uses pretrained GloVe embeddings \cite{Pennington:14} of dimensionality 50.
The argument move is then processed through the neural network to generate a fixed-size embedding.
\begin{figure}[!thb]
\begin{center}
  \includegraphics[scale=0.37]{neural_network_models.png}
  \end{center}
  \caption{Neural network models used in this study: individual models (a); joint multi-task model (b).}
\label{fig:neural_network_models}
\end{figure}
The handcrafted features are concatenated to the neural network embeddings, and the final feature vector is  input to a softmax classifier to output the final prediction (see Figure \ref{fig:neural_network_models}(a)).

In our prior work \cite{Lugini:18b} on argument component classification for discussions, we used oversampling to alleviate the class imbalance present in argumentation labels (which is also present in the Discussion Tracker corpus). However, since our Discussion Tracker experiments also include 3 task multi-task learning, oversampling with respect to argumentation labels might have negative impact on other tasks.
To address class imbalance, instead of using oversampling, we manually set class weights to impact the loss function, trying to increase the importance of the less frequent labels (evidence, explanation).
The class weights were set by roughly approximating the frequency of labels in the corpus used in our prior work \cite{Lugini:18b}:
the previous corpus contained almost twice as many claims than evidence, and almost twice as many explanations as evidence, prompting us to use the weights \textit{(claims: 1, evidence: 2, explanations: 4)}.

Table \ref{tab:results} shows the cross-validation results.
As we can see from Row 1, the difference between accuracy and f-score indicates that the model performs differently for the three argumentation labels.
The f-score for claims, evidence, and explanations are respectively 0.776, 0.565, and 0.164.
While specifying class weights at training time helped, this shows that there is ample room for improvement.

\subsubsection{Specificity}
As we showed in our previous study \cite{Lugini:17}, using the off-the-shelf Speciteller tool~\cite{Li:15} for predicting sentence specificity performed poorly when applied to text-based classroom discussions. We were able to significantly improve classification results by proposing features and models explicitly developed for classroom discussions.
Like our previous work on argumentation, however, the corpus is not publicly available.
To provide a  baseline for the Discussion Tracker corpus, we evaluated specificity prediction performance using the same model as described in Section \ref{sec:case_study_argumentation}
By using this model we achieved very similar performance to the model we proposed in Lugini and Litman \shortcite{Lugini:17} at a fraction of the computational cost. The main difference between the two models is the use of a convolutional neural network instead of a recurrent neural network.

As we can see in Row 3 of Table \ref{tab:results}, the small variation across the three performance metrics indicates consistent performance for all three specificity labels.
Additionally, kappa and f-score show that the specificity model is much more accurate than the one for argumentation.
\begin{table*}[!htb]
\centering
\begin{tabular}{cllccc}
\textbf{Row} & \textbf{Experiment} &Model & \textbf{Accuracy} & \textbf{Kappa} & \textbf{Macro F} \\ \hline
1 & Argument Move &Individual& 0.669 & 0.343 & 0.502 \\
2 & Argument Move &Joint & 0.673 & 0.365 & 0.516 \\ \hline
3 & Specificity &Individual& 0.703 & 0.750 & 0.695 \\
4 & Specificity &Joint & 0.706 & 0.751 & 0.698 \\ \hline
\end{tabular}
\caption{Neural classification results (both individual and joint models) for argument discourse unit prediction tasks.}
\label{tab:results}
\end{table*}

\subsubsection{Collaboration}
Since collaboration was not annotated in our previously used corpus of classroom discussions (unlike argumentation and specificity),
we do not have a prior  prediction model to draw upon as a baseline. We instead use
Naive Bayes to model collaboration, both for model simplicity and because it is a typical baseline model for NLP tasks.
The feature vector we use is bag of words (BOW) on each turn, tokenized using NLTK's word tokenizer. We filter out  stop words and use tokens that occur once to approximate unknown words.

We experimented with using TF-IDF weighting on  bag of words
and using different Naive Bayes variants.  Results from the best performing configurations  are reported in Table \ref{tab:collabResults}.

As shown in Row 1, we found that the best configuration  for predicting the four collaboration labels  was Multinomial Naive Bayes with TF-IDF features. Note that  accuracy is much higher than both kappa and macro f-score, likely reflecting the skewed distribution among the four classes. These baseline results show the difficulty in distinguishing between all of the collaboration annotations in the Discussion Tracker corpus.
We also explored a simpler binary version of our classification task (Row 2), which  reduced the class skew while still making a pedagogically useful distinction.
In particular, during discussions with teachers where we visualized the collaboration annotations in the corpus that came from their particular classrooms, we found that teachers were very curious about whether students were introducing new information into the discussion or building off of what was previously said. Therefore we  experimented on how well a classifier could distinguish student turns labeled `New' from the other collaboration annotations. We found that Gaussian Naive Bayes without TF-IDF features performed the best. While the results improved compared to  predicting the original 4 classes, there is still  room for improvement. In sum, determining the  collaboration labels for  student turns  is difficult for our simple  Naive Bayes with BOW baseline method.

Finally, although the collaboration experiment was performed using only the manually-annotated argumentative subset of student turns (as is typical of system component evaluations), an end-to-end system would in addition need to first (or jointly) automatically separate the argumentative and non-argumentative turns, before classifying the collaboration labels for the argumentative turns. Using the same approach as for predicting collaboration labels, we find that a Gaussian Naive Bayes model with TF-IDF performs the best at this task (Row 3 in Table \ref{tab:collabResults}).

\begin{table*}[bht]
    \centering
    \begin{tabular}{l l l r r r}
        \textbf{Row} & \textbf{Experiment} & \textbf{Model} & \textbf{Accuracy} & \textbf{Kappa} & \textbf{Macro F} \\ \hline
        1& Collaboration (all 4 labels) & Multinomial W/TF-IDF & 0.504 & 0.086 & 0.254 \\
        2& Collaboration (`New' vs Other)  & Gaussian w/ BOW & 0.623 & 0.217 & 0.604 \\ \hline
        3& Argumentative vs Non-Argumentative  & Gaussian W/TF-IDF & 0.785 &0.513 & 0.756 \\ \hline

    \end{tabular}
    \caption{Naive Bayes classification results for turn-level prediction tasks.}
    \label{tab:collabResults}
\end{table*}

\subsection{Learning through Multiple Annotations}
\label{sec:results}
In this section, we describe an experiment that extends our previous multi-task learning study \cite{Lugini:18b}, by using three rather than two tasks for the learning, by using the new Discussion Tracker corpus,
and by providing annotation and associated benchmark results that can be replicated and extended by others in the future. More specifically, we test the following research hypotheses:
by modeling argument component, specificity, and collaboration information simultaneously, we can develop a single model that outperforms individual classifiers for (H1) argument component, and (H2) specificity.
\begin{figure}[!bt]
\begin{center}
  \includegraphics[scale=0.4]{Collaboration_argumentation_distributions.png}
  \end{center}
  \caption{Distribution of collaboration labels for different argumentation categories}
\label{fig:collaboration_label_distribution}
\end{figure}
These hypotheses are motivated by our observation of differences between collaboration label distributions across argumentative moves.
However, given the different unit of analysis for the annotation of collaboration (turn) versus argumentation and specificity (argument discourse unit), for the multi-task learning setting the collaboration annotations have been converted to BIO
format in order to have one annotation per argument move\footnote{If a turn labeled ``\textit{New}'' for collaboration is segmented into two argument moves, the collaboration label is converted into ``\textit{B-New}'' for the first argument move and ``\textit{I-new}'' for the second.}.
For example, we observed that the most frequent collaboration labels for claims are \textit{B-extension $(43.9\%)$, B-new $(37.0\%)$ and B-challenge $(11.1\%)$}.
Looking at evidence, the most frequent collaboration labels are \textit{I-extension $(47.8\%)$, I-new $(20.8\%)$ and B-extension $(14.7\%)$}.
Lastly, the most frequent collaboration codes for explanations are \textit{I-extension $(63.9\%)$, I-new $(23.2\%)$ and I-challenge $(12.2\%)$}.
The complete label distributions are shown in Figure \ref{fig:collaboration_label_distribution}.

With the goal of exploiting the potential relationships between argumentation, specificity, and collaboration, we developed a single joint model trained through multi-task learning.
This model consists of the same convolutional neural network (see Section \ref{sec:case_study_argumentation}) along with the same handcrafted feature set, with the difference that the final feature vector is used as input to three softmax classifiers simultaneously: one for argument component, one for specificity and one for collaboration.
In this setting the representation for an argument move is entirely shared between the three tasks (see Figure \ref{fig:neural_network_models}(b)).
The final loss of the model is the sum of the individual cross-entropy losses: we chose an unweighted sum so that we can understand potential relationships between the three prediction tasks; if the goal is that of maximizing performance, one of the tasks can be favored by increasing its weight in the loss function.

Table \ref{tab:results} shows the results of our experiments.
Rows 1 and 2 relate to our hypothesis H1, and we can see an improvement in accuracy, kappa and f-score. The performance improvements achieved through the joint model, though, are not yet statistically significant.
Although differences exist across argumentation labels for different collaborative moves, our joint model is not able to optimally capture them. This may be due to the low performance of the collaboration classifier: if the collaboration model cannot reliably capture collaboration information, it cannot properly inform the argumentation classifier.
We believe that increasing the performance of the individual classifiers and using learned weights in the loss function  will result in a more effective joint model. Rows 3 and 4 relate to our hypothesis H2.
Like for argumentation, the results on specificity show the joint model outperforming the single-task one in all metrics, though the difference was not statistically significant.\footnote{Recall that while collaboration was  annotated at the turn level, the joint model uses the BIO converted (ADU) representation. We thus do not investigate whether the joint ADU model improves the individual turn-level collaboration prediction.}

Although the current results show a limited gain in performance of the joint model over the individual ones, the Discussion Tracker corpus allows the research community to further analyze inter-dependencies between argumentation, specificity and collaboration, and develop more effective models to take advantage of these dependencies.

\subsection{Other Potential Corpus Uses}

Going beyond classification, our corpus can also be used in conjunction with other publicly available corpora.
Daxenberger et al. \shortcite{Daxenberger:17} for example performed a qualitative analysis to understand the difference in conceptualization of claims across multiple datasets. None of the datasets analyzed, however, includes transcripts of spoken dialogues.
The Discussion Tracker corpus can be used in a similar way, for example, to study the different conceptualization of argument components between spoken multi-party discussions and online multi-party dialogues or written essays.
Additionally, the corpus could also support educational research, which has taken interest in classroom discourse since the 1970's \cite{howe2013classroom,mercer2014study}. Howe et al.'s \shortcite{howe2019teacher} recent study of 72 elementary classroom environments established statistically significant relationships between positive learning outcomes and student talk dimensions similar to the annotations we include in our corpus: participation (how much and how many students speak), elaboration (similar to our extensions), and querying (similar to our challenge/probe category).
The corpus metadata can also be used to support the investigation of issues of educational equity (e.g., gender and racial) in collaborative argumentation research \cite{godley2019,howe1997gender,kelly2008race}.

\section{Future Corpus Extensions}
Over the next three years of the Discussion Tracker project, we will be collecting and annotating new classroom data that will more than triple
the size of this first release of our corpus. In these future corpus extensions, we will also include new information in our transcripts, namely time stamps and  phenomena specific to spoken data (including filled pauses like ``uh''). This will allow for more investigation on the similarities and differences between spoken and written synchronous collaborative argumentation.\footnote{Although the inclusion of audio files would contribute greatly to these endeavors and might be possible in future releases, standard IRB regulations for research on minors in authentic learning environments would require costly de-identification.}

\section{Summary}
By releasing the Discussion Tracker corpus, we hope to contribute to collaborative argument mining research concerned with multi-party synchronous argumentative discourse collected in authentic environments. The 29 transcriptions included in the Discussion Tracker corpus contain multi-party argumentation along with annotations for collaboration at the turn level and annotations for argument move type and specificity at the argument discourse unit level. We created performance baselines for a variety of individual classification tasks, and demonstrated the potential use of the simultaneous annotations by exploring multi-task learning as method for improving  baseline performance. We believe that the Discussion Tracker corpus will be a useful resource for others, not only because it provides challenging multi-party collaborative argumentation data for future NLP research, but also because it provides multiple simultaneous annotations that can allow for a wider variety of learning approaches.

\section{Acknowledgements}
This work was supported by the National Science Foundation (EAGER 1842334 and 1917673)
and in part by the University of Pittsburgh Center for Research Computing through the resources provided.

\section{Bibliographical References}
\label{main:ref}

\bibliographystyle{lrec}


text:
\section{Introduction}

The origin of superconductivity in copper oxides is still not understood after more than three decades of investigation. Adding to the challenge is that the cuprates are subject to a host of spin, charge, and nematic instabilities whose spectroscopic signatures are difficult to identify and disentangle from those of superconductivity itself \cite{keimer2015quantum}. Establishing what other order parameters may be present, and whether they are related to important aspects of cuprate phenomenology (e.g., the pseudogap) is therefore a major focus of current research.

Many recent studies have focused on instabilities that are periodic or quasiperiodic in real space, such as spin density waves, charge density waves, or more exotic intertwined states such as the proposed pair density wave (PDW) \cite{berg2009a,berg2009b}. However, an important sub-category of candidate orders are those that occur at $Q \sim 0$, i.e., phenomena that are uniform in space and have signatures in the average spectroscopic properties of the material. Many experimental studies have reported evidence for such order interpreted in terms of broken time reversal symmetry \cite{kaminski2002spontaneous,he2011single,xia2008polar,fauque2006magnetic,mook2008observation,
baledent2011evidence,de2012evidence,mangin2014characterization,mangin2015intra,mangin2017b,
tang2018orientation,li2008unusual,li2011magnetic,croft2017no,zhao2017global}, inversion symmetry \cite{zhao2017global,lim2020observation}, rotational symmetry \cite{hinkov2008electronic,daou2010broken,sato2017thermodynamic,achkar2016nematicity,
lawler2010intra,mesaros2011topological,fujita2014simultaneous,mukhopadhyay2019evidence,
ishida2020divergent}, chiral symmetry \cite{lim2020observation}, mirror symmetry\cite{delatorre2021}, as well as anapole symmetry that may be related to proposals of ``loop current" order \cite{kaminski2002spontaneous,fauque2006magnetic,mook2008observation,baledent2011evidence,
de2012evidence,mangin2014characterization,mangin2015intra,mangin2017b,tang2018orientation,
li2008unusual,li2011magnetic,croft2017no,scagnoli2011observation}.
However, some of these experiments still remain controversial and new experimental schemes capable of detecting exotic valence band order near $Q \sim 0$ are needed.

Resonant soft x-ray scattering (RSXS) is a momentum-resolved spectroscopic technique that is capable of coupling to a variety of ordered valence band states, in principle those comprising higher order electric and magnetic multipoles \cite{fink2013resonant,lovesey2005}. The technique is sensitive to the symmetry of valence band order through the structure of the scattering tensor that arises when the incident photon energy is tuned to atomic resonances such as the Cu $L_3$ edge. Signatures of $Q \sim 0$ order appear in RSXS in two ways, either through the resonance behavior of Bragg reflections, which represent the behavior of phenomena with the same periodicity as the structural lattice, or through the average optical properties of the material detected in specular reflectivity from the material surface.

Bi$_{2.1}$Sr$_{1.9}$CaCu$_2$O$_{8+x}$ (Bi-2212), the material that we focus on in this study, has a base-centered orthorhombic average crystal structure \cite{di2007x}. Some Bragg reflections in this material---for example, the $(0,0,3)$ reflection---are extinguished by symmetry due to the equivalence of the Cu atoms in different layers. This extinction arises because of the approximate spherical symmetry of the Cu atomic density in the scattering regime dominated by Thomson scattering. However, near resonance where scattering is tensorial, this extinction is no longer guaranteed if local multipole moments are coherently arranged in different layers \cite{murakami1998}. Measuring the RSXS signal near forbidden $(0,0,L)$ reflections on resonance, where $L$ is odd, is therefore a highly sensitive way to potentially detect $Q \sim 0$ order, in principle even if it involves anapole or other exotic symmetries. Such an approach yields extremely high sensitivity, since it is effectively a difference measurement between two nearly equivalent sites, which as tensor objects are less likely to cancel as they do in simple Thomson scattering \cite{murakami1998}. Partial information about the symmetry of the scattering tensors can be attained by measuring the polarization dependence of the scattering.

\begin{figure*}
\includegraphics[width=0.8\linewidth]{fig1.png}
\caption{\label{fig1}Resonance profiles of the structural $(0,0,2)$ Bragg peak. (a) Temperature dependence of the $(0,0,2)$ intensity measured with with circularly polarized light at APS. The integrated intensity was obtained by fitting the  peak with a Lorentzian. The XAS spectrum (gray background) is shown for comparison. (b) Temperature dependence of the $(0,0,2)$ intensity using linearly polarized light at ALS. The integrated intensity was obtained from Voigt fits. The XAS (gray background) is, again, shown for reference. (c) Representative momentum scans with taken at $T=102$ K with circularly polarized light, with individual data points shown in black and the fit curves shown as colored lines. (d) Representative, individual momentum scans with taken at $T=30$ K with linearly polarized light using the same color scheme as in panel (c).
}
\end{figure*}

Here, we present RSXS measurements of the forbidden $(0,0,3)$ reflection on the Cu $L_3$ edge in Bi-2212 above and below $T^\ast$. Intriguing oscillations of the scattering intensity and resonance features of forbidden and specular reflections, that change with both temperature and polarization, are observed. These intriguing results should influence future thinking about what competing orders may be present in the cuprates.

\section{Experiment}

Single crystals of bilayer Bi-2212 were grown using floating zone methods described previously \cite{wen2008}.
An optimally doped crystal with $T_c=91$ K and $T^\ast\sim200$ K was chosen because $T_c$ and  $T^\ast$ are split and accessible with our experiment. The sample surface was prepared by cleaving in air using Scotch tape and immediately transferring the sample into the vacuum system.

RSXS measurements were performed near the Cu $L_{3}$ edge as in many past studies \cite{abbamonte2004crystallization, abbamonte2005spatially, damascelli2014charge, comin2015broken, ghiringhelli2012long, comin2016resonant, frano2020charge}. Experiments were carried out at the Advanced Light Source (ALS) beamline 4.0.2 and the Advanced Photon Source (APS) Sector 29.
The incident beam at APS has circular polarization while the beam at ALS has vertical polarization, with both setups employing polarization-integrating detectors. Comparison between the two facilities therefore provides partial, though not complete, information about the symmetry of the scattering tensor, which could be deduced through appropriate modeling that we will not undertake here.

Otherwise, the setups at the two beamlines were comparable. We performed Cu $L_3$ edge x-ray absorption (XAS) measurements in total electron yield mode at APS and total fluorescence yield at ALS (Fig. 1(a),(b)), and then used the measured edge position to correct for slight differences in the beam energy between the two facilities. The degree of second harmonic contamination near the Cu $L_3$ edge, i.e., 1870 eV photons in the nominally 935 eV beam, was also nearly the same---about 1-2\% of the fundamental intensity at both facilities. Both setups employed energy-integrated detectors, without an energy analyzer. The APS setup used a photodiode detector with an angular resolution of 3$^{\circ}$. The ALS setup used the same type of detector, but with an angular resolution of 0.15$^{\circ}$. In addition, a charge-coupled device (CCD) detector was available at ALS, as discussed further below. All measurements were carried out using a photodiode unless otherwise specified.

In this study we denote momenta using orthorhombic units, i.e., Miller indices $(H,K,L)$ designate a momentum transfer $Q=2 \pi (H/a,K/b,L/c)$, where $a = b = 5.4 \AA$ and $c=30.9 \AA$. The sample was oriented so that the $(1,1,0)$ direction, which is $45^{\circ}$ from the structural  supermodulation, was placed in the horizontal scattering plane. The intensity units were calibrated between the two facilities by matching the $(0,0,2)$ peak intensity on resonance at 24 K at APS and at 30 K at ALS, compensating for differences in flux and detector efficiency between the two facilities.

RSXS instruments are highly suitable for detecting $Q\sim0$ order, as they are capable of measuring both Bragg peaks and specular reflectivity. The former appear as sharp peaks at integer values of $(H,K,L)$ while the latter forms a continuous rod along the $(0,0,L)$ direction, as we discuss further below.

\section{Results}

\begin{figure*}
\includegraphics[width=1\linewidth]{fig2.png}
\caption{\label{fig2}RSXS momentum scans in the vicinity of the forbidden Bragg peak, $(0,0,3)$, for two different polarizations, a variety of temperatures, and a variety of different incident energies across the Cu $L_3$ edge. The black points are the experimental data and the colored lines represent fits using Eq. 1. (a)-(d) Measurements using circular polarization at APS. (e)-(g) Measurements using linear polarization at ALS.
}
\end{figure*}

We started by performing a routine measurement of the $(0,0,2)$ Bragg peak. This is a normal, allowed structural Bragg reflection expected to show no changes with temperature. Therefore, such measurements are useful for validating our subsequent measurements on forbidden Bragg peaks and specular reflections, and also serve as an opportunity to compare performances between the two facilities.

The results are summarized in Fig. \ref{fig1}. We performed momentum scans through the $(0,0,2)$ reflection at a variety of incident energies across the Cu $L_3$ edge at several temperatures, as shown in the line curves in Fig. \ref{fig1}(c)-(d). The results are summarized as resonance profiles in Figs. \ref{fig1}(a)-(b), which show the integrated intensity of the $(0,0,2)$ Bragg peak as a function of incident energy, compared to the XAS spectrum. The profile forms an asymmetric peak that remains strong off the resonance visible at both facilities, which arises from interference between resonant scattering and non-resonant Thomson scattering. This shape is typical of anomalous scattering from a structural reflection whose existence is not tied to subtle details in the electronic structure \cite{abbamonte2006charge}. Figs. \ref{fig1}(a)-(b) show that, due to the drifts in the beam current and other instabilities in the beamline, the overall intensity could drift by as much as 20\% during a weeks-long experimental run (see in particular Fig. \ref{fig1}(a)). Apart from overall intensity drifts, the results at the two facilities are highly consistent. This confirms that the configurations of the two beamlines are the same and that we can compare the results within an absolute intensity of 20\% (one caveat is that the harmonic contribution may be more unstable than the fundamental energy, as discussed further below). No temperature dependence of the $(0,0,2)$ reflection was observed when cooling through  $T^\ast$, confirming expectations that the overall crystal structure does not change at $T^\ast$.

Having validated the setups, we performed the same measurements near the $(0,0,3)$ point in momentum space. This reflection is forbidden by the crystal structure so, to a first approximation, no Bragg scattering is expected, apart from a weak second harmonic background from the $(0,0,6)$ reflection diffracting $\sim$1870 eV photons. Note that harmonic scattering can be identified because it is energy-independent, since there is no resonance in Bi-2212 at around 1870 eV. However, as discussed in the introduction, a true $(0,0,3)$ reflection may become weakly allowed if there is some symmetry breaking in the valence band that makes Cu atoms in two of the layers inequivalent.

Measurements near $(0,0,3)$ are summarized in Fig. \ref{fig2}, which shows individual momentum scans through the $(0,0,3)$ location at a variety of incident energies across the Cu $L_3$ edge at several temperatures between 24 K and 300 K. Identical scans were done at APS and ALS, i.e., with beams having circular and vertical polarization, respectively.

Starting with circular polarization at 24 K (Fig. \ref{fig2}(a)), the spectrum shows a weak reflection at $L=3$ sitting on a strong specular reflectivity background that is $L$-independent. The specular intensity is strongly energy-dependent, showing an enhancement at the Cu $ L_3$ edge, which is expected because the optical constants of the material are strongly energy-dependent. The peak near the $(0,0,3)$ position, however, was found to be energy-independent through the Cu $L_3$ edge. This observation is consistent with pure harmonic scattering, i.e., diffraction of 1870 eV background light from the $(0,0,6)$ reflection. This measurement provides a useful measure of the degree of harmonic contamination in the beam.

At an increased temperature of 103 K, three new features appear in the data, as shown in Fig. \ref{fig2}(b). First, periodic oscillations are observed with a period of $\delta L = 0.02$ r.l.u. (reciprocal lattice units). These oscillations resemble interference fringes from a thin film with a thickness of 150 nm, though the sample has a thickness of 100 $\mu$m, was freshly cleaved, and no known layer was present. Also, the fringes are not discernible at 24 K.
These oscillations are highly energy-dependent near the Cu $L_3$ edge, as shown in Fig. \ref{fig2}(b), identifying them as arising from the Bi-2212 crystal and not an extrinsic overlayer. Second, the intensity of the $L$-independent specular reflectivity increases by a factor of three near the Cu $L_3$ resonance, compared to the intensity at 24 K. This enhancement was not observed in the off-resonant signal and appears to signify a change in the $Q=0$ optical constants near the Cu $L_3$ resonance. This change would be expected if there were some form of $Q=0$ valence band order in the material. Third, a true $(0,0,3)$ Bragg feature appears that was not obviously discernible at 24 K. The peak interferes with the oscillations, and appears in the data as a distortion to the fringes near the $L=3$ location (Fig. 2(b)). This distortion is strongly energy-dependent, indicating that this is a true $(0,0,3)$ reflection has acquired a resonant component not observed at 24 K.

At $T=$ 195 K, as shown in Fig. \ref{fig2}(c), the fringes and the specular reflectivity are still strong on resonance, with intensity similar to 103 K. The (003) feature is also still present, though is somewhat weaker than at 103 K.

Finally, at room temperature (Fig. 2(d)), the fringes are no longer clearly visible. The specular reflectivity is significantly reduced, though it is still stronger than at 24 K. The $(0,0,3)$ reflection is now absent again. The remaining Bragg feature is, again, an energy-independent harmonic signal from the $(0,0,6)$ reflection, similar to what was observed at 24 K.

These results are surprising. There is no clear reason for the presence of interference oscillations, since the measurements were performed on a freshly cleaved single crystal on which no known thin film is present. Further, the changes in the specular rod and the $(0,0,3)$ reflection at intermediate temperatures suggest some alteration of the $Q=0$ optical properties of the material in the temperature range 100 K $<T<$ 200 K, i.e., between $T_c$ and $T^\ast$.
Moreover, the $(0,0,3)$ reflection is only visible in this same temperature range and only on resonance, suggesting it could indicate some kind of $Q \sim 0$ electronic ordering in the material.

To assess the reproducibility of these results, and to explore how they depend on the beam polarization, we repeated the experiment at ALS using linearly polarized light. We used the same sample as at APS, but we cleaved it again to achieve a fresh surface and remove any possible thin film that may have been present.

At $T=30$ K, as before, a weak but clear peak near $(0,0,3)$ was observed (Fig. \ref{fig2}(e)). As at APS, the peak is energy-independent, suggesting it is again coming from the $(0,0,6)$ reflection diffracting the harmonic content in the beam.
The intensity is similar to that at APS, confirming expectations that the harmonic content at the two facilities is similar.
Despite the new cleave, clear oscillations were again observed, this time with a period $\delta L = 0.0085$ r.l.u., that resemble interference fringes from a film with thickness 360 nm. The oscillations were strongly energy-dependent near the Cu $L_3$ edge demonstrating, again, that they are a property of the Bi-2212 crystal, though the periodicitiy differs from what was observed with circularly polarized light by more than a factor of two.
Both the $(0,0,3)$ peak and the oscillations sit on a specular reflectivity background that is strongly energy dependent. Strangely, the energy dependence of the fringes and the specular reflectivity are both very different from what was observed using circularly polarized x-rays at APS. We will summarize this difference quantitatively below.

Increasing the temperature to 150 K (Fig. \ref{fig2}(f)), the spectrum changes very little. The peak near the $(0,0,3)$ position is still present and energy-independent, indicating that it is purely due to harmonic scattering. However, the intensity is lower than at 30 K. This change is most likely due to changes in the harmonic content in the beam, which is more subject to subtle alignment drifts than the fundamental. Oscillations are still present with the same intensity and period as at 30 K. This behavior is very different from what was observed with circular polarization at APS, where the oscillations are barely visible at 24 K but are strong at 103 K. The reflectivity has the same intensity and energy dependence as at 30 K.

Warming to 300 K, the peak near $(0,0,3)$ still energy-independent (Fig. \ref{fig2}(g)), indicating that it arises solely from harmonic scattering at all temperatures. In other words, a true $(0,0,3)$ reflection is not observed at any temperature with linearly polarized x-rays. The oscillations are still present at this temperature but are weaker than at 150 K by a factor of two. The specular reflectivity is unchanged, and is essentially temperature-independent in this polarization.

\begin{figure*}
\includegraphics[width=1\linewidth]{fig3.pdf}
\caption{\label{fig3}
Resonance profiles of the primary features in the RSXS measurements shown in Fig. 2.
(a) Integrated intensity of the $(0,0,3)$ peak plotted as a function of incident beam energy taken at several temperatures with circularly polarized light at APS. (b) Same plot as panel (a) but measured with linear polarization at ALS. (c) Amplitude of the interference fringes plotted as a function of incident beam energy at several temperatures with circularly polarized light. (d) Same plot as panel (c) but measured with linear polarization. (e) Integrated intensity of the parabolic, specular background plotted as a function of incident beam energy taken at several temperatures with circularly polarized light. (f) Same plot as panel (e) but measured with linear polarization. The XAS spectrum (gray background) is shown in all panels for comparison.
}
\end{figure*}

\section{Summary Fits}

A summary of all results near the $(0,0,3)$ location in momentum space using both polarizations is shown in Fig. \ref{fig3}. To create this figure, we fit all the individual curves in Fig. \ref{fig2} using a fit function consisting of a Lorentzian for the Bragg peak, a sinusoidal function for the oscillations, and a parabola for the specular component,

\begin{eqnarray}
I\left( q_L \right) = \frac{A_I}{\pi}\frac{\sigma}{(q_L-q_c)^2 + \sigma^2}
+ O_s\sin\frac{2\pi(q_L-q_0)}{\Delta q} \\ \nonumber
+ aq_L^2 + bq_L+c.
\label{fit_function}
\end{eqnarray}

\noindent
In this expression, $A_I$ represents the integrated intensity of the Bragg peak, $\sigma$ is its width, $O_s$ is the amplitude of the oscillations, and $\Delta q$ represents the oscillation period. This fit function allows us to decompose the experimental data into different components so they can be examined separately.

Figs. \ref{fig3}(a)-(b) show the $(0,0,3)$ Bragg peak resonance profiles, i.e., the integrated intensity of the peak as a function of incident energy compared to the XAS spectrum. For the circularly polarized beam at APS, a resonant $(0,0,3)$ reflection is present only in the temperature range $T_c < T < T^\ast$, suggesting some $Q \sim 0$ ordering in which Cu atoms in different layers become inequivalent, may be present in the vicinity of the pseudogap temperature. This signature is not present at room temperature or below $T_c$, suggesting it may be somehow connected to the
pseudogap state. For the linearly polarized beam at ALS, however, this resonant feature is not present at any temperature, suggesting that the $Q\sim0$ order exhibits some multipolar symmetry that couples only to circularly polarized light.

The same kind of resonance plots for the interference fringes is shown in Figs. \ref{fig3}(c)-(d), which show the amplitude of the oscillations as a function of incident energy, again compared to the XAS spectrum. Oscillations are observed at all temperatures and polarizations. For circularly polarized light, the oscillations are strongest in the temperature range 100 K $<T<$ 200 K when the resonant $(0,0,3)$ reflection is present, but are barely visible at room temperature or below $T_c$. However, for linearly polarized light, the oscillations are present at all temperatures, though the amplitude is lower at room temperature.

The energy dependence of the oscillations is very different in linear and circular polarization. In
the circular case, the oscillations resonate near the peak of the edge, which is characteristic of valence band order \cite{abbamonte2005spatially, damascelli2014charge, ghiringhelli2012long, fink2011phase, achkar2012distinct, miao2017high, blanco2013momentum, thampy2014rotated, tabis2014charge, wu2012charge, wang2020high,boschini2021dynamic,chen2019charge,blanco2014resonant,he2016observation,jang2017superconductivity,bluschke2019adiabatic,peng2018re,da2015charge,
chaix2017dispersive,he2018persistent,betto2020imprint,forst2014melting}. By contrast, in linear polarization the resonance shows an inflection point characteristic of structural scattering \cite{abbamonte2006charge}, similar to the $(0,0,2)$ reflection shown in Fig. \ref{fig1}. The different energy profiles suggest that whatever is giving rise to these oscillations is a combination of structural and valence band effects, with the former scattering strongly in linear polarization and the latter scattering strongly in circular polarization.

The behavior of the featureless, specular component of the data is summarized in Figs. \ref{fig3}(e)-(f). The behavior is similar to that of the fringes. For circularly polarized light, the reflectivity is strongest in the temperature range $T_c <T< T^\ast$, where the resonant $(0,0,3)$ reflection is present, exhibiting the behavior of valence band scattering. For linearly polarized light, the behavor looks like that of pure structural scattering. The strong similarity between the oscillatory and parabolic contributions suggest that these two effects have a common origin and may be considered as a single phenomenon. The overall behavior of the reflectivity and the fringes points to some form of extrinsic structural inhomogeneity, perhaps related to the detailed surface morphology or the way the material cleaves, interplaying in a nontrivial way with intrinsic electronic instabilities in the material.

\begin{figure*}
\includegraphics[width=0.8\linewidth]{fig4.pdf}
\caption{\label{fig4}Position and momentum dependence of the oscillations observed in Fig. 2. (a) Microscope image of the cleaved sample after RSXS measurements. The adjacent panel shows a color map of the total electron yield (TEY) at the peak of the Cu $L_3$ edge from the same sample, taken as the beam is scanned across the surface.
The dark red square in the microscope image shows the position of the beam used for measurements at ALS. The three rectangular boxes show the size and location of the beam spot at APS used for several measurements.
(b) Momentum scans in the $(0,0,L)$ direction for the three different beam locations, A, B, and C, indicated in panel (a). The overall shape and periodicity of the oscillations strongly depends on the location of the beam on the surface.
(c) In-plane momentum map at $L = 2.996$ r.l.u. near the Cu $L_3$ edge (6.5 eV off resonance) taken with a CCD detector at ALS. The rectangular boxes, D, E, F, and G, show the ranges of integration in the $(H,K)$ plane for a selection of in-plane momentum values used in panel (d). (d) Momentum scans in the $(0,0,L)$ direction for several different in-plane momenta indicated in panel (c). The oscillations are always visible but their period and overall shape strongly depends on the in-plane momentum chosen.
}
\end{figure*}

One speculation is that, when the crystal cleaves, it gives rise to some electronic, self-organized layer near the surface of the material whose thickness depends on details of the fracture dynamics or local residual strain. In this case, one might expect the amplitude and period of the fringes to vary significantly on macroscopic scales over the sample surface. We decided to test this possibility by examining the extent to which the oscillations depend on where the x-ray beam strikes the surface.
Using circularly polarized x-rays at APS, we studied the behavior of the fringes at 103 K as the beam is scanned across the sample surface. The results are shown in Figs. \ref{fig4}(a)-(b).
Fig. \ref{fig4}(a) shows a microscope image of the sample after this cleave next to a map of the total electron yield for the beam striking different locations, which is sensitive to local surface morhphology. We focused our measurements on a visually flat region of the sample surface where the electron yield was high, and then carried out momentum scans in the $(0,0,L)$ direction at several locations denoted A, B and C. The results are summarized in Fig. \ref{fig4}(b). Oscillations are present in all curves, but the periodicity and overall shape is very different for different beam  positions. This observation supports the conclusion that the oscillations, while resonant and seemingly indicating a valence band effect, depend on details of the local surface structure and therefore are also partly an extrinsic phenomenon.

Another way to view this effect is to examine the off-specular scattering, i.e., the nonzero values of $H$ or $K$ which corresponds to having a small in-plane component of the momentum transfer. Such a measurement would reveal whether the fringes correlate with different in-plane periodicities of the crystal surface.

To obtain information about the off-specular scattering, we repeated the measurement at ALS with linearly polarized light at $T=30$ K, this time using a CCD area detector, which provides a 2D image of near-specular scattering corresponding to small in-plane momenta. The results are shown in Figs. \ref{fig4}(c)-(d).
Fig. \ref{fig4}(c) shows the $(H,K)$ plane at a fixed momentum $L = 2.996$ r.l.u.. Surprisingly, the results do not show a simple symmetric peak at $H=K=0$. Rather, an intricate structure is observed that involves a significant amount of off-specular scattering. This structure is likely a convolution of two different effects: (1) coherent off-specular scattering due to non-trivial in-plane periodicities, and (2) specular scattering from a surface that is not flat or is corrugated because of morphology of the cleaved surface. Fig. \ref{fig4}(d) shows how the fringes vary with $H$ and $K$, i.e., how the periodicity of the fringes in the $L$ direction depends on the in-plane momentum for several different $(H, K)$ values denoted as D, E, F and G in Fig. \ref{fig4}(c). Again, the fringes are present for all values of $(H,K)$, but the periodicity varies and depends on the chosen in-plane momentum. Evidently, the period of the fringes does not just depend on where the beam hits the sample surface, but also on the form of the surface morphology at any given point. This measurement supports a conclusion that the oscillations are strongly dependent on the detailed local structure of the surface, and so involve some interaction between valence instabilities of the surface and extrinsic effects resulting from the way the crystal cleaves.

\section{Discussion}
To summarize, we performed RSXS measurements near the forbidden Bragg reflection $(0,0,3)$ to explore whether some evidence might be found for $Q=0$ electronic ordering in optimally doped Bi-2212. We found that the $(0,0,3)$ reflection, while forbidden by normal x-ray extinction rules, becomes allowed on resonance when scattering circularly polarized x-rays in the temperature range $T_c < T < T^\ast$. Its resonance properties are typical of valence band order \cite{abbamonte2005spatially, damascelli2014charge, ghiringhelli2012long, fink2011phase, achkar2012distinct, miao2017high, blanco2013momentum, thampy2014rotated, tabis2014charge, wu2012charge, wang2020high,boschini2021dynamic,chen2019charge,blanco2014resonant,he2016observation,jang2017superconductivity,bluschke2019adiabatic,peng2018re,da2015charge,
chaix2017dispersive,he2018persistent,betto2020imprint,forst2014melting}, suggesting some
$Q \sim 0$ order may be present in this narrow temperature regime near the pseudogap temperature.
Outside this temperature range, or using linearly polarized x-rays, this reflection remains forbidden. This result might indicate that the order exhibits a multipolar symmetry that couples only to circularly polarized light. Further studies, perhaps using azimuthal rotations or a polarimeter, are needed to reproduce this result and identify the detailed symmetry.

In addition, we observed highly reproducible periodic oscillations that resemble interference fringes from a thin film, though no layered structure is known to be present on the material. The resonance properties of these fringes, in circular polarization, resemble those of the $(0,0,3)$ reflection, suggesting they are a valence band effect. However, in linear polarization these oscillations exhibit the resonance behavior of a structural phenomenon. These fringes are present at all temperatures and polarizations, even after several cleaves of the sample surface. However, their periodicity varies greatly across the sample surface and for different values of the in-plane momentum. We suggest that these fringes are due to some interaction between intrinsic valence band instabilities in the material and extrinsic structural morphologies of the sample surface.

Concerning the question of $Q=0$ order, our results do not conclusively demonstrate the existence of ordered multipoles or loop currents in the pseudogap regime. However, they demonstrate that some nontrivial valence band ordering effect is taking place and further studies are warranted that could disentangle intrinsic ordering phenomena from effects influenced by the surface properties. An ideal approach would be to perform RSXS studies with a highly focused beam that could investigate a single location on the sample surface down to a sub-micron dimension whose local properties are better defined. This could be accomplished, for example, at the Coherent Soft X-ray (CSX) Scattering facility of the National Synchrotron Light Source II or any other coherent or microbeam instruments currently operational worldwide.

\begin{acknowledgements}
We thank C. M. Varma for inspiring us to undertake this study, and S. L. Cooper for carefully reading the manuscript. This work was supported by the Center for Quantum Sensing and Quantum Materials, an Energy Frontier Research
Center funded by the U. S. Department of Energy, Office of Science, Basic Energy Sciences under Award DE-SC0021238. P. A. gratefully acknowledges support from the EPiQS program of the Gordon and Betty Moore Foundation, Grant No. GBMF9452. Crystal growth was supported by DOE Grant No. DE-SC0012704.

\end{acknowledgements}

redpajama_set_name: RedPajamaArXiv