\section{Introduction}

Misinformation on social media has become a major focus of public debate and academic research. Previous works have demonstrated that misinformation is widespread on social media platforms and that misinformation diffuses significantly farther, faster, deeper, and more broadly than the truth \cite[\eg,][]{Vosoughi.2018}. Concerns about misinformation on social media have been rising in recent years, particularly given its potential impacts on elections \cite{Aral.2019,Bakshy.2015,Grinberg.2019}, public health \cite{Broniatowski.2018}, and public safety \cite{Starbird.2017}. Major social media providers (\eg, Twitter, Facebook) thus have been called upon to develop effective countermeasures to combat the spread of misinformation on their platforms.

Previous approaches to identify misinformation on social media can roughly be grouped into two categories: (i) human-based systems in which (human) experts or fact-checking organizations (\eg, \url{snopes.com}, \url{politifact.com}, \url{factcheck.org}) determine the veracity of information \cite[\eg,][]{Hassan.2017, Shao.2016}; and (ii) machine learning-based systems that can automatically classify veracity \cite[\eg,][]{Ma.2016,Qazvinian.2011}. In the latter category, machine learning models are typically trained to classify misinformation using content-based features (\eg text, images, video), context-based features (\eg, time, location), or based on propagation patterns (\ie, how misinformation circulates among users). Yet despite the rise of fact-checking websites \cite{Reporterlab.2018} and advances in machine learning-based detection, misinformation is still widespread on social media \cite{Vosoughi.2018}.

This indicates that current fact-checking approaches are insufficient, and complementary approaches are necessary to effectively combat misinformation on social media \cite{Moravec.2019}.

Informed by these findings, Twitter has recently launched ``Birdwatch,'' a new attempt to address misinformation on social media by harnessing the ``wisdom of crowds.'' Birdwatch is a community-driven approach to identify misleading tweets on Twitter. The idea is that the user base on Twitter provides a wealth of knowledge that can help in the fight against misinformation. On Birdwatch, users can identify tweets they believe are misleading and write (textual) notes that provide \emph{context} to the tweet. They can also add links to their sources of information. A key feature of Birdwatch is that it also uses a rating mechanism that allows users to rate the quality of other participants' notes. These ratings should help to identify the context that people will find most helpful and raise its visibility to other users. While the Birdwatch feature is currently in pilot phase and only available via a separate Birdwatch website, Twitter's goal is that community-written notes will be visible directly on tweets, available to everyone on Twitter.

\textbf{Research goal: }
In this work, we provide a holistic analysis of the new Birdwatch pilot on Twitter. We empirically analyze how users interact with Birdwatch and study factors that make community-created fact checks more likely to be perceived as helpful or unhelpful by other users. Specifically, we address the following research questions:
\begin{itemize}
\item \textbf{(RQ1)} \emph{What are specific reasons due to which Birdwatch users report tweets?}
\item \textbf{(RQ2)} \emph{How do Birdwatch notes for tweets categorized as being misleading vs. not misleading differ in terms of their content characteristics? (\eg, sentiment, length)}
\item \textbf{(RQ3)} \emph{Are tweets from Twitter accounts with certain characteristics (\eg, politicians, accounts with many followers) more likely to be fact-checked on Birdwatch?}
\item \textbf{(RQ4)} \emph{Which characteristics of Birdwatch notes are associated with greater helpfulness for other users?}
\item \textbf{(RQ5)} \emph{Does the level of consensus between users vary depending on the level of social influence of the source tweet?}
\end{itemize}

\textbf{Methodology:} To address our research questions, we collect \emph{all} Birdwatch notes and ratings from the Birdwatch website between the introduction of the feature on January 23, 2021, and the end of March 2021. This comprehensive dataset contains a total number of \num{4644} Birdwatch notes and \num{12761} ratings.
We use text mining methods to extract content characteristics from the text explanations of the Birdwatch notes (\eg sentiment, length). In addition, we employ the Twitter historical API to collect information about the source tweets (\ie, the fact-checked tweets) referenced in the Birdwatch notes. We then perform an empirical analysis of the observational data and use regression models to understand how (i) the user categorization, (ii) the content characteristics of the text explanation, (iii) the social influence of the source tweet are associated with the helpfulness of Birdwatch notes.

\textbf{Contributions:} To the best of our knowledge, this study is the first to present a thorough empirical analysis of Twitter's new Birdwatch feature. Our work yields the following main findings:
\begin{enumerate}
\item Users file a larger number of Birdwatch notes for misleading than for not misleading tweets. Misleading tweets are primarily reported because of factual errors, lack of important context or because they contain unverified claims. Not misleading tweets are primarily reported because they are perceived as factually correct or because they clearly refer to personal opinion or satire.
\item Birdwatch notes filed for misleading vs. not misleading tweets differ in terms of their content characteristics. For tweets reported as misleading, authors of Birdwatch notes use a more negative sentiment, more complex language, and tend to write longer text explanations.
\item Tweets from influential politicians (on both sides of the political spectrum) are more likely to be fact-checked on Birdwatch. For many of these accounts, Birdwatch users overwhelmingly report misinformation.
\item Birdwatch notes reporting misleading tweets receive more votes and are more helpful for other Birdwatch users.
Birdwatch notes are perceived as being particularly helpful if they provide trustworthy sources and embed a positive sentiment.
\item The level of consensus between users depends on the social influence of the author of the source tweet. For influential users with many followers, Birdwatch notes receive more total votes (helpful \& unhelpful) but a lower helpfulness ratio (\ie, a lower level of consensus). Also, Birdwatch notes for influential users are particularly likely to be seen as incorrect.
\end{enumerate}

\textbf{Implications:} Our findings have direct implications for the Birdwatch platform and future attempts to implement community-based approaches to combat misinformation on social media.
We show that users perceive a relatively high share of community-created fact checks (Birdwatch notes) as being informative, clear and helpful. Here we find that encouraging users to provide \emph{context} (\eg, by linking trustworthy sources) and avoid the use of inflammatory language is crucial for users to perceive community-based fact checks as helpful. Despite showing promising potential, our analysis also suggests that the Birdwatch's community-driven approach faces challenges concerning opinion speculation, biased views, and polarized opinions among the user base -- in particular with regards to influential accounts with high numbers of followers.

\section{Background}

\begin{table*}[]
\footnotesize
\def\em{\em}
\begin{tabular}{lp{0.35\textwidth}p{0.09\textwidth}p{0.35\textwidth}lr}
\toprule
\multicolumn{2}{l}{\textbf{Source tweet}} & \multicolumn{2}{l}{\textbf{Birdwatch note}} & \multicolumn{2}{l}{\textbf{Ratings}} \\
\cmidrule(lr){3-4} \cmidrule(lr){5-6}
 && \textbf{Categorization} &  \textbf{Text explanation}  & $\boldsymbol{\mathit{HVotes}}$ & $\boldsymbol{\mathit{Votes}}$ \\
\midrule
\#1&``\texttt{I am happy to work with Republicans on this issue where there’s common ground, but you almost had me murdered 3 weeks ago so you can sit this one out. Happy to work w/ almost any other GOP that aren’t trying to get me killed. In the meantime if you want to help, you can resign. https://t.co/4mVREbaqqm}''  & Misleading & ``\texttt{While I can understand AOC being worried she may have been hurt by the Capitol Protesters she cannot claim Ted Cruz tried to have her killed without providing proof. That's dangerous misinformation by falsely accusing Ted Cruz of a serious crime.}'' & 47 & 59 \\
\addlinespace
\#2&``\texttt{Rush Limbaugh had a regular radio segment where he would read off the names of gay people who died of AIDS and celebrate it and play horns and bells and stuff.}'' & Misleading & ``\texttt{These segments are fictitious and have never happened.}'' & 3 & 24 \\
\addlinespace
\#3&``\texttt{Every single American should be OUTRAGED by this: Democrats just voted to ban voter ID nationwide and force every state to permanently expand mail-in voting.}''& Misleading & ``\texttt{The bill does not ban voter id requirements. But it does allow voters who dont show an ID to submit a signed statement of their elgibility: \url{https://www.congress.gov/bill/116th-congress/house-bill/1}  \url{https://www.cnn.com/2021/03/03/politics/fact-check-pence-election-hr1-democrats-elections/index.html}}'' & 11 & 12 \\
\addlinespace
\#4&``\texttt{The vast majority of Americans believe in universal background checks. As a gun owner myself, I firmly support the Second Amendment but I also believe we have to be willing to make some changes for the greater good.  Read my full statement on \#HR8 here: \url{https://t.co/UcrOJt8A0M}}'' & Not misleading & ``\texttt{Universal background checks enjoy high levels of public support; a 2016 representative survey found 86\% of registered voters in the United States supported the measure. \url{https://news.gallup.com/poll/1645/guns.aspx}}''
`` & 7 & 7 \\
\addlinespace
\#5&``\texttt{Congratulations to @PeteButtigieg on becoming the second openly gay member of a President’s Cabinet.  Welcome to the club! \url{https://t.co/BbKK6iTBW1}}'' & Not misleading & ``\texttt{This is correct. Ric Grenell was the first openly gay cabinet member. \url{https://thehill.com/changing-america/respect/diversity-inclusion/484026-trump-names-the-first-openly-gay-person-to-a?amp}}'' & 5 & 6\\

\bottomrule
\end{tabular}
\caption{Examples of Birdwatch notes and ratings. The column $\boldsymbol{\mathit{HVotes}}$ refers to the number of users who have responded ``Yes'' to the rating question ``Is this note helpful?'' The column $\boldsymbol{\mathit{Votes}}$ refers to the total number of ratings a Birdwatch note has received (helpful \& unhelpful).}
\label{tbl:examples}
\end{table*}

\subsection{Misinformation on Social Media}

Social media has become a prevalent platform for consuming and sharing information online \cite{Bakshy.2015, Wattal.2010}. It is estimated that almost 62\% of the adult population consume news via social media platforms, and this proportion is expected to increase further \cite{Pew.2016}. As any user can share information on social media \cite{Shore.2018}, quality control for the content has essentially moved from trained journalists to regular users \cite{Kim.2019}. The inevitable lack of oversight from experts makes social media vulnerable to the spread of misinformation \cite{Shao.2016}. Social media platforms have indeed been observed to be a medium that disseminates vast amounts of misinformation \citep{Vosoughi.2018}. Several works have studied diffusion characteristics of misinformation on social media \cite[\eg,][]{Friggeri.2014,Vosoughi.2018}, finding that misinformation spreads significantly farther, faster, deeper, and more broadly than the truth. The presence of misinformation on social media has detrimental consequences on how opinions are formed and the offline world \cite{Allcott.2017,Bakshy.2015,Bessi.2015,DelVicario.2016,Oh.2013}. Misinformation thus threatens not only the reputation of individuals and organizations, but also society at large.

Previous research has identified several reasons why misinformation is widespread on social media. On the one hand, misinformation pieces are often intentionally written to mislead other users \cite{Wu.2019}. It is thus difficult for users to spot misinformation from the content itself. On the other hand, the vast majority of social media users do not fact-check articles they read \cite{Geeng.2020,Vo.2018}. This indicates that social media users are often in a hedonic mindset and avoid cognitive reasoning such as verification behavior \cite{Moravec.2019}. A recent study further suggests that the current design of social media platforms may discourage users from reflecting on accuracy \cite{Pennycook.2021}.

The spread of misinformation can also be seen as a social phenomenon. Online social networks are characterized by homophily \cite{McPherson.2001}, polarization \cite{Levy.2021}, and echo chambers \cite{Barbera.2015}. In these information environments with low content diversity and strong social reinforcement, users tend to selectively consume information that shares similar views or ideologies while disregarding contradictory arguments \cite{Ecker.2010}
These effects can even be exaggerated in the presence of repeated exposure: once misinformation has been absorbed, users are less likely to change their beliefs even when the misinformation is debunked \cite{Pennycook.2018}.

\subsection{Identification of Misinformation}

\textbf{Identification via expert's verification:} Misinformation can be manually detected by human experts. In traditional media (\eg, newspapers), this task is typically carried out by human editors and expert journalists who verify information before it is published \cite{Kim.2019}. However, this approach does not scale to the volume and speed of content that is generated in social media \cite{Tschiatschek.2018}.
This has given rise to a number of third-party fact-checking organizations such as \url{snopes.com} and \url{politifact.com} that thoroughly investigate and debunk rumors \cite{Wu.2019,Vosoughi.2018}. The fact-checking assessments are consumed and broadcast by social media users like any other type of news content and are supposed to help users to identify misinformation on social media \cite{Shao.2016}. However, previous research suggests that such platforms are likely to have minimal real-world effects. In particular, Vosoughi et al. \cite{Vosoughi.2018} show that \emph{fact-checked} false rumors spread significantly more pronounced than the truth.

\textbf{Identification using machine learning methods:}
Previous research has intensively focused on the problem of misinformation detection via means of supervised machine learning \cite{Castillo.2011,Ducci.2020, Kwon.2017,Kwon.2013,Vosoughi.2017}. Researchers typically collect posts and their labels from social media platforms such as Twitter and Sina Weibo, and then train a classifier based on diverse sets of features. For instance, {content-based approaches} aim at directly detecting misinformation based on its content, such as text, images, and video \cite[\eg,][]{Yang.2012}. However, different from traditional text categorization tasks, misinformation posts are deliberately made seemingly real and accurate. The predictive power of the actual content in detecting misinformation is thus limited \cite{Wu.2019}. Other common information sources include context-based features (\eg, time, location), or propagation patterns (\ie, how misinformation circulates among users) \cite[\eg,][]{Kwon.2014,Wu.2016}. Here the underlying intuition is that the direction of information exchange can reveal community structures and personal characteristics \cite{Wu.2019}. For instance, temporal patterns and posting frequency can be used as features in supervised misinformation detection systems \cite{Kwon.2014}. Yet also with these features, the lack of ground truth labels poses serious challenges when training a supervised machine learning classifier -- in particular in the early stages of the spreading process \cite{Wu.2019}. Hence, some works also use unsupervised learning methods to detect misinformation \cite[\eg,][]{Chu.2010}.

Altogether, significant efforts have been made to identify misinformation on social media. However, the fact that it is still widespread \cite[\eg][]{Vosoughi.2018} indicates that current fact-checking systems are insufficient, and complementary approaches are necessary to combat misinformation effectively.
For this purpose, Twitter recently launched ``Birdwatch,'' a new attempt to address misinformation by harnessing the ``wisdom of crowds.''

\section{What is Birdwatch?}

On January 15, 2021, Twitter has launched the ``Birdwatch'' feature, a new approach to address misinformation on social media by harnessing the ``wisdom of crowds.'' Birdwatch is a community-driven approach to identify misleading tweets circulating on Twitter. On Birdwatch, users can identify tweets they believe are misleading and write (textual) notes that provide \emph{context} to the tweet. Birdwatch also features a rating mechanism that allow users to rate the quality of other users' notes. These ratings are supposed to help to identify the context that people will find most helpful and raise its visibility to other users.
While the Birdwatch feature is currently in pilot phase and only available via a separate Birdwatch website\footnote{The Birdwatch website is available via \url{birdwatch.twitter.com}}, Twitter's goal is that community-written notes will be visible directly on tweets, available to everyone on Twitter.

\textbf{Birdwatch notes:}
Users can add Birdwatch notes to any tweet they come across and think might be misleading. Notes are composed of (i) multiple-choice questions that allows users to state why a tweet might or might not be misleading; (ii) an open text field where uses can explain why they believe a tweet is misleading, as well as link to relevant sources. The maximum number of characters in the open text field is 280. Here, each URL counts as only 1 character towards the 280 character limit. After it's submitted, the note is available on the Birdwatch site for other users to read and rate. Birdwatch notes are public, and anyone can browse the Birdwatch website to see them.

\textbf{Ratings:}
Users on Birdwatch can rate the helpfulness of notes from other users. These ratings are supposed to help to identify which notes are most helpful and to allow Birdwatch to raise the visibility of those found most helpful by a wide range of contributors. Tweets with Birdwatch notes are highlighted with a Birdwatch icon that helps users to find them. Users can then click on the icon to read all notes others have written about that tweet and rate their quality (\ie, whether or not the Birdwatch note is helpful). Notes rated by the community to be particularly helpful then receive a currently rated helpful badge.

\textbf{Illustrative examples:} Tbl.~\ref{tbl:examples} presents five examples of Birdwatch notes and helpfulness ratings. The table shows that many Birdwatch notes report tweets that refer to a political topic (\eg, voting rights). We also see that the text explanations differ regarding their informativeness. Some text explanations are longer and link to additional sources that are supposed to support the comments of the author. There are also differences regarding how other users perceive the Birdwatch notes. For example, Birdwatch note \#1 in Tbl.~\ref{tbl:examples} received 47 helpful votes out of 59 total votes, whereas 7 out of 7 users rated Birdwatch note \#4 as helpful. Note that there are also some users trying to abuse Birdwatch by falsely asserting that a tweet is misleading. In such situations, other users can downvote the note through Birdwatch's rating system. For example, the (false) assertion in Birdwatch note \#2 in Tbl.~\ref{tbl:examples} received only 3 helpful votes out of 27 total votes.

\section{Data}

\subsection{Data Collection}

We downloaded \emph{all} Birdwatch notes and ratings between the introduction of the feature on January 23, 2021, and the end of March 2021 from the Birdwatch website. This comprehensive dataset contains a total number of \num{4644} Birdwatch notes and \num{12761} ratings. Each Birdwatch note has a unique id (\emph{noteId}) and each rating refers to a single Birdwatch note. We merged the ratings with the Birdwatch notes using this \emph{noteId} field. The result is a single dataframe in which each Birdwatch note corresponds to one observation for which we know the number of helpful and unhelpful votes from the rating data.

\subsection{Key Variables}

We now present the key variables that we extracted from the Birdwatch data. All of these variables will be empirically analyzed in the next sections:

\begin{itemize}

\item \emph{Misleading: } A binary indicator of whether a tweet has been reported as being misleading by the author of the Birdwatch note ($=1$; otherwise $=0$).

\item \emph{Text explanation: } The user-entered text explanation (max 280 characters) to explain why a tweet is misleading or not misleading.

\item \emph{Trustworthy sources: } A binary indicator of whether the author of the Birdwatch note has responded ``Yes'' to the note writing question ``Did you link to sources you believe most people would consider trustworthy?'' ($=1$; otherwise $=0$).

\item \emph{HVotes: } The number of other users who have responded ``Yes'' to the rating question  ``Is this note helpful?''
\item \emph{Votes: } The total number of ratings a Birdwatch note has received by other users (helpful \& unhelpful).
\end{itemize}

\subsection{Content Characteristics}

We use text mining methods to extract the following content characteristics from the text explanations in the Birdwatch notes:
\begin{itemize}
\item \emph{Sentiment: } We calculate a sentiment score that measures the extent of positive vs. negative emotions in Birdwatch notes. Our computation follows a dictionary-based approach as in \cite{Vosoughi.2018}.
Here we use the NRC emotion lexicon \cite{Mohammad.2013}, which classifies English words into positive and negative emotions. The fraction of words in the text explanations related to positive and negative emotions is then aggregated and averaged to create a vector of positive and negative emotion weights that sum to one. The sentiment score is then defined as the difference between positive and negative emotion scores.
\item \emph{Text complexity: } We calculate a text complexity score, specifically, the Gunning-Fog index \cite{Gunning.1968}. This index estimates the years of formal education necessary for a person to understand a text upon reading it for the first time: $0.4 \times (\mathit{ASL} + 100 \times n_{\text{wsy} \geq 3} / n_{\text{w}})$, where $\mathit{ASL}$ is the average sentence length (number of words), $n_{\text{w}}$ is the total number of words, and $n_{\text{wsy} \geq 3}$ is the number of words with three syllables or more. A higher value thus indicates greater complexity.
\item \emph{Word count: } We determine the length of the text explanations in Birdwatch notes as given by the number of words.
\end{itemize}

Our text mining pipeline is implemented in R~4.0.2 using the packages \texttt{quanteda} (Version 2.0.1) and \texttt{sentimentr} (Version 2.7.1) for text mining and the built-in NRC emotion lexicon.

\subsection{Twitter Historical API}

Each Birdwatch note addresses a single tweet, \ie, the tweet that has been categorized as being misleading or not misleading by the author of the Birdwatch note. We used the Twitter historical API to map the \emph{tweetID} referenced in each Birdwatch note to the source tweet and collected the following information about each source tweet:
\begin{itemize}
\item \emph{Account name: } The name of the Twitter account for which the Birdwatch note has been reported.
\item \emph{Followers: } The number of followers, \ie, the number of accounts that follow the author of the source tweet on Twitter.
\item \emph{Followees: } The number of followees, \ie, the number of accounts whom the author of the source tweet follows on Twitter.
\item \emph{Account age: } The age of the author of the source tweet's account (in years).
\item \emph{Verified: } A binary dummy indicating whether the account of the source tweet has been officially verified by Twitter ($=1$; otherwise $=0$).
\end{itemize}

\section{Empirical Analysis}

\subsection{Analysis of Birdwatch Notes (RQ1 \& RQ2)}

In this section, we analyze how users categorize misleading and not misleading tweets in Birdwatch notes. We also explore the reasons because of which Birdwatch users report tweets and how Birdwatch notes for misleading vs. not misleading differ in terms of their content characteristics (\eg, sentiment).

\textbf{Categorization of tweets in Birdwatch notes: } Fig.~\ref{fig:trustworthy} shows that users file a larger number of Birdwatch notes for misleading than for not misleading tweets. Out of all Birdwatch notes, \SI{87.96}{\percent} refer to misleading tweets, whereas the remaining \SI{12.04}{\percent} refer to not misleading tweets. Fig.~\ref{fig:trustworthy} further suggests that authors of Birdwatch notes are slightly more likely to report that they have linked to trustworthy sources for misleading (\SI{63.96}{\percent}) than for not misleading tweets (\SI{62.92}{\percent}).

\begin{figure
				\centering

				{\includegraphics[width=.8\linewidth]{figures/barplot_trustworthy.pdf}}

				\caption{Number of users who responded ``Yes'' to the question ``Did you link to sources you believe most people would consider trustworthy?''}
				\label{fig:trustworthy}
\end{figure}

\textbf{Why do users report misleading tweets? } Authors of Birdwatch users also need to answer a checkbox question (multiple selections possible) on why they perceive a Birdwatch note as being misleading. Fig.~\ref{fig:ccdf_notes_misleading} shows that misleading tweets are primarily reported because of factual errors (\SI{30}{\percent}), lack of important context (\SI{28}{\percent}), or because they contain unverified claims as fact (\SI{25}{\percent}). Birdwatch notes reporting outdated information (\SI{5.35}{\percent}), satire (\SI{5}{\percent}), or manipulated media (\SI{3}{\percent}) are relatively rare. Only \SI{3}{\percent} of Birdwatch notes are reported because of other reasons.

\textbf{Why do users report not misleading tweets? } Fig.~\ref{fig:ccdf_notes_notMisleading} shows that not misleading tweets are primarily reported because they are perceived as factually correct (\SI{62}{\percent}), or because they clearly refer to personal opinion (\SI{16}{\percent}) or satire (\SI{12}{\percent}). Birdwatch users only rarely report tweets containing information that was correct at the time of writing but is now outdated (\SI{1}{\percent}). Only \SI{8}{\percent} of Birdwatch notes are reported because of other reasons.

\textbf{Content characteristics: } The complementary cumulative distribution functions (CCDFs) in Fig.~\ref{fig:ccdf_notes_content} visualize how Birdwatch notes for misleading vs. not misleading tweets differ in terms of their content characteristics. We find that Birdwatch notes reporting misleading tweets embed a higher proportion of negative emotions than tweets reported as being not misleading.
For tweets reported as being misleading, authors of Birdwatch notes also use slightly less complex language and tend to write longer text explanations.

\begin{figure
	\centering
	{\includegraphics[width=\linewidth]{figures/barplot_classification_misleading}}
	\caption{Number of Birdwatch notes per checkbox answer option in response to the question ``Why do you believe this tweet may be misleading?''}
	\label{fig:ccdf_notes_misleading}
\end{figure}

\begin{figure
				\centering

				{\includegraphics[width=\linewidth]{figures/barplot_classification_notMisleading}\label{fig:cascade_size_ccdf}}

				\caption{Number of Birdwatch notes per checkbox answer option in response to the question ``Why do you believe this tweet is not misleading?''}
	\label{fig:ccdf_notes_notMisleading}
\end{figure}

\subsection{Analysis of Source Tweets (RQ3)}

We now explore whether tweets from Twitter accounts with certain characteristics (\eg, politicians, accounts with many followers) are more likely to be fact-checked on Birdwatch.

\textbf{Most reported accounts: }
Tbl.~\ref{tbl:accounts_notes} reports the names of the ten Twitter accounts that have gathered the most Birdwatch notes. We find that many of these accounts belong to influential politicians from both sides of the political spectrum (\eg, \emph{AOC}, \emph{SenTedCruz}). For most of these accounts, Birdwatch users overwhelmingly report misinformation. For instance, \num{138} Birdwatch notes have been reported for Democratic Congresswoman Alexandria Ocasio-Cortez (\emph{AOC}), out of which \SI{91.30}{\percent} have been categorized as being misleading tweets. We also observe many reports of misleading tweets for other prominent public figures such as Donald Trump Jr. (\emph{DonaldJTrumpJr}) and NYT best-selling author Candace Owens (\emph{RealCandanceO}). The only Twitter account in the list in Tbl.~\ref{tbl:accounts_notes} for which the majority of tweets have been reported as being not misleading is the account of Steven Crowder (\emph{scrowder}), an American-Canadian conservative political commentator, media host, and comedian.

\begin{table
\centering
\begin{tabular}{>{\em}lrr}
  \toprule
\textbf{\emph{Account name}} & \textbf{\#Notes} & \textbf{Misleading (in \%)} \\
  \midrule
AOC & 138 & 91.30 \\
  mtgreenee &  78 & 100.00 \\
  laurenboebert &  40 & 92.50 \\
  AlexBerenson &  37 & 97.30 \\
  POTUS &  35 & 71.43 \\
  RealCandaceO &  25 & 92.00 \\
  SenTedCruz &  24 & 87.50 \\
  Jim\_Jordan &  22 & 81.82 \\
  scrowder &  22 & 45.45 \\
  DonaldJTrumpJr &  20 & 85.00 \\
\bottomrule
\end{tabular}
\caption{Top~10 Twitter accounts with the highest number of Birdwatch notes.
}
\label{tbl:accounts_notes}
\end{table}

\textbf{Account characteristics:}
Fig.~\ref{fig:ccdf_social_influence} plots the CCDFs for different characteristics of Twitter accounts, namely (a) the number of followers, (b) the number followees, and (c) the account age (in years). The plots show that tweets reported as being misleading tend to have a lower number of followers and a lower number of followees. We observe only minor differences with regard to the account age. Fig.~\ref{fig:source_verified} further shows that a majority of Birdwatch notes are reported for tweets from verified accounts. Here we find only minor differences for misleading vs. not misleading tweets.

\begin{figure}[H]
\captionsetup{position=top}
				\centering

				\subfloat[Sentiment]{\includegraphics[width=.425\linewidth]{figures/ccdf_classification_sentiment}}
				\hspace{0.2cm}
				\subfloat[Text complexity]{\includegraphics[width=.425\linewidth]{figures/ccdf_classification_FOG}}\\

				\subfloat[Word count]{\includegraphics[width=.425\linewidth]{figures/ccdf_classification_word_count}}

				\caption{Complementary cumulative distribution functions (CCDFs) for (a) sentiment, (b) text complexity, and (c) word count in Birdwatch notes.}
	\label{fig:ccdf_notes_content}
\end{figure}

\begin{figure}[H]
\captionsetup{position=top}
				\centering

				\subfloat[Followers]{\includegraphics[width=.425\linewidth]{figures/ccdf_source_tweet_source_followers_count}}
				\hspace{0.2cm}
				\subfloat[Followees]{\includegraphics[width=.425\linewidth]{figures/ccdf_source_tweet_source_friends_count}}\\

				\subfloat[Account age]{\includegraphics[width=.425\linewidth]{figures/ccdf_source_tweet_source_account_age}}

				\caption{Complementary cumulative distribution functions (CCDFs) for (a) followers, (b) followees, and (c) account age (in years).}
	\label{fig:ccdf_social_influence}
\end{figure}

\begin{figure
				\centering

				{\includegraphics[width=.8\linewidth]{figures/barplot_verified.pdf}}

				\caption{Share of tweets originating from verified accounts.}
				\label{fig:source_verified}
\end{figure}

\subsection{Analysis of Ratings (RQ4)}

\textbf{Helpfulness: }
Fig.~\ref{fig:ccdf_helpfulness} plots the CCDFs for the helpfulness ratio ($HVotes/Votes$) and the total number of helpful and unhelpful votes ($Votes$). We find that Birdwatch Notes reporting misleading tweets tend to receive both a higher helpfulness ratio and a higher number of total votes.
In Fig.~\ref{fig:ccdf_trustworthy}, we visualize the CCDFs separately for Birdwatch notes in which the authors have / have not reported that they have linked to trustworthy sources. We find that linking to trustworthy sources yields a higher helpfulness ratio and more votes for not misleading tweets. In the case of misleading tweets, linking trustworthy sources yields a higher helpfulness ratio but a smaller total number of votes.

\begin{figure
\captionsetup{position=top}
				\centering

				\subfloat[Helpfulness ratio]{\includegraphics[width=.45\linewidth]{figures/ccdf_ratioHelpful_classification}}
				\hspace{0.35cm}
				\subfloat[Total votes]{\includegraphics[width=.45\linewidth]{figures/ccdf_totalVotes_classification}}

				\caption{Complementary cumulative distribution functions (CCDFs) for (a) helpfulness ratio and (b) total votes. }
	\label{fig:ccdf_helpfulness}
\end{figure}

\textbf{Why do users find Birdwatch notes helpful? } Users rating a Birdwatch note helpful need to answer a checkbox question (multiple selections possible) on why they perceive a Birdwatch note as being helpful. Fig.~\ref{fig:reasons_helpful} shows that users find that Birdwatch notes are particularly helpful if they are (i) informative (\SI{31}{\percent}), (ii) clear (\SI{24}{\percent}), and (iii) provide good sources (\SI{19}{\percent}). To a lesser extent, users also value empathetic Birdwatch notes (\SI{13}{\percent}) and Birdwatch notes that provide unique context (\SI{11}{\percent}). Only (\SI{1}{\percent}) of Birdwatch notes are perceived as helpful because of other reasons.

\begin{figure
\captionsetup{position=top}
\centering
\vspace{-.1cm}
	\subfloat[Not misleading]{\label{fig:ccdf_grouped_basic_emotions}
\begin{tabular}[c]{@{}ll@{}}
\includegraphics[width=.475\linewidth]{figures/ccdf_ratioHelpful_notMisleading_trustworthySources.pdf}&
\includegraphics[width=.475\linewidth]{figures/ccdf_totalVotes_notMisleading_trustworthySources}
\\%
\end{tabular}%
} \hfill
\subfloat[Misleading]{\label{fig:ccdf_grouped_derived_emotions}
\begin{tabular}[b]{@{}ll@{}}
\includegraphics[width=.475\linewidth]{figures/ccdf_ratioHelpful_misleading_trustworthySources.pdf}&
\includegraphics[width=.475\linewidth]{figures/ccdf_totalVotes_misleading_trustworthySources.pdf}
\\%
\end{tabular}%
} \hfill
\caption{Complementary cumulative distribution functions (CCDFs) for helpfulness ratio and total votes. Here we distinguish between Birdwatch notes for which the author has answered ``Yes'' or ``No'' to the question ``Did you link to sources you believe most people would consider trustworthy?''
}
	\label{fig:ccdf_trustworthy}
\end{figure}

\begin{figure
	\centering
	{\includegraphics[width=\linewidth]{figures/barplot_helpfulness_misleading.pdf}}
	\caption{Number of ratings per checkbox answer option in response to the question ``What about this note was helpful to you?''}
	\label{fig:reasons_helpful}
\end{figure}

\textbf{Why do users find Birdwatch notes unhelpful? } Fig.~\ref{fig:reasons_unhelpful} shows that users find Birdwatch notes unhelpful (i) because of opinion speculation or bias (\SI{25}{\percent}), (ii) if sources are missing or unreliable (\SI{20}{\percent}), (iii) if key points are missing (\SI{15}{\percent}), (iv) if the Birdwatch note is argumentative or inflammatory (\SI{13}{\percent}), or (v) if the Birdwatch note is incorrect (\SI{10}{\percent}). In some cases, users also perceive a Birdwatch note as unhelpful because it is off-topic (\SI{6}{\percent}) or hard to understand (\SI{3}{\percent}). Only few Birdwatch notes are perceived as unhelpful because of spam harassment (\SI{3}{\percent}), outdated information (\SI{2}{\percent}), or other reasons (\SI{3}{\percent}).

\begin{figure
	\centering
	{\includegraphics[width=\linewidth]{figures/barplot_helpfulness_notMisleading.pdf}}
	\caption{Number of ratings per checkbox answer option in response to the question ``Help us understand why this note was unhelpful.''}
	\label{fig:reasons_unhelpful}
\end{figure}

\subsection{Regression Model for Helpfulness (RQ4 \& RQ5)}

We now address the question of ``what makes a Birdwatch note helpful?''
To address this question, we employ regression analyses using helpfulness as the dependent variable and (i) the user categorization, (ii) the content characteristics of the text explanation, (iii) the social influence of the source tweet as explanatory variables.

\textbf{Model specification: } Following previous research modeling helpfulness \cite[\eg][]{Yin.2016}, we model the number of helpful votes, $HVotes$, as a binomial variable with probability parameter $\theta$ and $Votes$ trials:

\vspace{-.7\baselineskip}
{\small
\begin{flalign}
\logit&(\theta) =  \,\beta_0
	+ \underbrace{\beta_1 \,\mathit{Misleading}
	+ \beta_2 \,\mathit{Trustworthy Sources}}_{\text{User categorization}} \label{eq:theta}  \\
	&+ \underbrace{\beta_3 \,\mathit{Text Complexity}
	+ \beta_4 \,\mathit{Sentiment}
	+ \beta_5 \,\mathit{Word Count}}_{\text{Text explanation}}  \nonumber  \\
	&+ \underbrace{\beta_6 \,\mathit{Account Age}
	+ \beta_7 \,\mathit{Followers}

	+ \beta_8 \,\mathit{Followees}
	+ \beta_{9} \,\mathit{Verified}}_{\text{Source tweet}} + \, \varepsilon ,  \nonumber
\end{flalign}
\begin{flalign}
HVotes \sim Binomial[Votes, \theta], & \hskip0.22\textwidth \label{eq:binom}
\end{flalign}
}
\normalsize
with intercept $\beta_0$ and error term $\varepsilon$. We estimate Eq.~\ref{eq:theta} and Eq.~\ref{eq:binom} using maximum likelihood estimation and generalized linear models \cite{Wooldridge.2010}. To facilitate the interpretability of our findings, we z-standardize all variables, so that we can compare the effects of regression coefficients on the dependent variable measured in standard deviations.

We use the same set of explanatory variables as in Eq.~\ref{eq:theta} to model the total number of votes (helpful and unhelpful). The dependent variable is $Votes$, which results in the model

\vspace{-.7\baselineskip}
{\small
\begin{flalign}
\log&({\mathup{E}(Votes \mid ^*)})  = \,\beta_0
	+ \underbrace{\beta_1 \,\mathit{Misleading}
	+ \beta_2 \,\mathit{Trustworthy Sources}}_{\text{User categorization}} \label{eq:regression_votes}  \\
	&+ \underbrace{\beta_3 \,\mathit{Text Complexity}
	+ \beta_4 \,\mathit{Sentiment}
	+ \beta_5 \,\mathit{Word Count}}_{\text{Text explanation}}  \nonumber  \\
	&+ \underbrace{\beta_6 \,\mathit{Account Age}
	+ \beta_7 \,\mathit{Followers}

	+ \beta_8 \,\mathit{Followees}
	+ \beta_{9} \,\mathit{Verified}}_{\text{Source tweet}} + \, \varepsilon ,  \nonumber

\end{flalign}
}
\normalsize
with intercept $\beta_0$ and error term $\varepsilon$. We estimate Eq.~\ref{eq:regression_votes} via a negative binomial regression with log-transformation. The reason is that the number of votes denotes count data with overdispersion (\ie, variance larger than the mean).

\textbf{Coefficient estimates: } The parameter estimates in Fig.~\ref{fig:regression_helpfulness} show that Birdwatch notes reporting misleading tweets are significantly more helpful. The coefficient for \emph{Misleading} is \num{0.539} ($p<0.001$), which implies that the odds of Birdwatch notes reporting misleading tweets are \SI{71.43}{\percent} ($e^{0.539} \approx 1.71$) higher than for Birdwatch notes reporting not misleading tweets. Similarly, we find that the odds for Birdwatch notes providing trustworthy sources are \SI{100.77}{\percent} higher than for Birdwatch notes that do not provide trustworthy sources (coef: 0.697; $p<0.001$). We also find that content characteristics of Birdwatch notes are important determinants regarding their helpfulness: the coefficients for \emph{Text complexity}, \emph{Word count}, and \emph{Sentiment} are positive and statistically significant ($p<0.001$). Hence, Birdwatch notes are estimated to be more helpful if they embed positive sentiment, and if they use more words and more complex language. A one standard deviation change in the explanatory variable increases the odds of a helpful vote by \SI{15.03}{\percent} for \emph{Text complexity}, \SI[retain-explicit-plus]{18.41}{\percent} for \emph{Word count}, and \SI[retain-explicit-plus]{20.68}{\percent} for \emph{Sentiment}. We also find that the social influence of the account that has posted the source tweet plays an important role regarding the helpfulness of Birdwatch notes. Here the largest effect size is estimated for the number of followers. A one standard deviation change in this variable decreases the odds of a helpful vote by \SI[retain-explicit-plus]{24.34}{\percent}. Hence, there is a lower level of consensus for tweets from high-follower accounts.

We observe a different pattern for the negative binomial model with the total number of votes (helpful and unhelpful) as the dependent variable. Here we find a negative and statistically significant coefficient for \emph{Account age} ($p<0.001$) and positive and statistically significant coefficients for \emph{Verified} and \emph{Followers} ($p<0.001$). A one standard deviation increase in the \emph{Account age} is expected to decrease the total number of votes by \SI{14.10}{\percent}. A one standard deviation in the number of followers increase is expected to increase the total number of votes by \SI{117.28}{\percent}. Birdwatch notes reporting tweets from verified accounts are expected to receive \SI{47.11}{\percent} more votes.
Altogether, we thus find that the total number of votes primarily depends on the social influence of the source tweet rather than the characteristics of the Birdwatch note itself. In contrast, the odds of a Birdwatch note to be perceived as helpful depends on both the social influence of the source tweet and the characteristics of the Birdwatch note. Birdwatch notes reported for accounts with high social influence receive more total votes but are less likely to be perceived as being helpful.

\begin{figure}
    \centering
        \centering
				{\includegraphics[width=\linewidth]{figures/regression_coef_helpful_votes}}

				\caption{Regression results for helpfulness ratio and total votes as dependent variables (DV). Reported are standardized parameter estimates and \SI{95}{\percent} confidence intervals.}
				\label{fig:regression_helpfulness}
\end{figure}

\begin{figure*
\captionsetup{position=top}
				\centering

				\subfloat[Subcategories: Helpful]{\includegraphics[width=.95\linewidth]{figures/regression_coef_helpful_subcategories}\label{fig:regression_subcategories_helpful}}

				\\
				\subfloat[Subcategories: Unhelpful]{\includegraphics[width=.95\linewidth]{figures/regression_coef_notHelpful_subcategories}\label{fig:regression_subcategories_NotHelpful}}

				\caption{Regression results for subcategories of (a) helpfulness and (b) unhelpfulness as dependent variables (DV). Reported are standardized parameter estimates and \SI{95}{\percent} confidence intervals.}
	\label{fig:regression_subcategories}
\end{figure*}

\textbf{Analysis for subcategories of helpfulness: } We repeated our regression analysis for different subcategories of helpfulness. For this purpose, we estimate the effect of the explanatory variables on the frequency of the responses of Birdwatch users to the question ``What about this note was helpful/unhelpful to you?'' The coefficient estimates in Fig.~\ref{fig:regression_subcategories} show that Birdwatch notes reporting misleading tweets are more informative but not necessarily more empathetic. We also find that these Birdwatch notes are less likely to be spam harassment, opinion speculation or hard to understand. We again find that linking to trustworthy sources is crucial regarding helpfulness. Unsurprisingly, the largest effect size is observed for \emph{Trustworthy sources} on the dependent variable \emph{Good sources}. However, linking trustworthy sources also makes it more likely that a Birdwatch note is perceived to provide unique context, to be informative, to be empathetic, and to be clear. We find that Birdwatch notes that are longer, more complex, and embed a more positive sentiment in the text explanation are perceived as more helpful across almost all subcategories of helpfulness. Also, more positive sentiment in the text explanations makes Birdwatch notes less likely to be perceived as argumentative. Our analysis further suggests reasons because of which fact checks for high-follower accounts are perceived as less helpful: Birdwatch notes for these accounts are more likely to be seen as being incorrect or opinion speculation.

\textbf{Model checks: } We conducted several checks to validate the robustness of our results: (1) We followed common practice in regression analysis and checked that variance inflation factors as an indicator of multicollinearity were below five. This check led to the desired outcome. (2)~We controlled for non-linear relationships by adding quadratic terms for each explanatory variable to our regression models. (3) We estimated separate regressions for misleading vs. not misleading tweets. In all cases, our results are robust and consistently support our findings.

\section{Discussion}

\textbf{Fact-checking guidelines: } Our findings can help to formulate guidelines for users on how to write more helpful fact checks. Our analysis suggests that a relatively high share of community-created fact checks are already perceived as being informative, clear and helpful. Here we find that it is most important to encourage users to provide \emph{context} in their text explanations. Users should be encouraged to link trustworthy sources in their Birdwatch notes. Furthermore, we show that content characteristics matter. Here, we find that Birdwatch notes are estimated to be more helpful if they embed more positive vs. negative emotions. Users should thus avoid employing inflammatory language. We also find that Birdwatch notes that use more words and complex language are perceived as more helpful. It should thus be encouraged that users provide profound explanations and exhaust the full character limit to explain why they perceive a tweet as being misleading or not misleading.

\textbf{Challenges: } There are many challenges involved in building a community-driven system for fact-checking social media posts. Birdwatch will only be effective if the tweet context it produces is found to be helpful and appropriate by a wide range of people with diverse views. Yet our analysis suggests that the high level of polarization and biased views among the user base poses a major challenge. A substantial share of Birdwatch notes is perceived as being opinion speculation or bias. We also observe that the level of consensus between users is significantly lower if misinformation is reported for tweets from high-follower accounts (\ie, users with high social influence). Further challenges may arise when it comes to making Birdwatch resistant to manipulation attempts and harassment. For instance, conspiracists may denounce attempts to debunk false information as acts of misinformation. However, our analysis shows that -- up to now -- only a relatively small share of community-created fact checks are perceived as being spam harassment or abuse of the fact-checking feature.

\textbf{Avenues for future research: }
More research is necessary to address the above challenges regarding polarization and biased views among Twitter users. Here future research should conduct additional analyses to better understand \emph{which} groups of users engage in fact-checking and the accuracy of the community-created fact checks. In addition, it will be of utmost importance to understand the effect the Birdwatch feature has on the user behavior on the overall social media platform (\ie, Twitter). It has to be ensured that the Birdwatch feature does not yield adverse side effects. Previous experience of Facebook with ``flags'' suggests that adding warning labels to some posts can make all other (unverified) posts seem more credible \cite{BBC.2017}. A possible reason is that users might have a false sense of security because they might assume something was fact-checked when it wasn't at all.
Further research is thus necessary to understand the effects of community-created fact checks on the diffusion of both the fact-checked tweet itself as well as other misleading information circulating on Twitter.

\section{Conclusion}

Twitter has recently launched Birdwatch, a community-driven approach to identify misinformation on social media by harnessing the ``wisdom of crowds.'' This study is the first to empirically analyze how users interact with this new feature.
We find that encouraging users to provide \emph{context} (\eg, by linking trustworthy sources) and avoid the use of inflammatory language is crucial for users to perceive community-based fact checks as helpful.
However, despite showing promising potential, our analysis also suggests that Birdwatch's community-driven approach faces challenges concerning opinion speculation, biased views, and polarized opinions among the user base -- in particular with regards to influential accounts with high numbers of followers. Our findings are relevant both for the Birdwatch platform and future attempts to implement community-based approaches to combat misinformation on social media.

\begin{acks}
We thank Twitter for granting us access to their data.
\end{acks}

\bibliographystyle{ACM-Reference-Format-no-doi-abbrv}
\balance