text:
\section{Introduction}

\subsection{Outcome misclassification--a major source of bias in randomized experiments}\label{subsec: introduction}

For inferring causal effects of a treatment (intervention), conducting a randomized experiment (trial) is the gold standard. In a randomized experiment, treatments are randomly assigned according to some randomization design (i.e., assignment mechanism) and downstream statistical inference can be conducted based only on randomization (i.e., randomization-based inference) without imposing any super-population models on the subjects (e.g., assuming that subjects are i.i.d. realizations from some data-generating models); see \citet{rosenbaum2002observational, rosenbaum2010design}, \citet{imbens2015causal}, and \citet{athey2017econometrics} for detailed discussions. However, it is well-known that although randomization can make a causal conclusion immune to potential bias caused by confounders (measured or unmeasured) or model misspecification, it cannot remove potential bias resulting from measurement error in outcome (\citealp{arnold2016negative}; Chapter 9; \citealp{hernan2020causal}), which is typically called \textit{outcome misclassification} when the outcome is binary (\citealp{carroll2006measurement, buonaccorsi2010measurement, yi2017statistical}).

Outcome misclassification commonly exists in randomized experiments (trials). For example, common outcomes in randomized clinical trials are cancer diagnosis results (cancerous versus non-cancerous), blood test results (normal versus abnormal), antibody test results (positive versus negative), among many others. It is widely recognized that these binary clinical outcomes in practice can be misclassified (misdiagnosed) due to one or several of the following factors: 1) technical factors such as the limitations and imperfection of the diagnosis equipment and methods (\citealp{wittram2004ct}); 2) physician-side factors such as misinterpretations of the clinical data or patient's symptoms (\citealp{walsh1995clinical}); 3) patient-side factors such as patients' abnormal activities before the diagnosis and anxiety during the diagnosis (\citealp{ogedegbe2008misdiagnosis}). Another major type of outcome misclassification is reporting bias in self-reported binary outcomes, especially those concerning sensitive topics such as mental illness (\citealp{knauper1994diagnosing}), sexuality variables (\citealp{catania1986questionnaire}), or unhappiness with a service or product (\citealp{wood2008empirical}).

In general, there are at least two ways that outcome misclassification, in particular systematic outcome misclassification (i.e., differential outcome misclassification by treatment status), can severely bias a causal conclusion drawn from a randomized experiment (trial): 1) In randomized trials without or with inadequate blinding, systematically larger effects over subjective outcomes (either subject-reported or investigator-assessed) often exist due to differential outcome misclassification from knowledge of treatment status, as found by a large systematic review of clinical trials (\citealp{wood2008empirical}). Such exaggerated reporting of health improvements documented in previous non-blinded trials can be due to either placebo effects (i.e., treated subjects tend to overstate health improvements due to psychological factors) or courtesy bias (i.e., treated subjects tend to not fully state their unhappiness with the treatment as an attempt to be polite toward the investigators). However, blinding is not practical in a wide range of randomized experiments and therefore bias due to outcome misclassification may be inevitable for many such studies (\citealp{arnold2016negative}). 2) In some studies, receiving the treatment can help or prevent the detection of the outcome. For example, some clinical trials have found that the treatments they investigated can change the volume of the study organ or the pattern of the study tumor, therefore diseases may be easier or harder to be detected among treated subjects than control subjects in such cases (\citealp{lucia2007finasteride, redman2008finasteride}). That is, even if blinding was effectively carried out in a randomized trial, potential bias due to systematic outcome misclassification may still exist.

\subsubsection{Example: a puzzle from the Prostate Cancer Prevention Trial (PCPT)}\label{subsubsec: PCPT}

Prostate cancer is one of the most common cancers in men. According to \citet{siegel2020colorectal}, in the United States, prostate cancer is estimated to account for 21\% of the new cancer cases diagnosed in men in 2020. Since the development of prostate cancer is a long-term process, many studies have focused on the prevention of prostate cancer. Among these studies, the Prostate Cancer Prevention Trial (PCPT) (\citealp{thompson2003influence}) is especially influential because it is ``the first study to show that a drug can reduce a man's chances of developing prostate cancer" (\citealp{nih}). In the PCPT, within each of the 221 study sites, men 55 years of age or older with no evidence of prostate cancer are randomly assigned to finasteride (5 mg per day) (i.e., the treatment) or placebo (i.e., control). The primary outcome is whether the man is diagnosed with prostate cancer throughout the following seven years. Of the 9060 men included in the final analysis, 803 of the 4368 in the finasteride group and 1147 of the 4692 in the placebo group were diagnosed with prostate cancer (any grades); see Table 1 in \citet{thompson2003influence}. Applying the Mantel-Haenszel test (randomization-based inference) to the 221 strata (study sites), the two-sided p-value under Fisher's sharp null of no treatment effect is $4.66 \times 10^{-13}$. According to the two-sided level 0.05 prespecified by the PCPT (\citealp{thompson2003influence}), a treatment effect of finasteride on the prevention of prostate cancer was detected.

Meanwhile, the PCPT also identified a controversial and seemingly contradictive phenomenon: taking finasteride may increase the risk of high-grade prostate cancer (i.e., tumor with Gleason score $\geq 7$). Specifically, of the 9017 men with available Gleason score, 280 of the 4350 in the finasteride group and 236 of the 4667 the placebo group were diagnosed with high-grade prostate cancer. Applying the Mantel-Haenszel test, the two-sided p-value under Fisher's sharp null is $6.79 \times 10^{-3}$, which is also statistically significant at the prespecified level 0.05. Therefore, taking finasteride was shown to increase the risk of high-prostate cancer.

Can finasteride prevents prostate cancer but meanwhile promotes high-grade prostate cancer? This seeming contradiction puzzled many scholars when the results were first published in \textit{New England Journal of Medicine} in 2003 (\citealp{thompson2003influence}). Several follow-up studies have pointed out that this puzzle may be due to the potential bias caused by misclassification of the prostate cancer status or severity (e.g., \citealp{lucia2007finasteride, redman2008finasteride, shepherd2008does}). So a natural question is: can we develop a model-free causal inference framework to help explain this puzzle from the perspective of outcome misclassification?

\subsection{Our contributions}\label{subsec: contributions}

Although outcome misclassification commonly exists in randomized experiments (trials) and may severely distort a downstream causal inference, it has been routinely ignored in practical research. To the best of our knowledge, until now there is no unified framework for addressing outcome misclassification in randomized experiments without imposing any additional assumptions to a randomized experiment. Although there is extensive previous literature on addressing outcome misclassification in statistical inference in causal or association studies, all of these previous approaches require assuming that subjects are realizations (typically i.i.d. realizations) from some super-population models--either parametric, semiparametric, or nonparametric models. For representative works, see \citet{quade1980effects}, \citet{magder1997logistic}, \citet{lyles2005extending}, \citet{carroll2006measurement}, \citet{kuchenhoff2006general}, \citet{shepherd2008does}, \citet{buonaccorsi2010measurement}, \citet{gilbert2016misclassification}, \citet{yi2017statistical}, \citet{beesley2020statistical}, among many others. These model-based approaches have been successfully applied in various settings and greatly contributed to practical research. However, directly applying or adjusting these model-based approaches to randomized experiments may introduce unnecessary bias to a model-free randomization-based inference. On the one hand, for an approach based on some parametric or semiparametric model, the statistical inference may be biased due to model misspecification (\citealp{magder1997logistic}). On the other hand, even if an approach adopts a flexible nonparametric model, there may still exist potential bias due to: 1) The existence of unmeasured features that are involved in the outcome misclassification mechanism. For example, whether a patient's disease is correctly diagnosed or not highly depends on a doctor's expertise and experience on such disease, which may be difficult to measure or quantify (\citealp{weiss2003empirical}). 2) The violation of the i.i.d. assumption. For example, laboratories routinely calibrate their measuring instruments using the results and values of previously measured outcomes (\citealp{buonaccorsi2010measurement}). In such cases, whether some subject's outcome is correctly classified or not may depend on other subjects' outcomes, making the i.i.d. assumption unrealistic. Therefore, new ideas that intrinsically differ from all of these model-based approaches are necessary for many randomized experiments.

In this paper, we develop a model-free and finite-population-exact framework for randomized experiments subject to outcome misclassification via integer programming. Our framework provides a unified approach to address the following four common questions concerning the design, analysis, and validation of a randomized experiment:
\begin{itemize}
\item \textbf{Design Stage:} \textbf{Q1}--Given the planned randomization design, the planned sample size, and some assumed effect size, how accurate is accurate enough for the outcome measurement if we would like to ensure that the causal conclusion based on the measured outcomes agrees with that based on the true outcomes?

\item \textbf{Analysis Stage:} After obtaining the measured outcomes and getting a causal conclusion based on them, two central questions are: \textbf{Q2}--How sensitive is the causal conclusion drawn from the measured outcomes to outcome misclassification? \textbf{Q3}--Is that causal conclusion more sensitive to false positives vs. false negatives among the treated vs. control subjects?

\item \textbf{Validation Stage:} \textbf{Q4}--If researchers plan to conduct some validation studies on outcome misclassification, how to choose a validation subset among the study subjects to make a validation study more efficient?
\end{itemize}

A central concept of our framework is called \textit{warning accuracy}, which is defined as the finite-population-exact threshold such that the causal conclusion based on the measured outcomes may differ from that based on the true outcomes if the accuracy of the measured outcomes did not surpass that threshold. We show how warning accuracy, its related information \textit{sensitivity weights}, and its dual concept \textit{design accuracy} can help address \textbf{Q1--4} without adding any additional assumptions to a randomized experiment. To handle the computational difficulty encountered in computing warning accuracy for large-scale experiments called the ``curse of symmetry," we propose a computational approach to adaptively formulate a corresponding integer programming problem with respect to the randomization design. Our computational approach leverages some intrinsic features of various randomization designs and recent advances in erasing symmetry in integer programming (\citealp{fogarty2016discrete, fogarty2017randomization}), which can be of independent interest. Our framework covers both Fisher's sharp null and Neyman's weak null and works for a wide range of randomization designs (i.e., assignment mechanisms) in randomized experiments, including completely randomized experiments, stratified and finely stratified experiments, paired experiments, and so on. Our framework can also be applied to matched or stratified observational studies that adopt randomization-based inference. We illustrate our new framework through studying the well-known puzzle in the PCPT introduced in Section~\ref{subsubsec: PCPT}.

One important remark is that all model-based approaches in the previous literature on outcome misclassification can still be very useful as long as the models and assumptions are appropriately imposed, and researchers can still do further model-based approaches after adopting our framework. The central spirit of our framework is: without making any additional assumptions to a randomized experiment, or before imposing any additional assumptions to a randomized experiment to conduct a model-assisted analysis, what useful information concerning outcome misclassification can we learn from the data?

\section{Review}

\subsection{Randomization-based inference with binary outcome}

Suppose that there are $I\geq 1$ strata (or blocks, or study centers), and there are $n_{i}$ subjects in stratum $i$, $i=1,\dots,I$, so $N=\sum_{i=1}^{I}n_{i}$ subjects in total. Let $Z_{ij}$ be the treatment indicator of subject $j$ in stratum $i$: $Z_{ij}=1$ if subject $j$ in stratum $i$ received treatment and $Z_{ij}=0$ otherwise. Suppose that in stratum $i$ a fixed number $m_{i}$ of subjects are designed to receive the treatment, and $n_{i}-m_{i}$ subjects receive controls, i.e, $\sum_{j=1}^{n_{i}}Z_{ij}=m_{i}$ where $m_{i}\in \{ 1,\dots, n_{i}-1\}$. Let $\mathbf{Z}=(Z_{11},\dots, Z_{In_{I}})$ and $\mathcal{Z}=\{ \mathbf{Z}\in \{0, 1\}^{N}:\sum_{j=1}^{n_{i}}Z_{ij}=m_{i}, i=1,\dots, I\}$ denote all possible treatment allocations. In a randomized experiment, the treatment assignments are random within each stratum (physical randomization), i.e.,
\begin{equation}\label{eqn: randomization assumption}
    P(\mathbf{Z}=\mathbf{z}\mid \mathcal{Z})=\prod_{i=1}^{I}{n_{i}\choose m_{i}}^{-1}, \quad \forall \mathbf{z}\in \mathcal{Z}.
\end{equation}
In an observational study, the randomization assumption (\ref{eqn: randomization assumption}) is often assumed after adjusting for covariates using matching or subclassification (\citealp{rosenbaum2002observational}). The above notations cover a wide range of randomized experiments and observational studies. When $I=1$, the study is called a completely randomized experiment. For $I\geq 2$, the study is a general stratified or blocked randomized experiment or observational study. When $n_{i}=2$ for all $i$, the study is a paired randomized experiment or pair-matched observational study. When $m_{i}=1$ and $n_{i}=n\geq 3$ for all $i$, the study is a randomized experiment or observational study with multiple controls. When $m_{i}=1$ for all $i$ while $n_{i}$ may vary with $i$, the study is a randomized experiment or observational study with variable controls. When $\min\{m_{i}, n_{i}-m_{i}\}=1$ for all $i$, the study is a finely stratified experiment (\citealp{fogarty2018mitigating}) or an observational study with full matching (\citealp{rosenbaum1991characterization, hansen2004full}). See \citet{imbens2015causal} and \citet{rosenbaum2002observational, rosenbaum2010design} for detailed introductions and definitions of various types of randomized experiments/trials and observational studies.

 Let $Y_{ij} \in \{0,1\}$ be the true outcome of subject $ij$ and $\mathbf{Y}=(Y_{11}, \dots, Y_{In_{I}})$. Following the potential outcomes framework (\citealp{neyman1923application, rubin1974estimating}), let $Y_{ij}(1)$ and $Y_{ij}(0)$ be the potential (true) outcome under treated and that under control of subject $j$ in stratum $i$ respectively, therefore $Y_{ij}=Y_{ij}(1)Z_{ij}+Y_{ij}(0)(1-Z_{ij})$. In randomization-based inference for randomized experiments or observational studies, potential outcomes are fixed values and the only probability distribution that enters statistical inference is the randomization assumption (\ref{eqn: randomization assumption}) (\citealp{rosenbaum2002observational, imbens2015causal, athey2017econometrics}). Fisher's sharp null hypothesis of no treatment effect (\citealp{fisher1937design}) asserts that
\begin{equation*}
    H_{0, \text{sharp}}: Y_{ij}(1)=Y_{ij}(0), \quad \forall i, j.
\end{equation*}
The commonly used test statistics in randomization-based inference with binary outcome are Fisher's exact test (when $I=1$) and the Mantel-Haenszel test (when $I\geq 1$), which is defined as $T_{\text{M-H}}(\mathbf{Z}, \mathbf{Y})=\sum_{i=1}^{I}\sum_{j=1}^{n_{i}}Z_{ij}Y_{ij}$ (\citealp{mantel1959statistical}). The Mantel-Haenszel test reduces to Fisher's exact test when $I=1$ and reduces to McNemar's test when $n_{i}=2$ for all $i$ (\citealp{cox2018analysis}). When $N$ is sufficiently large, the following finite-population central limit theorem holds under $H_{0, \text{sharp}}$ under mild conditions (\citealp{hajek1960limiting, li2017general}):
\begin{equation*}
    \frac{T_{\text{M-H}}-E(T_{\text{M-H}})}{\sqrt{\text{Var}(T_{\text{M-H}}) }}\xrightarrow{\mathcal{L}} N(0,1),
\end{equation*}
where, under $H_{0, \text{sharp}}$,
\begin{equation*}
    E(T_{\text{M-H}})=\sum_{i=1}^{I}\Big(\frac{m_{i}}{n_{i}}\sum_{j=1}^{n_{i}}Y_{ij}\Big), \ \text{Var}(T_{\text{M-H}})=\sum_{i=1}^{I}\frac{m_{i}(\sum_{j=1}^{n_{i}}Y_{ij})(n_{i}-\sum_{j=1}^{n_{i}}Y_{ij})(n_{i}-m_{i})}{n_{i}^{2}(n_{i}-1)}.
\end{equation*}
In a two-sided level-$\alpha$ testing procedure, we reject $H_{0, \text{sharp}}$ if and only if $\frac{\{T_{\text{M-H}}-E(T_{\text{M-H}})\}^{2}}{\text{Var}(T_{\text{M-H}})}>\chi^{2}_{1, 1-\alpha}$, where $\chi^{2}_{1, 1-\alpha}$ is $1-\alpha$ quantile of chi-squared distribution with one degree of freedom.

Another extensively considered null hypothesis is Neyman's weak null hypothesis of no average treatment effect (\citealp{neyman1923application}):
\begin{equation*}
    H_{0, \text{weak}}:\frac{1}{N} \sum_{i=1}^{I}\sum_{j=1}^{n_{i}}(Y_{ij}(1)-Y_{ij}(0))=0.
\end{equation*}
Let $\tau_{i}=n_{i}^{-1}\sum_{j=1}^{n_{i}}Y_{ij}(1)-n_{i}^{-1}\sum_{j=1}^{n_{i}}Y_{ij}(0)$, $H_{0, \text{weak}}$ can be rewritten as $H_{0, \text{weak}}: \sum_{i=1}^{I}\frac{n_{i}}{N}\cdot \tau_{i}=0$. A commonly used test statistic for testing $H_{0, \text{weak}}$ is the Neyman estimator (also known as the difference-in-means estimator) (\citealp{neyman1923application})
\begin{equation*}
    T_{\text{Neyman}}(\mathbf{Z}, \mathbf{Y})=\sum_{i=1}^{I}\frac{n_{i}}{N} \cdot \hat{\tau}_{i}, \text{ where $\hat{\tau}_{i}=\frac{1}{m_{i}}\sum_{j=1}^{n_{i}}Z_{ij}Y_{ij}-\frac{1}{n_{i}- m_{i}}\sum_{j=1}^{n_{i}}(1-Z_{ij})Y_{ij}$.}
\end{equation*}
Under $H_{0, \text{weak}}$, we have $E(T_{\text{Neyman}})=0$ and $\text{Var}(T_{\text{Neyman}})=\sum_{i=1}^{I} (n_{i}/N)^{2}\cdot \text{Var}(\hat{\tau}_{i})$, where
    $\text{Var}(\hat{\tau}_{i})=\frac{S_{t, i}^{2}}{m_{i}}+\frac{S_{c, i}^{2}}{n_{i}-m_{i}}-\frac{S_{ i}^{2}}{n_{i}}$, with $S_{t, i}^{2}=(n_{i}-1)^{-1}\sum_{j=1}^{n_{i}}\big(Y_{ij}(1)-n_{i}^{-1}\sum_{j=1}^{n_{i}}Y_{ij}(1)\big)^{2}, S_{c, i}^{2}=(n_{i}-1)^{-1}\sum_{j=1}^{n_{i}}\big(Y_{ij}(0)-n_{i}^{-1}\sum_{j=1}^{n_{i}}Y_{ij}(0)\big)^{2}$, and $S_{i}^{2}=(n_{i}-1)^{-1}\sum_{j=1}^{n_{i}}\big(Y_{ij}(1)-Y_{ij}(0)-\tau_{i} \big)^{2}$. When $N$ is sufficiently large, we have the following finite-population central limit theorem holds under mild conditions (\citealp{hajek1960limiting, li2017general}):
\begin{equation*}
    \frac{T_{\text{Neyman}}}{\sqrt{\text{Var}(T_{\text{Neyman}})}} \xrightarrow{\mathcal{L}} N(0,1).
\end{equation*}
Since $S_{i}^{2}$ involves $\tau_{i}$ which cannot be identified from the observed data, researchers typically adopt the following conservative estimator $\widehat{\text{Var}}(T_{\text{Neyman}})$ for $\text{Var}(T_{\text{Neyman}})$ (\citealp{neyman1923application}): $\widehat{\text{Var}}(T_{\text{Neyman}})=\sum_{i=1}^{I} (\frac{n_{i}}{N})^{2}\cdot (\frac{\widehat{S}_{t, i}^{2}}{m_{i}}+\frac{\widehat{S}_{c, i}^{2}}{n_{i}-m_{i}})=\sum_{i=1}^{I} (\frac{n_{i}}{N})^{2} \large(\frac{1}{m_{i}(m_{i}-1)}\sum_{j=1}^{n_{i}}Z_{ij}(Y_{ij}^{*}-\frac{1}{m_{i}}\sum_{j=1}^{n_{i}}Z_{ij}Y_{ij}^{*})^{2}+\frac{1}{(n_{i}-m_{i})(n_{i}-m_{i}-1)}\sum_{j=1}^{n_{i}}(1-Z_{ij})(Y_{ij}^{*}-\frac{1}{n_{i}-m_{i}}\sum_{j=1}^{n_{i}}(1-Z_{ij})Y_{ij}^{*})^{2})$. For example, in a two-sided $\alpha$-level testing procedure, we reject $H_{0, \text{weak}}$ if and only if we have $T_{\text{Neyman}}^{2}/\widehat{\text{Var}}(T_{\text{Neyman}})> \chi^{2}_{1, 1-\alpha}$. See \citet{imbens2015causal} for more details.

\subsection{Some basic concepts about outcome misclassification}

As discussed in Section~\ref{subsec: introduction}, in practice the measured outcomes $\mathbf{Y}^{*}=(Y_{11}^{*}, \dots, Y_{In_{I}}^{*})$ may be subject to misclassification (i.e., $\mathbf{Y}^{*} \neq \mathbf{Y})$, and a causal conclusion based on $\mathbf{Y}^{*}$ may differ from that based on the true outcomes $\mathbf{Y}$. In outcome misclassification and binary classification literature, the measured outcome $Y^{*}_{ij}$ is said to be a \textit{true positive} if $(Y^{*}_{ij}, Y_{ij})=(1, 1)$, a \textit{false positive} if $(Y^{*}_{ij}, Y_{ij})=(1, 0)$, a \textit{true negative} if $(Y^{*}_{ij}, Y_{ij})=(0, 0)$, or a \textit{false negative} if $(Y^{*}_{ij}, Y_{ij})=(0, 1)$. Let $TP, FP, TN$ and $FN$ denote the total number of subjects that lie in the above four categories respectively: $TP=|\{(i,j): (Y^{*}_{ij}, Y_{ij})=(1, 1)\}| =\sum_{i=1}^{I}\sum_{j=1}^{n_{i}}Y_{ij}^{*}Y_{ij}$, $FP=|\{(i,j): (Y^{*}_{ij}, Y_{ij})=(1, 0)\}|=\sum_{i=1}^{I}\sum_{j=1}^{n_{i}}Y_{ij}^{*}(1-Y_{ij})$, $TN=|\{(i,j): (Y^{*}_{ij}, Y_{ij})=(0, 0)\}|=\sum_{i=1}^{I}\sum_{j=1}^{n_{i}}(1-Y_{ij}^{*})(1-Y_{ij})$, $FN=|\{(i,j): (Y^{*}_{ij}, Y_{ij})=(0, 1)\}|=\sum_{i=1}^{I}\sum_{j=1}^{n_{i}}(1-Y_{ij}^{*})Y_{ij}$. One of the most fundamental and widely used measures of the precision of $\mathbf{Y}^{*}$ is \textit{accuracy}, which is defined as the proportion of correct classification among $\mathbf{Y}^{*}$ under the true outcomes $\mathbf{Y}$ (denoted as $\mathcal{A}(\mathbf{Y}^{*}\mid \mathbf{Y})$):
\begin{equation}\label{equa: warning accuracy}
    \mathcal{A}(\mathbf{Y}^{*}\mid \mathbf{Y})=\frac{TP+TN}{TP+FP+TN+FN}=\frac{\sum_{i=1}^{I}\sum_{j=1}^{n_{i}}\mathbf{1}(Y_{ij}^{*}=Y_{ij})}{N}.
\end{equation}
Note that accuracy $\mathcal{A}(\mathbf{Y}^{*}\mid \mathbf{Y})$ is a finite-population-exact concept and is therefore especially compatible with randomization-based inference framework which does not require assuming any super-population models on the study subjects.

\section{A Model-free and Finite-population-exact Framework}

In this section, we introduce several model-free and finite-population-exact quantities to help address the four common questions \textbf{Q1--4} listed in Section~\ref{subsec: contributions}. In Section~\ref{subsec: introducing the warning accuracy}, we give the motivations, definitions, and explanations of those quantities. In Section~\ref{subsec: usages of all the concepts}, we conclude how to use those quantities to address \textbf{Q1--4}.

\subsection{Warning accuracy, sensitivity weights, and design accuracy}\label{subsec: introducing the warning accuracy}

After obtaining a causal conclusion (i.e., rejecting null hypothesis or not) based on the measured outcomes (assuming no outcome misclassification), a natural question is: how many misclassified outcomes are needed at least to overturn that causal conclusion? We call such number as \textit{minimal alteration number}. Or equivalently, what is the threshold such that the causal conclusion based on the measured outcomes may differ from that based on the true outcomes if the accuracy of the measured outcomes did not surpass that threshold? We call this threshold the \textit{warning accuracy}, for which a rigorous definition is given below.
\begin{Definition}[Warning Accuracy]\label{def: warning accuracy}
Let $\mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y}^{*})$ and $\mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y})$ denote the level-$\alpha$ hypothesis testing result (rejecting the null hypothesis of no treatment effect or not) based on the measured outcomes $\mathbf{Y}^{*}$ and that based on the true outcomes $\mathbf{Y}$ respectively, and $\mathcal{A}(\mathbf{Y}^{*}\mid \mathbf{Y})$ the accuracy of $\mathbf{Y}^{*}$ under $\mathbf{Y}$. The warning accuracy given level $\alpha$, the treatment indicators $\mathbf{Z}$, and the measured outcomes $\mathbf{Y}^{*}$, is defined as
\begin{equation}\label{eqn: warning accuracy}
    \mathcal{WA}=\max_{\mathbf{Y}: \mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y}^{*})\neq \mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y}) }\mathcal{A}(\mathbf{Y}^{*}\mid \mathbf{Y}).
\end{equation}
Then the corresponding minimal alteration number is $(1-\mathcal{WA})\times N$.
\end{Definition}
We here give some remarks on the warning accuracy $\mathcal{WA}$ defined in Definition~\ref{def: warning accuracy}. First, it immediately follows from Definition~\ref{def: warning accuracy} that if the (unknown) actual outcome accuracy $\mathcal{A}(\mathbf{Y}^{*}\mid \mathbf{Y}) > \mathcal{WA}$, we have $\mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y}^{*})=\mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y})$ while if $\mathcal{A}(\mathbf{Y}^{*}\mid \mathbf{Y}) \leq  \mathcal{WA}$, it may happen that $\mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y}^{*})\neq \mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y})$. Therefore, other things being equal, a smaller warning accuracy $\mathcal{WA}$ indicates less sensitivity to outcome misclassification. Second, the warning accuracy is model-free and finite-population-exact as it does not require assuming that subjects are realizations from some super-population model.

We now use two simple examples to illustrate why reporting the warning accuracy, as a complement of the p-value based on measured outcomes, can greatly benefit a randomized experiment (or an observational study using randomization-based inference) subject to outcome misclassification.

\begin{Example}\label{exp: 1}
Consider a complete randomized experiment with one treated subject and 1000 controls. Without loss of generality, suppose that the treatment indicators $\mathbf{Z}=(1,0,\dots,0)$. Assume that the corresponding measured outcomes $\mathbf{Y}^{*}=(1,0,\dots,0)$. Then the p-value under Fisher's sharp null based on $\mathbf{Y}^{*}=(1,0,\dots,0)$ is 1/1001<0.001, which will be considered as very strong evidence of treatment effect for many scientific journals. However, if the true outcomes $\mathbf{Y}=(0, 0, \dots, 0)$, the true p-value will be one (no evidence), even if $\mathbf{Y}^{*}$ and $\mathbf{Y}$ only differ by one case of misclassification. In this case, the warning accuracy is $1000/1001$, implying high sensitivity to outcome misclassification, even if the p-value based on $\mathbf{Y}^{*}$ is very statistically significant.
\end{Example}
\begin{Example}\label{exp: 2}
We consider the following two stratified randomized experiments with equal total sample size and the same prespecified level of 0.05 (one-sided). It is easy to check that study 1 has a smaller one-sided p-value but larger warning accuracy while study 2 has a larger one-sided p-value but smaller warning accuracy.
\begin{table}[H]
\begin{center}
\begin{tabular}{ c c c c c c }
   Study 1 & Stratum 1 & Stratum 2 & Stratum 3   & p-value & Warning Accuracy \\
 $\mathbf{Z}$ & (1 0 1) & (0 0 1 0 0 0) & (0 0 0 1 0 0 0 0)  & \multirow{2}{*}{1/144} & \multirow{2}{*}{16/17} \\
  $\mathbf{Y}^{*}$ & (1 0 1) & (0 0 1 0 0 0) & (0 0 0 1 0 0 0 0) &   &  \\
   Study 2 & Stratum 1 & Stratum 2 & Stratum 3 &  p-value & Warning Accuracy  \\
   $\mathbf{Z}$ & (1 0 0 0 0 0 0) & (1 1 0 1 0) & (0 0 1 1 0)  & \multirow{2}{*}{1/100} & \multirow{2}{*}{15/17} \\
  $\mathbf{Y}^{*}$ & (0 0 0 0 0 0 0) & (1 1 0 1 0) & (0 0 1 1 0)  &   &
 \end{tabular}\label{tab: toy example}
 \end{center}
\end{table}
\end{Example}
In summary, Example~\ref{exp: 1} shows that even a causal conclusion (based on measured outcomes) with a very small p-value can be very sensitive to outcome misclassification (exhibits high warning accuracy). Example~\ref{exp: 2} implies that a causal conclusion with a smaller p-value is not necessarily less sensitive to outcome misclassification than that with a larger p-value. Therefore, when outcome misclassification is a concern, reporting the warning accuracy can provide useful information about sensitivity to outcome misclassification. Such information cannot be covered by p-value based on measured outcomes and does not require any additional assumptions.

However, only reporting warning accuracy has two limitations: 1) Warning accuracy is a worst-case scenario sensitivity analysis for outcome misclassification. Without any additional information, there is nothing wrong with a worst-case scenario sensitivity analysis as it is universally valid. However, if there exists additional prior information or expert knowledge, only looking at warning accuracy may be conservative. 2) Warning accuracy itself does not provide any information about if a causal conclusion is in general more sensitive to a false positive versus a false negative in the treated versus the control group. We propose another concept called \textit{sensitivity weights} to overcome these two limitations, which is built on the definition of warning accuracy and the following concept called a \textit{sensitive set}. A sensitive set refers to a minimal (in terms of cardinality) set of subjects such that if the outcomes of those subjects were misclassified, the causal conclusion based on the measured outcomes will be overturned.
\begin{Definition}[Sensitive Sets]\label{def: sensitive sets}
Under the setting of Definition~\ref{def: warning accuracy}, let
\begin{equation*}
   \widetilde{\mathbf{Y}}=(\widetilde{Y}_{11}, \dots, \widetilde{Y}_{In_{I}})\in \argmax_{\mathbf{Y}: \mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y})\neq \mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y}^{*}) }\mathcal{A}(\mathbf{Y}^{*}\mid \mathbf{Y})
\end{equation*}
be an optimal solution to the optimization problem associated with the definition of warning accuracy. Then $\mathcal{S}=\{ij: \widetilde{Y}_{ij}\neq Y_{ij}^{*}\}$ is called a sensitive set.
\end{Definition}
The motivation of Definition~\ref{def: sensitive sets} is intuitive: to solve the optimization problem involved in Definition~\ref{def: warning accuracy}, an optimization solver has to identify the subjects such that the causal conclusion is especially sensitive to potential misclassified outcomes of those subjects and generate cases of misclassified outcomes for these subjects. Therefore, as suggested by its name, a sensitive set $\mathcal{S}$ is a collection of subjects whose outcomes being misclassified or not are particularly influential to the causal conclusion. In practice, there may exist more than one sensitive set as the solution to the optimization problem involved in Definition~\ref{def: warning accuracy} may not be unique. Therefore, a sensitive set itself may not be an intrinsic quantify. However, as we will show in Section~\ref{sec: computing}, for the most widely used tests such as the Mantel-Haenszel test and the difference-in-means estimator, these different sensitive sets can, in general, be transformed to each other by some decomposition of within-strata and between-strata permutations, and share the same quantities called \textit{sensitivity wights}--the key to answer \textbf{Q3} and \textbf{Q4}.

\begin{Definition}[Sensitivity Weights]\label{def: sensitivity weights}
Under the setting of Definition~\ref{def: warning accuracy}, let $\mathcal{S}$ be a sensitive set. Then there is a set of sensitivity weights defined as the following $2 \times 2$ table:
\begin{center}
    \begin{tabular}{ccc}
  \hline
  Sensitivity Weights & False Positives & False Negative   \\
  \hline
   Treated   & $w_{T, FP}$ & $w_{T, FN}$\\
   Control & $w_{C, FP}$ & $w_{C, FN}$   \\
 \hline
 \end{tabular}
\end{center}
where $w_{T, FP}$, $w_{T, FN}$, $w_{C, FP}$, and $w_{C, FN}$ denote the sensitivity weight of false positives in the treated group, that of false negatives in the treated group, that of false positives in the control group, and that of false negatives in the control group respectively, defined as
\begin{align*}
    &w_{T, FP}=\frac{|\{ij: Z_{ij}=1, Y_{ij}^{*}=1, \widetilde{Y}_{ij}=0\}|}{|\mathcal{S}|}, \quad w_{T, FN}=\frac{|\{ij: Z_{ij}=1, Y_{ij}^{*}=0, \widetilde{Y}_{ij}=1\}|}{|\mathcal{S}|},\\
    &w_{C, FP}=\frac{|\{ij: Z_{ij}=0, Y_{ij}^{*}=1, \widetilde{Y}_{ij}=0\}|}{|\mathcal{S}|}, \quad w_{C, FN}=\frac{|\{ij: Z_{ij}=0, Y_{ij}^{*}=0, \widetilde{Y}_{ij}=1\}|}{|\mathcal{S}|}.
\end{align*}
\end{Definition}
The rationale of sensitivity weights is straightforward: since the causal conclusion is particularly sensitive to potential cases of outcome misclassification among the subjects in a sensitive set, then the proportion of each of the four types of outcome misclassification (i.e., false positives/negatives in the treated/control group) within a sensitive set offers a sensible quantification about if the causal conclusion is in particular sensitive to certain type of outcome misclassification. For example, in Example~\ref{exp: 1}, it is clear that the sensitivity weight of a false positive in the treated group is the dominant term according to Definition~\ref{def: sensitivity weights}, suggesting that the causal conclusion is especially sensitive to a false positive in the treated group and therefore should be given priorities when conducting a validation study.

Not only can our framework benefit the analysis and validation of a randomized experiment or an observational study based on randomization-based inference, but also it can provide a benchmark for outcome accuracy at the design stage of such studies. Specifically, by introducing a dual concept of warning accuracy called \textit{design accuracy}, under each assumed effect size, we can calculate the expected threshold such that the causal conclusion based on the measured outcomes is guaranteed to agree with that based on the true outcomes as long as the outcome accuracy surpasses that threshold. We call such procedure in a design stage as \textit{accuracy calculation}.
\begin{Definition}[Design Accuracy]\label{def: Design Accuracy}
Let $\mathbf{Y}\sim (p_{0}, p_{1})$ denote that $Y_{ij}\mid Z_{ij}=0 \sim \text{Bern}(p_{0})$ and $Y_{ij}\mid Z_{ij}=1 \sim \text{Bern}(p_{1})$. The design accuracy given level $\alpha$, the treatment indicators $\mathbf{Z}$, and the assumed effect size $(p_{0}, p_{1})$, is defined as
\begin{equation*}
    \mathcal{DA}=E_{\mathbf{Y}\sim (p_{0}, p_{1})}\Big\{ \max_{\mathbf{Y}^{*}: \mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y}^{*})\neq \mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y}) }\mathcal{A}(\mathbf{Y}^{*}\mid \mathbf{Y})\Big\}.
\end{equation*}
\end{Definition}
As indicated in Definition~\ref{def: Design Accuracy}, design accuracy can be seen as a dual concept of warning accuracy. For warning accuracy, measured outcomes are given and we use an optimization solver to manipulate the unknown true outcomes to conduct a worst-case scenario sensitivity analysis. In contrast, for design accuracy, we generate each set of true outcomes according to some assumed effect size and manipulate measured outcomes for each generated set of true outcomes to conduct a worst-case scenario accuracy calculation. Ideally, we would like outcome measurement to be as accurate as possible. However, there is typically a trade-off between budget and outcome accuracy -- the more accurate the outcome measurement we would like to achieve, the more budget and resource we may need to spend on. Design accuracy provides a sensible benchmark for outcome accuracy without modeling the outcome misclassification mechanism.

\subsection{Strengthening a randomized experiment with warning accuracy, sensitivity weights, and design accuracy}\label{subsec: usages of all the concepts}

Putting all the messages delivered in Section~\ref{subsec: introducing the warning accuracy} together, we here conclude how to use warning accuracy, sensitivity weights, and design accuracy to strengthen the design, analysis, and validation of a randomized experiment (trial) subject to outcome misclassification.

\textbf{Design Stage:} To address \textbf{Q1}, design accuracy can serve as a worst-case scenario benchmark for outcome accuracy if we would like the causal conclusion based on measured outcomes to agree with that based on true outcomes. Specifically, after fixing the sample size and randomization design, researchers can specify various effect sizes $(p_{0}, p_{1})$ and generate 1000 simulated sets of true outcomes $\mathbf{Y}$ according to $Y_{ij} \mid Z_{ij}=0 \sim \text{Bern}(p_{0})$ and $Y_{ij} \mid Z_{ij}=1 \sim \text{Bern}(p_{1})$. Then researchers compute the 1000 corresponding optimal values of the following optimization problem:
\begin{equation*}
    \max_{\mathbf{Y}^{*}: \mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y}^{*})\neq \mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y}) }\mathcal{A}(\mathbf{Y}^{*}\mid \mathbf{Y}),
\end{equation*}
and take their average as the approximate design accuracy defined in Definition~\ref{def: Design Accuracy}. For various prespecified effect sizes $p_{0}$ and $p_{1}$, design accuracy can serve as a benchmark for how accurate their outcomes measurement should ideally be in order to ensure the causal conclusion based on measured outcomes and that drawn from true outcomes agree with each other.

\textbf{Analysis Stage:} To address \textbf{Q2}, after reporting the p-value based on the measured outcomes, researchers can then report the warning accuracy under the prespecified level. A low warning accuracy implies a causal conclusion's high insensitivity (robustness) to outcome misclassification. If the warning accuracy is relatively high instead, researchers should report the sensitivity weights and check if the dominant term among the four types of outcome misclassification (false positives/negatives in the treated/control group) agrees with that based on prior information and/or expert knowledge or not (we will illustrate such procedure in detail in Section~\ref{sec: PCPT analysis}). If this is the case, then it indeed can happen that the causal conclusion drawn from the measured outcomes may be misleading even if the unknown actual outcome accuracy equals or is close to that high warning accuracy. Otherwise, we would expect the actual accuracy needed to overturn the causal conclusion based on the measured outcomes to be lower, or even much lower, than that high warning accuracy. To address \textbf{Q3}, we report the four sensitivity weights and observe if there is any dominant term among the four sensitivity weights. Such a term would be a strong sign that the causal conclusion based on the measured outcomes is particularly sensitive to that type of outcome misclassification.

\textbf{Validation Stage:} If the warning accuracy is relatively high meanwhile the dominant term among the four sensitivity weights is expected in practice, then researchers may want to select a subset of study subjects to validate if their measured outcomes were correctly classified or not. When choosing a validation subset, it makes sense to give priorities to 1) the type of outcome misclassification with the dominant sensitivity weight and 2) the subjects that belong to a sensitive set. This offers a sensible answer to \textbf{Q4}.

\section{Computing Warning Accuracy via Adaptively Reformulating an Integer Quadratic Program}\label{sec: computing}

\subsection{Original problem formulation and the ``curse of symmetry"}\label{subsec: original problem}

When the sample size $N$ is large, it is typically infeasible to calculate warning accuracy by hand as in Section~\ref{subsec: introducing the warning accuracy}. A general strategy for tackling a computationally extensive problem involving integers such as calculating warning accuracy (\ref{eqn: warning accuracy}) is to appropriately formulate the problem into an integer programming problem and apply a state-of-the-art optimization solver to solve it. We first consider testing Fisher's sharp null $H_{0, \text{sharp}}$ with the routinely used Mantel-Haenszel test, which reduces to Fisher's exact test when $I=1$. If $H_{0, \text{sharp}}$ was rejected by the  Mantel Haenszel test at level $\alpha$, i.e., if $\frac{\{T_{\text{M-H}}-E(T_{\text{M-H}})\}^{2}}{\text{Var}(T_{\text{M-H}})}>\chi^{2}_{1, 1-\alpha}$, by Definition~\ref{def: warning accuracy}, the warning accuracy $\mathcal{WA}$ is the optimal value of the following integer quadratically constrained linear programming problem:
\begin{equation*}
     \begin{split}
        \underset{\mathbf{Y}} {\text{maximize}} \quad &\frac{1}{N}\sum_{i=1}^{I}\sum_{j=1}^{n_{i}}Y_{ij}^{*}Y_{ij}+\frac{1}{N}\sum_{i=1}^{I}\sum_{j=1}^{n_{i}}(1-Y_{ij}^{*})(1-Y_{ij}) \quad \quad (\text{P}0)\\
         \text{subject to}\quad & [T_{\text{M-H}}(\mathbf{Z}, \mathbf{Y})-E\{T_{\text{M-H}}(\mathbf{Z}, \mathbf{Y})\}]^{2}-\chi^{2}_{1, 1-\alpha} \cdot \text{Var}\{T_{\text{M-H}}(\mathbf{Z}, \mathbf{Y})\}\leq 0, \\
         &Y_{ij} \in \{0,1\}. \quad \forall i, j
     \end{split}
 \end{equation*}
 The objective function of (P0) comes from Definition~\ref{def: warning accuracy} and a simple observation that $\sum_{i=1}^{I}\sum_{j=1}^{n_{i}}\mathbf{1}(Y_{ij}^{*}=Y_{ij})=\sum_{i=1}^{I}\sum_{j=1}^{n_{i}}Y_{ij}^{*}Y_{ij}+\sum_{i=1}^{I}\sum_{j=1}^{n_{i}}(1-Y_{ij}^{*})(1-Y_{ij})$, which is a linear function in $\mathbf{Y}$ given $\mathbf{Y}^{*}$. The inequality constraint of (P0) comes from the fact that $\mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y})\neq \mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y}^{*})=``Reject"$ if and only if $\frac{\{T_{\text{M-H}}-E(T_{\text{M-H}})\}^{2}}{\text{Var}(T_{\text{M-H}})}\leq \chi^{2}_{1, 1-\alpha}$, which can be rewritten as a quadratic constraint in $\mathbf{Y}$ as in (P0).

 Instead if $H_{0, \text{sharp}}$ was accepted by the Mantel-Haenszel test (i.e., $\mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y}^{*})=``Accept"$), to compute the warning accuracy, we just need to solve a simple variant of (P0) with replacing the ``$\leq 0$" with the ``$\geq 0$" in the constraint. For calculating the warning accuracy under Neyman's weak null, we just need to replace the inequality constraint in (P0) with $\{T_{\text{Neyman}}(\mathbf{Z}, \mathbf{Y})\}^{2}-\chi^{2}_{1,1-\alpha} \cdot \widehat{\text{Var}}\{T_{\text{Neyman}}(\mathbf{Z}, \mathbf{Y})\} \leq 0$ (or $\geq 0$), which is still a quadratic constraint in $\mathbf{Y}$. See the online supplementary materials for more details.

 Formulating the warning accuracy computing problem as the integer program (P0) seems straightforward and attractive. However, this seemingly reasonable integer program formulation can easily make the computation infeasible because of the so-called ``curse of symmetry" in the integer programming literature (\citealp{margot2010symmetry}). For clarity of the notations needed in the following text, without loss of generality, in Section~\ref{subsec: original problem} we re-organize subjects within the same stratum such that for each stratum $i$, we have $Z_{ij_{1}}\leq Z_{ij_{2}}$ and $Y_{ij_{1}}^{*}\leq Y_{ij_{2}}^{*}$ as long as $j_{1}\leq j_{2}$. Following the typical notations in group theory (\citealp{scott2012group}), we let $(ij, ij^{\prime})$ denote the permutation from $\{0, 1\}^{N}$ to $\{0, 1\}^{N}$ such that the subject $ij$'s value exchange with subject $ij^{\prime}$'s value while all other subjects' values remain the same. Define the following permutation group from $\{0, 1\}^{N}$ to $\{0, 1\}^{N}$. $G_{\text{within}}=\{\text{All possible compositions of any $(ij, ij^{\prime})$ s.t. $Y_{ij}^{*}=Y_{ij^{\prime}}^{*}$ and $Z_{ij}=Z_{ij^{\prime}}$}\}$. Let $g\mathbf{Y}$ denote that the group $g$ acts on $\mathbf{Y}$. It is clear that for any $g \in G_{\text{within}}$, we have $\mathcal{A}(\mathbf{Y}^{*}\mid g\mathbf{Y})=\mathcal{A}(\mathbf{Y}^{*}\mid \mathbf{Y})$ and $\mathcal{D}_{\alpha}(\mathbf{Z}, g\mathbf{Y})= \mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y})$, i.e., the integer program (P0) is invariant under any transformation $g \in G_{\text{within}}$. We call this property \textit{within-strata symmetry}. Then for each stratum $i$, let $\mathbf{\Lambda}_{i}=(\Lambda_{i}^{00}, \Lambda_{i}^{01}, \Lambda_{i}^{10}, \Lambda_{i}^{11})=(\sum_{j=1}^{n_{i}}(1-Z_{ij})(1-Y_{ij}^{*}), \sum_{j=1}^{n_{i}}(1-Z_{ij})Y_{ij}^{*}, \sum_{j=1}^{n_{i}}Z_{ij}(1-Y_{ij}^{*}), \sum_{j=1}^{n_{i}}Z_{ij}Y_{ij}^{*})$ denotes the measured $2\times 2$ table of stratum $i$. Let $S$ denote the number of unique $2\times 2$ tables among $\{\mathbf{\Lambda}_{i}, i=1,\dots, I\}$, and let $P_{s}$ denote the number of strata with the measured $2\times 2$ tables equalling the $s$-th unique table $\mathbf{\Lambda}_{[s]}=(\Lambda_{[s]}^{00}, \Lambda_{[s]}^{01}, \Lambda_{[s]}^{10}, \Lambda_{[s]}^{11}), s=1,\dots, S$, therefore $\sum_{s=1}^{S}P_{s}=I$. For two strata $i$ and $i^{\prime}$ with $n_{i}=n_{i^{\prime}}$, we let $(i, i^{\prime})$ denote the permutation from $\{0, 1\}^{N}$ to $\{0, 1\}^{N}$ such that the stratum $i$'s values exchange with subject $i^{\prime}$'s values while all other subjects' values remain the same. Define the following permutation group $G_{\text{between}}=\{\text{All possible compositions of any $(i, i^{\prime})$ s.t. $\mathbf{\Lambda}_{i}=\mathbf{\Lambda}_{i^{\prime}}$}\}$ from $\{0, 1\}^{N}$ to $\{0, 1\}^{N}$. It is clear that for any $g \in G_{\text{between}}$, we also have $\mathcal{A}(\mathbf{Y}^{*}\mid g\mathbf{Y})=\mathcal{A}(\mathbf{Y}^{*}\mid \mathbf{Y})$ and $\mathcal{D}_{\alpha}(\mathbf{Z}, g\mathbf{Y})= \mathcal{D}_{\alpha}(\mathbf{Z}, \mathbf{Y})$, i.e., the integer program (P0) is invariant under any transformation $g \in G_{\text{between}}$. We call such property as \textit{between-strata symmetry}. To illustrate, we classify all the study subjects into four types based on the treatment indicator $Z$ and measured outcome $Y^{*}$ as in Table~\ref{tab: four types}. Then the two types of symmetry, within-strata symmetry and between-strata symmetry, are shown in Table~\ref{tab: two types of symmetry}.

 \begin{table}[ht]
      \caption{Four types of subjects based their treatment indicators and measured outcomes.}
     \label{tab: four types}
     \centering
    \begin{tabular}{c| c c }
   & $Y^{*}=1$ & $Y^{*}=0$ \\
 \hline
 \multirow{2}{*}{$Z=1$} & \semaphore{red}{red} & \semaphore{red}{blue} \\
 \multirow{2}{*}{$Z=0$} & \semaphore{blue}{red} & \semaphore{blue}{blue} \\
 \end{tabular}
 \end{table}

 \begin{table}[ht]
      \caption{Illustration of the two types of symmetry: between-strata symmetry (e.g., stratum 1 and stratum 2) and within-strata symmetry (e.g., subject 1 and subject 2 in stratum 3).}
     \label{tab: two types of symmetry}
     \centering
     \begin{tabular}{c| c c c}
   & Subject 1 & Subject 2 & Subject 3 \\
 \hline
 \multirow{2}{*}{Stratum 1} & \semaphore{red}{red} & \semaphore{red}{blue} & \semaphore{blue}{blue} \\
 \multirow{2}{*}{Stratum 2} & \semaphore{red}{red} & \semaphore{red}{blue} & \semaphore{blue}{blue} \\
  \multirow{2}{*}{Stratum 3} & \semaphore{red}{blue} & \semaphore{red}{blue}  & \semaphore{blue}{blue} \\
 \dots  \dots  & \dots    & \dots   &\dots \\
 \end{tabular}

 \end{table}

 Putting the above discussions together, define the permutation group $G=\{\text{All possible}$\\
 $\text{compositions of elements from $G_{\text{within}}$ and $G_{\text{between}}$}\}$. For any $g \in G$, the integer program (P0) is invariant under transformation $g$. Note that \begin{equation*}
      |G|=|G_{\text{between}}|\times |G_{\text{within}}|=\prod_{s=1}^{S}P_{s}!\times \prod_{i=1}^{I}\Lambda^{00}_{i}!\Lambda^{01}_{i}!\Lambda^{10}_{i}!\Lambda^{11}_{i}!,
 \end{equation*}
which is an extremely large number if $N$ is large, indicating an extremely high degree of symmetry of program (P0) and resulting in computational infeasibility of solving (P0) due to the so-called ``curse of symmetry," which refers to a general fact that an integer program is typically computationally infeasible if its variables can be permuted in many ways (e.g., $|G|$ is large) without changing the structure of the problem (\citealp{margot2010symmetry}).

\subsection{Two types of randomization designs and an adaptive reformulation strategy}\label{subsec: adaptive}

In Section~\ref{subsec: adaptive}, we propose a general strategy to solve the ``curse of symmetry" encountered when calculating the warning accuracy according to the original problem formulation (P0). The core idea of our strategy is to reformulate the integer program (P0) with respect to an intrinsic characteristic of various randomization designs -- whether within-strata symmetry dominates between-strata symmetry for that randomization design or vice visa. Specifically, we classify many commonly used randomization designs into the following two types:
\begin{itemize}
    \item Type I randomization designs: those in which within-strata symmetry dominates between-strata symmetry (i.e., $|G_{\text{within}}|\gg|G_{\text{between}}|$). This type of randomization designs include some commonly used randomization designs such as completely randomized experiments ($|G_{\text{between}}|=1$) and stratified/blocked randomized experiments or observational studies with most strata/blocks being large.
    \item Type II randomization designs: those in which between-strata symmetry dominates within-strata symmetry (i.e., $|G_{\text{between}}|\gg|G_{\text{within}}|$). This type of randomization designs include some widely used randomization designs such as paired randomized experiments or observational studies ($|G_{\text{within}}|=1$), randomized experiments or observational studies with multiple controls, randomized experiments or observational studies with variable controls, finely stratified experiments, and observational studies with full matching.
\end{itemize}

\subsubsection{Type I: within-strata symmetry dominates between-strata symmetry}\label{subsubsec: type I}

We first show how to reformulate the integer program (P0) for type I randomization designs. Note that the accuracy (the objective function of (P0)) can be determined by the $I$ $2\times 2$ tables $\{\sum_{j=1}^{n_{i}}\mathbf{1}(Y_{ij}^{*}=b, Y_{ij}=c): b, c \in \{0,1\}\}$ ($i=1,\dots,I$), and the quadratic constraint's feasibility can be determined by the $I$ $2\times 2$ tables $\{\sum_{j=1}^{n_{i}}\mathbf{1}(Z_{ij}=a, Y_{ij}=c): b, c \in \{0,1\}\}$ ($i=1,\dots,I$). Therefore, both the objective function and the quadratic constraint's feasibility can be determined by the $I$ $2\times 2 \times 2$ tables $\{\sum_{j=1}^{n_{i}}\mathbf{1}(Z_{ij}=a, Y_{ij}^{*}=b, Y_{ij}=c): a, b, c \in \{0,1\}\}$ ($i=1,\dots,I$). Because $\mathbf{Z}$ and $\mathbf{Y}^{*}$ have been observed, for each stratum $i$, the $i$-th $2\times 2 \times 2$ table only has the following four degrees of freedom: $\Upsilon_{i}^{00}=\sum_{j=1}^{n_{i}}\mathbf{1}(Z_{ij}=0, Y_{ij}^{*}=0, Y_{ij}=1)$, $\Upsilon_{i}^{01}=\sum_{j=1}^{n_{i}}\mathbf{1}(Z_{ij}=0, Y_{ij}^{*}=1, Y_{ij}=1)$, $\Upsilon_{i}^{10}=\sum_{j=1}^{n_{i}}\mathbf{1}(Z_{ij}=1, Y_{ij}^{*}=0, Y_{ij}=1)$, and $\Upsilon_{i}^{11}=\sum_{j=1}^{n_{i}}\mathbf{1}(Z_{ij}=1, Y_{ij}^{*}=1, Y_{ij}=1)$. Let $\mathbf{\Upsilon}=(\Upsilon_{1}^{00}, \Upsilon_{1}^{01}, \Upsilon_{1}^{10}, \Upsilon_{1}^{11}, \dots, \Upsilon_{I}^{00}, \Upsilon_{I}^{01}, \Upsilon_{I}^{10}, \Upsilon_{I}^{11})$. Then (P0) can be reformulated as the following integer quadratic program (P1)
\begin{equation*}
     \begin{split}
        \underset{\mathbf{\Upsilon}} {\text{max}} \quad &N^{-1}\sum_{i=1}^{I}(\Upsilon_{i}^{01}+\Upsilon_{i}^{11}-\Upsilon_{i}^{00}-\Upsilon_{i}^{10})+N^{-1}\sum_{i=1}^{I}\sum_{j=1}^{n_{i}}(1-Y_{ij}^{*}) \quad \quad (\text{P}1)\\
         \text{s.t.}\quad & \Big[ \sum_{i=1}^{I}(\Upsilon_{i}^{10}+\Upsilon_{i}^{11})-\sum_{i=1}^{I}\Big\{\frac{m_{i}}{n_{i}}(\Upsilon_{i}^{00}+\Upsilon_{i}^{01}+\Upsilon_{i}^{10}+\Upsilon_{i}^{11}) \Big\} \Big]^{2}\\
   &\quad \quad \quad \quad - \chi^{2}_{1, 1-\alpha} \cdot \sum_{i=1}^{I}\frac{m_{i}(\Upsilon_{i}^{00}+\Upsilon_{i}^{01}+\Upsilon_{i}^{10}+\Upsilon_{i}^{11})(n_{i}-\Upsilon_{i}^{00}-\Upsilon_{i}^{01}-\Upsilon_{i}^{10}-\Upsilon_{i}^{11})(n_{i}-m_{i})}{n_{i}^{2}(n_{i}-1)}\leq 0, \\
        &0 \leq \Upsilon_{i}^{00} \leq \sum_{j=1}^{n_{i}}(1-Z_{ij})(1-Y_{ij}^{*}), \quad \forall i \\
        &0 \leq \Upsilon_{i}^{01} \leq \sum_{j=1}^{n_{i}}(1-Z_{ij})Y_{ij}^{*}, \quad \forall i \\
        &0 \leq \Upsilon_{i}^{10} \leq \sum_{j=1}^{n_{i}}Z_{ij}(1-Y_{ij}^{*}), \quad \forall i \\
        &0 \leq \Upsilon_{i}^{11} \leq \sum_{j=1}^{n_{i}}Z_{ij}Y_{ij}^{*}, \quad \forall i \\
        &\mathbf{\Upsilon} \in \mathbb{Z}^{4I}.
     \end{split}
 \end{equation*}
Note that in (P1), in addition to rewriting the objective function and constraint of (P0) in terms of new variables $\mathbf{\Upsilon}$, we also add the bounding constraints for $\mathbf{\Upsilon}$. It is clear that there is no within-strata symmetry anymore in (P1), which was previously a major source of symmetry in the original formulation (P0) for type I randomization designs.

\subsubsection{Type II: between-strata symmetry dominates within-strata symmetry}

Our strategy of reformulating the integer program (P0) for type II randomization designs is inspired by both the observations discussed in Section~\ref{subsubsec: type I} and an idea from \citet{fogarty2016discrete}. Specifically, in \citet{fogarty2016discrete}, for handling the ``curse of symmetry" in randomization inference for the average
treatment effect with binary outcomes in a finely stratified experiment or an observational study with full matching, instead if manipulating the potential outcomes based on the treatment indicators and observed outcomes, the authors proposed to list all possible $2 \times 2 \times 2$ tables (in terms of treatment indicators, observed outcomes, and potential outcomes) and manipulate the number of each possible $2 \times 2 \times 2$ that enter into randomization inference. Although in this paper we are studying a totally different problem, we can combine this general philosophy proposed in \citet{fogarty2016discrete} and the arguments in Section~\ref{subsubsec: type I} to reformulate the integer program (P0) for type II randomization designs.

Specifically, for the $s$-th unique $2 \times 2$ table $\mathbf{\Lambda}_{[s]}=(\Lambda_{[s]}^{00}, \Lambda_{[s]}^{01}, \Lambda_{[s]}^{10}, \Lambda_{[s]}^{11}), s=1,\dots, S$, let $\widetilde{n}_{s}$ and $\widetilde{m}_{s}$ denote its number of total study subjects and its number of treated subjects. Moreover, for the $s$-th unique table $\mathbf{\Lambda}_{[s]}$, there are $\widetilde{N}_{s}=(\Lambda_{[s]}^{00}+1) (\Lambda_{[s]}^{01}+1)(\Lambda_{[s]}^{10}+1)(\Lambda_{[s]}^{11}+1)$ possible unique $2\times 2 \times 2$ tables $\{\sum_{j=1}^{\widetilde{n}_{s}}\mathbf{1}(Z_{ij}=a, Y_{ij}^{*}=b, Y_{ij}=c): a, b, c \in \{0,1\}\}$ ($s=1,\dots,S$). Let $d_{sp}$ be the number of the $p$-th unique $2\times 2 \times 2$ table $\Delta_{sp}$ for the $s$-th unique table $\mathbf{\Lambda}_{[s]}$, $s=1,\dots,S$ and $p=1, \dots, \widetilde{N}_{s}$. Since $\mathbf{Z}$ and $\mathbf{Y}^{*}$ are given, each $2\times 2 \times 2$ table $\Delta_{sp}$ can be uniquely determined by four numbers $\Delta_{sp}^{00}=\sum_{j=1}^{\widetilde{n}_{s}}\mathbf{1}(Z_{ij}=0, Y_{ij}^{*}=0, Y_{ij}=1)$, $\Delta_{sp}^{01}=\sum_{j=1}^{\widetilde{n}_{s}}\mathbf{1}(Z_{ij}=0, Y_{ij}^{*}=1, Y_{ij}=1)$, $\Delta_{sp}^{10}=\sum_{j=1}^{\widetilde{n}_{s}}\mathbf{1}(Z_{ij}=1, Y_{ij}^{*}=0, Y_{ij}=1)$, and $\Delta_{sp}^{11}=\sum_{j=1}^{\widetilde{n}_{s}}\mathbf{1}(Z_{ij}=1, Y_{ij}^{*}=1, Y_{ij}=1)$. Then we can reformulate the original integer program (P0) into the following quadratic integer program (P2):
\begin{equation*}
     \begin{split}
        \underset{d_{sp}}{\text{max}} \quad & N^{-1}\sum_{s=1}^{S}\sum_{p=1}^{\widetilde{N}_{s}}d_{sp}(\Delta_{sp}^{01}+\Delta_{sp}^{11}-\Delta_{sp}^{00}-\Delta_{sp}^{10})+N^{-1}\sum_{i=1}^{I}\sum_{j=1}^{n_{i}}(1-Y_{ij}^{*}) \quad \quad (\text{P}2)\\
         \text{s.t. }\quad & \Big[ \sum_{s=1}^{S}\sum_{p=1}^{\widetilde{N}_{s}}d_{sp}(\Delta_{sp}^{10}+\Delta_{sp}^{11})-\sum_{s=1}^{S}\sum_{p=1}^{\widetilde{N}_{s}}d_{sp}\Big\{\frac{\widetilde{m}_{s}}{\widetilde{n}_{s}}(\Delta_{sp}^{00}+\Delta_{sp}^{01}+\Delta_{sp}^{10}+\Delta_{sp}^{11}) \Big\} \Big]^{2}\\
   &\quad - \chi^{2}_{1, 1-\alpha} \cdot \sum_{s=1}^{S}\sum_{p=1}^{\widetilde{N}_{s}}d_{sp}\frac{\widetilde{m}_{s}(\Delta_{sp}^{00}+\Delta_{sp}^{01}+\Delta_{sp}^{10}+\Delta_{sp}^{11})(\widetilde{n}_{s}-\Delta_{sp}^{00}-\Delta_{sp}^{01}-\Delta_{sp}^{10}-\Delta_{sp}^{11})(\widetilde{n}_{s}-\widetilde{m}_{s})}{\widetilde{n}_{s}^{2}(\widetilde{n}_{s}-1)}\leq 0, \\
        & \sum_{p=1}^{\widetilde{N}_{s}} d_{sp} = P_{s}, \quad \forall s \\
        & d_{sp}\geq 0, \quad \forall s, p \\
        & d_{sp}\in \mathbb{Z}. \quad \forall s, p
     \end{split}
 \end{equation*}
Note that in (P2), $(\Delta_{sp}^{00}, \Delta_{sp}^{01}, \Delta_{sp}^{10}, \Delta_{sp}^{11})$ are fixed numbers and the decision variables are $d_{sp}$ ($s=1,\dots,S$ and $p=1, \dots, \widetilde{N}_{s}$), i.e., the number of the unique $2 \times 2 \times 2$ table $(\Delta_{sp}^{00}, \Delta_{sp}^{01}, \Delta_{sp}^{10}, \Delta_{sp}^{11})$ that enters into inference. It is clear that there is neither within-strata symmetry nor between-strata symmetry in (P2) and therefore surpasses the formulation (P1) in terms of erasing symmetry. However, the dimension of the decision variables in (P2) is $\sum_{s=1}^{S}\widetilde{N}_{s}=\sum_{s=1}^{S}(\Lambda_{[s]}^{00}+1) (\Lambda_{[s]}^{01}+1)(\Lambda_{[s]}^{10}+1)(\Lambda_{[s]}^{11}+1)$, which is typically very high for type I randomization designs, but is typically not high for type II randomization. Therefore, for type II randomization designs, it is appropriate to reformulate (P0) as (P2). While for type I randomization designs, it is more appropriate to reformulate (P0) as (P1).

\section{Real Data Application: Understanding the Puzzle in the PCPT}\label{sec: PCPT analysis}

We now apply our newly developed framework to understand the puzzle in the PCPT described in Section~\ref{subsubsec: PCPT}. Following \citet{thompson2003influence}, the prespecified level is 0.05. We apply the efficient computation strategy developed in Section~\ref{sec: computing} to calculate warning accuracy (equivalently, minimal alteration number) and sensitivity weights for the two binary outcomes of interest in the PCPT: prostate cancer indicator (cancer vs. no cancer) and high-grade prostate cancer (high-grade prostate cancer vs. no cancer or low-grade cancer). The total computation was finished in a few seconds using the optimization solver \textsf{Gurobi} (version 9.1). The results are reported in Table~\ref{tab: normal and severe cancer} below.

\begin{table}[ht]
\caption{The p-values, the values of warning accuracy, and the sensitivity weights for the two binary outcomes of interest in the PCPT under Fisher's sharp null hypothesis of no treatment effect and level 0.05.}
\label{tab: normal and severe cancer}
\small
\centering
\begin{tabular}{c|cc|cc}
  \hline
   Outcome & \multicolumn{2}{c|}{Prostate Cancer}  & \multicolumn{2}{c}{High-Grade Prostate Cancer}\\
  (Sample Size N) & \multicolumn{2}{c|}{($N=9060$)}  & \multicolumn{2}{c}{($N=9017$)}\\
   \hline
   Relative Risk & \multicolumn{2}{c|}{ 0.75 (Protective Factor)}  & \multicolumn{2}{c}{1.27 (Risk Factor)} \\
   \hline
   p-value & \multicolumn{2}{c|}{$4.66 \times 10^{-13}$}  & \multicolumn{2}{c}{$6.79 \times 10^{-3}$}  \\
   \hline
  Reject $H_{0}$ or Not & \multicolumn{2}{c|}{Yes}  & \multicolumn{2}{c}{Yes} \\
   \hline
   Causal Conclusion & \multicolumn{2}{c|}{Prevents Prostate Cancer} & \multicolumn{2}{c}{Promotes High-Grade Cancer} \\
   \hline
Warning Accuracy & \multicolumn{2}{c|}{98.37\%}  & \multicolumn{2}{c}{99.88\%} \\
   \hline
    Minimal Alteration \# & \multicolumn{2}{c|}{147}  & \multicolumn{2}{c}{11} \\
   \hline
Sensitivity Weights  & False Positives & False Negatives  & False Positives & False Negatives  \\
   \hline
   Finasteride  & 0 & 132/147   & 2/11 & 0  \\
   Placebo & 15/147 & 0   & 0 & 9/11  \\
 \hline
 \end{tabular}
\end{table}

We now give an interpretation of the results in Table~\ref{tab: normal and severe cancer}. First, according to the reported values of warning accuracy and minimal alteration number, in a worst-case scenario sensitivity analysis, the causal conclusion concerning the prevention effect of Finasteride on prostate cancer is less sensitive to outcome misclassification than the causal conclusion concerning the promotion effect on high-grade prostate cancer (the two values of warning accuracy differ by 1.5\%). A 1.5\% difference in warning accuracy is nontrivial as the sample size is large (over 90,000 study subjects) and it corresponds to a 136 difference in minimal alteration number. In other words, to alter the causal conclusion concerning the prevention effect, it requires 147/11 $\approx$ 13.4 times more cases of outcome misclassification than that required by the causal conclusion concerning the promotion effect.

Second, we leverage the reported sensitivity weights and related prior information and expert knowledge to further investigate sensitivity to outcome misclassification for each of the two causal conclusions. From the reported sensitivity weights, we learn that for both of the causal conclusions, the major concern is false negatives: the dominant term among the sensitivity weights for the prevention effect is false negatives among the Finasteride (treated) group and that for the promotion effect is false negatives among the placebo (control) group. For each of the two causal conclusions, is it plausible that the dominant term among the sensitivity weights is the dominant term among the four actual numbers of outcome misclassification cases?

We now use prior information and expert knowledge to shed light on this issue for the two outcomes of interest. We define the following notations: $N_{T}$--number of treated subjects; $N_{C}$--number of control subjects; $p_{T,1}$ (or $p_{T,0}$)--proportion of positive (or negative) true outcomes among the treated subjects; $p_{C,1}$ (or $p_{C,0}$)--proportion of positive (or negative) true outcomes among the control subjects; $\pi_{T, 1|0}$ (or $\pi_{T, 0|1}$)--proportion of false positives (or false negatives) among the treated subjects with true outcomes being negative (or positive); $\pi_{C, 1|0}$ (or $\pi_{C, 0|1}$)--proportion of false positives (or false negatives) among the control subjects with true outcomes being negative (or positive). Then the total number of each of the four types of outcome misclassification (false positives/negatives among the treated/control group) can be decomposed into the product of the above three terms, as shown in Table~\ref{tab: decomposition}.
\begin{table}[ht]
\caption{Decomposition of the total number of outcome misclassification cases for each of the four types of outcome misclassification.}\label{tab: decomposition}
    \centering
    \begin{tabular}{c|cc}
  \hline
  Misclassification Cases & False Positives & False Negatives   \\
  \hline
  Finasteride (Treated)   & $N_{T}\cdot p_{T, 0}\cdot \pi_{T, 1|0}$ & $N_{T}\cdot p_{T, 1}\cdot \pi_{T, 0|1}$   \\
  Placebo (Control) & $N_{C}\cdot p_{C, 0}\cdot \pi_{C, 1|0}$ & $N_{C}\cdot p_{C, 1}\cdot \pi_{C, 0|1}$  \\
 \hline
 \end{tabular}
\end{table}

Meanwhile, we have the following related prior information or expert knowledge:
\begin{itemize}
    \item (i) We have $N_{T}\approx N_{C}$ by design.
    \item (ii) Even if there are misclassified outcomes, it may still be sensible to get some sense of the values of $(p_{T, 0}, p_{T, 1}, p_{C, 0}, p_{C, 1})$ using measured outcomes. Based on measured outcomes, for the prostate cancer (all grades) outcome, $p_{T, 0}\approx 82\%, p_{T, 1}\approx 18\%, p_{C, 0}\approx 76\%, p_{C, 1}\approx 24\%$. For the high-grade prostate cancer outcome, $p_{T, 0}\approx 94\%, p_{T, 1}\approx 6\%, p_{C, 0}\approx 95\%, p_{C, 1}\approx 5\%$. Here ``$\approx$" means a very rough estimation.

    \item (iii) Finasteride substantially improves detection of prostate cancer (all-grades). According to \citet{nih}: ``Finasteride has several effects on the prostate that allow better detection of prostate cancers. The drug shrinks the prostate, reducing its size and volume and increasing the chance that a biopsy will find existing cancers." Meanwhile, Finasteride also greatly improves accuracy in prostate cancer grading at biopsy (\citealp{redman2008finasteride}). Therefore, we expect $\pi_{C, 0|1}\gg \pi_{T, 0|1}$ for both of the two outcomes.
\end{itemize}

Therefore, according to (i), (ii), and (iii), we expect that: 1) For the prostate cancer outcome, false negatives among the treated (Finasteride) group (i.e., the dominant term among the four sensitivity weights) should not be the actual dominant term among the four types of outcome misclassification, as at least we expect that the number of false negatives among the treated should not dominate the number of false negatives among controls. Consequently, we expect that for the prostate cancer outcome, the actual accuracy needed to overturn the causal conclusion about the prevention effect is lower than the warning accuracy of 98.37\%. 2) For the high-grade prostate cancer outcome, false negatives among the control (placebo) group (i.e., the dominant term among the four sensitivity weights) could be the actual dominant term among the four types of outcome misclassification, as we at least expect that the number of false negatives among controls should dominate the number of false negatives among the treated. Although whether the number of false negatives among controls dominates the number of false positives among the treated/control group needs further investigation and information, unlike the prostate cancer outcome (prevention effect), for the high-grade prostate cancer outcome we currently do not have evidence that the actual accuracy needed to overturn the causal conclusion about the promotion effect is lower than the reported warning accuracy of 99.88\%.

Putting all the arguments together, based on the experimental data, the information provided using our new approach, and related expert knowledge, we have evidence that the causal conclusion that Finasteride prevents prostate cancer is more reliable and the causal conclusion that Finasteride promotes high-grade prostate cancer is less convincing and may be due to systematic (differential) outcome misclassification. Our finding supports some previous arguments proposed by domain scientists and biostatisticians (e.g., \citealp{lucia2007finasteride, redman2008finasteride, shepherd2008does}).

\section{Summary}

Randomized experiments (trials) are considered the gold standard for causal inference. Through physical randomization (e.g., flipping a fair coin), a causal conclusion drawn from a randomized experiment cannot be biased by confounding or model misspecification. However, outcome misclassification commonly exists in practice and can severely distort causal inferences in randomized experiments, especially for systematic (differential) outcome misclassification. All the previous literature on outcome misclassification requires imposing some additional assumptions to a randomized experiment such as the i.i.d. assumption
and/or some modeling assumptions, which can be violated in practice.

In this paper, we proposed a model-free and finite-population-exact framework for analyzing a randomized experiment subject to outcome misclassification, covering the design stage, analysis stage, and validation stage. The highlight of our framework is that it does not require any additional assumptions and meanwhile can provide some useful information to help researchers analyze a randomized experiment in a more comprehensive way. Our framework is built on efficient computation of large-scale integer programming, and our adaptive reformulation strategy for handling the ``curse of symmetry" encountered in randomization inference with binary outcomes could shed light on many other problems in causal inference with binary outcomes.

\begin{center}
{\large\bf Acknowledgement}
\end{center}

This work is supported by NIH grant R01AI131771. We would like to thank Professor Thomas Lumley, Professor Bryan Shepherd, Professor Dylan Small, and the participants in the Center for Causal Inference (CCI) seminar for helpful comments and suggestions.

\bibliographystyle{apalike}

redpajama_set_name: RedPajamaArXiv


text:
Home » Skillest in the "New York Times"
Skillest in the "New York Times"
by Baden Schaff
It's official. Skillest has hit the big time! We were incredibly excited to be featured in The New York Times on Wednesday 7/7/21. Paul Sullivan wrote a wonderful piece on the growth of the platform, how COVID has accelerated golfers willingness to learn golf online and just how powerful it is as a learning model. But he focused on something we don't often talk about at Skillest.
As you probably know we are obsessed with passionate golfers who want to take their handicap from 18 to 10 or 7 to 4, but did you know that Tour players are using Skillest to find coaches and then interact with them while on tour?
Over the weekend Lucas Herbert won the Irish Open for his second win on the European Tour and he attributes his success to the ability to use Skillest to interact with his coach, Dom Azzopardi over the past year. "It's 10:30 p.m. in the evening here, and Lucas is about to go and practice at 8:30 a.m., so the time zones make it so different," Azzopardi said. "Instead, I wake up and see his swings, view them, draw lines on them and do a voice-over. It's just been a really easy way to communicate."
Herbert went on to say..
Not having his coach on the range or caddying for him was different at first. But the connection through the app has worked well.
"I'm quite visual," Herbert said. "I like to see what I want to change, what's going well, in front of me. The app is good for that. I can put a picture to my mind to see and a voice to guide me."
Skillest Co-Founder Baden Schaff was also quoted… "We've tripled in size in the past 12 months," said Baden. "I've always known that it was right for the elite players in the game. They've always interacted to a degree like this with their coaches. What's more exciting is the average person has more interaction with their coach and is getting what elite players have always had. The elite players get better because they have constant feedback from the best coaches in the world," he said. "When an average player comes back every three or four weeks, you don't progress because you don't hold on to what you're working on. The elite players have the ability to come back the next day and the day after that. That's why they get better."
You can read the rest of this great article here and be sure to head to Skillest to get your Tour experience.
previousStudent in Iceland makes incredible swing change!
nextBelgium's best swings analysed!
Golf Drills at Home
Golf Skills: 10 Skills To Improve Your Golf Scores
A Guide to Finding the Ideal Golf Impact Position
Step-By-Step Proper Golf Grip Guide: What You Need to Know
What Length Putter Do I Need, and Why Does It Matter?
Student Results (6)
Skillest connects you with the best golf coaches across the world for online golf coaching, training, putting tips and swing video analysis.
Download in App Store
Download in Google Play

redpajama_set_name: RedPajamaCommonCrawl


Baltimore General Manager Ozzie Newsome raised the possibility of linebacker/defensive end Terrell Suggs might not return to the Ravens next season because Suggs has a $12.4 million salary cap figure. Suggs' contract includes a base salary of $7.8 million. Suggs, 31, has 94.5 sacks in his 11-year career and had 10 last season.
However, Suggs had only one sack over his final eight games. Suggs has talked about wanting a contract extension this offseason. However, Newsome said the team would have to make a decision that could include letting go of Suggs.
"That's a decision we'll talk about. Terrell is a real good football player," Newsome said.


text:
Resource Title:
2000: John Goodlad
SHARE THIS RESOURCE ON
Starting as a teacher in a rural, one-room school in Canada, John Goodlad since has been involved in an array of education reform programs and projects, and has engaged in large-scale studies of education change, schooling and teacher education. At the age of 79, in addition to advancing a comprehensive program of research and development directed to the simultaneous renewal of schooling and teacher education, Goodlad continues to inquire about education's mission in a democratic society. He has authored, co-authored or edited more than 30 books and numerous chapters, papers and articles; he also has received many distinguished awards. Goodlad has taught at all grade levels and in a variety of institutions. He is professor emeritus of education and co-director of the Center for Educational Renewal at the University of Washington in Seattle and president of the independent Institute for Educational Inquiry.
AUTHOR(S): Unspecified
RESOURCE TYPE: Unspecified
EDUCATION LEVEL: Unspecified
STATE(S): Unspecified
Education Commission of the States is the trusted source for comprehensive knowledge and unbiased resources on education policy issues ranging from early learning through postsecondary education.
Subscribe to our publications and stay informed.
Need more information? Contact one of our policy experts.


New interviews with Mary Kuryla, author of "Freak Weather"
Mary Kuryla's Freak Weather, the winner of the AWP's 2016 Grace Paley Prize in Short Fiction, has been generating a good deal of buzz since its publication this November. Rising up from the underbelly of Los Angeles, the thirteen stories in this debut collection feature a cast of provocative and unapologetic women—from would-be rodeo queens and mail order brides to pediatric ICU nurses and hippie housewives.
While People has lauded this "imaginative, unpredictable debut short story collection," interviews and craft pieces consider how Kuryla bucks gender norms with her misbehaving female protagonists and reveal how performance and screenwriting have shaped Freak Weather. In an interview with Electric Literature, Kuryla explains that "There's something like a bad boy thing" in post-Vietnam literature. "We might feel like their behavior is really shocking, but there's something that really stays with us about the burden these men are under. I was really interested in telling a female version of that, because I just didn't see any. I was interested in what it would be like to have a female character behaving really badly."
In a craft piece for Necessary Fiction the author elaborates on her process, as her stories have been adapted for stage and screen and in turn, reworked. Kuryla notes that the collection's titular story, "Freak Weather," "served as the foundation for a feature film that I wrote and directed, and which premiered at the Toronto International Film Festival . . . [H]aving gone through the process of rehearsing with the film's actress (Jacqueline McKenzie), hearing the lines spoken and the actions embodied, followed by shooting and editing the film, then viewing it on a big screen with large audiences, I was able to witness the gaps and inconsistencies in the original story." After seeing the film, she "returned later to the story with absolute clarity and revised it based on this experience of seeing it performed and rendered onscreen."
As Kuryla explains in an interview with CRAFT Literary, her training as a filmmaker and screenwriter had a great impact on her fiction: "I was very influenced early on by the films of Chantal Ackerman and Rainer Werner Fassbinder and Godard, and by feminist film theory. All of this was very emboldening because it was also working at the boundaries of form. The literary world seemed softer at that time, in the '90s. I wanted the turbulence of Laurence Sterne. Screenwriting seemed so much more interactive and exciting."


Click on the image to view a full-size PDF.
Purchase three dozen personalized Titleist Pro V1®, Pro V1x® or AVX™ golf balls between March 22 – April 22, 2019 and receive one dozen of the same ball type at no charge.
For more information and to submit an order please contact The Home Course Head Golf Professional Sean Howland at (253) 964-0520 or sean.howland@thehomecourse.com.


17 October 2017 - This week, the final regional preparatory meeting for the upcoming Global Judicial Integrity Network was held in Namibia for Anglophone and Lusophone African countries. From 16 to 17 October, Chief Justices and senior judges from 17 countries gathered to add their voices to what will be the world's first-ever global platform dedicated exclusively to upholding judicial integrity and preventing corruption within the justice system once launched in April 2018.
The meeting - jointly hosted by UNODC and the Office of the Judiciary of the Republic of Namibia - follows similar regional events held over the past year in Asia (Bangkok), Europe (Vienna), Francophone Africa (Ouagadougou) and Latin America (Panama). As with these gatherings, this final meeting looks to add to the overall structure of the Network. Critically, it allowed those present to share their needs and expectations in terms of capacity-building support, advisory services, tools, networking opportunities and other resources and how best to meet these.
In their opening statements, the Chief Justice of Namibia, H.E. Peter Shivute, and UNODC Southern Africa Regional Representative, Zhuldyz Akisheva, both highlighted the critical importance of judicial integrity.
"It is no exaggeration to say that without integrity our judicial systems will disintegrate, and become empty shells that are unable to carry out their sacred constitutional mandate", noted Chief Justice Shivute. "To ensure that integrity becomes the bedrock of all our work and interactions as judicial officers, we must continually inculcate the values of honesty, fairness and justice in all that we do".
Adding to this, Ms. Akisheva said, "The judiciary plays a crucial role in combatting corruption, but in order to fulfil this, they themselves must be free from corruption. This final preparatory meeting provides members of the judiciaries in Anglophone and Lusophone speaking African countries an important forum to discuss challenges being faced and expectations towards what the Network will offer".
Reflecting the international linkages that will make the Global Judicial Integrity Network so valuable in helping share cross-border solutions to common problems, the meeting was also joined by the Honourable President Chief Justice of the Peoples' Supreme Court of Cuba, Ruben Remigio Ferro. In his keynote address the Chief Justice shared his thoughts on this issue. "The mission of judges is not to be simple enforcers of norms and dispositions to the case before them, but also implies the duty to do so with a clear notion and sense of what is fair". He also added that in his experience in Cuba, "exercising judicial power implies, in turn, the commitment to comply with judicial duty".
The meeting proved highly fruitful and over the two days participants shared a vast array of innovative approaches which have been taken to effectively implement Codes of Conduct and hold judges accountable. Attendees also spoke of specific challenges which need to be overcome, including continuous attempts of the executive and the legislature in some countries to interfere with judicial decision-making and judicial appointments, among others. Ultimately the participants welcomed the proposal to launch a Global Judicial Integrity Network as a platform to allow for further exchange around issues of mutual concern, to access resources and capacity building opportunities and to develop new tools.
In addition to this preparatory meeting, Swakopmund is also host to two other important regional judicial events this week. The inaugural Southern African Judicial Administrators Association and the Southern African Chief Justices Forum's Annual General Meeting are both taking place, providing further focus on this key group.
The creation of the Global Judicial Integrity Network is part of UNODC's Global Programme for the Implementation of the Doha Declaration, which was adopted by the 13th United Nations Congress on Crime Prevention and Criminal Justice in 2015. The Global Programme provides support and technical assistance to Member States across the world in areas of judicial integrity, prisoners' rehabilitation, youth crime prevention, and education for justice, all with the objective of promoting a culture of lawfulness and boosting respect for the rule of law.


Gaurav brings with him over 6 years of experience in Investment Banking and Financial Services Industry across different verticals - Finance, Accounts, Mortgage and Taxation. Currently represent Dexter Capital Advisors for its south India operation.
He is a start-up enthusiast and worked with start-ups and corporate firms like Shop Diaries, United Trading Company, American Home Mortgage Servicing and Dirk India Pvt. Ltd.
Guarav is an MBA (Finance), PGDBF (Finance) and certified investment professional in Capital Market, Money Market & Risk Management by NSE India. By qualification, a Company Secretary, and CFA from CFA Institute, USA.


Three 3.40 mm Vertically Oriented Diamonds Framed by an Open Pear 14k Gold Frame in This Fabulous 3-Stone Diamond Pendant With 1/2 carat of Diamonds. Choose 14k White, Rose or Yellow Gold. NOTE: The very facets that create the beautiful sparkle in a loose gemstones or gemstones set in jewelry may create optical illusion white or dark/black spots and areas, or uneven coloring when a gemstone is photographed. This effect also applies to photographing highly reflective metals such as gold. Single dimension photographs can not accurately display the true beauty and life in a gemstone or gemstone jewelry. This Genuine Gemstone Jewelry must be seen in person to be appreciated.


https://thedandydons.com/wp-content/uploads/2013/08/renaissance-aberdeen-fc.jpg 360 900 Dandy https://thedandydons.com/wp-content/uploads/2018/11/dandii.png Dandy2013-08-16 12:32:112015-07-10 22:07:36Aberdeen F.C, the Renaissance!!
Played one, won one. Rejoice!
How to get more people to Pittodrie?


When did you last look closely in your pet's mouth? We invite you for a free complete oral exam.
The essential quarterly prophylaxes against the "Parks Worm" are now offered as a yearly package at a significant discount.
Conducted for video Veterinary Center Ramat Hasharon as photographs into a television show that took place the..
To Dr. Doron Neri, Zeev Gans and all the vet center team thank you for the dedicated treatment, the concern, a..


The three winners were among 15 startups competing in a pitch competition on Thursday after a day of meetings with industry experts helping them to hone their pitches, gain direct feedback on their products and technologies, and network with investors and corporate leaders.
Unlike many products that are 'Hawaiian' in branding only, naturally caffeine-free, zero-calorie Shaka Tea is built around māmaki - a plant that only grows in the Hawaiian islands, said cofounder Bella Hughes, who wowed the FoodBytes! judges with her unique concept and strong sales traction, but also with Shaka's business model, which supports local regenerative agriculture and economic development through the cultivation of māmaki.
North Dakota-based Bushel has built a software platform that allows grain elevators to digitally connect with their growers to deliver near real-time information that informs better business decisions, said cofounder Jake Joraanstad.
Toast Ale impressed the audience and the judges with its innovative approach to fighting food waste: brewing craft beer from unsold bread from local bakeries. It also donates 100% of its profits to charities reducing food waste.
"We replace one third of the malted barley in beer with surplus bread, but the rest of the process is exactly the same," said 'chief toaster' Rob Wilson on the FoodBytes! stage.
"We use fresh surplus bread that would otherwise go to waste... so with a slice of surplus bread in every one of our craft beers, we think we're the best thing since... you know what."
The winners - whittled down from several hundred applicants - will each have the opportunity to pitch and network with c-level food and agriculture executives at Rabobank's annual food & agriculture summit in New York City in December, participate in an innovation day with ADM's food and beverage product advisors, and gain access to legal consultation from Polsinelli, go-to-market consultation from The Intertwine Group, and recruiting consultation from Creative Alignments. They will also have access to Rabobank research reports.
Management team: Exceptional domain and entrepreneurial experience.
Business model: Companies with a clear understanding of their costs and margin structures, competitors, and ability to scale.
Market opportunity: High growth market opportunities and high barriers to entry.
Product innovation: Game-changing approaches to solving problems.
Production validation and traction: Evidence companies are building a successful business and are able to raise external funds.
Sustainability and social impact: Concept is focused on the triple bottom line: people, profit and planet.
Watch all the pitches from FoodBytes! San Francisco HERE, and checkout FoodNavigator-USA next week for coverage of the event and interviews with mentors, judges, and startups.
Hawaii-based Shaka Tea has created the first Hawaiian ready-to-drink iced tea, and the first brewed with māmaki, a plant that only grows in the Hawaiian islands, supporting regenerative agriculture and economic development.
New York based Toast Ale produces pale ale from surplus bread.
Illinois-based The Worthy Company has developed the Worthy Blendie Bowl, which combines fruits, vegetables, legumes and chia to create a nutritious on the go snack or mini meal.
California-based GEM has developed 'chewable bites of daily nutrition' made from 13 algae and plant-based ingredients that provide 15+ vitamins and minerals per bite.
Delaware-based Taali has developed a snack made from popped water lily seeds.
California-based SnapDNA has developed what it claims is the fastest food pathogen test in the industry, eliminating the need to culture bacteria prior to analysis, enabling sample-to-answer test results in one-hour for key pathogens; listeria, salmonella, and E. coli. Its RNA/DNA-based patented technology is being tested by the USDA, which will validate SnapDNA products for use by government agencies including the FDA, FSIS, and CDC, says the company.
Colorado-based Bond Pet Foods is taking a single celled micro-organism and adding DNA sequences (which can be 3D printed using synthetic biology techniques) which effectively instruct the organism to produce the proteins found in chicken after being fed sugars and other nutrients in a big fermentation tank. The resulting protein can then be harvested and mixed with other foods to make petfood.
California-based Planetarians has developed a process to upcycle defatted sunflower seeds to create flour with twice the protein and three times the fiber for the same price as wheat flour.
Israeli-based Hargol FoodTech has built the world's first commercial grasshopper farm for human food consumption.
North Dakota-based Bushel has built a software platform that allows grain elevators to digitally connect with their growers to deliver near real-time information that informs better business decisions.
California-based Osmo Systems has built a water quality sensing system (osmobot) to help shrimp farmers monitor ponds online.
India-based CropIn Technology Solutions is on a mission to "digitize every farm" enabling farmers to archive patterns and predict trends.
California-based Corumat makes 100% plant-based, compostable, packaging that it claims generates 60% less CO2 during production than plastics traditionally used to make red party cups.
Canada-based Healthy Cow has developed ProPreg, a combination of pre-and probiotics designed to support and augment an optimal microbiome in dairy cows.


Cypress, CA – November 5, 2018 – Yamaha Motor Europe (YME) today introduced the all-new 700 Adventure Touring motorcycle during an exclusive press conference in Milan, Italy, and Yamaha Motor Corporation, U.S.A. (YMUS) announced that this exciting all-new motorcycle would also be produced for the U.S. market. Production for the European model will take place in France, while production for the U.S. model will take place at Yamaha's factory in Japan. Due to differing government regulatory standards and factory production line schedules, the all-new Ténéré 700 is scheduled to arrive at U.S. dealerships in the second half of 2020.


As the need for mobile data continues to increase, phone companies are trying to meet the demand. ATT&T is one of multiple companies to offer rollover data. This works similarly to rollover minutes from the days when the number of minutes a user could use per month was a priority.
Customers may find bug fixes and performance enhancements to be more important than viewing rollover data because the app only has a rating of 2.5/5. There are quite a few complaints saying the app doesn't work for Windows Phone at all. Hopefully the recent updates have made the app more stable.


On a hazy summer morning, Sheila and I drive 30 miles to the east end of Canton, South Dakota to see a fence. The fence in the distance holds one of South Dakota's most painful secrets. Located between the fourth and fifth holes of the Hiawatha Golf Course, the fence surrounds a cemetery where 121 Native Americans are buried who were held at the Hiawatha Asylum for Insane Indians. The only memorial is a large granite stone with a bronze plaque naming the individuals. The Indian Office decided that individual grave markers were too expensive.
The grounds of the asylum are now home to the Hiawatha Golf Course, a gun club, and a 4-H fairground. The cemetery is very close to Sioux Falls where I live. To get to the cemetery, Sheila and I borrow a cart from the local golf shop. We head out on a paved path, past the first hole, and over a little bridge. Then straight up the hill when we finally see the worn, weathered split rail fence in the distance, we pull over. Signs warn the golfers not to play within the boundary of the fence. Black, red, white, and yellow strips of ribbon blow in the summer breeze. Each ribbon on the fence is left by tribal members to remember and honor their ancestors. The ribbons mark the four directions.
The Hiawatha Asylum for Insane Indians opened in 1903 and closed its doors in 1934. Some current researchers call the asylum the "little Dachau" of the Dakotas. The large brick building, with pointed roofs, a grand front porch, and a swing set on the front lawn, appears hospitable at first glance. But reservation agents once shipped hundreds of Indians from reservations to the asylum and warehoused them within its walls. Many inmates never saw their family members again.
The asylum's first director had no medical background and in 1929 when the US government sent Dr. Samuel Silk to inspect conditions he filed a report: "Three patients were found padlocked in rooms. One was sick in bed, supposed to be suffering from a brain tumor, being bedridden and helpless…a boy about 10 years of age was in a strait jacket lying in his bed…one patient who had been in the hospital six years was padlocked in a room and, according to the attendant, had been secluded in this room for nearly three years." Really, it was a prison, disguised as a hospital. It was a warehouse for the "crazy" Indians. However, few Indians were crazy; most were forced off the reservation and sent to Hiawatha because they were viewed as troublemakers by the Bureau of Indian Affairs agents.
Back in the early 1900s, railroad cars would stop at the great iron gate and fence so visitors could see the "crazy" Indians paraded in front of them. Approximately 374 Indians were brought to the Asylum; most could not communicate with each other because they couldn't speak each other's language. Women were strapped to beds, children neglected, and men left in their own filth. Chamber pots filled to the brim and were left in hallways to fester. When the asylum closed in 1934, the Canton Chamber of Commerce pleaded to keep it open because it provided revenue for the town.
I go to this spot two or three times a year. The juxtaposition of golf course with cemetery gives me the creeps. The US Government cared so little for these 121 inmates that they were buried en masse and never given proper funerals.
The proximity of the burial site only intensifies the horrors of its distant past. Would I have done anything differently than the residents of Canton did? I come from Scandinavian ancestors who populated towns just like Canton, and I work at Augustana University where at least one of the buildings contains bricks from the original asylum. What would I have known about this place? Would I have heard the cries of those who came to find their relatives among the inmates?
Sheila and I walk around the faint outlines of plots where individuals were buried. Crosses made from simple branches strapped together with twine mark two of the graves. The deaths of these people are not shadows of a distant past. Family members visit every year to honor their loved ones. They are reminders to me of the present–of feeling the horrible dissonance of walking around a cemetery located on a golf course. We watch someone tee off. We take a few more pictures and decide to return to the clubhouse.
The fence recedes from our view as we drive away in our borrowed golf cart. The fairways are bone dry; we need rain to bring life back to the brown, brittle grasses. The house of the asylum's superintendent has been moved and renamed the Newton Hills Lodge in Newton Hills State Park. It can be rented for large gatherings. Nothing but the cemetery remains to remind people what happened.
Listen to Confluences, by Gary Pederson, joining the sounds of Native American flute and the harmonica in both harmonious and dissonant ways. It is based on the Hiawatha Cemetery painting and uses folk and hymn tunes to evoke both comfort and distress.


text:
Q: habilitar y deshabiltar div con hover

  <input type="text" >
  Input 1

  <input type="text" >
  Input 2

  <input type="text" >
  Input 3

  <input type="text" >
  Input 4

Quiero deshabilitar un grupo de divs al poner el mouse sobre ellos pero quiero volver a habilitarlos cuando el mouse no este sobre ellos. El problema es que una vez deshabilitados no puedo volver a habilitar esos divs.
  $('.desabilitar').hover(
           function () {
              $('.desabilitar').prop('disabled', true);
           },
           function () {
              $('.desabilitar').prop('disabled', false);
           }
        );

A: Lo que tu debes hacer es deshabilitar los inputs dentro del div y puedes lograrlo con $('.desabilitar :input').prop('disabled', true);

$('.desabilitar').hover(
     function () {
        $('.desabilitar :input').prop('disabled', true);
     },
     function () {
        $('.desabilitar :input').prop('disabled', false);
     }
  );

  <input type="text" >
  Input 1

  <input type="text" >
  Input 2

  <input type="text" >
  Input 3

  <input type="text" >
  Input 4

Si lo que necesitas es deshabilitar el/los inputs DENTRO del div que haces hover debes aplicar

$('.desabilitar').hover(
   function () {
      $(this).children('input').prop('disabled', true);
   },
   function () {
      $(this).children('input').prop('disabled', false);
   }
);

  <input type="text" >
  Input 1

  <input type="text" >
  Input 2

  <input type="text" >
  Input 3

  <input type="text" >
  Input 4

A: Según entiendo disabled no es un propiedad estándar de div pero sí de input, esto por disable es para controles y un div es un elemento estructural cuya finalidad es agrupar elementos.
Dado que la clase "deshabilitar" está aplicada al div, para hacer el mínimo de cambios al código del OP se deberá usar algo como $(this).children('input') para inhabilitar el input que es hijo del div sobre el que se hace el "hover".

$('.desabilitar').hover(
     function () {
        $(this).children('input').prop('disabled', true);
     },
     function () {
        $(this).children('input').prop('disabled', false);
     }
  );

  <input type="text" >
  Input 1

  <input type="text" >
  Input 2

  <input type="text" >
  Input 3

  <input type="text" >
  Input 4

Referencia
7.5.4 Grouping elements: the DIV and SPAN elements

A: No entendí bien quieres deshabilitar el input o todo el div? sie es el div debes apuntar al hijo del div que quieres deshabilitar o habilitar en este caso tu fubncion quedaria asi
$('.desabilitar').hover(
    function () {
       $('.desabilitar input').prop('disabled', true);
    },
    function () {
       $('.desabilitar input').prop('disabled', false);
    }
 );

redpajama_set_name: RedPajamaStackExchange