text:
\section*{Receiver operating characteristic (ROC) movies, universal ROC (UROC) curves, and coefficient of predictive ability (CPA)}}

\bigskip

{\centering \subsection*{Tilmann Gneiting$^{1,2}$ and Eva--Maria Walz$^{2,1}$}}

\medskip

\begin{center}

$^{1}$Computational Statistics, Heidelberg Institute for Theoretical Studies, Heidelberg, Germany

\smallskip

$^{2}$Institute for Stochastics, Karlsruhe Institute of Technology, Karlsruhe, Germany

\bigskip

\today

\bigskip

\section*{Main Text}

\end{center}

\begin{abstract}
Throughout science and technology, receiver operating characteristic
(ROC) curves and associated area under the curve (AUC) measures
constitute powerful tools for assessing the predictive abilities of
features, markers and tests in binary classification problems. Despite
its immense popularity, ROC analysis has been subject to a fundamental
restriction, in that it applies to dichotomous (yes or no) outcomes
only.  Here we introduce ROC movies and universal ROC (UROC) curves
that apply to just any ordinal or real-valued outcome, along with a
new, asymmetric coefficient of predictive ability (CPA) measure.  CPA
equals the area under the UROC curve, and admits appealing
interpretations in terms of probabilities and rank based covariances.
For binary outcomes CPA equals AUC, and for pairwise distinct outcomes
CPA relates linearly to Spearman's rank correlation coefficient.  ROC
movies, UROC curves, and CPA nest and generalize the tools of
classical ROC analysis, and are bound to supersede them in a wealth of
applications.  Their usage is illustrated in data examples from
biomedicine and meteorology, where CPA yields new insights in the
WeatherBench comparison of the predictive performance of convolutional
neural networks and physical-numerical models for weather prediction.
\end{abstract}
\section{Introduction}  \label{sec:introduction}

Originating from signal processing and psychology, popularized in the
1980s (Hanley and McNeil, 1982; Swets, 1988), and witnessing a surge of usage
in machine learnin (Bradley, 1997; Huang and Ling, 2005; Fawcett, 2006;
	Flach, 2016), receiver operating characteristic or relative operating
characteristic (ROC) curves and area under the ROC curve (AUC)
measures belong to the most widely used quantitative tools in science
and technology.  Strikingly, a Web of Science topic search for the
terms ``receiver operating characteristic'' or ``ROC'' yields well
over 15,000 scientific papers published in calendar year 2019
alone. In a nutshell, the ROC curve quantifies the potential value of
a real-valued classifier score, feature, marker, or test as a
predictor of a binary outcome.  To give a classical example,
Fig.~\ref{fig:ROCC_survival} illustrates the initial levels of two
biomedical markers, serum albumin and serum bilirubin, in a Mayo
Clinic trial on primary biliary cirrhosis (PBC), a chronic fatal
disease of the liver (Dickson et al., 1989).  While patient records
specify the duration of survival in days, traditional ROC analysis
mandates the reduction of the outcome to a binary event, which here we
take as survival beyond four years.  Assuming that higher marker
values are more indicative of survival, we can take any threshold
value to predict survival if the marker exceeds the threshold, and
non-survival otherwise. This type of binary classifier yields true
positives, false positives (erroneous predictions of survival), true
negatives, and false negatives (erroneous predictions of
non-survival).  The ROC curve is the piecewise linear curve that plots
the true positive rate, or sensitivity, versus the false positive
rate, or one minus the specificity, as the threshold for the
classifier moves through all possible values.

Despite its popularity, ROC analysis has been subject to a fundamental
shortcoming, namely, the restriction to binary outcomes.  Real-valued
outcomes are ubiquitous in scientific practice, and investigators have
been forced to artificially make them binary if the tools of ROC
analysis are to be applied.  In this light, researchers have been
seeking generalizations of ROC analysis that apply to just any type of
ordinal or real-valued outcome in natural ways (Etzioni et al., 1999;
	Heagerty et al., 2000; Bi and Benett, 2003; Pencina and D'Agostino 2004; Heagerty and Zheng, 2005; Rosset et al., 2005;
	Mason and Weigel, 2009; Hern{\'a}ndez-Orallo, 2013).  Still, notwithstanding decades of
scientific endeavor, a fully satisfactory generalization has been
elusive.

\begin{figure}[t]
	\centering
	\includegraphics[width = \textwidth]{ROCC_survival_sm}
	\caption{Traditional ROC curves for two biomedical markers, serum
		albumin and serum bilirubin, as predictors of patient survival
		beyond a threshold value of 1462 days (four years) in a Mayo Clinic
		trial.  (a), (c) Bar plots of marker levels conditional on survival
		or non-survival. The stronger shading results from overlap.  For
		bilirubin, we reverse orientation, as is customary in the biomedical
		literature.  (b) ROC curves and AUC values. The crosses correspond
		to binary classifiers at the feature thresholds indicated in panels
		(a) and (c), respectively. \label{fig:ROCC_survival}}
\end{figure}

In this paper, we propose a powerful generalization of ROC analysis,
which overcomes extant shortcomings, and introduce novel data science
tools in the form of the ROC movie, the universal ROC (UROC) curve,
and an associated asymmetric, rank based coefficient of predictive
ability (CPA) measure — tools that apply to just any linearly ordered
outcome, including both binary, ordinal, mixed discrete-continuous and
continuous variables.  The ROC movie comprises the sequence of the
traditional, static ROC curves as the linearly ordered outcome is
converted to a binary variable at successively higher thresholds.  The
UROC curve is a weighted average of the individual ROC curves that
constitute the ROC movie, with weights that depend on the class
configuration, as induced by the unique values of the outcome, in
judiciously predicated, well-defined ways.  CPA is a weighted average
of the individual AUC values in the very same way that the UROC curve
is a weighted average of the individual ROC curves that comprise the
ROC movie.  Hence, CPA equals the area under the UROC curve.  This set
of generalized tools reduces to the standard ROC curve and AUC when
applied to binary outcomes.  Moreover, key properties and relations
from conventional ROC theory extend to ROC movies, UROC curves, and
CPA in meaningful ways, to result in a coherent toolbox that properly
extends the standard ROC concept.  For a graphical preview, we return
to the survival data example from Fig.~\ref{fig:ROCC_survival}, where
the outcome was artificially made binary.  Equipped with the new set
of tools we no longer need to transform survival time into a specific
dichotomous outcome.  Figure \ref{fig:ROCM_survival} shows the ROC
movie, the UROC curve, and CPA for the survival dataset.

The remainder of the paper is organized as follows.  Section
\ref{sec:binary} provides a brief review of conventional ROC analysis
for dichotomous outcomes.  The key technical development is in
Sections \ref{sec:ROC_movies} and \ref{sec:CPA}, where we introduce
and study ROC movies, UROC curves, and the asymmetric, rank based CPA.
To illustrate the practical usage and relevance of the newly developed
tools, real data examples from survival analysis and weather
prediction are presented in Section \ref{sec:examples}.  We monitor
recent progress in numerical weather prediction (NWP) and shed new
light on a recent comparison of the predictive abilities of
convolutional neural networks (CNNs) vs.~traditional NWP models.  The
paper closes with a discussion in Section \ref{sec:discussion}.

\begin{figure}[t]
	\centering
	\animategraphics[label=myAnim7,height=70mm]{7}{imgeps4_out}{}{}
	\caption{ROC movies, UROC curves, and CPA for two biomedical markers,
		serum albumin and serum bilirubin, as predictors of patient survival
		(in days) in a Mayo Clinic trial.  The ROC movies show the
		traditional ROC curves for binary events that correspond to patient
		survival beyond successively higher thresholds.  The numbers at
		upper left show the current value of the threshold in days, at upper
		middle the respective relative weight, and at bottom right the AUC
		values.  The threshold value of 1462 days recovers the traditional
		ROC curves in Fig.~\ref{fig:ROCC_survival}.  The movie ends in a
		static screen with the UROC curves and CPA values for the two
		markers. \label{fig:ROCM_survival}}
\end{figure}

\section{Receiver operating characteristic (ROC) curves and area under the curve (AUC) for binary outcomes}
\label{sec:binary}

Before we to introduce ROC movies, UROC curves, and CPA, it is
essential that we establish notation and review the classical case of
ROC analysis for binary outcomes, as described in review articles and
monographs by Hanley and McNeil (1982), Swets (1988),
Bradley (1997), Pepe (2003), Fawcett (2006), and Flach (2016), among others.

\subsection{Binary setting}  \label{sec:setting}

Throughout this section we consider bivariate data of the form
\begin{equation}  \label{eq:binary}
(x_1,y_1), \ldots, (x_n,y_n) \, \in \, \real \times \{ 0, 1 \},
\end{equation}
where $x_i \in \real$ is a real-valued classifier score, feature, marker,
or covariate value, and $y_i \in \{ 0, 1 \}$ is a binary outcome, for
$i = 1, \ldots, n$.  Following the extant literature, we refer to $y =
1$ as the positive outcome and to $y = 0$ as the negative outcome, and
we assume that higher values of the feature are indicative of stronger
support for the positive outcome.  Throughout we assume that there is
at least one index $i \in \{ 1, \ldots, n \}$ with $y_i = 0$, and a
further index $j \in \{ 1, \ldots, n \}$ with $y_j = 1$.

\subsection{Receiver operating characteristic (ROC) curves}  \label{sec:ROCC}

We can use any threshold value $x \in \real$ to obtain a hard
classifier, by predicting a positive outcome for a feature value $>
x$, and predicting a negative outcome for a feature value $\leq x$.
If we compare to the actual outcome, four possibilities arise.  True
positive and true negative cases correspond to correctly classified
instances from class 1 and class 0.  Similarly, false positive and
false negative cases are misclassified instances from class 1 and
class 0, respectively.

Considering the data \eqref{eq:binary}, we obtain the respective
\textit{true positive rate}, \textit{hit rate}\/ or
\textit{sensitivity}\/ (se),
\[
{\rm se}(x) =
\frac{\frac{1}{n} \sum_{i=1}^n \mathbbm{1} \{ x_i > x, \hspace{0.2mm} y_i = 1 \}}{\frac{1}{n} \sum_{i=1}^n \mathbbm{1} \{ \hspace{0.2mm} y_i = 1 \}},
\]
and the \textit{false negative rate}, \textit{false alarm rate}\/ or
one minus the \textit{specificity}~(sp),
\[
1 - {\rm sp}(x) =
\frac{\frac{1}{n} \sum_{i=1}^n \mathbbm{1} \{ x_i > x, \hspace{0.2mm} y_i = 0 \}}{\frac{1}{n} \sum_{i=1}^n \mathbbm{1} \{ \hspace{0.2mm} y_i = 0 \}},
\]
at the threshold value $x \in \real$, where the indicator $\mathbbm{1} \{ A
\}$ equals one if the event $A$ is true and zero otherwise.

Evidently, it suffices to consider threshold values $x$ equal to any
of the unique values of $x_1, \ldots, x_n$ or some $x_0 < x_1$.  For
every $x$ of this form, we obtain a point
\[
(1 - {\rm sp}(x), {\rm se}(x))
\]
in the unit square.  Linear interpolation of the respective discrete
point set results in a piecewise linear curve from $(0,0)$ to $(1,1)$
that is called the \textit{receiver operating characteristic}\/ (ROC)
\textit{curve}.  For a mathematically oriented, detailed discussion of
the construction see Section 2 of Gneiting and Vogel (2018).

\subsection{Area under the curve (AUC)}  \label{sec:AUC}

The area under the ROC curve is a widely used measure of the
predictive potential of a feature and generally referred to as the
\textit{area under the curve}\/ (AUC).

In what follows, a well-known interpretation of \AUC \ in terms of
probabilities will be useful. To this end, we define the function
\begin{equation}  \label{eq:s}
s(x,x') = \mathbbm{1} \{ x < x' \} + \frac{1}{2} \mathbbm{1} \{ x = x' \},
\end{equation}
where $x, x' \in \real$.  For subsequent use, note that if $x$ and
$x'$ are ranked within a list, and ties are resolved by assigning
equal ranks within tied groups, then $s(x,x') = s(\rk(x),\rk(x'))$.

We now change notation and refer to the feature values in class $i \in
\{ 0, 1 \}$ as $x_{ik}$ for $k = 1, \ldots, n_i$, where $n_0 =
\sum_{i=1}^n \mathbbm{1} \{ y_i = 0 \}$ and $n_1 = \sum_{i=1}^n \mathbbm{1} \{ y_i =
1 \}$, respectively.  Thus, we have rewritten \eqref{eq:binary} as
\begin{equation}  \label{eq:binary2class}
(x_{01},0), \ldots, (x_{0n_0},0), (x_{11},1), \ldots, (x_{1n_1},1) \, \in \, \real \times \{ 0, 1 \}.
\end{equation}
Using the new notation, Result 4.10 of Pepe (2003) states that
\begin{equation}  \label{eq:MWU}
\textrm{AUC} = \frac{1}{n_0 n_1} \, \sum_{i=1}^{n_0} \sum_{j = 1}^{n_1} s(x_{0i},x_{1j}).
\end{equation}
In words, AUC equals the probability that under random sampling a
feature value from a positive instance is greater than a feature value
from a negative instance, with any ties resolved at random.  Expressed
differently, AUC equals the tie-adjusted probability of concordance in
feature--outcome pairs, where we define instances $(x,y) \in \real^2$
and $(x',y') \in \real^2$ with $y \not= y'$ to be
\textit{concordant}\/ if either $x > x'$ and $y > y'$, or $x < x'$ and
$y < y'$.  Similarly, instances $(x,y)$ and $(x',y')$ with $y \not=
y'$ are \textit{discordant}\/ if either $x > x'$ and $y < y'$, or $x <
x'$ and $y > y'$.

Further investigation reveals a close connection to Somers' $D$, a
classical measure of ordinal association (Somers, 1962).  This
measure is defined as
\[
D = \frac{n_c - n_d}{n_0 n_1},
\]
where $n_0 n_1$ is the total number of pairs with distinct outcomes
that arise from the data in \eqref{eq:binary2class}, $n_c$ is the
number of concordant pairs, and $n_d$ is the number of discordant
pairs.  Finally, let $n_e$ be the number of pairs for which the
feature values are equal.  The relationship \eqref{eq:MWU} yields
\[
\AUC = \frac{n_c}{n_0 n_1} + \frac{1}{2} \frac{n_e}{n_0 n_1},
\]
and as $n_0 n_1 = n_c + n_d + n_e$, it follows that
\begin{equation}  \label{eq:SomersD}
\AUC = \frac{1}{2} \left( D + 1 \right)
\end{equation}
relates linearly to Somers' $D$. To give an example, suppose that the real-valued outcome $Y$ and the
features $X$, $X'$ and $X''$ are jointly Gaussian.  Specifically, we
assume that the joint distribution of $(Y, X, X', X'')$ is
multivariate normal with covariance matrix
\begin{align}  \label{eq:MVN}
\begin{pmatrix}
1 & 0.8 & 0.5 & 0.2 \\
0.8 & 1 & 0.8 & 0.5 \\
0.5 & 0.8 & 1 & 0.8 \\
0.2 & 0.5 & 0.8 & 1 \\
\end{pmatrix}
.
\end{align}
In order to apply classical ROC analysis, the real-valued outcome $Y$
needs to be converted to a binary variable, namely, an event of the
type $Y_\theta = \mathbbm{1} \{ Y \geq \theta \}$ of $Y$ being greater than
or equal to a certain threshold value $\theta$.  Figure \ref{fig:ROCC}
shows ROC curves for the features $X$, $X'$ and $X''$ as a predictor
of the induced binary variable $Y_1$, based on a sample of size $n =
400$.  The AUC values for $X$, $X'$ and $X''$ as a predictor of $Y_1$
are .91, .72 and .61, respectively.  While the choice of the threshold
value $\theta = 1$ is arbitrary and artificial, as is often the case
in practice, its specification is mandated by the limitations of
classical ROC analysis.

\begin{figure}[t]
	\centering
	\includegraphics[width = 105mm]{ROCC_sim}
	\caption{Traditional ROC curves and AUC values for the features $X$,
		$X'$ and $X''$ as predictors of the binary outcome $Y_1 = \mathbbm{1} \{ Y
		\geq 1 \}$ in the simulation example of Section \ref{sec:AUC}, based
		on a sample of size $n = 400$.  \label{fig:ROCC}}
\end{figure}

\subsection{Key properties}  \label{sec:initialsummary}

A key requirement for a persuasive generalization of classical ROC
analysis is the reduction to ROC curves and AUC if the outcomes are
binary.  Furthermore, well established desirable properties from ROC
analysis ought to be retained.  To facilitate judging whether the
generalizations in Sections \ref{sec:ROC_movies} and \ref{sec:CPA}
satisfy these desiderata, we summarize key properties of ROC curves
and AUC in the following (slightly informal) listing.
\begin{itemize}
	\item[(1)] The ROC curve and AUC are straightforward to compute and
	interpret, in the (rough) sense of \textit{the larger the better}.
	\item[(2)] AUC attains values between 0 and 1 and relates linearly to
	Somers' $D$.  For a perfect feature, $\AUC = 1$ and $D = 1$; for a
	feature that is independent of the binary outcome, $\AUC =
	\frac{1}{2}$ und $D = 0$.
	\item[(3)] The numerical value of AUC admits an interpretation as the
	probability of concordance for feature--outcome pairs.
	\item[(4)] The ROC curve and AUC are purely rank based and, therefore,
	invariant under strictly increasing transformations. Specifically,
	if $\phi : \real \to \real$ is a strictly increasing function, then
	the ROC curve and AUC computed from
	\begin{equation}  \label{eq:binaryT}
	(\phi(x_1),y_1), \ldots, (\phi(x_n),y_n) \, \in \, \real \times \{ 0, 1 \}
	\end{equation}
	are the same as the ROC curve and AUC computed from
	\eqref{eq:binary}.
\end{itemize}
As an immediate consequence of the latter property, ROC curves and AUC
assess the discrimination ability or \textit{potential}\/ predictive
ability of a classifier, feature, marker, or test (Wilks, 2019).
Distinctly different methods are called for if one seeks to evaluate a
classifier's \textit{actual}\/ value in any given applied setting (Adams and Hands, 1999; Hern{\'a}ndez-Orallo et al., 2012; Ehm et al., 2016).

\section{ROC movies and universal ROC (UROC) curves for real-valued outcomes}
\label{sec:ROC_movies}

As noted, traditional ROC analysis applies to binary outcomes only.
Thus, researchers working with real-valued outcomes, and desiring to
apply ROC analysis, need to convert and reduce to binary outcomes, by
thresholding artificially at a cut-off value.  Here we propose a
powerful generalization of ROC analysis, which overcomes extant
shortcomings, and introduce novel data analytic tools in the form of
the ROC movie, the universal ROC (UROC) curve, and an associated
asymmetric, rank based coefficient of predictive ability (CPA) measure
--- tools that apply to just any linearly ordered outcome, including
both binary, ordinal, mixed discrete-continuous and continuous
variables.

\subsection{General real-valued setting}  \label{sec:real}

Generalizing the binary setting in \eqref{eq:binary}, we now consider
bivariate data of the form
\begin{equation}  \label{eq:real}
(x_1,y_1), \ldots, (x_n,y_n) \, \in \, \real \times \real,
\end{equation}
where $x_i$ is a real-valued point forecast, regression output, feature, marker, or covariate value, and
$y_i$ is a real-valued outcome, for $i = 1, \ldots, n$.  Throughout we
assume that there are at least two unique values among the outcomes
$y_1, \ldots, y_n$.

The crux of the subsequent development lies in a conversion to a
sequence of binary problems.  To this end, we let
\[
z_1 < \cdots < z_m
\]
denote the $m \leq n$ distinct order statistics of $y_1, \ldots, y_n$,
and we define
\[
n_c = \sum_{i=1}^n \mathbbm{1} \{ y_i = z_c \}
\]
as the number of instances among the outcomes $y_1, \ldots, y_n$ that
equal $z_c$, for $c = 1, \ldots, m$, so that $n_1 + \cdots + n_m = n$.
We refer to the respective groups of instances as \textit{classes}.

Next we transform the real-valued outcomes $y_1, \ldots, y_n$ into
binary outcomes $\mathbbm{1} \{ y_1 \geq \theta \}, \ldots, \mathbbm{1} \{ y_n \geq
\theta \}$ relative to a threshold value $\theta \in \real$.  Thus,
instead of analysing the original problem in \eqref{eq:real}, we
consider a series of binary problems.  By construction, only values of
$\theta$ equal to the distinct order statistics $z_2 < \cdots < z_m$
result in nontrivial, unique sets of binary outcomes.  Therefore, we
consider $m - 1$ derived classification problems with binary data of
the form
\begin{equation}  \label{eq:real2binary}
(x_1, \mathbbm{1} \{ y_1 \geq z_{c+1} \}), \ldots, (x_n, \mathbbm{1} \{ y_n \geq z_{c+1} \})
\, \in \, \real \times \{ 0, 1 \},
\end{equation}
where $c = 1, \ldots, m - 1$.  As the derived problems are binary, all
the tools of traditional ROC analysis apply.

In the remainder of the section we describe our generalization of ROC
curves for binary data to ROC movies and universal ROC (UROC) curves
for real-valued data.  First, we argue that the $m - 1$ classical ROC
curves for the derived data in \eqref{eq:real2binary} can be merged
into a single dynamical display, to which we refer as a ROC movie
(Definition \ref{defn:ROCM}).  Then we define the UROC curve as a
judiciously weighted average of the classical ROC curves of which the
ROC movie is composed (Definition \ref{defn:UROC}).

Finally, we introduce a general measure of potential predictive
ability for features, termed the coefficient of predictive ability
(CPA).  CPA is a weighted average of the AUC values for the derived
binary problems in the very same way that the UROC curve is a weighted
average of the (classical) ROC curves that comprise the ROC movie.
Hence, CPA equals the area under the UROC curve (Definition
\ref{defn:CPA}).  Alternatively, CPA can be interpreted as a weighted
probability of concordance (Theorem \ref{thm:CPA2L}) or in terms of
rank based covariances (Theorem \ref{thm:CPA2cov}).  CPA reduces to
AUC if the outcomes are binary, and relates linearly to Spearman's
rank correlation coefficient if the outcomes are continuous (Theorems
\ref{thm:Spearman} and \ref{thm:Spearman.m}).

\subsection{ROC movies}  \label{sec:ROCM}

We consider the sequence of $m - 1$ classification problems for the
derived binary data in \eqref{eq:real2binary}.  For $c = 1, \ldots, m
- 1$, we let $\ROC_c$ denote the associated ROC curve, and we let
$\AUC_c$ be the respective AUC value.

\begin{definition}  \label{defn:ROCM}
	For data of the form \eqref{eq:real}, the ROC \textit{movie}\/ is the
	sequence $(\ROC_c)_{c = 1, \ldots, m-1}$ of the ROC curves for the
	induced binary data in \eqref{eq:real2binary}.
\end{definition}

If the original problem is binary there are $m = 2$ classes only, and
the ROC movie reduces to the classical ROC curve.  In case the outcome
attains $m \geq 3$ distinct values the ROC movie can be visualized by
displaying the associated sequence of $m - 1$ ROC curves.  In medical
survival analysis, the outcomes $y_1, \ldots, y_n$ in data of the form
\eqref{eq:real} are survival times, and the analysis is frequently
hampered by censoring, as patients drop out of studies.  In this
setting, Etzioni et al. (1999) and Heagerty et al. (2000) introduced the
notion of time-dependent ROC curves, which are classical ROC curves
for the binary indicator $\mathbbm{1} \{ y_i \geq t \}$ of survival through
(follow-up) time $t$.  If the survival times considered correspond to
the unique values in $y_1, \ldots, y_n$, the sequence of
time-dependent ROC curves becomes a ROC movie in the sense of
Definition \ref{defn:ROCM}.  If the number $m \leq n$ of classes is
small or modest, the generation of the ROC movie is straightforward.
Adaptations might be required as $m$ grows, and we tend to this
question in Section \ref{sec:NWP}.

\begin{figure}[t]
	\centering
	\animategraphics[label=myAnim7,height=75mm]{28}{imgeps3_reduced}{0}{}
	\caption{ROC movies and UROC curves for the features $X$, $X'$ and
		$X''$ as predictors of the real-valued outcome $Y$ in the simulation
		example of Section \ref{sec:AUC}, based on the same sample as in
		Fig.~\ref{fig:ROCC}.  In the ROC movies, the number at upper left
		shows the threshold under consideration, the number at upper center
		the relative weight $w_c/\max_{l = 1, \ldots, m-1} w_l$ from
		\eqref{eq:w}, and the numbers at bottom right the respective AUC
		values.  \label{fig:ROCM}}
\end{figure}

We have implemented ROC movies, UROC curves, and CPA within the
\texttt{uroc} package for the statistical programming language R
(R Core Team, 2020) where the \texttt{animation} package of
Xie (2013) provides functionality for converting R images into a
GIF animation, based on the external software \texttt{ImageMagick}.
The \texttt{uroc} package can be downloaded from
\url{htpps://github.com/evwalz/uroc}.  In addition, a Python
(Python, 2019) implementation is available at
\url{https://github.com/evwalz/rocm_python_nwp}.  Returning to the
example of Section \ref{sec:AUC}, Fig.~\ref{fig:ROCM} compares the
features $X$, $X'$ and $X''$ as predictors of the real-valued outcome
$Y$ in a joint display of the three ROC movies and UROC curves, based
on the same sample of size $n = 400$ as in Fig.~\ref{fig:ROCC}.  In
the ROC movies, the threshold $z = 1.00$ recovers the traditional ROC
curves in Fig.~\ref{fig:ROCC}.

\subsection{Universal ROC (UROC) curves}  \label{sec:UROC}

Next we propose a simple and efficient way of subsuming a ROC movie
for data of the form \eqref{eq:real} into a single, static graphical
display.  As before, let $z_1 < \cdots < z_m$ denote the distinct
values of $y_1, \ldots, y_n$, let $n_c = \sum_{i=1}^n \mathbbm{1} \{ y_i =
z_c \}$, and let $\ROC_c$ denote the (classical) ROC curve associated
with the binary problem in \eqref{eq:real2binary}, for $c = 1, \ldots,
m - 1$.

By Theorem 5 of Gneiting and Vogel (2018), there is a natural bijection
between the class of the ROC curves and the class of the cumulative
distribution functions (CDFs) of Borel probability measures on the
unit interval.  In particular, any ROC curve can be associated with a
non-decreasing, right-continuous function $R : [0,1] \to [0,1]$ such
that $R(0) = 0$ and $R(1) = 1$.  Hence, any convex combination of the
ROC curves $\ROC_1, \ldots, \ROC_{m-1}$ can also be associated with a
non-decreasing, right-continuous function on the unit interval.  It is
in this sense that we define the following; in a nutshell, the UROC
curve averages the traditional ROC curves of which the ROC movie is
composed.

\begin{definition}  \label{defn:UROC}
	For data of the form \eqref{eq:real}, the \textit{universal receiver
		operating characteristic}\/ (UROC) \textit{curve} is the curve
	associated with the function
	\begin{equation}  \label{eq:UROC}
	\sum_{c = 1}^{m-1} w_c \, \ROC_c
	\end{equation}
	on the unit interval, with weights
	\begin{equation}  \label{eq:w}
	w_c = \left. \left( \, \sum_{i=1}^c n_i \sum_{i=c+1}^m n_i \right) \right/
	\left( \sum_{i=1}^{m-1} \sum_{j=i+1}^m ( \hspace{0.2mm} j-i) \hspace{0.2mm} n_i n_j \right)
	\end{equation}
	for $c = 1, \ldots, m - 1$.
\end{definition}

Importantly, the weights in \eqref{eq:w} depend on the data in
\eqref{eq:real} via the outcomes $y_1, \ldots, y_n$ only.  Thus, they
are independent of the feature values and can be used meaningfully in
order to compare and rank features.  Their specific choice is
justified in Theorems \ref{thm:CPA2L} and \ref{thm:CPA2cov} below.
Clearly, the weights are nonnegative and sum to one.  If $m = n$ then
$n_1 = \cdots = n_m = 1$, and \eqref{eq:w} reduces to
\begin{equation}  \label{eq:wnoties}
w_c = 6 \, \frac{c(n-c)}{n(n^2-1)} \quad \textrm{for} \quad c = 1, \ldots, n - 1;
\end{equation}
so the weights are quadratic in the rank $c$ and symmetric about the
inner most rank(s), at which they attain a maximum.  As we will see,
our choice of weights has the effect that in this setting the area
under the UROC curve, to which we refer as a general coefficient of
predictive ability (CPA), relates linearly to Spearman's rank
correlation coefficient, in the same way that AUC relates linearly to
Somers' $D$.

In Fig.~\ref{fig:ROCM} the UROC curves appear in the final static
screen, subsequent to the ROC movies.  Within each ROC movie, the
individual frames show the ROC curve $\ROC_c$ for the feature
considered.  Furthermore, we display the threshold $z_c$, the
\textit{relative}\/ weight from \eqref{eq:w} (the actual weight
normalized to the unit interval, i.e., we show $w_c/\max_{l = 1,
	\ldots, m-1} w_l$), and $\AUC_c$, respectively, for $c = 1, \ldots,
m - 1$.  Once more we emphasize that the use of ROC movies, UROC
curves, and CPA frees researchers from the need to select ---
typically, arbitrary --- threshold values and binarize, as mandated by
classical ROC analysis.

Of course, if specific threshold values are of particular substantive
interest, the respective ROC curves can be extracted from the ROC
movie.  Furthermore, in comparing competing features relative to
particular thresholds, weightings other than \eqref{eq:w} can be
considered, and it can be useful to plot the respective values of
$\AUC_c$ versus the threshold value $z_c$.  Displays of this type have
been introduced and studied by Rosset et al. (2015).

\section{Coefficient of predictive ability (CPA)}  \label{sec:CPA}

We proceed to define the coefficient of predictive ability (CPA) as a
general measure of potential predictive ability.

\begin{definition}  \label{defn:CPA}
	For data of the form \eqref{eq:real} and weights $w_1, \ldots,
	w_{m-1}$ as in \eqref{eq:w}, the \textit{coefficient of predictive
		ability} (CPA) is defined as
	\begin{equation}  \label{eq:CPA}
	\CPA = \sum_{c=1}^{m-1} w_c \, \AUC_c.
	\end{equation}
	In words, CPA equals the area under the UROC curve.
\end{definition}

Importantly, ROC movies, UROC curves, and CPA satisfy a fundamental
requirement on any generalization of ROC curves and AUC, in that they
reduce to the classical notions when applied to a binary problem,
whence $m = 2$ in \eqref{eq:UROC} and \eqref{eq:CPA}, respectively.

\subsection{Interpretation as a weighted probability}  \label{sec:interpretation}

We now express CPA in terms of pairwise comparisons via the function
$s$ in \eqref{eq:s}.  To this end, we usefully change notation for the
data in \eqref{eq:real} and refer to the feature values in class $c
\in \{ 1, \ldots, m \}$ as $x_{ck}$, for $k = 1, \ldots, n_c$.  Thus,
we rewrite \eqref{eq:real} as
\begin{equation}  \label{eq:real2class}
(x_{11}, z_1), \ldots, (x_{1n_1},z_1), \, \ldots, \, (x_{m1}, z_m),
\ldots, (x_{mn_m},z_m) \, \in \, \real \times \real,
\end{equation}
where $z_1 < \cdots < z_m$ are the unique order statistics of $y_1,
\ldots, y_n$ and $n_c = \sum_{i=1}^n \mathbbm{1} \{ y_i = z_c \}$, for $c =
1, \ldots, m$.

\begin{theorem}  \label{thm:CPA2L}
	For data of the form \eqref{eq:real2class},
	\begin{equation}  \label{eq:CPA2L}
	{\rm CPA} =
	\frac{\sum_{i=1}^{m-1} \sum_{j=i+1}^m \sum_{k=1}^{n_i} \sum_{l=1}^{n_j} ( \hspace{0.2mm} j-i) \, s(x_{ik},x_{jl})}
	{\sum_{i=1}^{m-1} \sum_{j=i+1}^m ( \hspace{0.2mm} j-i) \, n_i n_j}.
	\end{equation}
\end{theorem}

\begin{proof}
	By \eqref{eq:MWU}, the individual AUC values satisfy
	\[
	\AUC_c =
	\frac{1}{\sum_{i=1}^c n_i \sum_{i=c+1}^m n_i}
	\sum_{i=1}^c \sum_{j=c+1}^m \sum_{k=1}^{n_i} \sum_{l=1}^{n_j} s(x_{ik},x_{jl})
	\]
	for $c = 1, \ldots, m - 1$.  In view of \eqref{eq:w} and \eqref{eq:CPA}, summation yields
	\begin{align*}
	\textrm{CPA}
	& = \sum_{c=1}^{m-1} w_c \, \textrm{AUC}_c \\
	& = \frac{\sum_{c=1}^{m-1} \sum_{i=1}^c \sum_{j=c+1}^m \sum_{k=1}^{n_i} \sum_{l=1}^{n_j} s(x_{ik},x_{jl})}{\sum_{i=1}^{m-1} \sum_{j=i+1}^m ( \hspace{0.2mm} j-i) \, n_i n_j} \\
	& = \frac{\sum_{i=1}^{m-1} \sum_{j=i+1}^m \sum_{k=1}^{n_{i}} \sum_{l=1}^{n_j} ( \hspace{0.2mm} j-i) \, s(x_{ik},x_{jl})}{\sum_{i=1}^{m-1} \sum_{j=i+1}^m ( \hspace{0.2mm} j-i) \, n_i n_j},
	\end{align*}
	as claimed.
\end{proof}

Thus, CPA is based on pairwise comparisons of feature values, counting
the number of concordant pairs in \eqref{eq:real2class}, adjusting to
a count of $\frac{1}{2}$ if feature values are tied, and weighting a
pair's contribution by a class based distance, $j - i$, between the
respective outcomes, $z_j > z_i$.  In other words, CPA equals a
weighted probability of concordance, with weights that grow linearly
in the class based distance between outcomes.

The specific form of CPA in \eqref{eq:CPA2L} invites comparison to a
widely used measure of discrimination in biomedical applications,
namely, the C \textit{index} (Harrell et al., 1996; Pencina and D'Agostino, 2004)
\begin{equation}  \label{eq:C}
{\rm C} =
\frac{\sum_{i=1}^{m-1} \sum_{j=i+1}^m \sum_{k=1}^{n_{i}} \sum_{l=1}^{n_j} s(x_{ik},x_{jl})}{\sum_{i=1}^{m-1} \sum_{j=i+1}^m n_i n_j}.
\end{equation}
If the outcomes are binary, both the C index and CPA reduce to AUC.
While CPA can be interpreted as a weighted probability of concordance,
C admits an interpretation as an unweighted probability, whence Mason and Weigel, (2009) recommend its use for administrative purposes.
However, the weighting in \eqref{eq:CPA2L} appears to be more
meaningful, as concordances between feature--outcome pairs with
outcomes that are far apart tend to be of greater practical relevance
than concordances between pairs with alike outcomes.  While CPA admits
the appealing, equivalent interpretation \eqref{eq:CPA} in terms of
binary AUC values and the area under the UROC curve, relationships of
this type are unavailable for the C index.

Subject to conditions, the C index relates linearly to Kendall's rank
correlation coefficient (Somers, 1962; Pencina and D'Agostino, 2004; Mason and Weigel, 2009).  In Section \ref{sec:Spearman} we
demonstrate the same type of relationship for CPA and Spearman's rank
correlation coefficient, thereby resolving a problem raised by Heagerty and Zheng (2005, p. 95). Just as the C index bridges and
generalizes AUC and Kendall's coefficient, CPA bridges and nests AUC
and Spearman's coefficient, with the added benefit of appealing
interpretations in terms of the area under the UROC curve and rank
based covariances.

\subsection{Representation in terms of covariances}  \label{sec:CPA2cov}

The key result in this section represents CPA in terms of the
covariance between the class of the outcome and the mid rank of the
feature, relative to the covariance between the class of the outcome
and the mid rank of the outcome itself.

The mid rank method handles ties by assigning the arithmetic average
of the ranks involved (Woodbury, 1940; Kruskal, 1958).  For instance, if the third to seventh
positions in a list are tied, their shared \textit{mid rank}~is
$\frac{1}{5} (3 + 4 + 5 + 6 + 7) = 5$.  This approach treats equal
values alike and guarantees that the sum of the ranks in any tied
group is unchanged from the case of no ties.  As before, if $y_i =
z_j$, where $z_1 < \cdots < z_m$ are the unique order statistics of
$y_1, \ldots, y_n$ in \eqref{eq:real}, we say that the
\textit{class}\/ of $y_i$ is $j$.  In brief, we express this as
$\cl(y_i) = j$.  Similarly, we refer to the mid rank of $x_i$ within
$x_1, \ldots, x_n$ as $\mrk(x_i)$.

\begin{theorem}  \label{thm:CPA2cov}
	Let the random vector\/ $(X,Y)$ be drawn from the empirical
	distribution of the data in \eqref{eq:real} or \eqref{eq:real2class}.
	Then
	\begin{equation}  \label{eq:CPA2cov}
	{\rm CPA} = \frac{1}{2} \left( \frac{{\rm cov}({\rm cl}(Y),\overline{\rm rk}(X))}
	{{\rm cov}({\rm cl}(Y),\overline{\rm rk}(Y))} + 1 \right).
	\end{equation}
\end{theorem}

\begin{proof}
	Suppose that the law of the random vector\/ $(X,Y)$ is the empirical
	distribution of the data in \eqref{eq:real}.  Based on the equivalent
	representation in \eqref{eq:real2class}, we find that
	\[
	\frac{\cov(\cl(Y),\mrk(X))}{\cov(\cl(Y),\mrk(Y))}
	= \frac{\sum_{i=1}^m \sum_{k=1}^{n_i} i \hspace{0.2mm} \hspace{0.2mm} \mrk(x_{ik}) - \frac{1}{2} \hspace{0.2mm} (n+1) \sum_{i=1}^m i \hspace{0.2mm} n_i}
	{\sum_{i=1}^m i \hspace{0.2mm} n_i \left( \sum_{j=0}^{i-1} n_j + \frac{1}{2} (n_i+1) \right) - \frac{1}{2} \hspace{0.2mm} (n+1) \sum_{i=1}^m i \hspace{0.2mm} n_i},
	\]
	where $n_0 = 0$.  Consequently, we can rewrite \eqref{eq:CPA2cov} as
	\begin{equation}  \label{eq:num.denom}
	{\rm CPA} = \frac{\sum_{i=1}^m \sum_{k=1}^{n_i} i \hspace{0.2mm} \hspace{0.2mm} \mrk(x_{ik})
		+ \sum_{i=1}^m i \hspace{0.2mm} n_i \left( \sum_{j=0}^{i-1} n_j + \frac{1}{2} n_i - n - \frac{1}{2} \right)}
	{\sum_{i=1}^m i \hspace{0.2mm} n_i \left( 2 \hspace{0.2mm} \sum_{j=0}^{i-1} n_j + n_i - n \right)}.
	\end{equation}

	We proceed to demonstrate that the numerator and denominator in
	\eqref{eq:CPA2L} equal the numerator and denominator in
	\eqref{eq:num.denom}, respectively.  To this end, we first compare
	feature values within classes and note that
	\[
	\sum_{i=1}^m \sum_{k=1}^{n_i} \sum_{l=1}^{n_i} i \hspace{0.2mm} s(x_{il},x_{ik})
	= \sum_{i=1}^m i \sum_{k=1}^{n_i} \left( n_i - k + \frac{1}{2} \right)
	= \frac{1}{2} \sum_{i=1}^m i \hspace{0.2mm} n_i^2;
	\]
	for if the feature values in class $i$ are all distinct, the largest
	one exceeds $n_i - 1$ others, the second largest exceeds $n_i - 2$
	others, and so on, and analogously in case of ties.  We now show the
	equality of the numerators in \eqref{eq:CPA2L} and
	\eqref{eq:num.denom}, in that
	\begin{align*}
	& \sum_{i=1}^{m-1} \sum_{j=i+1}^m \sum_{k=1}^{n_i} \sum_{l=1}^{n_j} ( \hspace{0.2mm} j-i) \, s(x_{ik},x_{jl}) \\
	& = \sum_{i=1}^{m-1} \sum_{j=i+1}^m \sum_{k=1}^{n_i} \sum_{l=1}^{n_j} j \hspace{0.2mm} s(x_{ik},x_{jl})
	- \sum_{i=1}^{m-1} \sum_{j=i+1}^m \sum_{k=1}^{n_i} \sum_{l=1}^{n_j} i \hspace{0.2mm} s(x_{ik},x_{jl}) \\
	& \hspace{20mm} + \sum_{j=1}^{m-1} \sum_{i=j+1}^m \sum_{k=1}^{n_j} \sum_{l=1}^{n_i} j \hspace{0.2mm} s(x_{ik},x_{jl})
	- \sum_{j=1}^{m-1} \sum_{i=j+1}^m \sum_{k=1}^{n_j} \sum_{l=1}^{n_i} j \hspace{0.2mm} s(x_{ik},x_{jl}) \\
	& = \sum_{i=1}^m \sum_{\substack{j=1 \\ j \neq i}}^m \sum_{k=1}^{n_i} \sum_{l=1}^{n_j} j \hspace{0.2mm} s(x_{ik},x_{jl})
	- \sum_{i=1}^{m-1} \sum_{j=i+1}^m \sum_{k=1}^{n_i} \sum_{l=1}^{n_j} i \left( s(x_{jl},x_{ik}) + s(x_{ik},x_{jl}) \right) \\
	& = \sum_{j=1}^m \sum_{l=1}^{n_j} j \left( \hspace{0.2mm} \mrk(x_{jl}) - \frac{1}{2} \right) - \sum_{i=1}^m \sum_{k=1}^{n_i} \sum_{l=1}^{n_i} i \hspace{0.2mm} s(x_{il},x_{ik})
	- \sum_{i=1}^{m-1} \sum_{j=i+1}^m i \hspace{0.2mm} n_i n_j \\
	& = \sum_{i=1}^m \sum_{k=1}^{n_i} i \hspace{0.2mm} \mrk(x_{ik)} - \frac{1}{2} \sum_{i=1}^m i \hspace{0.2mm} n_i - \frac{1}{2} \sum_{i=1}^m i \hspace{0.2mm} n_i^2
	- n \sum_{i=1}^{m-1} i \hspace{0.2mm} n_i + \sum_{i=1}^{m-1} i \hspace{0.2mm} n_i \sum_{j=0}^i n_j \\
	& = \sum_{i=1}^m \sum_{k=1}^{n_i} i \hspace{0.2mm} \mrk(x_{ik)} - \frac{1}{2} \sum_{i=1}^m i \hspace{0.2mm} n_i - \frac{1}{2} \sum_{i=1}^m i \hspace{0.2mm} n_i^2
	- n \sum_{i=1}^m i \hspace{0.2mm} n_i + \sum_{i=1}^m i \hspace{0.2mm} n_i \sum_{j=0}^i n_j \\
	& = \sum_{i=1}^m \sum_{k=1}^{n_i} i \hspace{0.2mm} \mrk(x_{ik)}
	+ \sum_{i=1}^m i \hspace{0.2mm} n_i \left( \sum_{j=0}^{i-1} n_j + \frac{1}{2} n_i - n - \frac{1}{2} \right).
	\end{align*}
	As for the denominators,
	\begin{align*}
	& \sum_{i=1}^{m-1} \sum_{j=i+1}^m ( \hspace{0.2mm} j-i) \, n_i n_j \\
	& \hspace{4mm} = \sum_{i=1}^{m-1} \sum_{j=i+1}^{m} j \hspace{0.2mm} n_i n_j - \sum_{i=1}^{m-1} \sum_{j=i+1}^m i \hspace{0.2mm} n_i n_j \\
	& \hspace{4mm} = \sum_{i=1}^m i \hspace{0.2mm} n_i \sum_{k=0}^{i-1} n_k - n \sum_{i=1}^{m-1} i \hspace{0.2mm} n_i + \sum_{i=1}^{m-1} i \hspace{0.2mm} n_i \sum_{k=1}^i n_k \\
	& \hspace{4mm} = 2 \sum_{i=1}^m i n_i \sum_{k=0}^{i-1} n_k - n \sum_{i=1}^{m-1} i \hspace{0.2mm} n_i + \sum_{i=1}^{m-1} i \hspace{0.2mm} n_i^2
	+ \sum_{i=1}^{m-1} i \hspace{0.2mm} n_i \sum_{k=0}^{i-1} n_k - \sum_{i=1}^m i \hspace{0.2mm} n_i \sum_{k=0}^{i-1} n_k \\
	& \hspace{4mm} = 2 \sum_{i=1}^m i \hspace{0.2mm} n_i \sum_{k=0}^{i-1} n_k - n \sum_{i=1}^{m-1} i \hspace{0.2mm} n_i + \sum_{i=1}^{m-1} i \hspace{0.2mm} n_i^2 - nmn_m + mn_m^2 \\
	& \hspace{4mm} = 2 \sum_{i=1}^m i \hspace{0.2mm} n_i \sum_{k=0}^{i-1} n_k - n \sum_{i=1}^m i \hspace{0.2mm} n_i + \sum_{i=1}^m i \hspace{0.2mm} n_i^2 \\
	& \hspace{4mm} = \sum_{i=1}^m i \hspace{0.2mm} n_i \left( 2 \hspace{0.2mm} \sum_{j=0}^{i-1} n_j + n_i - n \right),
	\end{align*}
	whence the proof is complete.
\end{proof}

Interestingly, the representation \eqref{eq:CPA2cov} in terms of rank
and class based covariances appears to be new even in the special case
when the outcomes are binary, so that CPA reduces to AUC.  The
representation also sheds new light on the asymmetry of CPA, in that,
in general, the value of CPA changes if we transpose the roles of the
feature and the outcome.  In stark contrast to customarily used
measures of bivariate association and dependence, which are
necessarily symmetric (Ne\v{s}lehov\'{a}, 2007; Reshef et al., 2011; Weihs et al., 2018), CPA is directed and aimed at
quantifying predictive potential.

\subsection{Relationship to Spearman's rank correlation coefficient}  \label{sec:Spearman}

Spearman's rank correlation coefficient $\rho_{\rm S}$ for data of the
form \eqref{eq:real} is generally understood as Pearson's correlation coefficient
applied to the respective ranks (Spearman, 1904).  In case there are no ties in
either $x_1, \ldots, x_n$ nor $y_1, \ldots, y_n$, the concept is
unambiguous, and Spearman's coefficient can be computed as
\begin{equation}  \label{eq:def_rho_S}
\rho_{\rm S} = 1 - \frac{6}{n(n^2-1)} \sum_{i=1}^n \left( \rk(x_i) - \rk(y_i) \right)^2,
\end{equation}
where $\rk(x_i)$ denotes the rank of $x_i$ within $x_1, \ldots, x_n$,
and $\rk(y_i)$ the rank of $y_i$ within $y_1, \ldots, y_n$,

In this setting CPA relates linearly to Spearman's rank correlation
coefficient $\rho_{\rm S}$, in the very same way that AUC relates to
Somers' $D$ in eq.~\eqref{eq:SomersD}.

\begin{theorem}  \label{thm:Spearman}
	In the case of no ties,
	\begin{equation}  \label{eq:rho_S}
	{\rm CPA} = \frac{1}{2} \left( \hspace{0.2mm} \rho_{\rm S} + 1 \right).
	\end{equation}
\end{theorem}

Indeed, in case there are no ties, both mid ranks and classes reduce
to ranks proper, and then \eqref{eq:rho_S} is readily identified as a
special case of \eqref{eq:CPA2cov}.  For an alternative proof, in the
absence of ties the weights $w_{c}$ in \eqref{eq:w} are of the form
\eqref{eq:wnoties}.  The stated result then follows upon combining the
defining equation \eqref{eq:UROC}, the equality stated at the bottom
of the left column of page 4 in Rosset et al. (2005), as well as
equation (5) in the same reference.

Note that CPA becomes symmetric in this case, as its value remains
unchanged if we transpose the roles of the feature and the outcome.
Furthermore, if the joint distribution of a bivariate random vector
$(X,Y)$ is continuous, and we think of the data in \eqref{eq:real} as
a sample from the respective population, then, by applying Definition
\ref{defn:CPA} and Theorem \ref{thm:Spearman} in the large sample
limit, and taking \eqref{eq:wnoties} into account, we (informally)
obtain a population version of CPA, namely,
\begin{equation}  \label{eq:S2CPA}
\CPA
= 6 \int_0^1 \alpha (1 - \alpha) \, \AUC_\alpha \, \rm{d} \alpha
= \frac{1}{2} \left( \hspace{0.2mm} \rho_{\rm S} + 1 \right),
\end{equation}
where $\AUC_\alpha$ is the population version of AUC for $(X, \mathbbm{1} \{
Y \geq q_\alpha \})$, with $q_\alpha$ denoting the $\alpha$-quantile
of the marginal law of $Y$.  We defer a rigorous derivation of
\eqref{eq:S2CPA} to future work and stress that, as both $X$ and $Y$
are continuous here, their roles can be interchanged.

Under the assumption of multivariate normality, the population version
of Spearman's $\rho_{\rm S}$ is related to Pearson's correlation
coefficient $r$ as
\begin{equation}  \label{eq:P2S}
\rho_{\rm S} = \frac{6}{\pi} \arcsin \frac{r}{2};
\end{equation}
see, e.g., Kruskal (1958).  Returning to the example in Section
\ref{sec:AUC}, where $(Y, X, X', X'')$ is jointly Gaussian with
covariance matrix \eqref{eq:MVN}, Table \ref{tab:MVN} states, for each
feature, the population values of Pearson's correlation coefficient
$r$, Spearman's rank correlation coefficient $\rho_{\rm S}$ and CPA
relative to the real-valued outcome $Y$, as derived from
\eqref{eq:S2CPA} and \eqref{eq:P2S}.  In Fig.~\ref{fig:ROCM} the CPA
values for the features appear along with the UROC curves in the final
static screen, subsequent to the ROC movie.  The empirical values show
the expected approximate agreement with the population quantities in
the table.

\renewcommand{\arraystretch}{1.3}
\begin{table}[t]
	\centering
	\caption{Population values of Pearson's correlation coefficient $r$,
		Spearman's rank correlation coefficient $\rho_{\rm S}$ and CPA for
		the features $X$, $X'$, $X''$ relative to the real-valued outcome
		$Y$, where $(Y, X, X', X'')$ is Gaussian with covariance matrix
		\eqref{eq:MVN}.  \label{tab:MVN}}
	\begin{tabular}{cccc}
		\toprule
		Feature  & $r$ & $\rho_{\rm S}$ & CPA \\
		\toprule
		$X$   &  0.800 & 0.786 & 0.893 \\
		$X'$  &  0.500 & 0.483 & 0.741 \\
		$X''$ &  0.200 & 0.191 & 0.596 \\
		\bottomrule
	\end{tabular}
\end{table}

Suppose now that the values $y_1, \ldots, y_n$ of the outcomes are
unique, whereas the feature values $x_1, \ldots, x_n$ might involves
ties.  Let $p \geq 0$ denote the number of tied groups within $x_1,
\ldots, x_n$.  If $p = 0$ let $V = 0$.  If $p \geq 1$, let $v_j$ be
the number of equal values in the $j$th group, for $j = 1, \ldots, p$,
and let
\[
V = \frac{1}{12} \sum_{j=1}^p \left( v_j^3 - v_j \right).
\]
Then Spearman's \textit{mid rank adjusted}\/ coefficient $\rho_{\rm
	M}$ is defined as
\begin{equation}  \label{eq:def_rho_M}
\rho_{\rm M} = 1 - \frac{6}{n(n^2-1)} \left( \sum_{i=1}^n \left( \, \mrk(x_i) - \rk(y_i) \right)^2 + V \right),
\end{equation}
where $\mrk$ is the aforementioned mid rank.  As shown by
Woodbury (1940), if one assigns all possible combinations of
integer ranks within tied sets, computes Spearman's $\rho_{\rm S}$ in
\eqref{eq:def_rho_S} on every such combination and averages over the
respective values, one obtains the formula for $\rho_{\rm M}$ in
\eqref{eq:def_rho_M}.

The following result reduces to the statement of Theorem
\ref{thm:Spearman} in the case $p = 0$ when there are no ties in $x_1,
\ldots, x_n$ either.

\begin{theorem}  \label{thm:Spearman.m}
	In case there are no ties within\/ $y_1, \ldots, y_n$,
	\begin{equation}  \label{eq:rho_M}
	{\rm CPA} = \frac{1}{2} \left( \hspace{0.2mm} \rho_{\rm M} + 1 \right).
	\end{equation}
\end{theorem}

\begin{proof}
	As noted, $\rho_{\rm M}$ arises from $\rho_{\rm S}$ if one assigns all
	possible combinations of integer ranks within tied sets, computes
	$\rho_{\rm S}$ on every such combination and averages over the
	respective values.  In view of \eqref{eq:num.denom}, if there are no
	ties in $y_1, \ldots, y_n$, averaging $\frac{1}{2} \left( \hspace{0.2mm}
	\rho_{\rm S} + 1 \right)$ over the combinations yields $\frac{1}{2}
	\left( \hspace{0.2mm} \rho_{\rm M} + 1 \right)$, which equals CPA by
	\eqref{eq:CPA2cov}.
\end{proof}

The relationships \eqref{eq:SomersD}, \eqref{eq:rho_S} and
\eqref{eq:rho_M} constitute but special cases of the general,
covariance based representation \eqref{eq:CPA2cov}.  In this light,
CPA provides a unified way of quantifying potential predictive ability
for the full gamut of dichotomous, categorical, mixed
discrete-continuous and continuous types of outcomes.  In particular,
CPA bridges and generalizes AUC, Somers' $D$ and Spearman's rank
correlation coefficient, up to a common linear relationship.

\subsection{Computational issues}

We turn to a discussion of the computational costs of generalized ROC
analysis for a dataset of the form \eqref{eq:real} or
\eqref{eq:real2class} with $n$ instances and $m \leq n$ classes.

It is well known that a traditional ROC curve can be generated from a
dataset with $n$ instances in $O(n \log n)$ operations (Fawcett, 2006, Algorithm 1).  A ROC movie comprises $m - 1$
traditional ROC curves, so in a naive approach, ROC movies can be
computed in $O(mn \log n)$ operations.  However, our implementation
takes advantage of recursive relations between consecutive component
curves ROC$_{i-1}$ and ROC$_i$.  While a formal analysis will need to
be left to future work, we believe that our algorithm has
computational costs of $O(n \log n)$ operations only.  For the
vertical averaging of the component curves in the construction of UROC
curves, we partition the unit interval into 1,000 equally sized
subintervals.

Importantly, CPA can be computed in $O(n \log n)$ operations, without
any need to invoke ROC analysis, by sorting $x_1, \ldots, x_n$ and
$y_1, \ldots, y_n$, computing the respective mid ranks and classes,
and plugging into the rank based representation \eqref{eq:num.denom}.

\subsection{Key properties: Comparison to traditional ROC analysis}  \label{sec:summary}

We are now in a position to judge whether the proposed toolbox of ROC
movies, UROC curves, and CPA constitutes a proper generalization of
traditional ROC analysis.  To facilitate the assessment, the
subsequent statements admit immediate comparison with the key insights
of classical ROC analysis, as summarized in Section
\ref{sec:initialsummary}.

We start with the trivial but important observation that the new tools
nest the notions of traditional ROC analysis.  This is not to be taken
for granted, as extant generalizations do not necessarily share this
property (Bi and Benett, 2003; Hern{\'a}ndez-Orallo, 2013).
\begin{itemize}
	\item[(0)] In the case of a binary outcome, both the ROC movie and the
	UROC curve reduce to the ROC curve, and CPA reduces to AUC.
	\item[(1)] ROC movies, the UROC curve and CPA are straightforward to
	compute and interpret, in the (rough) sense of \textit{the larger
		the better}.
	\item[(2)] CPA attains values between 0 and 1 and relates linearly to
	the covariance between the class of the outcome and the mid rank of
	the feature, relative to the covariance between the class and the
	mid rank of the outcome.  In particular, if the outcomes are
	pairwise distinct, then $\CPA = \frac{1}{2} \left( \hspace{0.2mm} \rho_{\rm M}
	+ 1 \right)$, where $\rho_{\rm M}$ is Spearman's mid rank adjusted
	coefficient \eqref{eq:def_rho_M}. If the outcomes are binary, then
	$\CPA = \frac{1}{2} \left( D + 1 \right)$ in terms of Somers' $D$.
	For a perfect feature, $\CPA = 1$, $\rho_{\rm M} = 1$ under pairwise
	distinct and $D = 1$ under binary outcomes.  For a feature that is
	independent of the outcome, $\CPA = \frac{1}{2}$, $\rho_{\rm M} = 0$
	under pairwise distinct and $D = 0$ under binary outcomes.
	\item[(3)] The numerical value of CPA admits an interpretation as a
	weighted probability of concordance for feature--outcome pairs, with
	weights that grow linearly in the class based distance between
	outcomes.
	\item[(4)] ROC movies, UROC curves, and CPA are purely rank based and,
	therefore, invariant under strictly increasing
	transformations. Specifically, if $\phi : \real \to \real$ and $\psi
	: \real \to \real$ are strictly increasing, then the ROC movie, UROC
	curve, and CPA computed from
	\begin{equation}  \label{eq:realT}
	(\phi(x_1),\psi(y_1)), \ldots, (\phi(x_n),\psi(y_n)) \, \in \, \real \times \real
	\end{equation}
	are the same as the ROC movie, UROC curve, and CPA computed from the
	data in \eqref{eq:real}.
\end{itemize}
We iterate and emphasize that, as an immediate consequence of the
final property, ROC movies, UROC curves, and CPA assess the
discrimination ability or \textit{potential}\/ predictive ability of a
point forecast, regression output, feature, marker, or test.  Markedly
different techniques are called for if one seeks to assess a
forecast's \textit{actual}\/ value in any given applied problem (Ben Bouall\`egue et al., 2015; Ehm et al., 2016).

\section{Real data examples}  \label{sec:examples}

In the following examples from survival analysis and numerical weather
prediction the usage of ROC movies, UROC curves, and CPA is
demonstrated.  We start by returning to the survival example from
Section \ref{sec:introduction}, where the new set of tools frees
researchers form the need to artificially binarize the outcome.  Then
the usage of CPA is highlighted in a study of recent progress in
numerical weather prediction (NWP), and in a comparison of the
predictive performance of NWP models and convolutional neural
networks.

\subsection{Survival data from Mayo Clinic trial}  \label{sec:survival}

In the introduction, Figures \ref{fig:ROCC_survival} and
\ref{fig:ROCM_survival} serve to illustrate and contrast traditional
ROC curves, ROC movies and UROC curves.  They are based on a classical
dataset from a Mayo Clinic trial on primary biliary cirrhosis (PBC), a
chronic fatal disease of the liver, that was conducted between 1974
and 1984 (Dickson et al., 1989).  The data are provided by various R
packages, such as \texttt{SMPracticals} and \texttt{survival}, and
have been analyzed in textbooks (Fleming and Harrington, 1991; Davison, 2003).  The
outcome of interest is survival time past entry into the study.
Patients were randomly assigned to either a placebo or treatment with
the drug D-penicillamine.  However, extant analyses do not show
treatment effects (Dickson et al., 1989), and so we follow previous
practice and study treatment and placebo groups jointly.

We consider two biochemical markers, namely, serum albumin and serum
bilirubin concentration in mg/dl, for which higher and lower levels,
respectively, are known to be indicative of earlier disease stage,
thus supporting survival.  Hence, for the purposes of ROC analysis we
reverse the orientation of the serum bilirubin values. Given our goal
of illustration, we avoid complications and remove patient records
with censored survival times, to obtain a dataset with $n = 161$
patient records and $m = 156$ unique survival times. The proper
handling of censoring is beyond the scope of our study, and we leave
this task to subsequent work.  For a discussion and comparison of
extant approaches in the context of time-dependent ROC curves see
Blanche et al. (2013).

The traditional ROC curves in Fig.~\ref{fig:ROCC_survival} are
obtained by binarizing survival time at a threshold of 1462 days,
which is the survival time in the data record that gets closest to
four years. The ROC movies and UROC curves in
Fig.~\ref{fig:ROCM_survival} are generated directly from the survival
times, without any need to artificially pick a threshold. The CPA
values for serum albumin and serum bilirubin are $.73$ and $.77$,
respectively, and contrary to the ranking in
Fig.~\ref{fig:ROCC_survival}, where bilirubin was deemed superior,
based on outcomes that were artificially made binary. Our new tools
free researchers from the need to binarize, and still they allow for
an assessment at the binary level, if desired.  For example, the ROC
curves and AUC values from Fig.~\ref{fig:ROCC_survival} appear in the
ROC movie at a threshold value of 1462 days. In line with current uses
of AUC in a gamut of applied settings, CPA is particularly well suited
to the purposes of feature screening and variable selection in
statistical and machine learning models (Guyon and Elisseeff, 2003). Here, AUC
and CPA demonstrate that both albumin and bilirubin contribute to
prognostic models for survival (Dickson et al., 1989, Fleming and Harrington, 1991).

\begin{figure}[t]
	\includegraphics[width=1\textwidth]{CPA_HRES_PST_sm}
	\caption{Temporal evolution of CPA for forecasts from European Centre
		for Medium-Range Weather Forecasts (ECMWF) high-resolution model at
		lead times from one to five days in comparison to the simplistic
		persistence forecast. The weather variables considered are (a)
		surface (2-meter) temperature, (b) surface wind speed and (c)
		24-hour precipitation accumulation. The CPA values refer to a domain
		that covers Europe and twelve-month periods that correspond to
		January--December (solid and dotted lines), April--March, July--June
		and October--September (dotted lines only), based on gridded
		forecast and observational data from January 2007 through December
		2018.\label{fig:CPA_HRES_PST}}
\end{figure}

\begin{figure}[t]
	\centering
	\animategraphics[label=myAnim7,height=75mm]{28}{Test4_reduced3}{}{}
	\caption{ROC movies, UROC curves, and CPA for ECMWF high-resolution
		(HRES) and persistence forecasts of 24-hour precipitation
		accumulation over Europe at a lead time of five days in calendar
		year 2018.  In the ROC movies, the number at upper left shows the
		threshold at hand in the unit of millimeter, the number at upper
		center the relative weight $w_c/\max_{l = 1, \ldots, m-1} w_l$ from
		\eqref{eq:w}, and the numbers at bottom right the respective AUC
		values.  \label{fig:ROCM_HRES_PST}}
\end{figure}

\subsection{Monitoring progress in numerical weather prediction (NWP)}  \label{sec:NWP}

Here we illustrate the usage of CPA in the assessment of recent
progress in numerical weather prediction (NWP), which has experienced
tremendous advance over the past few decades (Bauer et al., 2015; Alley et al., 2019; Ben Bouall\`egue et al., 2019). Specifically, we consider forecasts of
surface (2-meter) temperature, surface (10-meter) wind speed and
24-hour precipitation accumulation initialized at 00:00 UTC at lead
times from a single day (24 hours) to five days (120 hours) ahead from
the high-resolution model operated by the European Centre for
Medium-Range Weather Forecasts (ECMWF, 2012) which is generally
considered the leading global NWP model.  The forecast data are
available at \url{https://confluence.ecmwf.int/display/TIGGE}.  As
observational reference we take the ERA5 reanalysis product
(Hersbach, 2019) which can be downloaded from
\url{https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview}. We
use forecasts and observations from $279 \times 199 = 55,521$ model
grid boxes of size $0.25^\circ \times 0.25^\circ$ each in a geographic
region that covers Europe from 25.0$^\circ$ W to 44.5$^\circ$ E in
latitude and 25.0$^\circ$ N to 74.5$^\circ$ N in longitude.  The time
period considered ranges from January 2007 to December 2018.

In Figure \ref{fig:CPA_HRES_PST} the forecasts from ECMWF
high-resolution run are compared to a reference technique, namely, the
persistence forecast.  The persistence forecast is simply the most
recent available observation for the weather quantity of interest; as
such, it does not depend on the lead time.  The CPA values are
computed on rolling twelve-month periods that correspond to
January--December, April--March, July--June or October--September,
typically comprising $n = 365 \times 55,521 = 20,265,165$ individual
forecast cases.  The ECMWF forecast has considerably higher CPA than
the persistence forecast for all lead times and variables considered.
For the persistence forecast CPA fluctuates around a constant level;
for the ECMWF forecast CPA improves steadily, attesting to continuing
progress in NWP (Bauer et al., 2015; Alley et al., 2019; Ben Bouall\`egue et al., 2019; Haiden et al., 2019).  To place these findings further into context, recall
that CPA is a weighted average of AUC values for binarized outcomes at
individual threshold values.  Weightings other than in \eqref{eq:w}
are feasible, and relate to current practice for performance
monitoring and headline scores at ECMWF, which involve AUC for
binarized outcomes at a single threshold (Ben Bouall\`egue et al., 2019, Haiden et al., 2019), which obtains all of the weight, while the other
thresholds get ignored.  The new CPA measure preserves the spirit and
power of classical ROC analysis, and frees researchers from the need
to binarize real-valued outcomes.

The ROC movies, UROC curves, and CPA values in
Fig.~\ref{fig:ROCM_HRES_PST} compare the ECMWF high-resolution
forecast to the persistence forecast for 24-hour precipitation
accumulation at a lead time of five days in calendar year 2018.  As
noted, this record comprises more than 20 million individual forecast
cases, and there are $m = 35,993$ unique values of the outcome.  We
certainly lack the patience to watch the full sequence of $m - 1$
screens in the ROC movie.  A pragmatic solution is to consider a
subset $C \subseteq \{ 1, \ldots, m - 1 \}$ of indices, so that
$\ROC_c$ is included in the ROC movie (if and) only if $c \in C$.
Specifically, we set positive integer parameters $a \leq m-1$ and $b$
such that the ROC movie comprises at least $a$ and at most $a + b$
curves.  Let the integer $s$ be defined such that $1 + (a - 1) s \leq
m - 1$ $< 1 + as$, and let $C_a = \{ 1, 1 + s, \ldots, 1 + (a-1)s \}$,
so that $|C_a| = a$.  Let $C_b = \{ c : n_c \geq n/b \}$; evidently,
$|C_b| \leq b$.  Finally, let $C = C_a \cup C_b$ so that $a \leq |C|
\leq a + b$.  We have made good experiences with choices of $a = 400$
and $b = 100$, which in Fig.~\ref{fig:ROCM_HRES_PST} yield a ROC movie
with 401 screens.

\subsection{WeatherBench: Convolutional neural networks (CNNs) vs.~NWP models}  \label{sec:WeatherBench}

As noted, operational weather forecasts are based on the output of
global NWP models that represent the physics of the atmosphere.
However, the grid resolution of NWP models remains limited due to
finite computing resources (Bauer et al., 2015).  Spurred by the ever
increasing popularity and successes of machine learning models,
alternative, data-driven approaches are in vigorous development, with
convolutional neural networks (CNNs, LeCun et al. (2015)) being a
particularly attractive starting point, due to their ease of
adaptation to spatio-temporal data.  Rasp et al. (2020) introduce
WeatherBench, a ready-to use benchmark dataset for the comparison of
data-driven approaches, such as CNNs and a classical linear regression
(LR) based technique, to NWP models, such as the aforementioned HRES
model and simplified versions thereof, T63 and T42, which run at
successively coarser resolutions.  Furthermore, WeatherBench supplies
baseline methods, including both the persistence forecast and
climatological forecasts.

As evaluation measure for the various types of point forecasts,
WeatherBench uses the root mean squared error (RMSE).  In related
studies, the RMSE is accompanied by the anomaly correlation
coefficient (ACC), i.e., the normalized product moment between the
difference of the forecast at hand and the climatological forecast,
and the difference between the outcome and the climatological forecast
(Weyn et al., 2020).  However, as noted by Rasp et al. (2020), results in
terms of RMSE and ACC tend to be very similar.  Here we argue that,
due to its rank based character and emphasis on potential
predictability, CPA is a more suitable companion measure to RMSE than
ACC.

\begin{figure}[t]
	\includegraphics[width=\textwidth]{both.pdf}
	\caption{Predictive ability of WeatherBench three days ahead forecasts
		of 850 hPa temperature in 2017 and 2018 at different latitudes in
		terms of (a) RMSE and (b) CPA.  HRES, T63, and T42 indicate NWP
		models run at decreasing grid resolution that are compared to the
		CNN, linear regression (LR), and persistence forecast
		(Rasp et al., 2020).  Note that RMSE is negatively oriented (the
		smaller the better), whereas CPA is positively oriented.
		\label{fig:RMSE_CPA}}
\end{figure}

Figure \ref{fig:RMSE_CPA} compares WeatherBench forecasts three days
ahead for temperature at 850 hPa pressure, which is at around 1.5 km
height, in terms of RMSE (in Kelvin) and CPA.  With reference to Table
2 of Rasp et al. (2020), we consider the persistence forecast, the
(direct) linear regression (LR) forecast, the (direct) CNN forecast,
the Operational IFS (HRES) forecast, and successively coarser versions
thereof (T63 and T42).  The panels display RMSE and CPA as a function
of latitude bands, from the South Pole at 90$^\circ$S to the equator
at 0$^\circ$ and the North Pole at 90$^\circ$N, for the WeatherBench
final evaluation period of the years 2017 and 2018.  CPA is computed
grid cell by grid cell.  In a given latitude band, RMSE and CPA are
averaged across all grid cells, which facilitates weightings, such as
the latitude based weighting that is employed in WeatherBench.  Note
that RMSE is negatively oriented (the smaller, the better), whereas
CPA is positively oriented (the closer to the ideal value of 1 the
better).

With respect to RMSE (Figure \ref{fig:RMSE_CPA}a) marked geographical
differences are visible.  In equatorial regions, where day-to-day
temperature variations are generally low, all forecasts have a low
RMSE and the range between the best-performing HRES forecast and the
simplistic persistence forecast is small.  The HRES forecast remains
best for all latitudes, followed by the T63 forecast.  The coarsest
dynamical model forecast, T42, shows a further deterioration as
expected, but with large outliers in the high latitudes of the southern
hemisphere and in the 30s of the northern hemisphere.  It is likely
that the lack of model orography creates large errors in areas of high
terrain such as the Antarctic plateau and the Himalayas.  Among the
data-driven forecasts, CNN is better than LR for all extratropical
latitudes.  Finally, persistence performs worst through all latitudes
with prominent peaks near 50$^\circ$S and 50$^\circ$N.  These are the
midlatitude storm track regions, where day-to-day changes are large
and impede good forecasts based on persistence.

The corresponding analysis using CPA (Figure \ref{fig:RMSE_CPA}b)
shows remarkable differences.  Most notable is the low CPA in the
tropics, which indicates poor performance of all forecasts.  In
contrast, the low RMSE suggests superior performance in this region.
As a rank based measure, CPA is independent of magnitude, and thus
provides a scale free assessment of predictability, well in line with
recent findings in meteorology (Kniffka et al., 2020).  Another striking
difference to RMSE is the large drop in CPA in the Furious Fifties of
the southern hemisphere, creating a large asymmetry with the northern
midlatitudes.  This area is almost entirely oceanic and characterized
by mobile low-pressure systems, the dynamical behaviour of which
appears to be difficult to learn under data-driven approaches.

We conclude that RMSE and CPA bring orthogonal facets of predictive
performance to researchers' attention, and encourage the usage of grid
cell averaged CPA to supplement RMSE as key performance measures in
WeatherBench.  While ACC is scale free as well, it is moment based
rather than rank based, and thus is more closely aligned with RMSE
than is CPA.  Similar recommendations of using CPA apply in many
applied settings, where predictions of a real-valued outcome are
evaluated, and a magnitude dependent measure, such as RMSE, is usefully
accompanied by a rank based criterion of predictive performance.  In
the special case of probabilistic classifiers for binary outcomes,
this corresponds to reporting both the Brier mean squared error
measure and AUC.  See Hern{\'a}ndez-Orallo et al. (2012) for a detailed, theoretically
oriented comparison of these and other performance measures under
binary outcomes.

\section{Discussion}  \label{sec:discussion}

We have addressed a long-standing challenge in data analytics, by
introducing a set of tools --- comprising receiver operating
characteristic (ROC) movies, universal ROC (UROC) curves, and a
coefficient of predictive ability (CPA) measure --- for generalized
ROC analysis, thereby freeing researchers from the need to
artificially binarize real-valued outcomes.  Throughout the paper, we
have assumed that predictors and features are real-valued.  However,
as all results and techniques discussed are rank based, they carry
through for linearly ordered, but not necessarily real-valued
features.  For generalizations of ROC analysis to categorical outcomes
that cannot be linearly ordered see Hand and Till (2001) and Section 9 of
Fawcett (2006).

ROC movies, UROC curves, and CPA reduce to the classical ROC curve and
AUC when applied to binary data.  Moreover, attractive properties of
ROC curves, such as invariance under strictly increasing
transformations and straightforward interpretability are maintained by
ROC movies and UROC curves.  In stark contrast to customarily used
measures of bivariate association and dependence (Reshef et al., 2011; Weihs et al., 2018), CPA is asymmetric, i.e., in general, its value changes
if the roles of the feature and the outcome are transposed.  Thus, the
new measure is directed and quantifies predictive ability of features
for linearly ordered outcomes.  Just like AUC, CPA is rank based and
admits an appealing interpretation as a weighted probability of
concordance.  CPA bridges and generalizes AUC, Somers' D and
Spearman's rank correlation coefficient, up to a common linear
relationship.

In view of the advent of dynamic graphics in mainstream scientific
publishing, we contend that ROC movies, UROC curves, and CPA are bound
to supersede traditional ROC curves and AUC in a wealth of
applications.  Open source code for their implementation in Python
(Python, 2019) and the R language and environment for statistical
computing (R Core Team, 2020) is available on GitHub at
\url{https://github.com/evwalz/rocm_python_nwp} and
\url{https://github.com/evwalz/uroc}.

\section*{acknowledgements}

	We thank Zied Ben Bouall\`egue, Timo Dimitriadis, Andreas Eberl,
	Dominic Edelmann, Andreas Fink, Peter Knippertz, Sebastian Lerch,
	Marlon Maranan, Florian Pappenberger, Johannes Resin, Peter Sanders,
	David Richardson, Philipp Zschenderlein and seminar participants at
	the European Centre for Medium-Range Weather Forecasts (ECMWF) and
	International Symposium on Forecasting (ISF) for advice, discussion
	and encouragement.  In particular, Peter Knippertz provided detailed
	comments on the WeatherBench example.  This work has been supported by
	the Klaus Tschira Foundation, by the Helmholtz Association, and by the
	Deutsche Forschungs\-ge\-mein\-schaft (DFG, German Research
	Foundation) -- Project-ID 257899354 -- TRR 165.  Tilmann Gneiting
	furthermore acknowledges travel support via the ECMWF Fellowship
	programme.

\section*{Conflict of interest}
The authors declare that they have no conflict of interest.

redpajama_set_name: RedPajamaArXiv


text:
Movies – TV – Reviews
Review - Still Alice
Posted on Tuesday, December 30th, 2014 at 12:25 pm. Written by Steve
Kristen Stewart and Julianne Moore in Still Alice
A super confident woman, top of her game, a linguistics professor, one day discovers herself grasping for a word while she's giving a lecture. This being the movies, where a cough in one scene leads to coughing up blood in the next, we automatically suspect she's got Alzheimer's. The title providing another nudge (why Still?). And so it turns out, in a movie that seems determined to put a polish on the disease of the week movie, and largely succeeds.
Polish number one is that it's not just any old Alzheimer's but familial Alzheimer's, in which the gene – should you have been unlucky enough to have inherited it from an affected parent – means you have 100 per cent chance of getting Alzheimer's yourself.
But really the claims for genre transcendence are made by the quality of cast that writers/directors Richard Glatzer and Wash Westmoreland have attracted to the project. Julianne Moore plays the unlucky Dr Alice Howland and in the scenes set in the doctor's consulting room, where the camera rests entirely on her face as she told one awful truth after another, the wisdom of that casting decision becomes obvious. Matching her in strength and subtlety is Alec Baldwin as her uxorious but never sappy husband, Kate Bosworth as her prickly daughter, Hunter Parrish as the largely superfluous son and Kristen Stewart, clearly making a decision to step back from the spotlight, as the youngest daughter, who finds herself promoted to more of a caring role as the rest of the family quietly shuffle backwards.
It's also an unusually nuanced film, and gives its victim far more agency than we're used to in this sort of thing. So, alongside gruesome scenes like the one in which Alice pisses her light grey joggers – not a good look – and fails to recognise her daughter, there are others where she clearly uses her advancing condition to her advantage, ducking out of a dreary dinner party, or reading her youngest daughter's diary and putting it down to "my illness".
They're an unusual duo, Glatzer and Westmoreland, who you might remember as the names behind 2001's The Fluffer, a well acted, quirky gay rom-com. And might not remember as the names behind 2013's The Last of Robin Hood, which cast an excellent Kevin Kline as Errol Flynn in his final skiddy years (and, incidentally, gave a small role to Errol's grandson, Sean). That's when they're not working as producer/consultants on America's Next Top Model.
That TV background will count against them in some quarters, where this film will be pegged as a disease of the weeker not worthy of even a first look. It's irrefutable: that is exactly what it is, and the plaintive piano and string quartet soundtrack isn't trying to deny it either. But no matter how mangey and emotionally manipulative, every dog has its day. And this, ladies and gents, is that canine.
© Steve Morrissey 2014
Tags: alec baldwin, alzheimer's, Hunter Parrish, Julianne Moore, kate bosworth, kristen stewart, richard glatzer, Still Alice, wash westmoreland
Run time: 101 min
Director: Richard Glatzer, Wash Westmoreland
Writers: Lisa Genova, Richard Glatzer
Stars: Julianne Moore, Kate Bosworth, Shane McRae
Trivia: Alice Howland, happily married with three grown children, is a renowned linguistics professor who starts to forget words. When she receives a devastating diagnosis, Alice and her family find their bonds tested.
Storyline Alice Howland, happily married with three grown children, is a renowned linguistics professor who starts to forget words. When she receives a devastating diagnosis, Alice and her family find their bonds tested.
Film of the Day - Nowhere Boy
See This - Persuasion
Film of the Day - The Princess Bride
Email Me @ steve@moviesteve.com
Twitter: @stevemoviesteve
LinkedIn: Steve Morrissey
© 20012 - 2019 All rights reserved. Powered by Foundation. Designed by Joe Wheater.

redpajama_set_name: RedPajamaCommonCrawl


One lucky Cannes Film Festival-goer will get to go into outer space with Leonardo DiCaprio. Page Six has exclusively learned one of the auction prizes at tonight's amfAR gala at Cannes will be a trip on Virgin Galactic's space flight with "The Great Gatsby" star. Seats on the world's first commercial space flight cost $200,000, but sources say this auction item was "priceless" because the highest bidder will be personally escorted by DiCaprio. Virgin boss Sir Richard Branson said he hopes to launch the first manned flight from his spaceport in New Mexico around Christmas this year. The ship will carry six passengers and two pilots at a time and soar to an altitude of 360,000 feet above the earth. The trip into space and back will last around 2 1/2 hours, and passengers must first go through three days of training. According to Virgin Galactic, 65,000 people have already put their names down on the waiting list for seats on the flights and celebrities reported to have bought tickets include Brad Pitt and Angelina Jolie, scientist Stephen Hawking and Ashton Kutcher. Expected guests at tonight's 20th anniversary amfAR bash at the Hotel du Cap, chaired by Harvey Weinstein and Kenneth Cole, include Nicole Kidman, Christoph Waltz, Jeremy Renner, Rosario Dawson, Jessica Chastain, Karolina Kurkova and Janet Jackson, with performances planned by Duran Duran, Shirley Bassey and Ellie Goulding.


text:
Kings Mountain girls soccer falls to Watauga in 3A playoffs
By Jacob Conley Special to The Star
Watauga scored inside the first two minutes of their playoff match with Kings Mountain and added a pair of second-half goals as they stunned the Lady Mountaineers 3-1 on Friday.
Lilly Suyao scored a goal in each half to collect a brace for the Lady Pioneers as the visitors brought their physical and technical style of play down the mountain from Boone.
"We aren't used to that kind of physicality," Kings Mountain head coach Erik Price said. "They weren't playing dirty, they were just physical. They were also very good at passing the ball. We lost to a good team."
Watauga (17-4) showed their skill early as they controlled the run of play for much of the first half. Suyao found the back of the net inside of two minutes when she sent home a pass inside the box from the foot of Shaelyn Sheaff for a 1-0 Lady Pioneer lead.
"I think giving up that early goal shook us," Price said. "I'm proud of the way we were able to settle down and come back and tie the game in the second half."
Watauga nearly doubled the lead moments later, as Suyao put another shot past goalie Baylee Mar, but she was whistled for offside.
From that point, Kings Mountain (17-3) began to gain momentum, nearly topped off with a Sarah Drennan header that was just wide of the goal. The Lady Mountaineers were only able to put one more shot on frame for the rest of the half, but they did get stellar defensive play from Reagan Buchanan on the back line to keep the deficit at 1-0.
Kings Mountain became the aggressor early in the second half, playing multiple balls into the box. That strategy paid off with a goal off a penalty kick from Drennan in the 50th minute that tied things up.
"Once we equalized, I felt pretty good," Price said. "But Watauga is so good that just a second of lapse on the defensive end can cost you. That's what happened to us."
Watauga scored what turned out to be the game-winning goal in the 58th minute, when Brelyn Sturgile converted off a throw-in from Sadye Franklin. The Lady Pioneers added an insurance goal a few minutes later when Sturgile played a pass to Suyao in the box and she finished for a 3-1 advantage.
Kings Mountain had a few more chances over the final 14 minutes of play, but they could get no closer.
Even with the loss, Price is proud of his team.
"They have nothing to be ashamed of," Price said. They played well and lost to a good team. This group has accomplished a lot during their four years here. The nine girls in the senior class have gone 59-0-1 in conference play. There were some tears tonight, but I am so proud of them."


With vaulted ceilings, plentiful sunlight and deck access, this room is the highlight of the home.
New roof and steel siding in 2012 and new furnace in 2018.
The functional kitchen offers wrap around cabinetry along with ideal counter seating.
You will have no problem enjoying the nice weather with not only a deck accessible through the sunroom, it also includes stairs down to the outdoor concrete patio.
The dramatic brick arched entry invites you into this exquisite home featuring a spacious 2-story front foyer, french doors leading to the den, living room with beautiful maple floors, continued impressive 2-story vault ceiling, picture perfect window display, beautiful restoration lighting & a warm gas fireplace.
The comfortable open flow layout continues into the show-stopping gourmet kitchen...custom designed with quality Omega cabinetry, fabulous chic lighting, subway tile back-splash, large center island with prep sink & seating for 6, sleek Cambria tops, high-end GE Cafe series appliances plus Thermador refrigerator, maple floors, sunny skylights & access to the backyard patio with beautiful wooded ravine views.
Add'l home highlights include: exceptionally tasteful decor throughout, quality finish kitchen & bath remodels, indoor/outdoor speaker system, new HVAC systems & water heater, spacious catch-all basement storage room, crisp/clean white woodwork/trim, steel siding with appealing brick accent, newer garage door, shingles '14 & the aggregate sidewalk & new patio.
The master suite displays an appealing tray ceiling, cozy fireplace & private bath with tile floor, whirlpool tub, separate shower, raised dual sink vanity, Corian top & a large walk-in closet.
On the upper level are 3 bedrooms, including a master bedroom with en suite master bath.
From the open concept kitchen and dining area, you enter into the cozy 4 seasons room, warmed with a gas fireplace.
In the basement you will find a plethora of storage space.
In the lower level is an additional bedroom, large laundry/bathroom and a second living space with daylight windows and a large stone fireplace.
Off the kitchen, you'll find your mudroom alongside a large spacious storage room and your guest bathroom.
Your spacious private office is tucked away from the living area so you can work in peace.
Upstairs, you'll have your oversized Master Suite complete with your walk-in tiled shower, your double vanity sinks, and very large closets!
Three more bedrooms are on this floor, as well as a convenient laundry room.
The entrance emphasizes high ceilings and a dramatic open staircase with skywalk overlooking great room and foyer.
Center island with bonus wine storage and seating, all new stainless appliances including a coveted double wall oven and gas cooktop.
Formal dining room leads to an open, bright kitchen that was definitely made for cooking!
Walk in closet with additional 13x12 closet space!
The east wing features a spacious master bedroom with lighted tray ceiling, full master bath with soaking tub and separate shower, and large walk-in closet.
The lower level offers a huge family room with gas fireplace, office/play room nook, two additional bedrooms, full bath and bonus storage room that could be finished into a future office/craft room.
Through the front door is a grand, open floor plan that combines the living room with custom fireplace, kitchen with impressive center island, quartz counter-tops, and coffee/wine bar and the dining area with patio doors to the backyard.
You are greeted by a beautiful porch and impressive landscaping as you walk up to the front door of this home.
Head downstairs, and you find the large family room area with attached wet bar.
Step out the sliding patio door off the dining to the 16' x 16', maintenance free deck and enjoy park views while grilling or sitting on your deck this summer season.
The kitchen has ample cabinetry, and wrap around island peninsula and plenty of space for barstool seating.
Entering the home you will note a nice sized foyer, with closet and access to the 3 stall attached garage.
Entering the mudroom you will find ample storage and space to unload , moving to the kitchen you will notice the beautiful and modern high end cabinetry, stainless steel appliances, Quartz counter tops, and a bright work space open to the dining room.
Master bedroom has vaulted ceilings with elegant decorative ceiling beams, private full bathroom, and walk-in closet.
There is a family room and full kitchen with attractive stainless steel appliances, and patio doors leading to an incredible paver patio looking out to the pond.
Dining room has fantastic views to the pond in the back yard, and is open to the spacious living room with an attractive stone gas fire place.
The screened porch, maintenance free deck, and paver block patio offer incredible views of the ravine during all seasons.
Main floor also offers a spacious mudroom with laundry and 1/2 bathroom.
Walkout basement offers a 4th bedroom, full bathroom, large storage room, and a cozy family room with a wet bar.
Step inside and find formal dining and living areas to your left and right, or step down the hallway to find an updated kitchen, informal dining, and family room combo perfect for entertaining.
Two additional bedrooms and full bath downstairs as well as a bonus room that would be ideal for a home office or fitness room.
Lovely and spacious 5+ bedrooms including 3 main floor bedrooms as well as the added convenience of a main level laundry room.
Additional full bathroom on main level and large foyer with gorgeous open staircase leads to a lower level that includes a massive family room with granite wet bar and gas fireplace.
Gas insert in fireplace, French doors open to sunroom with arched way to dining room, add additional charm and character.
Maintenance free fencing in back yard, front porch and decking.
Plenty of closet space and shelving galore.
In the kitchen you will find ample beautiful, custom-built oak cabinetry with updated counter tops, sleek black appliances and garage access.
The living room features a cozy brick, wood-burning fireplace accented by built-ins and flows into the spacious family room with patio doors leading to a park-like backyard with new chain link fence, mature trees, wonderful landscaping, and covered patio - perfect for entertaining!
Completing the main floor is the updated 1/2 bath with laundry and the office/den with closet and patio doors to the backyard.
Main level features a living room, dining room w/ built ins, kitchen with breakfast counter, master bedroom with 3/4 bath, 3 more bedrooms (one is presently being used as an office) all with hardwood floors (2 of the bedrooms have built in desks and dressers), a full bath and patio doors to the fenced backyard.
Lower level offers a large family room with gas fireplace, bar and egress window, large play room and another very large room that could be used as a 5th bedroom or would make a great game room or craft room.
This spacious rambler sits on almost 1/2 acre in a wonderful, mature neighborhood in Upper North Mankato.
So many things to fall in love with.... it could be the see-thru gas FP in the LR, or the double french doors leading to a large wooden deck, or the beautiful 3-season porch off the formal dining room that also has a woodburning FP; or the functional kitchen with center island and all stainless steel appliances.
Main level laundry and a double attached garage completes the picture.
Walk outside to the beautifully landscaped yard that is partially fenced.
Notice the upgrades such as the large pantry, quartz tops, lovely tiled back splash, soft close cabinets and drawers, modern hardware and fixtures, new stainless appliances and Bosch dishwasher.
Upper level shows off wide open kitchen, dining and living space.
Nice corner lot, large storage shed included.
Recently Sold Homes in North Mankato, MN have a median listing price of $265,000 and a price per square foot of $114. There are 69 active recently sold homes in North Mankato, Minnesota, which spend an average of 121 days on the market. Some of the hottest neighborhoods near North Mankato, MN are Prairie Lake, Marion Village, Morgan Square, Hazelwood Creek Estates, Springbrook. You may also be interested in recently sold homes in popular zip codes like 56003, 56001, or in neighboring cities, such as Mankato, Saint Peter, Madison Lake, Eagle Lake, Lake Crystal.


We are excited to announce a new three-day course as part of our Masters of the Ministry initiative, Technical Mobile Testing with Richard Bradshaw (aka FriendlyTester). The course will be taking place in Manchester on the 4th-6th December 2017.
Mobile testing is a huge space with not nearly enough learning opportunities available. When you bring the technical aspect of testing together with mobile testing it presents a whole new area where we have found that testers have struggled time and time again to understand the landscape to help them testing effectively and efficiently.
Having spent three years testing the very popular O2 app 'O2Priority' on Android and iOS he feels he has a lot to share and teach when it comes to testing on mobile. Especially the technical aspect, with the wealth of tools out there, where is a tester to start? Richard aims to shed light on this during the class.
"I've taught parts of this course on many separate occasions, but yet to bring it all together as a three day class. I'm really excited about this, for me it's taking my teaching/training journey to the next level. Three days is the perfect amount of time to provide an insightful learning experience to the level of detail that is demanded, I'm definitely up for the challenge. I had a fantastic three years testing on mobile, I can't wait to share all the tips, tricks and tools I used with the attendees of the class. It's going to be a fascinating three days."
You can read the full description of the course and book online.


Where and what to eat in Sorrento.
Here are a few suggestions for where you can eat well in Sorrento without breaking the bank..all personally vetted!
A full-day private tour via luxury vehicle with a professional driver, including a stop at the ruins of Pompeii, a wine tasting at a local cellar, and a drive up to the Mt. Vesuvius trail head.
A full-day food tour to sample some of the best products from the region of Campania, including olive oil directly at a mill, mozzarella from an artisan producer, and pizza at a traditional pizzeria!
A full-day private tour of southern Italy's cultural and gastronomic highlights: the ruins of Pompeii, pizza school, and olive oil. All via luxury vehicle with a professional driver.


Australia is a major copper (Cu) producer with mining and smelting operations at Olympic Dam in South Australia (SA) and Mount Isa in Queensland (Qld).
Jul 15, 2018 The world's 20 largest copper mines produce nearly 9 million metric tons of the metal a Diego Delso [CC BY-SA 4.0], Wikimedia Commons.
Aug 7, 2013 Southeast Asia is home to the largest gold mine in the world. Gold was first discovered in the gold and copper bearing Cadia region in 1851 In the #8 spot is Mponeng, owned by South African major AngloGold Ashanti.
An interesting look at some of South Africa's biggest open pit mines. tonnes of refined copper per year - enough to supply most of South Africa's copper needs.
South Australia is a major copper producer and exporter, exploring, developing and mining copper in the State.


Rooms To Go is growing again, and we are adding to our supervisory team in our sLakeland facility. This position is a Department Supervisor for 3rd Shift, which means you will will work our overnight shift. As part of the distribution center leadership team for Rooms To Go, you will have the opportunity to lead some of the best employees anywhere! Also while helping our sales team continue to provide our customers quality firnitshings at an affordable price. We believe that our Distribution Team is the backbone to our company, providing structure and support. We can't do this without you!