text:
\section{Introduction}

\section{Introduction}

Climate models are built around models of the atmosphere, which are based on the laws of thermodynamics and on Newton's laws of motion for air as a fluid. Since they were first developed in the 1960s \citep{Smagorinsky63, Smagorinsky65, Manabe65, Mintz65a, Kasahara67a}, they have evolved from atmosphere-only models, via coupled atmosphere-ocean models with dynamic oceans, to Earth system models (ESMs) with dynamic cryospheres and biogeochemical cycles \citep{Bretherton12a, ipcc13}. Atmosphere and ocean models compute approximate numerical solutions to the laws of fluid dynamics and thermodynamics on a computational grid. For the atmosphere, the computational grid currently consists of $O(10^7)$ cells, spaced $O(10~\mathrm{km})$--$O(100~\mathrm{km})$ apart in the horizontal; for the oceans, the grid consists of $O(10^8)$ cells, spaced $O(10~\mathrm{km})$ apart in the horizontal. But scales smaller than the mesh size of a climate model cannot be resolved, yet are essential for its predictive capabilities. The unresolved scales are modeled by a variety of semi-empirical parameterization schemes, which represent the dynamics on subgrid-scales as parametric functions of the resolved dynamics on the computational grid \citep{Stensrud07a}. For example, the dynamical scales of stratocumulus clouds, the most common type of boundary layer clouds, are $O(10~\mathrm{m})$ and smaller, which will remain unresolvable on the computational grid of global atmosphere models for the foreseeable future \citep{Wood12a,Schneider17a}. Similarly, the submesoscale dynamics of oceans that may be important for biological processes near the surface have length scales of $O(100~\mathrm{m})$, which will also remain unresolvable for the foreseeable future \citep{Fox-Kemper14a}. Such smaller-scale dynamics in the atmosphere and oceans must be represented in climate models through parameterization schemes. Additionally, ESMs contain parameterization schemes for many processes for which the governing equations are not known or are only poorly known, for example, ecological or biogeochemical processes.

All of these parameterization schemes contain parameters that are uncertain, and the structure of the equations underlying them is uncertain itself. That is, there is parametric and structural uncertainty \citep{Draper95}. For example, entrainment and detrainment rates are parameters or parametric functions of state variables such as the vertical velocity of updrafts. They control the interaction of convective clouds with their environment and affect cloud properties and climate. But how they depend on state variables is uncertain, as is the structure of the closure equations in which they appear \citep[e.g.,][]{Stainforth05a,Holloway09a,Neelin09a,Romps10a,Nie12a,de-Rooy13a}. Or, as another example, the residence times of carbon in different reservoirs (e.g., soil, litter, plants) control how rapidly and where in the biosphere carbon accumulates. They affect the climate response of the biosphere. But they are likewise uncertain, differing by $O(1)$ factors among models \citep{Friedlingstein06a,Friedlingstein14a,Friend14a,Bloom16a}. Typically, parameterization schemes are developed and parameters in them are estimated independently of the  model into which they are eventually incorporated. They are tested with observations from field studies at a relatively small number of locations. For processes such as boundary-layer turbulence that are computable if sufficiently high resolution is available, parameterization schemes are increasingly also tested with data generated computationally in local process studies with high-resolution models \citep[e.g.,][]{Jakob03a,Jakob10a}. After the parameterization schemes are developed and incorporated in a climate model or ESM, modelers adjust (``tune'') parameters to satisfy large-scale  physical constraints, such as a closed energy balance at the top of the atmosphere (TOA), or selected observational constraints, such as reproduction of the 20th-century global-mean surface temperature record. This model tuning process currently relies on  knowledge and intuition of the modelers about plausible ranges of the tunable parameters and about the effect of parameter changes on the simulated climate of a model \citep{Randall97a,Mauritsen12a, Golaz13a,Hourdin13a, Flato13a, Hourdin17a}. But because of the nonlinear and interacting multiscale nature of the climate system, the simulated climate can depend sensitively and in unexpected ways on settings of tunable parameters \citep[e.g.,][]{Suzuki13a,Zhao16a}. It also remains unclear to what extent the resulting parameter choice is optimal, or how uncertain it is. Moreover, typically only a minute fraction of the available observations is used in the tuning process, usually only highly aggregated data such as global or large-scale mean values accumulated over periods of years or more. In part, this may be done to avoid overfitting, but more importantly, it is done because the tuning process usually involves parameter adjustments by hand, which each must be evaluated by a forward integration of the model. This makes the tuning process tedious and precludes adjustments of a larger set of parameters to fit more complex observational datasets or a wider range of high-resolution process simulations. It also precludes quantification of uncertainties \citep{Schirber13a,Hourdin17a}.

Climate models have improved over the past decades, leading, for example, to better simulations of El Ni{\~n}o, storm tracks, and tropical waves \citep{Guilyardi09a,Hung13a,Flato13a}. Weather prediction models, the higher-resolution siblings of the climate models' atmospheric component, have undergone a parallel evolution. Along with data assimilation techniques for the initialization of weather forecasts, this has led to great strides in the accuracy of weather forecasts \citep{Bauer15a}. But the accuracy of climate projections has not improved as much, and unacceptably large uncertainties remain.  For example, if one asks how high ${\rm {CO}_{2}}$\ concentrations can rise before Earth's surface will have warmed $2^\circ\mathrm{C}$ above pre-industrial temperatures---the warming target of the 2015 Paris Agreement, of which about $1^\circ\mathrm{C}$ remains because about $1^\circ\mathrm{C}$ has already been realized---the answers range from 480 to 600~ppm across current climate models \citep{Schneider17a}. A ${\rm {CO}_{2}}$\ concentration of 480~ppm will be reached in the late 2030s or early 2040s; 600~ppm may not be reached before 2060 even if ${\rm {CO}_{2}}$\ emissions continue to increase rapidly. Between these extremes lie vastly different optimal policy responses and socioeconomic costs of climate change \citep{Hope15a}.

These large and long-standing uncertainties in climate projections have their root in uncertainties in parameterization schemes. Parameterizations of clouds dominate the uncertainties in physical processes \citep{Cess89a,Cess90a,Stephens05,Bony06,Soden06a,Vial13a,Webb13b,Brient16b}. There are uncertainties both in the representation of the turbulent dynamics of clouds and in the representation of their microphysics, which control, for example, the distribution of droplet sizes in a cloud, the fraction of cloud condensate that precipitates out, and the phase partitioning of cloud condensate into liquid and ice \citep[e.g.,][]{Stainforth05a,Jiang12a,Suzuki13a,Golaz13a,Bodas-Salcedo14a,Zhao16a,Kay16a}. Additionally, there are numerous other parameterized processes that contribute to uncertainties in climate projections. For example, it is not precisely known what fraction of the ${\rm {CO}_{2}}$\ that is emitted by human activities will remain in the atmosphere, and so it is uncertain which emission pathways will lead to a given atmospheric ${\rm {CO}_{2}}$\ concentration target \citep{Knutti08b,Meinshausen09a,Friedlingstein15a}. Currently, only about half the emitted ${\rm {CO}_{2}}$\ accumulates in the atmosphere. The other half is taken up by oceans and on land. It is unclear in particular what fraction of the emitted ${\rm {CO}_{2}}$\ terrestrial ecosystems will take up in the future \citep{Friedlingstein06a,Canadell07a,Knorr09a,Le-Quere13a,Todd-Brown13a,Friedlingstein14a,Friend14a}. Reducing such uncertainties through the traditional approach to developing and improving parameterization schemes---attempting to develop one ``correct'' global parameterization scheme for each process in isolation, on the basis of observational or computational process studies that are usually focused on specific regions---has met only limited success \citep{Jakob03a,Jakob10a,Randall13b}.

Here we propose a new approach to improving parameterization schemes. The new approach invests considerable computational effort up front to exploit global observations and targeted high-resolution simulations through the use of data assimilation and machine learning within physical, biological, and chemical process models. We first outline in broad terms how we envision ESMs to learn from global observations and targeted high-resolution simulations (section~\ref{s:overview}). Then we discuss in more concrete terms the framework underlying such learning ESMs (section~\ref{s:learning}). We illustrate the approach by learning parameters in a relatively simple dynamical system that mimics characteristics of the atmosphere and oceans (section~\ref{s:lorenz96}). We conclude with an outlook of the opportunities the framework we outline presents and of the research program that needs to be pursued to realize it (section~\ref{s:discussion}).

\section{Learning from Observations and Targeted High-Resolution Simulations}\label{s:overview}

\subsection{Information Sources for Parameterization Schemes}

Parameterization schemes can learn from two sources of information:
\begin{enumerate}
\item \emph{Global observations}. We live in the golden age of Earth observations from space \citep{LEcuyer15a}. A suite of satellites flying in the formation known as the A-train has been streaming coordinated measurements of the composition of the atmosphere and of physical variables in the Earth system. We have nearly simultaneous measurements of variables such as temperature, humidity, and cloud and sea ice cover, with global coverage for more than a decade \citep{Stephens02a,Jiang12a,Simmons16a,Stephens17a}. Space-based measurements of biogeochemical tracers and processes, such as measurements of column-average ${\rm {CO}_{2}}$\ concentrations and of photosynthesis in terrestrial ecosystems, are also beginning to become available \citep[e.g.,][]{Crisp04a,Yokota09a,Frankenberg11a,Joiner11a,Frankenberg14a,Bloom16a,Eldering17a,Liu17a,Sun17a}, and so are more detailed observations of the cryosphere \citep[e.g.,][]{Shepherd12a,Gardner13a,Vaughan13a}. Parameterization schemes can learn from such space-based global data, which can be augmented and validated with more detailed local observations from the ground and from field studies.
\item \emph{Local high-resolution simulations}. Some processes parameterized in ESMs are in principle computable, only the globally achievable resolution precludes their explicit computation. For example, the turbulent dynamics (though currently not the microphysics) of clouds can be computed with high fidelity in limited domains in large-eddy simulations (LES) with grid spacings of $O(10~\mathrm{m})$ \citep{Siebesma03,Stevens05a,Khairoutdinov09a,Matheou14a,Schalkwijk15a,Pressel15a,Pressel17a}. Increased computational performance has made LES domain widths of $O(10~\mathrm{km})$--$O(100~\mathrm{km})$ feasible in recent years, while the horizontal mesh size in atmosphere models has shrunk, to the point that the two scales have converged. Thus, while global LES that reliably resolve low clouds such as cumulus or stratocumulus will not be feasible for decades, it is possible to nest LES in selected grid columns of atmosphere models and conduct high-fidelity local simulations of cloud dynamics in them \citep{Schneider17a}. Local high-resolution simulations of ocean mesoscale turbulence or sea ice dynamics can be conducted similarly. Parameterization schemes can learn from such nested high-resolution simulations.
\end{enumerate}
Of course, both observations and high-resolution simulations have been exploited in the development of parameterization schemes for some time. For example, data assimilation techniques have been used to estimate parameters in parameterization schemes from observations. Parameters especially in cloud, convection, and precipitation parameterizations have been estimated by minimizing errors in short-term weather forecasts over timescales of hours or days \citep[e.g.,][]{Emanuel99a,Grell02a,Aksoy06a,Schirber13a,Ruiz13a,Ruiz15a}, or by minimizing deviations between simulated and observed longer-term aggregates of climate statistics, such as global-mean TOA radiative fluxes accumulated over seasons or years \citep[e.g.,][]{Jackson08a,Jarvinen10a,Neelin10a,Solonen12a,Tett13a}. High-resolution simulations have been used to provide detailed dynamical information such as vertical velocity and turbulence kinetic energy profiles in convective clouds, which are not easily available from observations. They have often been employed to augment observations from local field studies, and parameterization schemes have been fit to and evaluated with the observations and the high-resolution simulations used in tandem \citep[e.g.,][]{Liu01a,Siebesma03,Stevens05a,Siebesma07,Hohenegger11a,de-Rooy13a,Romps16a}. High-resolution deep convection-resolving simulations with $O(1~\mathrm{km})$ horizontal grid spacing and, most recently, LES with $O(100~\mathrm{m})$ horizontal grid spacing have also been nested in small, usually two-dimensional subdomains of atmospheric grid columns, as a parameterization surrogate that explicitly resolves some aspects of cloud dynamics \citep[e.g.,][]{Grabowski99a,Grabowski01a,Khairoutdinov01a,Randall03a,Khairoutdinov05a,Randall13b,Grabowski16a,Parishani17a}. Such multiscale modeling approaches, often called superparameterization, have led to markedly improved simulations, for example, of the Asian monsoon, of tropical surface temperatures, and of precipitation and its diurnal cycle, albeit at great computational expense \citep[e.g.,][]{Benedict09a,Pritchard09a,Pritchard09b,Stan10a,DeMott13a}. However, multiscale modeling relies on a scale separation between the global-model mesh size and the domain size of the nested high-resolution simulation \citep{E07a}. Multiscale modeling is computationally advantageous relative to global high-resolution simulations as long as it suffices for the nested high-resolution simulation to subsample only a small fraction of the footprint of a global-model grid column, and to extrapolate the information so obtained to the entire footprint on the basis of statistical homogeneity assumptions. As the mesh size of global atmosphere models shrinks to horizontal scales of kilometers---resolutions that are already feasible in short integrations or limited areas and that will become routine in the next decade \citep{Palmer14c,Ban15a,Ohno16a,Schneider17a}---the scale separation to the minimum necessary domain size of nested high-resolution simulations will disappear, and with it the computational advantage of multiscale modeling.

What we propose here combines elements of these existing approaches in a novel way. At its core are still parameterization schemes that are based on physical, biological, or chemical process models, whose mathematical structure is developed on the basis of theory, local observations, and, where possible, high-resolution simulations. But we propose that these parameterization schemes, when they are embedded in ESMs, learn directly from observations and high-resolution simulations that both sample the globe. High-resolution simulations are employed in a targeted way---akin to targeted or adaptive observations in weather forecasting \citep{Palmer98a,Lorenz98a,Bishop01a}---to reduce uncertainties where observations are insufficient to obtain tight parameter estimates. Instead of incorporating high-resolution simulations globally in a small fraction of the footprint of each grid column like in multiscale modeling approaches, the ESM we envision deploys them locally, in entire grid columns, albeit only in a small subset of them. High-resolution simulations can be targeted to grid columns selected based on measures of uncertainty about model parameters. If the nested high-resolution simulations feed back onto the ESM, this corresponds to a locally extreme mesh refinement; however, two-way nesting may not always be necessary \citep[e.g.,][]{Moeng07a,Zhu10a}. The model learns parameters from observations and from nested high-resolution simulations in a computationally intensive learning phase, after which it can be used in a computationally more efficient manner, like models in use today. Nonetheless, even in simulations of climates beyond what has been observed, bursts of targeted high-resolution simulations can continue to be deployed to refine parameters and estimate their uncertainties.

\subsection{Computable and Non-computable Parameters}

Learning from  high-resolution simulations and observations is aimed at determining two different kinds of parameters in parameterization schemes: \emph{computable} and \emph{non-computable} parameters. (Since parameters and parametric functions of state variables play essentially the same role in our discussion, we simply use the term parameter, with the understanding that this can include parametric functions and even nonparametric functions.) Computable parameters are those that can in principle be inferred from high-resolution simulations alone. They include parameters in radiative transfer schemes, which can be inferred from detailed line-by-line calculations; dynamical parameters in cloud turbulence parameterizations, such as entrainment rates, which can be inferred from LES; or parameters in ocean mixing parameterizations, which can be inferred from high-resolution simulations. Non-computable parameters are parameters that, currently, cannot be inferred from high-resolution simulations, either because computational limitations make it necessary for them to also appear in parameterization schemes in high-resolution simulations, or because the microscopic equations governing the processes in question are unknown. They include parameters in cloud microphysics parameterizations, which are still necessary to include in LES, and many parameters characterizing ecological and biogeochemical processes, whose governing equations are unknown. Cloud microphysics parameters will increasingly become computable through direct numerical simulation \citep{Devenish12a,Grabowski13a}, but ecological and biogeochemical parameters will remain non-computable for the foreseeable future. Both computable and non-computable parameters can, in principle, be learned from observations; the only restrictions to their identifiability come  from the well-posedness of the learning problem and its computational tractability. But only computable parameters can be learned from targeted high-resolution simulations. To be able to learn computable parameters, it is essential to represent non-computable aspects of a parameterization scheme consistently in the high-resolution simulation and in the parameterization scheme that is to learn from the high-resolution simulation. For example, radiative transfer and microphysical processes need to be represented consistently in a high-resolution LES and in a parameterization scheme if the parameterization scheme is to learn computable dynamical parameters such as entrainment rates from the LES.

This approach presents challenges for parameter learning, since it implies the need to use observational data and high-resolution simulations in tandem to improve model parameterizations. But it also presents an opportunity: in doing so, the reliability and predictive power of ESMs can be improved, and uncertainties in parameters and predictions can be quantified.

\subsection{Objectives: Bias Reduction and Exploitation of Emergent Constraints}

Computational tractability is paramount for the success of any parameter learning algorithm for ESMs \citep[e.g.,][]{Annan07a,Jackson08a,Neelin10a,Solonen12a}. The central issue is the number of times the objective function needs to be evaluated, and hence an ESM needs to be run, in the process of parameter learning. Standard parameter estimation and inverse problem approaches may require $O(10^5)$ function or  derivative evaluations to learn $O(100)$ parameters, especially if uncertainty in the estimates is also required \citep{Cotter13a}. This many forward integrations and/or derivative evaluations of ESMs are not feasible if each involves accumulation of longer-term climate statistics. Fast parameterized processes in climate models often exhibit errors within a few hours or days of integration that are similar to errors in the mean state of the model \citep{Phillips04a,Rodwell07a,Xie12a,Ma13a,Klocke14a}. This has given rise to hopes that it may suffice to evaluate objective functions by weather hindcasts over timescales of only hours, making many evaluations of an objective function feasible \citep{Aksoy06a,Ruiz13a,Wan14a}. But experience has shown that such short-term optimization may not always lead to the desired improvements in climate simulations \citep{Schirber13a}. Additionally, slower parameterized processes, for example, involving biogeochemical cycles or the cryosphere, require longer integration times to accumulate statistics entering any meaningful objective function. Therefore, we focus on objective functions involving climate statistics accumulated over windows that we anticipate to be wide compared with the $O(10~\mathrm{days})$ timescale over which the atmosphere forgets its initial condition. Then the accumulated statistics do not depend sensitively on atmospheric initial conditions. This reduces the onus of correctly assimilating atmospheric initial conditions in parameter learning, which would be required if one were to match simulated and observed trajectories, as in approaches that assimilate model parameters jointly with the state of the system by augmenting state vectors with parameters \citep[e.g.,][]{Dee05a,Aksoy06a,Anderson09a}. The minimum window over which climate statistics will need to be accumulated will vary from processes to process, generally being longer for slower processes (e.g., the cryosphere) than faster processes (e.g., the atmosphere). For slower processes whose initial condition is not forgotten over the accumulation window, it will remain necessary to correctly assimilate initial conditions.

The objective functions to be minimized in the learning phase can be chosen to directly minimize biases in climate simulations, for example, precipitation biases such as the longstanding double-ITCZ bias in the tropics  \citep{Lin07a,Li14a,Zhang15a,Adam16b,Adam17a}, or cloud cover biases such as the ``too few--too bright'' bias in the subtropics \citep{Webb01a,Zhang05b,Karlsson08a,Nam12a}. Because the sensitivity with which an ESM responds to increases in greenhouse gas concentrations correlates with the spatial structure of some of these biases in the models \citep[e.g.,][]{Tian15a,Siler17a}, minimizing regional biases will likely reduce uncertainties in climate projections, in addition to leading to more reliable simulations of the present climate. To minimize biases, the objective function needs to include mean-field terms penalizing mismatch between spatially and at least seasonally resolved simulated and observed mean fields, for example, of precipitation, ecosystem primary productivity, and TOA radiative energy fluxes.

Additionally, there is a growing literature on ``emergent constraints,'' which typically are fluctuation-dissipation relationships that relate measurable fluctuations in the present climate to the response of the climate system to perturbations \citep{Hall06a,Collins12a,Klein15a}. For example, how strongly tropical low-cloud cover covaries with surface temperature from year to year or even seasonally in the present climate correlates in climate models with the amplitude of the cloud response to global warming  \citep{Qu14a,Qu15a,Brient16b}. Therefore, the observable low-cloud cover covariation with surface temperature in the present climate can be used to constrain the cloud response to global warming. Or, as another example, how strongly atmospheric ${\rm {CO}_{2}}$\ concentrations covary with surface temperature in the present climate correlates in climate models with the amplitude of the terrestrial ecosystem response to global warming (e.g., the balance between ${\rm {CO}_{2}}$\ fertilization of plants and enhanced soil and plant respiration under warming) \citep{Cox13a,Wenzel14a}. Therefore, the observable ${\rm {CO}_{2}}$\ concentration covariation with surface temperature can be used to constrain the terrestrial ecosystem response to global warming. Such emergent constraints are usually used post facto, in the evaluation of ESMs. They lead to inferences about the likelihood of a model given the measured natural variations, and they therefore can be used to assess how likely it is that its climate change projections are correct \citep[e.g.,][]{Brient16b}. But emergent constraints usually are not used directly to improve models. In what we propose, they are used directly to learn parameters in ESMs and to reduce uncertainties in the climate response. To do so, covariance terms (e.g., between surface temperature and cloud cover or TOA radiative fluxes, or between surface temperature and ${\rm {CO}_{2}}$\ concentrations) need to be included in the objective function.

The choice of objective functions to be employed is key to the success of what we propose. The use of time-averaged statistics such as mean-field and covariance terms will make the objective functions smoother and hence reduce the computational cost of minimization, compared with minimizing objective functions that directly penalize mismatch between simulated and observed trajectories of the Earth system. From the point of view of statistical theory, the objective functions should contain the sufficient statistics for the parameters of interest, but what these are is not usually known a priori. In practice, the choice of objective functions will be guided by expertise specific to the relevant subdomains of Earth system science, as well as computational cost. Given that current ESM components such as clouds and the carbon cycle exhibit large seasonal biases \citep[e.g.,][]{Keppel-Aleks12a,Karlsson13a,Lin14b}, and their response to long-term warming in some respects resembles their response to seasonal variations \citep[e.g.,][]{Brient16b, Wenzel16a}, accumulating seasonal statistics in the objective functions suggests itself as a starting point.

\section{Machine Learning Framework for Earth System Models}\label{s:learning}

\subsection{Models and Data}

To outline how we envision parameterization schemes in ESMs to learn from diverse data, we first set up notation. Let $\vec{\theta}=(\vec{\theta}_c, \vec{\theta}_n)$ denote the vector of model parameters to be learned, consisting of computable parameters $\vec{\theta}_c$ that can be learned from high-resolution simulations, and non-computable parameters $\vec{\theta}_n$ that can only be learned from observations (for example, because high-resolution simulations themselves depend on $\vec{\theta}_n$). The parameters $\vec{\theta}$ appear in parameterization schemes in a model, which may be viewed as a map $\mathop{\mathcal{G}}$, parameterized by time $t$, that takes the parameters $\vec{\theta}$ to the state variables $\vec{x}$,
\begin{equation}
\vec{x}(t) = \mathop{\mathcal{G}}(\vec{\theta}, t).
\end{equation}
The state variables $\vec{x}$ can include temperatures, humidity variables, and cloud, cryosphere, and biogeochemical variables, and the map $\mathop{\mathcal{G}}$ may depend on initial conditions and time-evolving boundary or forcing conditions. The map $\mathop{\mathcal{G}}$ typically represents a global ESM. The state variables $\vec{x}$ are linked to observables $\vec{y}$ through a map $\mathop{\mathcal{H}}$ representing an observing system, so that
\begin{equation}
\vec{y}(t)=\mathop{\mathcal{H}}\bigl(\vec{x}(t)\bigr).
\end{equation}
The observables $\vec{y}$ might represent surface temperatures, ${\rm {CO}_{2}}$\ concentrations, or spectral radiances emanating from the TOA. The map $\mathop{\mathcal{H}}$ in practice will be realized through an observing system simulator, which simulates how observables $\vec{y}$ are impacted by a multitude of state variables $\vec{x}$. The actual observations (e.g., space-based measurements) are denoted by $\vec{\tilde y}$, so $\vec{y}(t) - \vec{\tilde y}(t)$ is the mismatch between simulations and observations. Since $\vec{y}$ is parameterized by $\vec{\theta}$, while $\vec{\tilde y}$ is independent of $\vec{\theta}$, mismatches between $\vec{y}$ and  $\vec{\tilde y}$ can be used to learn about $\vec{\theta}$.

Local high-resolution simulations nested in a grid column of an ESM may be viewed as a time-dependent map $\mathop{\mathcal{L}}$ from the state variables $\vec{x}$ of the ESM to simulated state variables $\vec{\tilde z}$,
\begin{equation}
\vec{\tilde z}(t) = \mathop{\mathcal{L}}(\vec{\theta}_n,t; \vec{x}).
\end{equation}
The map $\mathop{\mathcal{L}}$ is parameterized by non-computable parameters $\vec{\theta}_n$ and time $t$, and it can involve the time-history of the state variables $\vec{x}$ up to time $t$. The vector $\vec{\tilde z}$ contains statistics of high-resolution variables whose counterparts in the ESM are computed by parameterization schemes, such as the mean cloud cover or liquid water content in a grid box. The corresponding variables $\vec{z}$ in the ESM are obtained by a time-dependent map $\mathop{\mathcal{S}}$ that takes state variables $\vec{x}$ and parameters $\vec{\theta}$ to $\vec{z}$,
\begin{equation}
\vec{z}(t) = \mathop{\mathcal{S}}(\vec{\theta},t; \vec{x}).
\end{equation}
The map $\mathop{\mathcal{S}}$ typically represents a single grid column of the ESM with its parameterization schemes, taking as input $\vec{x}$ from the ESM. It is structurally similar to $\mathop{\mathcal{L}}$. Crucially, however, $\mathop{\mathcal{S}}$ generally depends on all parameters $\vec{\theta}=(\vec{\theta}_c, \vec{\theta}_n)$, while $\mathop{\mathcal{L}}$ only depends on non-computable parameters $\vec{\theta}_n$. Thus, the mismatch $\vec{z}(t) - \vec{\tilde z}(t)$ can be used to learn about the computable parameters $\vec{\theta}_c$.

The same framework also covers other ways of learning about parameterizations schemes from data. For example, the map $\mathop{\mathcal{G}}$ may represent a single grid column of an ESM, driven by time-evolving boundary conditions from reanalysis data at selected sites. Observations at the sites can then be used to learn about the parameterization schemes in the column \citep{Neggers12a}. Or, similarly, the map $\mathop{\mathcal{G}}$ may represent a local high-resolution simulation driven by reanalysis data, with parameterization schemes, e.g., for cloud microphysics, about which one wants to learn from observations.

\subsection{Objective Functions}

Objective functions are defined through mismatch between the simulated data $\vec{y}$ and observations $\vec{\tilde y}$, on the one hand, and simulated data $\vec{z}$ and high-resolution simulations $\vec{\tilde z}$, on the other hand. We define mismatches using time-averaged statistics because they do not suffer from sensitivity to atmospheric initial conditions; indeed, matching trajectories directly requires assimilating atmospheric initial conditions, which would make it difficult to disentangle mismatches due to errors in climatically unimportant atmospheric initial conditions from those due to parameterization errors. However, the time averages can still depend on initial conditions for slowly evolving components of the Earth system, such as ocean circulations or ice sheets.

We denote the time average of a function $\phi(t)$ over the time interval $[t_0,t_0+T]$ by
\begin{equation}\label{e:Tavg}
\langle \phi \rangle_T = \frac{1}{T} \int_{t_0}^{t_0+T} \phi(t) \, dt.
\end{equation}
The observational objective function can then be written in the generic form
\begin{equation}\label{e:obj_o}
J_o(\vec{\theta})=\frac12\| \langle \vec{f}(\vec{y})  \rangle_T - \langle \vec{f}(\vec{\tilde y})
\rangle_T \|_{\Sigma_y}^2
\end{equation}
with the 2-norm
\begin{equation}
\|\cdot\|_{\Sigma_y}=\|\Sigma_y^{-\frac12}\cdot\|
\end{equation}
normalized by error standard deviations and covariance information captured in $\Sigma_y$. The function
$\vec{f}$ of the observables typically involves first- and second-order quantities, for example,
\begin{equation}
\vec{f}(\vec{y}) = \left(
\begin{array}{c}
\vec{y}\\
y_i' y_j'
\end{array}
\right),
\label{e:of}
\end{equation}
where, for any observable $\phi$, $\phi'(t) = \phi(t) - \langle \phi \rangle_T$ denotes the fluctuation of $\phi$ about its mean $\langle \phi \rangle_T$. With $\vec{f}$ given by \eqref{e:of}, the objective function penalizes mismatch between the vectors of mean values $\langle \vec{y} \rangle_T$ and $\langle \vec{\tilde y}\rangle_T$ and between the covariance components $\langle y_i' y_j' \rangle_T$ and  $\langle \tilde y_i' \tilde y_j' \rangle_T$ for some indices $i$ and $j$. The least-squares form of the objective function \eqref{e:obj_o} follows from assuming an error model
\begin{equation}
\vec{f}(\vec{y})=\vec{f}(\vec{\tilde y})+\vec{\eta},
\end{equation}
with the matrix $\Sigma_y$ encoding an assumed covariance structure of the noise vector $\vec{\eta}$. The relevant components of $\Sigma_y$ may be chosen very small for quantities that are used as constraints on the ESM (e.g., the requirement of a closed global energy balance at TOA).

For the mismatch to high-resolution simulations, we accumulate statistics over an ensemble of high-resolution simulations in different grid columns of the ESM and at different times, possibly, but not necessarily, also accumulating in time. We denote the corresponding ensemble and time average by $\langle \phi \rangle_E$, and define an objective function analogously to that for the observations through
\begin{equation}\label{e:obj_hr}
J_s(\vec{\theta}_c)=\frac12\| \langle \vec{g}(\vec{z})  \rangle_E - \langle \vec{g}(\vec{\tilde z})
\rangle_E \|_{\Sigma_z}^2.
\end{equation}
Like the function $\vec{f}$ above, the function $\vec{g}$  typically involves first- and second-order quantities, and the least-squares form of the objective functions follows from the assumed covariance structure $\Sigma_z$ of the noise.

\subsection{Learning Algorithms}

Learning algorithms attempt to choose parameters $\vec{\theta}$ that minimize $J_o$ and $J_s$. However, minimization of $J_o$ and $J_s$ does not always determine the parameters uniquely, for example, if there are strongly correlated parameters or if the number of parameters to be learned exceeds the number of available observational degrees of freedom. In such cases, regularization is necessary to choose a good solution for the parameters among the multitude of possible solutions. This may be achieved in various ways: by adding to the least-squares objective functions \eqref{e:obj_o} and \eqref{e:obj_hr} regularizing penalty terms that incorporate prior knowledge about the parameters \citep{Engl96}, by Bayesian probabilistic regularization \citep{Kaipio05a}, or by restriction of the parameters to a subset, as in ensemble Kalman inversion \citep{Iglesias13a}.

All of these regularization approaches may be useful in ESMs. They involve different trade-offs between computational expense and the amount of information about the parameters they provide.
\begin{itemize}
\item Classical regularized least squares leads to an optimization problem that is typically tackled by gradient descent or Gauss-Newton methods, in which derivatives of the parameter-to-data map are employed \citep{Nocedal06a}. Such methods usually require $O(10^2)$ integrations of the forward model or evaluations of its derivatives with respect to parameters.
\item Bayesian inversions usually employ Markov chain Monte Carlo (MCMC) methods \citep{Brooks11a} and variants such as sequential Monte Carlo \citep{Del-Moral06a} to approximate the posterior probability density function (PDF) of parameters, given data and a prior PDF. A PDF of parameters provides much more information than a point estimate, and consequently MCMC methods typically require many more forward model integrations, sometimes on the order of $O(10^5)$. The computational demands can be decreased by an order of magnitude by judicious use of derivative information where available (see \citet{Beskos17a} and references therein) or by improved sampling strategies \citep[e.g.,][]{Jackson08a,Jarvinen10a,Solonen12a}. Nonetheless, the cost remains orders of magnitude higher than for optimization techniques.
\item Ensemble Kalman methods are easily parallelizable, derivative-free alternatives to the classical optimization and Bayesian approaches \citep{Houtekamer16a}. Although theory for them is less well developed, empirical evidence demonstrates behavior similar to derivative-based algorithms in complex inversion problems, with a comparable number of forward model integrations \citep{Iglesias16a}. Ensemble methods for joint state and parameter estimation have recently been systematically developed \citep{Bocquet13a,Bocquet14a,Carrassi17a}, and they are emerging as a promising way to solve inverse problems and to obtain qualitative estimates of uncertainty. However, numerical experiments have indicated that such uncertainty information is qualitative at best: the Kalman methods invoke Gaussian assumptions, which may not be justified, and even if the Gaussian approximation holds, the ensemble sizes needed for uncertainty quantification may not be practical \citep{Law12a,Iglesias13a}.
\end{itemize}

An important consideration is how to blend the information about parameters contained in the high-resolution simulations and in the observations. One approach is as follows, although others may turn out to be preferable. Minimizing the high-resolution objective function $J_s$ in principle gives the computable parameters $\vec{\theta}_c$ as an implicit function of the non-computable parameters $\vec{\theta}_n$. This implicit function may then be used as prior information to minimize  the observational objective function $J_o$ over $\vec{\theta}.$ Bayesian MCMC approaches may be feasible for fitting $J_s$, since the single column model $\mathop{\mathcal{S}}$ is relatively cheap to evaluate, and the ensemble of high-resolution simulations $\mathop{\mathcal{L}}$ needed may not be large. Although Bayesian approaches may not be feasible for fitting $J_o$, for which accumulation of statistics of the model $\mathop{\mathcal{G}}$ is required, this hierarchical approach does have the potential to incorporate detailed uncertainty estimates coming from the high-resolution simulations.

The choice of normalization (i.e., $\Sigma_y$ and $\Sigma_z$) in the objective functions plays a significant role in parameter learning, and learning about it has been demonstrated to have considerable impact on data assimilation for weather forecasts \citep{Dee95a,Stewart14a}. We will not discuss this issue in any detail, but note it may be addressed by the use of hierarchical Bayesian methodology and ensemble Kalman analogues. Nor will we dwell on the important issue of structural uncertainty---model error--- other than to note that this can, in principle, be addressed through the inverse problem approach advocated here: additional unknown parameters, placed judiciously within the model to account for model error, can be learned from data \citep{Kennedy01a,Dee05a}. The choice of normalization is especially important in this context as it relates to disentangling learning about model error from learning about the other parameters of interest.

Learning algorithms for ESMs can be developed further in several ways:
\begin{itemize}
\item Minimization of the objective functions $J_o$ and $J_s$ may be performed by online filtering algorithms, akin to those used in the initialization of weather forecasts, which sequentially update parameters as information becomes available \citep{Law15a}. This can reduce the number of forward model integrations required for parameter estimation, and it can allow parameterization schemes to learn adaptively from high-resolution simulations during the course of a global simulation.
\item Where to employ targeted high-resolution simulations can be chosen to optimize aspects of the learning process. The simplest approach would be to deploy them randomly, for example, by selecting regions with a probability proportional to their climatological cloud fraction for high-resolution simulations of clouds. More efficient would be techniques of optimal experimental design (see \citet{Alexanderian16a}
and references therein), within online filtering algorithms. With such techniques, high-resolution simulations could be generated to order, to update aspects of parameterization schemes that have the most influence on the global system with which they interact.
\end{itemize}
Progress along these lines will require innovation. For example, filtering algorithms need to be adapted to deal with strong serial correlations such as those that arise when averages $\langle \phi \rangle_{T_i}$ are accumulated over increasing spans $T_i < T_{i+1}$ and parameters are updated from one average $\langle \phi \rangle_{T_i}$ to a longer average $\langle \phi \rangle_{T_{i+1}}$. And optimal experimental design techniques require the development of cheap computational methods to evaluate sensitivities of the ESM to individual aspects of parameterization schemes.

\section{Illustration With Dynamical System}\label{s:lorenz96}

We envision ESMs eventually to learn parameters online, with targeted high-resolution simulations triggering parameter updates on the fly. Here we want to illustrate in off-line mode some of the opportunities and challenges of learning parameters in a relatively simple dynamical system. We use the Lorenz-96 model \citep{Lorenz96a}, which has nonlinearities resembling the advective nonlinearities of fluid dynamics and a multiscale coupling of slow and fast variables similar to what is seen in ESMs. The model has been used extensively in the development and testing of data assimilation methods \citep[e.g.,][]{Lorenz98a, Anderson01a, Ott04a}.

\subsection{Lorenz-96 Model}

The Lorenz-96 model consists of $K$ slow variables $X_k$ ($k=1, \dots, K$), each of which is coupled to $J$ fast variables $Y_{j,k}$ ($j=1,\dots, J$) \citep{Lorenz96a}:
\begin{align}
\frac{dX_k}{dt}     &= - X_{k-1}(X_{k-2} - X_{k+1}) - X_k + F - hc \bar{Y}_k, \label{e:l96_slow}\\
\frac{1}{c} \, \frac{dY_{j,k}}{dt} &= -b Y_{j+1,k}(Y_{j+2,k} - Y_{j-1, k}) - Y_{j,k} + \frac{h}{J} X_k. \label{e:l96_fast}
\end{align}
The overbar denotes the mean value over $j$,
\begin{equation}
\bar{Y}_k = \frac{1}{J} \sum_{j=1}^J Y_{j,k}.
\end{equation}
Both the slow and fast variables are taken to be periodic in $k$ and $j$, forming a cyclic chain with $X_{k+K} = X_k$, $Y_{j, k+K}=Y_{j, k}$, and $Y_{j+J,k} = Y_{j,k+1}$.  The slow variables $X$ may be viewed as resolved-scale variables and the fast variables $Y$ as unresolved variables in an ESM. Each of the $K$ slow variables $X_k$ may represent a property such as surface air temperature in a cyclic chain of grid cells spanning a latitude circle. Each slow variable $X_k$ affects the $J$ fast variables $Y_{j,k}$ in the grid cell, which might represent cloud-scale variables such as liquid water path in each of $J$ cumulus clouds. In turn, the mean value of the fast variables over the cell, $\bar{Y}_k$, feeds back onto the slow variables $X_k$. The strength of the coupling between fast and slow variables is controlled by the parameter $h$, which represents an interaction coefficient, for example, an entrainment rate that couples cloud-scale variables to their large-scale environment. Time is nondimensionalized by the linear-damping timescale of the slow variables, which we nominally take to be 1~day, a typical thermal relaxation time of surface temperatures \citep{Swanson97b}. The parameter $c$ controls how rapidly the fast variables are damped relative to the slow; it may be interpreted as a microphysical parameter controlling relaxation of cloud variables, such as a precipitation efficiency.  The parameter $F$ controls the strength of the external large-scale forcing, and $b$ the amplitude of the nonlinear interactions among the fast variables. Following \citet{Lorenz96a}, albeit relabeling parameters, we choose $K=36$, $J=10$, $h=1$, and $F=c=b=10$, which ensures chaotic dynamics of the system.

The quadratic nonlinearities in this dynamical system resemble advective nonlinearities, e.g., in the sense that they conserve the quadratic invariants (``energies'')  $\sum_{k} X_k^2$ and $\sum_{j} Y_{j, k}^2$ \citep{Lorenz98a}.  The interaction between the slow and fast variables conserves the ``total energy'' $\sum_k \left(X_k^2 + \sum_j Y_{j, k}^2\right)$. Energies are damped by the linear terms; they are prevented from decaying to zero by the external forcing $F$. Eventually, the system approaches a statistically steady state in which driving by the external forcing $F$ balances the linear damping.

Let $\langle \cdot \rangle_\infty$ denote a long-term time mean in the statistically steady state, and note that all slow variables $X_k$ are statistically identical, as are all fast variables $Y_{j,k}$, so we can use the generic symbols $X$ and $Y$ in statistics of the variables. Multiplication of \eqref{e:l96_slow} by $X_k$, using that all variables $X_k$ are statistically identical, and averaging shows that, in the statistically steady state, second moments of the slow variables satisfy
\begin{equation}\label{e:moments1}
\langle X^2 \rangle_\infty = F \langle X \rangle_\infty - hc \langle X \bar Y \rangle_\infty.
\end{equation}
Similarly, second moments of the fast variables satisfy
\begin{equation}\label{e:moments2}
\langle \overline{Y^2} \rangle_\infty = \frac{h}{J} \langle X \bar Y \rangle_\infty,
\end{equation}
where the overbar again denotes a mean value over the fast-variable index $j$. That is, the interaction coefficient $h$ can be determined from estimates of the one-point statistics $\langle \overline{Y^2} \rangle_\infty$ and $\langle X \bar Y \rangle_\infty$. Its inverse is proportional to the regression coefficient of the fast variables onto the slow: $h^{-1} \propto \langle X \bar Y \rangle_\infty/\langle \overline{Y^2} \rangle_\infty$. So the regression of the fast variables onto the slow can be viewed as providing an ``emergent  constraint'' on the system, insofar as the interaction coefficient $h$ affects the response of the system to perturbations (e.g., in $F$). Estimates of $\langle X^2 \rangle_\infty$ and $\langle X \rangle_\infty$ provide an additional constraint \eqref{e:moments1} on the parameters $F$ and $c$. Taking mean values of the dynamical equations \eqref{e:l96_slow} and \eqref{e:l96_fast} would provide further constraints on these parameters, as well as on $b$, in terms of two-point statistics involving shifts in $k$ and $j$, e.g., covariances of $X_k$ and $X_{k-1}$.

In what follows, we demonstrate the performance of learning algorithms in a perfect-model setting, first focusing on one-point statistics to show how to learn about parameters in the full dynamical system from them. Subsequently, we use two-point statistics to learn about parameters in a single ``grid column'' of fast variables only.

\subsection{Parameter Learning in Perfect-Model Setting}

We generate data from the dynamical system \eqref{e:l96_slow} and \eqref{e:l96_fast} with the parameters $\vec{\theta} = (F, h, c, b)$ set to $\vec{\tilde\theta} = (10, 1, 10, 10)$. The role of ``observations'' $\vec{\tilde y} = (\tilde X, \tilde Y)$ in the perfect-model setting is played by data $\tilde X$ and $\tilde Y$ generated by the dynamical system with parameters $\vec{\theta}$ set to their ``true'' values $\vec{\tilde\theta}$. That is, the dynamical system \eqref{e:l96_slow} and \eqref{e:l96_fast} with parameters $\vec{\theta}$ stands for the global model $\mathop{\mathcal{G}}$, the observing system map $\mathop{\mathcal{H}}$ is the identity, and the data $\tilde X$ and $\tilde Y$ generated by the dynamical system with parameters $\vec{\tilde \theta}$ is a surrogate for observations. The parameters $\vec{\theta}$ of the dynamical system are then learned by matching statistics $\langle \phi \rangle_T$ accumulated over $T = 100~\mathrm{days}$ (with 1~day denoting the unit time of the system),  using discrete sums in place of the time integral in the average \eqref{e:Tavg} and minimizing the ``observational'' objective function
\begin{equation}\label{e:obj_o1}
J_o(\vec{\theta})=\frac12\| \langle \vec{f}(X, Y)  \rangle_T - \langle \vec{f}(\tilde X, \tilde Y)
\rangle_\infty \|_{\Sigma}^2.
\end{equation}
The moment function to be matched,
\begin{equation}
\vec{f}(X, Y) = \left(
\begin{array}{c}
X\\
\bar Y\\
X^2\\
X\bar Y\\
\overline{Y^2}
\end{array}
\right),
\end{equation}
has an entry for each of the $K=36$ indices $k$, giving a vector of length $5K =180$. The noise covariance matrix $\Sigma$ is chosen to be diagonal, with entries that are proportional to the sample variances of the moments contained in the vector $\vec{f}$,
\begin{equation}\label{e:noise}
\mathop{\mathrm{diag}} \Sigma = r^2 \bigl[\mathop{\mathrm{var}}(X), \mathop{\mathrm{var}}({\bar Y}), \mathop{\mathrm{var}}({X^2}), \mathop{\mathrm{var}}({X\bar Y}), \mathop{\mathrm{var}}({\overline{Y^2}})\bigr].
\end{equation}
Here $\mathop{\mathrm{var}}(\phi)$ denotes the variance of $\phi$, and $r$ is an empirical parameter indicating the noise level. The variances $\mathop{\mathrm{var}}(\phi)$ and the ``true moments'' $\langle \vec{f}(\tilde X, \tilde Y)\rangle_\infty$ are estimated from a long (46,416~days) control simulation of the dynamical system with the true parameters $\vec{\tilde\theta}$.

As an illustrative example, we use normal priors for $(\theta_1, \theta_2, \theta_4) = (F, h, b)$, with mean values $(\mu_1, \mu_2, \mu_4) = (10, 0, 5)$ and variances $(\sigma^2_1, \sigma^2_2, \sigma^2_4) = (10, 1, 10)$. Enforcing positivity of $c$, we use a log-normal prior for $\theta_3 = c$, with a mean value $\mu_3 = 2$ and variance $\sigma_3^2 = 0.1$ for $\log c$ (i.e., a mean value of $7.4$ for $c$). We take the parameters a priori to be uncorrelated, so that the prior covariance matrix is diagonal.

\begin{figure*}[htb]
   \centering
   \includegraphics[width=160mm]{Lorenz-MCMC.pdf}
   \caption{Probability density functions for parameter learning in Lorenz-96 model. First row: Marginal potential energies (negative logarithm of posterior PDF) at different noise levels $r$. Each marginal potential energy is obtained by varying the parameter on the abscissa, while holding other parameters fixed at their true values. Second row: Marginal prior and posterior PDFs, estimated by MCMC from the full dynamics \eqref{e:l96_slow}--\eqref{e:l96_fast},  with noise level $r=0.5$ and accumulating over $T=100~\mathrm{days}$ for each sample. Third row: Marginal prior and posterior PDFs, estimated by MCMC from the fast dynamics \eqref{e:l96_fast} only, likewise with noise level $r'=0.5$ but accumulating over only $T'=20~\mathrm{days}$ for each sample.}
   \label{fig:lorenz96_mcmc}
\end{figure*}
To illustrate the landscape learning algorithms have to navigate, Figure~\ref{fig:lorenz96_mcmc} (top row) shows sections through the potential energy, defined as the negative logarithm of the posterior PDF,
\begin{equation}
U(\vec{\theta}) = J_o(\vec{\theta}) - \sum_{i=1}^4 \log p_i(\theta_i),
\end{equation}
where $p_i(\theta_i)$ is the prior PDF of parameter $\theta_i$. The figure shows the marginal potential energies obtained as one parameter at a time is varied and the objective function $J_o(\vec{\theta})$ is accumulated by forward integration, while the other parameters are held fixed at their true values. As the noise level $r$ increases, the contribution of the log-likelihood of the data ($\propto J_o$) is down-weighted relative to the prior, the posterior modes shift toward the prior modes, and the posterior is smoothed. Here the objective function $J_o(\vec{\theta})$ for each parameter setting is accumulated over a long period ($10^4$~days) to minimize sampling variability. However, even with this wide accumulation window, sampling variability remains in some parameter regimes and there noticeably affects $J_o(\vec{\theta})$. An example is the roughness around $c=17$, which appears to be caused by metastability on timescales longer than the accumulation window.  The roughness could be smoothed by accumulating over periods that are yet longer, or by averaging over an ensemble of initial conditions, but analogous smoothing might be impractical for ESMs. Time-averaged ESM statistics may exhibit similarly rough dependencies on some parameters \citep[e.g.,][]{Suzuki13a,Zhao16a}, although the dependence on other parameters appears to be relatively smooth \citep[e.g.,][]{Neelin10a}, perhaps because ESM parameters targeted for tuning are chosen for the smooth dependence of the climate state on them. Roughness of the potential energy landscape can present challenges for learning algorithms, which may get stuck in local minima. Note also the bimodality in $b$, which arises because the one-point statistics we fit cannot easily distinguish prograde wave modes of the system (which propagate toward increasing $k$) from retrograde modes \citep[cf.][]{Lorenz98a}.

\subsubsection{Bayesian Inversion}

We use the random-walk Metropolis (RWM) MCMC algorithm \citep{Brooks11a} for a full Bayesian inversion of parameters in the dynamical system \eqref{e:l96_slow}, \eqref{e:l96_fast}, thereby sampling from the posterior PDF. To reduce burn-in (MCMC spin-up) time, we initialize the algorithm close to the true parameter values with the result of an ensemble Kalman inversion (see below). The RWM algorithm is then run over $2200$ iterations, the first $200$ iterations are discarded as burn-in, and the posterior PDF is estimated by binning every other of the remaining 2000 samples. The objective function for each sample is accumulated over $T=100~\mathrm{days}$, using the end state of the previous forward integration as initial condition for the next one, without discarding any spin-up after a parameter update.

The resulting marginal posterior PDFs do not all peak exactly at the true parameter values, but the true parameter values lie in a region that contains most of the posterior probability mass (Figure~\ref{fig:lorenz96_mcmc}, second row). The posterior PDFs indicate the uncertainties inherent in estimating the parameters. The posterior PDF of $c$ has the largest spread, in terms of standard deviation normalized by mean, indicating relatively large uncertainty in this parameter. The uncertainty appears to arise from the roughness of the potential energy (Figure~\ref{fig:lorenz96_mcmc}, first row), which reflects inherent sensitivity of the system response to parameter variability; additional roughness of the posterior PDFs may be caused by sampling variability from finite-time  averages \citep{Wang14c}. For all four parameters, the posterior PDFs differ significantly from the priors, demonstrating the information content provided by the synthetic data. Finally, although these results have been obtained with $O(10^3)$ forward model integrations and objective function evaluations, more objective function evaluations may be required for more complex forward models, such as ESMs.

\subsubsection{Ensemble Kalman Inversion}

Ensemble Kalman inversion may be an attractive learning algorithm for ESMs when Bayesian inversion with MCMC is computationally too demanding. To illustrate its performance, we use the algorithm of \citet{Iglesias13a}, initializing ensembles of size $M$ with parameters drawn from the prior PDFs. In the analysis step of the Kalman inversion, we perturb the target data by addition of noise with zero mean and variance given by \eqref{e:noise}, that is, replacing $\langle \vec{f}(\tilde X, \tilde Y) \rangle_\infty$ by $\langle \vec{f}(\tilde X, \tilde Y) \rangle_\infty + \vec{\eta}^{(j)}$ with $\vec{\eta}^{(j)}\sim \mathcal N(0,\Sigma)$ for each ensemble member $j$. As in the MCMC algorithm, the objective function for each parameter setting is accumulated over $T=100~\mathrm{days}$, without discarding any spin-up after each parameter update. As initial state for the integration of the ensemble, we use a state drawn from the statistically steady state of a simulation with the true parameters.

\begin{table}[ht]
\centering
\begin{tabular}{l|ccc}
  \hline
\multicolumn{1}{c|}{Noise} & Mean ($M=10$) & Mean ($M=100$) & Std ($M=100$) \\
  \hline
$r=0.1$ & (9.62, 0.579, 9.37, 2.63) & (9.71, 0.992, 8.70, 9.95) & (0.023, 0.001, 0.104, 0.022) \\
$r=0.2$ & (9.57, 0.516, 7.90, 3.15) & (9.77, 0.994, 9.07, 10.04) & (0.107, 0.005, 0.524, 0.103) \\
$r=0.5$ & (9.77, 0.522, 9.29, 5.31) & (9.63, 0.982, 8.34, 9.93) & (0.295, 0.017, 1.477, 0.350) \\
$r=1.0$ & (9.70, 0.633, 7.68, 6.13) & (9.53, 0.952, 7.97, 9.37) & (0.385, 0.039, 1.964, 0.701) \\
   \hline
\end{tabular}
\caption{Ensemble means and standard deviations for the parameters $\theta=(F, h, c, b)$ obtained by ensemble Kalman inversions for different ensemble sizes $M$ and different noise levels $r$.}
\label{tab:EnKF_opt}
\end{table}
Table~\ref{tab:EnKF_opt} summarizes the solutions obtained by this ensemble Kalman inversion after $N_{\max}=25$ iterations, for different ensemble sizes $M$ and noise levels $r$. The ensemble mean of the Kalman inversion provides reasonable parameter estimates. But the ensemble standard deviation does not always provide quantitatively accurate uncertainty information. For example, for low noise levels, the true parameter values often lie more than two standard deviations away from the ensemble mean. The ensemble spread also differs quantitatively from the posterior spread in the MCMC simulations. In experiments in which we did not perturb the target data, the smaller ensembles ($M=10$) occasionally collapsed, with each ensemble member giving the same point estimate of the parameters. In such cases, the ensemble contains no uncertainty information, illustrating potential pitfalls of using ensemble Kalman inversion for uncertainty quantification. However, with the perturbed data and for larger ensembles, the ensemble standard deviation is qualitatively consistent with the posterior PDF estimated by MCMC (Figure~\ref{fig:lorenz96_mcmc}, second row). It provides some uncertainty information, especially for higher noise levels, for example, in the sense that the parameter $c$ is demonstrably the most uncertain (Table~\ref{tab:EnKF_opt} and Figure~\ref{fig:EnKFerr_lorenz96ms}b). Methods such as localization and variance inflation can help with issues related to ensemble collapse and can also be used to improve ensemble statistics more generally (see \cite{Law15a} and the references therein). However, systematic principles for their application with the aim of correctly reproducing Bayesian posterior statistics have not been found, and so we have not adopted this approach.

The ensemble Kalman inversion typically converges within a few iterations (Figure~\ref{fig:EnKFerr_lorenz96ms} indicates ${\lesssim}\,5$ iterations when $M=100$). Larger ensembles lead to solutions closer to the truth (Figure~\ref{fig:EnKFerr_lorenz96ms}a). Convergence within 5 iterations for ensembles of size 10 or 100 implies 50 or 500 objective function evaluations, representing substantial computational savings over the MCMC algorithm with 2000 objective function evaluations. These computational savings come at the expense of detailed uncertainty information. Where the optimal trade-off lies between computational efficiency, on the one hand, and precision of parameter estimates and uncertainty quantification, on the other hand, remains to be investigated.
\begin{figure}[htb]
   \centering
   \includegraphics{Lorenz-EnKF.pdf}
   \caption{Convergence of ensemble Kalman inversion. (a) Error norm $\|\vec{\hat\theta} - \vec{\tilde\theta}\|$ of ensemble-mean parameter estimate $\vec{\hat\theta}$ as a function of iteration, for different ensemble sizes $M$ and noise levels $r$. Dashed lines for $M=10$, and solid lines for $M=100$. Color coding of noise levels $r=0.1$, $0.2$, $0.5$, and $1.0$ as in Figure~\ref{fig:lorenz96_mcmc} (top row). (b) Ensemble-mean value $\hat\theta_i$ of the four parameters as a function of iteration (solid lines), with interquartile range of ensemble (shading from 25th to 75th percentile), for $M=100$ and $r=0.5$. The objective function for each parameter setting is accumulated over $T=100~\mathrm{days}$.}
   \label{fig:EnKFerr_lorenz96ms}
\end{figure}

\subsection{Parameter Learning From Fast Dynamics}

Finally, we investigate learning about parameters from the fast dynamics \eqref{e:l96_fast} alone. This is similar to learning about computable parameters from local high-resolution simulations, e.g., of clouds. That is, the fast dynamics \eqref{e:l96_fast} with the true parameters $\vec{\tilde\theta}$ stand for the high-resolution model $\mathop{\mathcal{L}}$, which generates data $\vec{\tilde z} = \tilde Y$, and the fast dynamics with parameters $\vec{\theta}$ play the role of the single-column model $\mathop{\mathcal{S}}$, which generates data $\vec{z} = Y$. We choose $k=1$ and fix $X_1=2.556$, a value taken from the statistically steady state of the full dynamics. There are three parameters to learn from the fast dynamics: $(\theta_2, \theta_3, \theta_4) = (h,c,b)$. The one-point statistics $(\bar Y_1,\overline{Y_1^2})$ of the fast variables are not enough to recover all three. Therefore, we consider the moment function
\begin{equation}
\vec{g}(Y) = \left(
\begin{array}{c}
Y_{j,1}\\
Y_{j,1}Y_{j',1}
\end{array}
\right), \qquad j, j' \in \{1, \dots, J\},
\end{equation}
containing all first moments and second moments (including cross-moments), giving a vector of length  $J+J(J+1)/2=65$. We minimize the ``high-resolution'' objective function
\begin{equation}\label{e:obj_o2}
J_s(\theta_2, \theta_3, \theta_4)=\frac12\| \langle \vec{g}(Y)  \rangle_{T'} - \langle \vec{g}(\tilde Y)
\rangle_{\infty} \|_{\Sigma'}^2,
\end{equation}
using a diagonal noise covariance matrix $\Sigma'$ with diagonal elements proportional to the variances of the statistics in $\vec{g}$, with a noise level $r'$  analogous to the noise covariance matrix \eqref{e:noise}. The variances of the statistics are estimated from a long control integration of the fast dynamics with fixed $X_1 = 2.556$. Because the fast variables $Y$ evolve more rapidly than the slow variables $X$, we accumulate statistics over only $T'=20~\mathrm{days}$.

Bayesian inversion with RWM, with the same priors and algorithmic settings as before and with noise level $r'=0.5$, again gives marginal posterior PDFs with modes close to the truth (Figure~\ref{fig:lorenz96_mcmc}, third row). The posterior PDFs exhibit similar multi-modality and reflect similar uncertainties and biases of posterior modes as those obtained from the full dynamics, especially with respect to the relatively large uncertainties in $c$ (cf.\ Figure~\ref{fig:lorenz96_mcmc}, second row).

These examples illustrate the potential of learning about parameters from observations and from local high-resolution simulations under selected conditions (here, for just one value of the slow variable $X_1$). An important question for future investigations is to what extent such results generalize to imperfect parameterization schemes, whose dynamics is usually not identical to the data-generating dynamics, so that structural in addition to parametric uncertainties arise. This issue can be studied for the Lorenz-96 system, for example, by using approximate models as parameterizations of the fast dynamics \citep[e.g.,][]{Fatkullin04a,Wilks05a,Crommelin08a}.

\section{Outlook}\label{s:discussion}

Just as weather forecasts have made great strides over the past decades thanks to improvements in the assimilation of observations \citep{Bauer15a}, climate projections can advance similarly by harnessing observations and modern computational capabilities more systematically. New methods from data assimilation, inverse problems, and machine learning make it possible to integrate observations and targeted high-resolution simulations in an ESM that learns from both and uses both to quantify uncertainties. As an objective of such parameter learning we propose the reduction of biases and exploitation of emergent constraints through the matching of mean values and covariance components between ESMs, observations, and targeted high-resolution simulations.

Coordinated space-based observations of crucial processes in the climate system are now available. For example, more than a decade's worth of coordinated observations of clouds, precipitation, temperature, and humidity with global coverage is available; parameterizations of clouds, convection, and turbulence can learn from them. Or, simultaneous measurements of ${\rm {CO}_{2}}$\ concentrations and photosynthesis are becoming available; parameterizations of terrestrial ecosystems can learn from them. So far, such observations have been primarily used to evaluate models and identify their deficiencies. Their potential to improve models has not yet been harnessed. Additionally, it is feasible to conduct faithful local high-resolution simulations of processes such as the dynamics of clouds or sea ice, which are in principle computable but are too costly to compute globally. Parameterizations can also learn from such high-resolution simulations, either online by nesting them in an ESM or offline by creating libraries of high-resolution simulations representing different regions and climates to learn from. Such a systematic approach to learning parameterizations from data allows the quantification of uncertainties in parameterizations, which in turn can be used to produce ensembles of climate simulations to quantify the uncertainty in predictions.

The machine learning of parameterizations in our view should be informed by the governing equations of subgrid-scale processes whenever they are known. The governing equations can be systematically coarse-grained, for example, by modeling the joint PDF of the relevant variables as a mixture of Gaussian kernels and generating moment equations for the modeled PDF from the governing equations \citep[cf.][]{Lappen01a,Golaz02a,Guo15a,Firl15a}. The closure parameters that necessarily arise in any such coarse-graining of nonlinear governing equations can then be learned from a broad range of observations and high-resolution simulations, as parametric or nonparametric functions of ESM state variables \citep[cf.][]{Parish16a}. The fineness of the coarse-graining (measured by the number of Gaussian kernels in the above example) can adapt to the information available to learn closure parameters. Such equation-informed machine learning will provide a more versatile means of modeling subgrid-scale processes than the traditional approach of fixing closure parameters ad hoc or on the basis of a small sample of observations or high-resolution simulations. Because parameterizations learned within the structure of the known governing equations respect the relevant symmetries and conservation laws to within the closure approximations, they likely have greater out-of-sample predictive power than unstructured parameterization schemes, such as neural networks that are fit to subgrid-scale processes without explicit regard for symmetries  and conservation laws \citep[e.g.,][]{Krasnopolsky13a}. Out-of-sample predictive power will be crucial if high-resolution simulations performed in selected locations and under selected conditions are to provide information globally and in changed climates. However, for non-computable processes whose governing equations are unknown, like many ecological or biogeochemical processes, more empirical, data-driven parameterization approaches may well be called for.

An ESM that is designed from the outset to learn systematically from observations and high-resolution simulations represents an opportunity to achieve a leap in fidelity of parameterization schemes and thus of climate projections. Such an ESM can be expected to have attendant benefits for weather forecasting, because weather forecasting models and the atmospheric component of ESMs are essentially the same. However, challenges lie along the path toward realization of such an ESM:
\begin{itemize}
\item We need innovation in learning algorithms. Our relatively simple example showed that parameters in a perfect-model setting can be learned effectively and efficiently by ensemble Kalman inversion. It remains to investigate questions such as the optimal ensemble size in Kalman inversions, how to adapt inversion algorithms to imperfect models, and how to quantify uncertainties. To increase computational efficiency, online filtering algorithms need to be developed that update parameters on the fly as Earth system statistics are being accumulated.
\item We need investigations of the best metrics to use when  learning parameterization schemes from observations or high-resolution simulations. For example, are least-squares objective functions the best ones to use? Which covariance components or other statistics should be included in the objective functions? There are  trade-offs between the number of covariance components that can be estimated from data and the information they can provide about parameterization schemes.
\item We need innovation in how learning from observations should interact with learning from targeted high-resolution simulations. How should high-resolution simulations be targeted? Where is the optimum trade-off between the added computational cost of conducting high-resolution simulations and the marginal information about parameterization schemes they provide?
\item We need innovation in parameterization schemes themselves, to design them such that they can learn effectively from diverse data sources and can be systematically refined when more information becomes available. It will be important to develop parameterizations that treat subgrid-scale motions (e.g., boundary layer turbulence, shallow convection, deep convection) in a unified manner, to eliminate artificial spectral gaps that do not exist in nature and to reduce the number of correlated parameters in the schemes \citep[e.g.,][]{Lappen01a, Lappen01b,Kohler11a,Suselj13a,Park14a,Park14b,Guo15a}. Novel approaches that exploit ideas ranging from stochastic parameterization to systematic coarse-graining likely have roles to play here \citep[e.g.,][]{Majda03a,Majda08a,Klein06a,Palmer05a,Palmer10a,Majda12a,Wouters13a,Lucarini14a,Wouters16b,Berner17a}. Furthermore, as the resolution of ESMs increases, it will also be necessary to revisit the common practice of modeling subgrid-scale dynamics in grid columns, because the lateral exchange of subgrid-scale information across grid columns will play increasingly important roles.
\end{itemize}

The time is right to seize the opportunities that the available global observations and our computational resources present. Fundamentally re-engineering atmospheric parameterization schemes, such as cloud and boundary layer parameterizations, will become a necessity as atmosphere models, within the next decade, reach horizontal grid spacings of 1--10~km and begin to resolve deep convection \citep{Schneider17a}. At such resolutions, common assumptions made in existing parameterization schemes, such as that clouds and the planetary boundary layer adjust instantaneously to changes in resolved-scale dynamics, will become untenable. Additionally, advances in high-performance computing (e.g., many-core computational architectures based on graphical processing units) will soon require a re-design of the software infrastructure of ESMs \citep{Bretherton12a,Schulthess15a,Schalkwijk15a}. So it is timely now to re-engineer ESMs and parameterization schemes, and design them from the outset so that they can learn systematically from observations and targeted high-resolution simulations.

\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{schematic-small.pdf}
\caption{Schematic of Earth system modeling framework that integrates global observing systems (OS) and targeted high-resolution simulations. Not only can parameterizations in ESMs and high-resolution simulations learn from observations. The same framework and the observing system simulators that it needs to encompass can also be used for observing system simulation experiments to assess the value and benefits of new observing systems. (Image credit for satellite: NASA.)}
\label{f:schematic}
\end{figure}
Integrating observations and targeted high-resolution simulations in an Earth system modeling framework would have multiple attendant benefits. Solving the inverse problems of learning about parameterizations from observations requires observing system simulators that map model state variables to observables (Figure~\ref{f:schematic}). The same observing system simulators, integrated in an Earth system modeling framework, can be used to answer questions about the value new observations would provide, for example, in terms of reduced uncertainties in ESMs. Addressing such questions in observing system simulation experiments (OSSEs) is increasingly required before the acquisition of new observing systems (e.g., as part of the U.S.\ Weather Research and Forecasting Innovation Act of 2017). They are naturally answered within the framework we propose.

\acknowledgments
We gratefully acknowledge financial support by Charles Trimble, by the Office Naval Research (grant N00014-17-1-2079), and by the President's and Director's Fund of Caltech and the Jet Propulsion Laboratory. We also thank V.\ Balaji, Michael Keller, Dan McCleese, and John Worden for helpful discussions and comments on drafts, and Momme Hell for preparing Figure~\ref{f:schematic}. The program code used in this paper is available at climate-dynamics.org/publications/. Part of this research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration.

redpajama_set_name: RedPajamaArXiv


Every year Connor Construction send hampers full of treats to their clients to say thank you for their support throughout the year. This year however, in a collaborative project, they decided to do something a bit more unique and personal, and send their clients some local ale with their own branded label on the front. The company came up with a name for the gift- 'The Tarmacers Tipple'.
They believed that this gift would not only show off some of the great local produce that Frome has to offer, but also reveal the talents of local Graphic Design students who were to take part in their label design competition. The Ale was supplied by Milk St Brewery which is a family-run craft brewery based in Frome. Using the finest local ingredients, they have been brewing unique and multi award-winning beers since 1999.
Connor Construction said: "The reason that we approached Frome College to design our labels, was to give aspiring designers the opportunity to add to their portfolios and gain some work experience. For the competition winner, we promised a £200 cash prize and a small write up on the label so that our clients would know who the creative genius was.
The Creative Genius, Sol Smith is a Year 13 student studying Graphic Design at Frome College. He designed the winning label in his spare time, demonstrating his commitment to the competition. We loved the way Sol put our logo in the hot air balloons and used a vibrant blue to match our branding. Sol explained that the countryside background he used is actually of Longleat. This makes it all the more special to include local references to Frome. Sol only has a few months left to study and would like to go on to complete an Art Foundation, hopefully gaining work in Graphic Design when he finishes. The reason Sol would like to pursue a career in this field is because he is very creative and is naturally very good at this particular subject. We can imagine Sol doing very well with his career ambitions thanks to his clever creativity and professionalism. We would highly recommend Sol Smith to any of our clients looking for a Graphic Designer and wishing to give someone starting out their career a chance.
We could not believe the talent at Frome College! And because of the fantastic design entries that we received, Ian Webb, Managing Director of Connor Construction, decided that the entries were of such a high standard, they should all receive a prize.
In joint 2nd Place - Winners of £50 each - Tiffany Lee & Millie Harrington.
We are really proud of how this local project turned out, not just because of the fantastic outcome and feedback from clients, but in how well we all did in working together. Currently, we have sent 100 kegs of local ale with Sol's design on the front to clients all over the UK. A big thank you to the students who took part, Doug Knight and his colleagues at Frome College for supporting the project, Frome Brewing Company for their help and delicious ale and Werner Van Wyke at Multi Packaging Solutions for his help and printing such great quality labels. We are excited to say that Frome College have invited us to do a design project with them once a year. What will we do next? We have a few ideas up our sleeves- and now we know all the right people to help us!


It's that time again...don't the years go by so much quicker as we get older! I hope you have all had an amazing 2011.
Thank you for all of your support this year, every comment, email and follower makes me smile with pure delight so I can't wait to come back in 2012 with more energy and enthusiasm and share all the things I am loving...please stick around and let me know if I'm inspiring you in your homes and wardrobes!
I hope you are surrounding yourself with love and maybe a little champagne tonight for the countdown...and your night sparkles!
This Christmas why not give your boyf, brother or bestie an uber stylish gift. The boys in our lives can be hard to shop for but try and look beyond vouchers and the like.
Think pocket squares, sunnies or a tres chic iPhone cover. Like Net A Porter which I curated the girls list from last week, Mr. Porter has a incredible range of gorgeous things to keep everyone looking sharp.


text:
Baseball, Men's Tennis, Sports, Women's Tennis
May 7, 12:52 am
Around the League: May 7
Men's tennis Gabriel McCarthy, assistant Sports editor A number of teams across the Pac-12 advanced to the second weekend of competition in the NCAA tournament. No. Read more...
Photo: (Andy Bao/Daily Bruin)
Sports, Women's Tennis
Four women's tennis Bruins to represent UCLA in NCAA championship
It's win or go home for the Bruins, but four players will have the opportunity to compete once more. Freshman Elysia Bolton, redshirt junior Jada Hart and seniors Gabby Andrews and Ayan Broomfield were selected to participate in the NCAA Division I Women's Tennis Championship in Orlando, Florida. Read more...
Photo: The Bruins' No. 5 duo – seniors Gabby Andrews and Ayan Broomfield – was selected to compete in doubles play at the NCAA Division I Women's Tennis Championship in Orlando, Florida beginning May 20 after UCLA women's tennis competes its NCAA run as a team. (Tanmay Shankar/Daily Bruin)
May 5, 7:41 pm
Women's tennis' sweep of LSU showcases high energy, sets mood for next round
The Bruins notched two sweeps to advance to the third round of the NCAA tournament. No. 7 seed UCLA women's tennis (20-7, 8-2 Pac-12) recorded a 4-0 sweep over LSU (17-12, 5-8 SEC) at the Los Angeles Tennis Center on Sunday. Read more...
Photo: Redshirt junior Jada Hart defeated LSU's Eden Richardson 6-4, 6-3 in No. 1 singles. Hart joined freshman Elysia Bolton in doubles and claimed a doubles win over LSU's Paris Corley and Anna Loughlan. (Andy Bao/Daily Bruin)
Women's tennis starts NCAA tournament with dominating sweep of Northern Arizona
The Bruins took all six first sets in singles en route to a sweep over the Lumberjacks. No. 7 UCLA women's tennis (19-7, 8-2 Pac-12) whitewashed Northern Arizona (15-9, 9-1 Big Sky) in the first round of the NCAA championships Saturday afternoon to set up a second round encounter against LSU (17-11, 5-8 SEC). Read more...
Photo: Senior Ayan Broomfield executed a drop shot to clinch the dual for UCLA women's tennis. The Bruin will host the LSU Tigers on Sunday afternoon in the second round of the NCAA championship. (Andy Bao/Daily Bruin)
May 3, 1:13 am
Women's tennis to host Northern Arizona in first round of NCAA championship
The Bruins will begin their bid for the NCAA title this weekend. No. 7 seed UCLA women's tennis (18-7, 8-2 Pac-12) will begin the NCAA championship by welcoming Northern Arizona (15-8, 9-1 Big Sky) to the Los Angeles Tennis Center on Saturday. Read more...
Photo: Redshirt junior Jada Hart has not dropped a doubles match since March 9 and will help lead UCLA women's tennis against Northern Arizona in the first round of the NCAA championship. (Tanmay Shankar/Daily Bruin)
April 30, 11:57 pm
Women's tennis doubles prove successful with both new and old pairings
The Bruins win twice as often when they pick up the doubles point. No. 9 UCLA women's tennis (18-7, 8-2 Pac-12) is .400 when it loses the doubles point, but .800 when it is the first team to put a point on the scoreboard. Read more...
Photo: Sophomore Abi Altick played alongside freshman Taylor Johnson at the Pac-12 championship and won all three of their matches to help UCLA women's tennis reach the finals before falling to Stanford. (Tess Horowitz/Daily Bruin)
April 28, 2:47 pm
Women's tennis succumbs to Stanford in Pac-12 championship despite doubles success
For the sixth straight time, the Bruins found the Cardinal too much to handle. "Today's match was a lot like last year's match," said coach Stella Sampras Webster. Read more...
Photo: Senior Ayan Broomfield injured her knee during her doubles match against Washington, but still played through her singles match to clinch a spot in the finals of the Pac-12 championship for UCLA women's tennis. The Bruins, however, fell to Stanford in the championship dual. (Tanmay Shankar/Daily Bruin)
« Prev 1 2 3 4 5 … 73 Next »

redpajama_set_name: RedPajamaCommonCrawl


Marvel vs. Capcom 3: Fate of Two Worlds is a crossover fighting game developed by Capcom. It features Capcom's own characters and characters from American comic book company Marvel Comics. It is the fifth installment of the Marvel vs. Capcom franchise, the eighth installment in the Vs. series, the first to be exclusive to consoles, and the second in the Vs. series to use three-dimensional graphics on a two-dimensional battle area. The game is produced and directed by Ryota Niitsuma, who had previously worked on Tatsunoko vs. Capcom: Ultimate All-Stars, and features the same fast-paced, over-the-top fighting mechanics as earlier games in the series, along with new methods of play designed to make the game more accessible to new players.
Marvel vs. Capcom 3 is a versus fighting game where up to two players compete in battle using characters with their own unique fighting styles and special attacks. The game features the same tag-based team feature as previous installments of the series, where each player chooses three characters that can be swapped at any point during a match, and utilizes the "Evolved Vs. Fighting System", a modified version of earlier systems seen in other Capcom Vs. titles. It is the first game in the franchise to feature three-dimensional character models as opposed to two-dimensional sprites, though gameplay remains restricted to two-dimensions, allowing characters to move only backward, forward, or straight up into the air.
The game uses a control scheme modeled after Tatsunoko vs. Capcom: Ultimate All-Stars, which Capcom believes will "knock down the wall of complicated controls and open up the field of strategic fighting to all comers" as well as a new "Exchange" button used to launch opponents into the air, switch characters while performing a combo, and slam the opponent into the ground when used by certain characters. Players can use each button to string together combination attacks, as well as perform special moves using a combination of button presses and joystick movement. As characters attack, their Special Gauge fills with energy which can be expended by the player to execute powerful Hyper Combos and Team Combos that involve multiple characters.


text:
Salesforce Tops Q4 Earnings Estimate, But Tepid Guidance Hits Shares After Hours
Salesforce.com posted stronger-than-expected fourth quarter earnings Monday and boosted its revenue guidance for the current financial year, but near-term earnings guidance was modestly weaker than analysts had forecast.
Martin Baccardax
Mar 4, 2019 4:36 PM EST
Salesforce.com Inc. (CRM) - Get Report posted stronger-than-expected fourth quarter earnings Monday and boosted its revenue guidance for the current financial year, but near-term earnings guidance was modestly weaker than analysts had forecast, extending declines for the stock in after-hours trading.
Salesforce said earnings for the three months ending in January came in at 70 per share, more than double the 28 cents recorded over the same period last year and well ahead of the Street consensus forecast of 46 cents. Group revenues, Salesforce said, rose 26% from last year to 3.6 billion and again topped the consensus estimate of $3.56 billion. Subscription and support revenues were also 26% higher from last year at $3.38 billion.
Salesforce is Jim Cramer's Real Money Stock of the Day
Real Money: Plenty of Salesforce Virtues, Here's 3 Trade Ideas
Real Money: Can Salesforce Stock Rally Further From Here?
Looking into 2019, Salesforce said it was raising its revenue forecast to a range of $15.95 billion to $16.05 billion and also sees first quarter revenue in the range of $3.67 billion to $3.68 billion, just shy of the $3.7 billion forecast. Non-GAAP earnings are forecast in the range of 60 to 61 cents per share for the first quarter, Salesforce said, a figure that was just shy of the 63 cent Street estimate, and between $2.74 and $2.76 for the full year, matching analysts' at the top end of the range.
"Our relentless focus on delivering innovation and customer success has fueled our growth and solidified our leadership in the enterprise," said co CEO Keith Block. "This is just the beginning, which is why we're now targeting $26 to $28 billion in revenue by FY23 - organically doubling our revenue again in the next four years."
Salesforce shares were marked 3.3% lower in after-hours trading following the earnings release, after declining 3.66% during the main trading session, indicating an opening bell price on Tuesday of $153.31 each, a move that would trim the stock's three-month decline to around 10.7%.
EarningsInvesting


text:
Q: Take only one value from variadic I want to ask through this simple example: how could I take the last value from a variadic pack and print(1, 2, 3.14) to call print(3,14)?
void print()
{
   cout<<"--empty--";
}

void print(x)
{
    std::cout<<"Last Value from variadic pac--" << x;
}

template <typename T, typename... Types>
void print(T var1, Types... var2)
{
   int x = // last value from variadic pac : 3.14
   print(x) // 3.14
}

int main()
{
    print(1, 2, 3.14);

    return 0;
}

A: With std::tuple, you might do
template <typename T, typename... Types>
void print(T var1, Types... var2)
{
   auto x = std::get<sizeof...(Types) - 1>(std::tie(var2...));
   print(x); // 3.14
}

Demo

A: Though you tagged your question up to c++14, I thought it worth noting that a c++17 solution is much simpler using a fold expression and constexpr if:
In its entirety:
template <typename... Types>
void print(Types... args)
{
    if constexpr (sizeof...(Types) == 0)
        std::cout << "Empty";
    else
        std::cout << (args, ...);
}

The fold expands to using the comma operator, which evaluates each argument and discards the result until the last one is reached.  The last item's evaluation becomes the value of the entire expression, which is printed.

redpajama_set_name: RedPajamaStackExchange


text:
using System.Net;
using System.Net.Http;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Newtonsoft.Json;

namespace ShtikLive.Clients
{
    public static class HttpClientExtensions
    {
        private const string ApplicationJson = "application/json";

        public static Task<HttpResponseMessage> PostJsonAsync(this HttpClient client, string uri, object obj, CancellationToken ct = default(CancellationToken))
        {
            var json = JsonConvert.SerializeObject(obj);
            var content = new StringContent(json, Encoding.UTF8, ApplicationJson);
            return client.PostAsync(uri, content, ct);
        }

        public static Task<HttpResponseMessage> PutJsonAsync(this HttpClient client, string uri, object obj, CancellationToken ct = default(CancellationToken))
        {
            var json = JsonConvert.SerializeObject(obj);
            var content = new StringContent(json, Encoding.UTF8, ApplicationJson);
            return client.PutAsync(uri, content, ct);
        }

        public static ValueTask<T> Deserialize<T>(this HttpResponseMessage response)
        {
            if (!response.IsSuccessStatusCode)
            {
                if (response.StatusCode == HttpStatusCode.NotFound)
                {
                    return new ValueTask<T>(default(T));
                }
                throw new UpstreamServiceException(response.StatusCode, response.ReasonPhrase);
            }

            async Task<T> DeserializeAsync()
            {
                var content = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
                return JsonConvert.DeserializeObject<T>(content);
            }

            return new ValueTask<T>(DeserializeAsync());
        }
    }
}

redpajama_set_name: RedPajamaGithub


text:
Global Bushcraft Symposium
GBS Interviews
Get The Videos
John Rhyder
John is a naturalist, woodsman, wildlife tracker and writer. He is both founder and Head Instructor at Woodcraft School. John is one of the most experienced authorities in the field of bushcraft in the UK, with an impressive pedigree in this instructing since 1997. He is the originator of the first accredited bushcraft programme in the UK and has since gained accreditation for related programmes too.
He has gained Accredited Practitioner through the Institute of Outdoor Learning (APIOL). John also designed and wrote the training and assessment criteria for the Institute of Outdoor Learning's Foundational Bushcraft Competency certificate. Additionally, he has written a Bushcraft Award for a leading youth organisation, the Woodcraft Folk.
John previously worked and taught in arboriculture, conservation, and habitat management. He has been teaching since 1994, making this a full-time career since then.
John is certified through CyberTracker conservation as Senior Tracker. As such, he is the first, and currently the only, person at this level in Northern Europe. John is also working as an evaluator assessing the CyberTracker system in this region.
As a writer John has contributed many articles for various publications and has authored three books. John's latest book, Track and Sign, was published by The History Press in March 2021. The same publisher has also re-published John's first book, Woodcraft, which was originally self-published in 2018. Additionally, he has produced a field guide containing life sized drawings of Britain's mammals and birds.
John has been lucky enough to travel widely and has studied and led bushcraft, tracking and wildlife tours in Romania, Germany, Norway, Sweden, France, Iberia, Africa, the USA and Poland.
See other GBS speaker profiles
GBS 2022 Main Sponsors
Follow The Global Bushcraft Symposium On Social Media
Global Bushcraft Symposium UK Ltd is a company incorporated in England and Wales with company number 12283200 and with its registered office at 20-22 Wenlock Road London, N1 7GU England.
Ticket Terms and Conditions
© 2023 Global Bushcraft Symposium


text:
The Truth About The Royal Family's Crowns And Tiaras
By Bernadette Roe/March 29, 2020 5:49 pm EST
The royal family occasionally does things that convince us they're just like the rest of us. For example, Queen Elizabeth will sometimes drive herself around with a corgi chilling in the back seat. Or Kate Middleton will be a proud Instagram parent, sharing an adorable video of her kids clapping in gratitude to all the doctors, nurses, and healthcare staff working around the clock right now.
But while the royals do casual things, we're not exactly fooled. All the corgis and personally driven cars in the world can't hide the fact that there are some amazing perks that come with being part of the royal family, like owning all of the dolphins in the U.K. (Yup, Queen Elizabeth really does own sea creatures.)
Another such hint of their glamorous lives comes from frequent sightings of the many crowns and tiaras worn by the Queen and her extended family. And like anything within the British monarchy, there are some intense rules regarding who can wear what and when.
Only some members of the royal family wear tiaras
There's a whole etiquette around tiaras, as Insider reported. Only married women can wear them or women who were born princesses. "Traditionally, tiaras are a sign of marriage. So typically they could be worn by a bride on her wedding day, or after she is married, she can wear one to any white tie event," Grant Harrold, director at the Royal School of Etiquette, told Insider. "Single ladies don't typically wear tiaras, unless they are born into the Royal Family as a princess."
For her wedding day, Kate Middleton wore the Cartier Halo Tiara, according to Vogue. Her sister-in-law, Meghan Markle, wore the Queen Mary Diamond Bandeau tiara, a 1932 tiara made in the Art Deco design, per Harper's Bazaar.
Tiaras traditionally symbolize a loss of innocence. According to a Town & Country interview with London jewelry expert Geoffrey Munn, "The tiara has its roots in classical antiquity and was seen as an emblem of the loss of innocence to the crowning of love." Hence the reason no one saw Middleton or Markle in a tiara until their wedding days. But that's not the only noteworthy detail about the fancy headpieces.
Tiaras can only be worn at certain times of the day
Even now, Kate Middleton — who's happily married to Prince William and the mother of the future king of England — can't walk out of Kensington Palace at 10 a.m. wearing a tiara. (Can't a woman live?!)
That's because there are specific times and events for tiaras. "Tradition dictates that tiaras are not worn before 6 p.m., though weddings are an exception. They are worn on formal white tie events and state occasions such as state banquets," royal expert Richard Fitzwilliams told Insider. "The decision on whether to wear a tiara is linked to the dress code of a particular event and also to personal preference."
Interestingly, tiaras, unlike crowns, are not emblematic of "inherited rank," according to Town & Country. It's actually the grandiosity of the event, hence "white tie," that allows for the wearing of tiaras. In fact, invitations would commonly indicate the importance of the event by stipulating: "Tiaras will be worn."
Here's another fact: Royals were never allowed to wear tiaras at hotels, according to Town & Country. That's not the only restriction either. For modern women, hair products can be a hazard to the delicate jewelry: "Hairspray immediately deadens the color and fire of precious stones," London jewelry expert Geoffrey Munn told the magazine. Some stylists will use a piece of cardboard to shield the tiara before applying hairspray, but having a lot of hair certainly helps hold the object in place.
Crowns come with their own rules, too
So, what's the difference between a crown and a tiara? For starters, tiaras are only worn by women, while crowns are worn by both men and women, as Town & Country pointed out. While a tiara functions basically as glamorous jewelry, a crown symbolizes royalty. A crown — according to what expert Geoffrey Munn told Town & Country — "is not jewelry but regalia which has some emblematic function associated with sovereignty and nobility."
Another fun fact: A small crown is called a coronet. According to Insider, the queen wore coronets while her father, King George VI, was still king. Usually, the only time men wear coronets is at a coronation, except in very rare circumstances. For instance, the last time Prince Charles wore a coronet was at his investiture in 1969.
According to Historic Royal Palaces, this coronet is on display in the Jewel House at The Tower of London for the first time. In fact, many of the Crown Jewels are on display at the Tower under the security of armed guards. Other parts of the queen's collection are housed at The Royal Collection Trust — though this technically encompasses all of the royal palaces, residencies, and art galleries, so the collections aren't housed in just one place, but rather, spread all over the United Kingdom.
If you ever do get a chance to check out the royal family's jewels, now you'll know more of the backstory behind them.


Keedie will be joining Lucy in the "Triopera."
I believe she's been in the TriOpera for a few weeks already. I've seen the pictures.
They have been rehearsing for a few weeks. The show doesn't open until May 23rd.
That's awesome! You'll have to let us know how it is!