text:
\section{Introduction}

\begin{table*}[t]
\setlength{\tabcolsep}{1em}
{\renewcommand{\arraystretch}{1.5
\begin{tabular}{|ll|c|ll|}
\hline
{\bf Thermodynamics} 	&    &   & {\bf Statistics}  &  \\
{\bf Quantity:}        &  {\bf Interpretation:} & & {\bf Quantity:}     &  {\bf Interpretation:} \\
\hline
\hline
$\beta = { T}^{-1}$ & Inverse temperature & $\leftrightarrow$ &

			$N$ &  Sample size \\
			${\bm \theta}$ & State variables/vector  & $\leftrightarrow$& ${\bm \theta}$ & Model parameters \\
			$X^N$ & Quenched disorder  & $\leftrightarrow$ & $X^N$ & Observations \\
			${ E}_X({\bm \theta})$ & State energy  & $\leftrightarrow$ & $\hat{H}_X({\bm \theta})$ & Cross entropy estiamtor \\
			${ E}_0$ & Disorder-averaged ground state energy  & $\leftrightarrow$ & $H_0$ & Shannon entropy \\
		    $\rho({\bm \theta})$ & Density of states  & $\leftrightarrow$ & $\varpi({\bm \theta})$ & Prior  \\
 ${Z}$  & Partition function  & $\leftrightarrow$    & $Z$    & Evidence  \\
 		    ${Z}^{-1} \rho\ \exp -\beta { E}_X$ & Normalized Boltzmann weight  & $\leftrightarrow$ & $\varpi({\bm \theta}|X^N)$ & Posterior  \\
		    \hline
${ F} = -\beta^{-1}\log {Z}$	& Free energy & $\leftrightarrow$& $F = -N^{-1} \log Z$
        	& Minus-log-evidence  \\
$ { U} =   \partial_\beta \beta { F}$ & Average energy  & $\leftrightarrow$&   $U = \partial_N N F$ 	& Minus-log-prediction  \\
 ${ C} = -\beta^2 \partial^2_\beta \beta { F}$ & Heat capacity & $\leftrightarrow$ & ${ C} = -N^2 \partial^2_N N {F}$ 	& \textit{Learning capacity} \\

	 ${ S} = \beta^2\partial_\beta F$ & Gibbs entropy & $\leftrightarrow$      	& ${S} = N^2\partial_N F$ & \textit{Statistical Gibbs entropy} \\
            \hline
\end{tabular}
\caption{{\bf Thermodynamic-Bayesian correspondence.} The top half of the table lists the correspondences that can be determined directly from the definition of the marginal likelihood as the partition function. The lower half of the table lists the implied thermodynamic expressions and their existing or proposed statistical interpretation. \label{Table:stat}}}
\end{table*}

Despite  significant advances in learning algorithms, fundamental questions remain about the mechanisms of learning and the relationships between learning algorithms.
How does model complexity affect learning?
Why do some models have anomalously good learning performance?
In this paper, we explore the phenomenology of learning by exploiting a correspondence between Bayesian inference and statistical mechanics.
This correspondence has been previously described by Jaynes, Balasubramanian, and many others \cite{Jaynes1957information,Jaynes2003,barndorff1997statistics,Balasubramanian1997,nishimori2001statistical,mezard2009information} and many methods from statistical physics have been adapted to statistics \cite{doucet2000sequential,hoffman2014no,chen2014stochastic,minka2001expectation,xing2002generalized,minka2005divergence,niinimaki2013annealed,liang2001real,inoue2001application,mezard2006reconstruction,zdeborova2016statistical}. We extend this correspondence by using the canonical bridge between statistical mechanics and thermodynamics to compute the standard thermodynamic potentials and properties of a learning system.
The correspondence  identifies two novel statistical quantities, a learning capacity and the Gibbs entropy.
These quantities generate  new insights into the mechanism of learning and new learning algorithms.

The analysis of a novel \textit{learning capacity} (corresponding to the heat capacity) reveals an interesting connection between the Akaike Information Criterion (AIC) of information-based inference and the equipartition theorem in statistical mechanics \cite{pathria1996statistical}.
In addition, the learning capacity also provides new insights into the  mechanism of learning. It has long been known that some high-dimensional models learn anomalously well. These models have been termed \textit{sloppy}  \cite{Machta:2013hl}. We demonstrate that the learning capacity both provides a natural definition for the sloppiness phenomenon as well as providing a mechanism: a statistical analogue of the well known freeze-out mechanism of statistical mechanics.
We hypothesize that this mechanism is responsible for the anomalously high predictive performance of many high-dimensional models.

We also propose that the Gibbs entropy provides a natural device for determining model multiplicity, \textit{i.e.}~counting indistinguishable distributions in the context of statistical inference.
This interpretation allows us to define a \textit{generalized principle of indifference} (GPI) for selecting a prior in absence of \textit{a priori
} information about parameters or models. The GPI unifies a number of known, but seemingly unconnected objective Bayesian methods, while also providing an algorithm applicable to small sample size and singular models where existing approaches fail. The GPI also resolves a number of troubling anomalies in objective Bayesian analysis, providing a natural resolution to the Lindley-Bartlett paradox in which larger models are automatically rejected.

The paper is organized as follows: In Sec.~\ref{sec:corr}, we define the correspondence. In Sec.~\ref{sec:res}, we compute the thermodynamics potentials and properties of inference in the analytically tractable large-sample-size limit and use these results to deduce the statistical meaning of each quantity. In Sec.~\ref{sec:app}, we compute the learning capacity and GPI prior for a number of example analyses to demonstrate that the results in Sec.~\ref{sec:res} generalize beyond the normal model.

\section{Defining the correspondence}
\label{sec:corr}

We assume that a true parameter value ${\bm \theta}_0$ is drawn from a known prior distribution $\varpi({\bm \theta})$. We observe $N$ samples $x^N\equiv\{x_1,...,x_N\}$ which are distributed like  $q(x|{\bm \theta}_0)$:
\begin{equation}
X_i\sim q(\cdot|{\bm \theta}_0),
\end{equation}
where we use capital $X$ to denote random variables and the symbol $\sim$ to denote \textit{distributed like}.
For simplicity, we will assume that the observations are independent and identically distributed, but the approach can be generalized.

The correspondence between statistical physics and Bayesian inference is clearest when expressions are written in terms of the empirical estimator of the cross-entropy (Eqn.~\ref{eqn:Ce}):
\begin{eqnarray}
\hat{H}({\bm \theta}) &\equiv& -\left<\log q(X|{\bm \theta})\right>_{X\in x^N},
\end{eqnarray}
where the angle brackets represent the empirical expectation (Eqn.~\ref{eqn:emp}).
The  marginal likelihood (\textit{i.e.}~evidence) can be written \cite{Balasubramanian1997}:
\begin{align}
Z(x^N)&\equiv \int_{\bm \Theta}\!\! {\rm d}{\bm \theta}\ \varpi({\bm \theta})\,e^{-N\hat{H}},\label{eqn:evidence}
\end{align}
which can  be directly compared to the partition function in the canonical ensemble \cite{Jaynes1957information,Jaynes2003,barndorff1997statistics,Balasubramanian1997,nishimori2001statistical,mezard2009information}. The model parameters ${\bm \theta}$ are  the variables that define the physical state vector, the cross entropy $\inliner{\hat{H}({\bm \theta})}$ is the energy $ E({\bm \theta})$, the prior $\varpi({\bm \theta})$ is  the density of states $\rho({\bm \theta})$
The data $x^N$ is  quenched disorder in the physical system.
The sample size $N$ is identified with the inverse temperature $\beta\equiv { T}^{-1}$. (Choosing $\beta \leftrightarrow N$ is only one of at least two proposals for the identification of the temperature. See App.~\ref{App:alttemp}.) This assignment is natural in the following sense: At small-sample-size $N$, many parameter values are consistent with the data, in analogy with the large range of states $\bm \theta$ occupied at high temperature ${ T}$ in the canonical ensemble. The analogy between different quantities is summarized in Tab.~\ref{Table:stat}.

\subsection{Application of thermodynamic identities}
To extend the previously proposed correspondence, we follow the standard prescriptions from statistical mechanics to compute thermodynamic potentials, properties, and variables for the system  \cite{gibbs1902elementary,pathria1996statistical}. These are shown in the lower half of Tab.~\ref{Table:stat}.
The thermodynamic quantities depend on the particular realization of the data $X^N$.
In the current context we are interested in the expectation over this \textit{quenched disorder} (\textit{i.e.}~data).
We define the disorder average with an overbar:
\begin{align}
\overline{f}(N) \equiv \left<\right. f(X^N,{\bm \theta}_0) \left.\right>_{X,{\bm \theta}_0},
\end{align}
where $X\sim q(\cdot|{\bm \theta}_0)$ and ${\bm \theta}_0\sim \varpi$.

\begin{table*
\setlength{\tabcolsep}{1em}
{\renewcommand{\arraystretch}{1.5
\begin{tabular}{|c|l|l|l|}
\hline
\textbf{Model} & \textbf{K-D-Free-particle} & \textbf{K-D-Normal-prior} & \textbf{K-D-singular}  \\
\hline
$\overline{F}$  & ${ E}_0 + \textstyle \frac{K}{2\beta } \log \frac{\beta}{\beta}_0$   & $H_0 + {\textstyle \frac{K}{2N}} \log{\textstyle \frac{N}{N_0}}$ & $H_0 + \textstyle \frac{\gamma}{2N} \log N+...$  \\
$\overline{U}$   & ${ E}_0 + \textstyle \frac{K}{2\beta }$   & $H_0 + \textstyle \frac{K}{2N }$ &  $H_0+\textstyle \frac{\gamma}{2N}+...$  \\
$\overline{C}$  &  $\textstyle \frac{K}{2 }$  &   $\textstyle \frac{K}{2 }$  & $\textstyle \frac{\gamma}{2}+...$  \\
$\overline{S}$  &  ${\textstyle \frac{K}{2}}\left(1-\log \frac{\beta}{\beta_0}\right)$  & ${\textstyle \frac{K}{2}}\left(1-\log \frac{N}{N_0}\right)$   & $-{\textstyle \frac{\gamma}{2}}\log N+...$  \\
\hline
\end{tabular}
\caption{{\bf Thermodynamic-Bayesian correspondence.} The thermodynamic quantities of a $K$-dimensional free particle with ground-state energy $E_0$ and thermal de-Broglie inverse temperature $\beta_0$, defined in the App.~\ref{App:freeparticle} are compared to a $K$-dimensional normal model with a conjugate prior.
Inspection reveals that the free particle is exactly equivalent to the normal model, identifying the parameters as described in Tab.~\ref{Table:stat}.
For the singular model, we supply only the leading order contributions in the large $N$ limit.
The learning coefficient $\gamma \le K$. The special case of  $\gamma = K$ is a regular model. \label{tab:results}}}
\end{table*}

\section{Results}
\label{sec:res}

We motivate interpretations of the thermodynamic quantities by developing the thermodynamic potentials in the normal model and the large-sample-size limit of singular models.
The similarity between these large-sample-size results and familiar results in statistical mechanics show that interpretations of the thermodynamic quantities of a statistical model can be deduced from the meaning of their physical counterparts.

\subsection{Models}

\subsubsection{Free particle and the normal model}

We compare a free particle in $K$-dimensions confined to a $K$-cube in statistical mechanics to a $K$-dimensional normal model with $K$ unknown means $\vec{\mu}$ and known variance $\sigma^2$ in Bayesian inference.
The true parameter $\vec{\mu}_0$ is drawn from a $K$-dimensional normal distribution  (the prior $\varpi$) with mean $\vec{\mu}_\varpi$ and variance $\sigma_\varpi^2$.
It will be convenient to define a critical sample size:
\begin{equation}
N_0 \equiv \sigma^2/\sigma^2_\varpi, \label{eqn:critss}
\end{equation}
 where the information content of the observations $x^N$ is equal to the information content of the prior. (See Appendix Sec.~\ref{app:norm}.)

 In the current context, we will be interested in the uninformative limit:  $N_0\rightarrow 0$.

\subsubsection{Singular models}

The normal model is representative of the large sample-size-limit of a regular Bayesian model of dimension $K$ (i.e. the Bernstein-von Mises theorem \cite{bernstein1927theory,lecam1953some,ghosh2012stability}).  For generality, we also study the large-sample-size limit of a singular model of dimension $K$.  Models are \textit{singular} when parameters are \textit{structurally unidentifiable} \cite{watanabe2009}:
\begin{equation}
q(x|{\bm \theta}_1)=q(x|{\bm \theta}_2)\ \ \ \ \ {\rm for}\ \ \ \ \   {\bm \theta}_1\ne {\bm \theta}_2, \label{eqn:unident}
\end{equation}
where the  unidentifiability cannot be removed by coordinate transformation. A regular model is the special case where all parameters are identifiable, the parameter manifold is continuous, and the Fisher information matrix (defined in Eqn.~ \ref{fisherMat}) is positive definite. Using exact asymptotic results for singular models \cite{watanabe2009}, the thermodynamic quantities for each model and limit are shown in Tab.~\ref{tab:results}.

\subsection{Thermodynamic potentials}

\subsubsection{Free energy} A relation between the partition function and Bayesian evidence has long been discussed \cite{Jaynes1957information,Jaynes2003,barndorff1997statistics,Balasubramanian1997,nishimori2001statistical,mezard2009information}. The free energy $F$ represents the Bayesian model preference: the minus-log-evidence per observation, or message length per observation. In Tab~\ref{tab:results} it is seen that $F$ breaks up into two parts. The first term is the code length per observation using the optimal encoding, $H_0$, and the second term is the length of the code required to encode the model parameters using the prior (per observation) \cite{Rissanen1978,Grunwald2007}. The model that maximizes the evidence and therefore minimizes $F$ is selected in the canonical approach to Bayesian model selection.

\subsubsection{Average energy}
The thermodynamic prescription for computing the average energy involves a derivative with respect to temperature (Tab.~\ref{Table:stat}). In the context of statistics, we will formally interpret this derivative using a finite difference definition, such that
\begin{equation}
U(x^N) \equiv -\left<\log q(x_i|x^{\ne i})\right>_{i=1..N},\label{eqn:loocv}
\end{equation}
where  $\inliner{q(x_i|x^{\ne i})\equiv Z(x^N)/Z(x^{\ne i})}$ is the Bayes-predictive distribution. The RHS is a well-known statistical object: the Leave-One-Out-Cross-Validation (LOOCV) estimator of model performance (See App.~\ref{App:crossval}).
The statistical interpretation of average energy $U$ is therefore the minus-expected-predictive-performance of the model (\textit{e.g.}~\cite{hastie01statisticallearning}).

As shown in the examples in Tab.~\ref{tab:results}, the averaged energy can be written as the sum of two contributions: The first term  $H_0$ is the  performance of the model if the true parametrization was known and corresponds to a ground state energy. The second term represents the loss associated with predicting a new observation $X$ using estimated model parameters and corresponds to the thermal energy. The loss term follows the typical behavior predicted by the equipartition theorem: \textit{there is a half $k_BT$ of thermal energy per harmonic degree of freedom} \cite{pathria1996statistical}. This is an important universal property of regular models in the large-sample-size limit: Independent of the detailed structure of the model, there is a universal predictive loss $\inliner{\frac{1}{2N}}$ per degree of freedom in the model. This universal predictive loss can be interpreted as the mechanism by which the Akaike Information Criterion (AIC) estimates the predictive performance \cite{akaike1773,BurnhamBook}.

\subsubsection{Learning capacity}

To study the predictive loss, it is natural to study the statistical quantity corresponding to the heat capacity.
The heat capacity measures the rate of increase in thermal energy with temperature ($\overline{C}$ in Tab.~\ref{Table:stat}).
The statistical analogue of the heat capacity, a \textit{learning capacity}, is a measure of the rate of increase in predictive performance with sample size. For the normal model:
\begin{equation}
\overline{C} = \textstyle\frac{1}{2} K, \label{eqn:equipart}
\end{equation}
as implied by the equipartition theorem. (See Tab.~\ref{tab:results}.)

To consider how this analogy generalizes to a generic statistical model, we use the large-sample-size limit asymptotic expression for the Bayesian evidence from Ref.~\cite{watanabe2009} to compute the learning capacity. (See Tab.~\ref{tab:results}.)
Like the normal model, the learning capacity for a singular model has the equipartition form but with an effective dimension:
\begin{equation}
\overline{C} = \textstyle\frac{1}{2} K_{\rm eff}, \label{eqn:equipart}
\end{equation}
where $K_{\rm eff}=\gamma$ is the learning coefficient defined by Watanabe \cite{watanabe2009}.
A regular model is a special case of this expression where $K_{\rm eff} = K$, the dimension of the parameter manifold. We therefore conclude that equipartition theorem describes the universal properties of regular statistical models in the large-sample-size limit: The learning capacity is half the number of degrees of freedom

.

\subsubsection{The Gibbs entropy}
In physics, the Gibbs entropy generalizes the Boltzmann formula: $S = \log \Omega$ where $\Omega$ is the number of accessible states. We propose that the Gibbs entropy has the analogous meaning in the context of Bayesian statistics: The Gibbs entropy is the log-number of models consistent with the data. The finite-sample-size expression for the entropy is
\begin{equation}
S(x^N) \equiv N(U-F),\label{eqn:loocv2}
\end{equation}
where  Eqn.~\ref{eqn:loocv} provides an explicit expression for $U$.
The Gibbs entropy of a normal model is shown in Fig.~\ref{Fig:S}.
Above the critical sample size $N_0$, the data is informative to the parameter values and therefore the number of models consistent with the data is reduced.
As a result the Gibbs entropy becomes increasingly negative as sample size $N$ grows.

\subsection{The principle of indifference}
In statistical physics, the density of states is known (\textit{i.e.}~measured) but in inference the selection of a prior is often subjective. The construction of an objective or uninformative prior is a long-standing problem in Bayesian statistics.
What insight does the proposed correspondence provide for prior choice?

Prior construction since Bayes and Laplace has often attempted to apply a \textit{Principle of Indifference}: All \textit{mutually exclusive} and \textit{exhaustive} possibilities should be assigned equal prior probability \cite{Laplace1812,Keynes1921}. One interpretation of this prescription is that it maximizes entropy  \cite{Jaynes1957information,Shore1980Axiomatic}.
\begin{figure}
  \centering
    \includegraphics{Sw240}
      \caption{{\bf Understanding Gibbs entropy.} The Gibbs entropy for the normal-model-with-prior is plotted as a function of sample size. The Gibbs entropy can be understood heuristically as the log ratio of the model consistent with the data to allowed models. At small sample size, the model structure determines the parametrization and therefore all models allowed are consistent with the data and there is zero Gibbs entropy. As the sample size grows beyond the critical sample size $N_0$, fewer and fewer of the allowed models are consistent with the data and the entropy decreases like $-\frac{1}{2}K\log N$. The non-positivity of the Gibbs entropy is a direct consequence of the normalization of the prior, which forces the Gibbs entropy to have a maximum value of zero. A prior determined by the generalized principle of indifference avoids this non-physical result. \label{Fig:S}}
\end{figure}
However, the principle of indifference is difficult to interpret in the context of continuous parameters, or across models of different dimension.
For example, are normal models with means $\mu$ and $\mu+{\rm d}\mu$ mutually exclusive (distinguishable)? Even if the mean were constrained to be an integer ($\mu \in \mathbb{Z}$) to define \textit{mutually exclusive}, the exhaustive condition is also problematic. Exhaustive would correspond to a uniform weighting over all integers. This vanishing prior weight ($1/\infty$) on the non-compact set  $\mathbb{Z}$ results in a paradoxical value for the evidence $Z\rightarrow 0$ and the rejection of the model irrespective of the data,  as described in Sec.~\ref{Sec:ModelSelection} (Lindley-Bartlett paradox).

\subsection{A generalized principle of indifference}
To define \textit{mutually exclusive} in a statistical context, we look for natural analogues to this problem in statistical physics.
A surprising result from the perspective of classical physics  is that Nature makes no distinction between states with identical particles exchanged (\textit{e.g.}~electrons) and counts only distinguishable states (the Gibbs paradox).
Following V.~Balasubramanian \cite{Balasubramanian1997}, we proposed that the concept of indistinguishability must be applied to objective Bayesian inference.
We take the \textit{mutually exclusive} criteria in the principle of indifference to refer to distributions which are mutually distinguishable at the experimental resolution available.
We propose  a \textit{generalized principle of indifference}: sets of indistinguishable models are each collectively assigned the weight of a single distinguishable model.

To study the weighting of each model, \textit{we must prepare the data using a different procedure}. We distribute $X^N$ according to an assumed true parameter ${\bm \theta}$: $X^N\sim q(\cdot|{\bm \theta})$, omitting the expectation over $\bm \theta$ \footnote{We must  make the important distinction between the \textit{inference prior} $\varpi$, used to perform inference, and the \textit{true prior}, used to generate the true parameters ${\bm \theta}_0$. This approach has a long and important precedent: \textit{e.g.}~\cite{watanabe2009,Kashyap1971}.}.
A generalized principle of indifference states that the prior $\varpi$ should be chosen such that:
\begin{equation}
\overline{S}({\bm \theta}; N,\varpi) \approx {\rm const}\ \ \ \forall \ \ \ \ {\bm \theta} \in \Theta, \label{Eqn:pofi}
\end{equation}
 at sample size $N$, where the Gibbs entropy is now a function of  ${\bm \theta}$.

 Eqn.~\ref{Eqn:pofi} realizes a statistically principled definition for the condition of equal model weighting on mutually exclusive models.

The correspondence also offers a natural mechanism for resolving statistical anomalies arising from the \textit{exhaustive} condition in the principle of indifference that gives rise to the Lindley-Bartlett paradox.
In statistical mechanics, the partition function $Z$ is not normalized by construction since the density of states $\rho$ is a density but not a probability density. Therefore, a natural solution
to statistical anomalies arising from the exhaustive condition is to re-interpret the objective inference prior as a \textit{density of models}.
To specify a consistent density of models between different parameter values and model families, we replace the  prior $\varpi({\bm \theta})$ with a model density $w({\bm \theta})$ such that:
\begin{equation}
\overline{S}({\bm \theta};N,w) \approx 0, \label{eqn:gpoi}
\end{equation}
assigning unit multiplicity to all parameters ${\bm \theta}$ and model families $I$.
(\textit{Technical note:} We avoid specifying Eqns.~\ref{Eqn:pofi} and \ref{eqn:gpoi} as  equalities since the condition is typically not exactly realizable for all ${\bm \theta}_0$ at finite sample size $N$. A precise formulation will be described elsewhere, but is analogous to the mini-max approach of Kashyap where the largest violation of the GPI condition is minimized~\cite{Kashyap1971}.)  Eqn.~\ref{eqn:gpoi}, and the resulting prior are reparametrization invariant (see App.~\ref{app:invariant}).
The prior $w$ will be improper, but none-the-less the normalization is well defined. We shall refer to Eqn.~\ref{eqn:gpoi} as the \textit{Generalized Principle of Indifference} which realizes both the mutually-exclusive and exhaustive conditions using a principled statistical approach, regardless of the nature of the parameter manifold. We will call the prior $w$ that satisfies Eqn.~\ref{eqn:gpoi} the \textit{GPI prior}.

\section{Applications}
\label{sec:app}

We have used the correspondence between thermodynamics and statistics to motivate the definition of the learning capacity and the Gibbs entropy (and resultant generalized principle of indifference). We now wish to investigate the statistical properties of these definitions. We will find that both these novel statistical objects provide new insight into statistics and learning.

\subsection{Learning Capacity}
\label{sec:learningCap}

\begin{figure}
  \centering
    \includegraphics[width=.50\textwidth]{learningCap2}
      \caption{{\bf The failure of equipartition.} The behaviors of the heat capacity and learning capacity are compared and related to the applicability or inapplicability of the Equipartition theorem in different regimes. \textbf{Panel A: Low-temperature freeze-out in a quantum system.} The heat capacity is plotted as a function of temperature. Equipartition predicts that the reduced heat capacity should be constant, equal to half the degrees of freedom in the system. Plateaus can clearly be observed at half-integer values, but the number of degrees of freedom is temperature dependent due to the discrete nature of quantum energy levels. At low temperature, some degrees of freedom are frozen out since the first excited state is thermally inaccessible.  This discrete topology of the energy levels implies anharmonicity in the potential and therefore failure of the equipartition theorem. \textbf{Panel B: High-temperature freeze-out in the Learning capacity.} Analogous to the statistical mechanics system, the statistical learning capacity transitions between half integer plateaus, reflecting a temperature-dependent number of degrees of freedom. At low sample size $N$ (high temperature), the parameters are completely specified  by model constraints (the prior) and therefore the parameters do not contribute to the learning capacity. At large sample size $N$, the parameters become data dominated and therefore the learning capacity is predicted by equipartition ($\frac{1}{2}K$).
      \label{Fig:DU}}
\end{figure}

The applicability and failure of the equipartition theorem are well understood phenomena in physics. At high or low temperature, degrees of freedom can become anharmonic, altering their contribution to the heat capacity \cite{pathria1996statistical}.
For instance, due to the discrete structure of the quantum energy levels, degrees of freedom can \textit{freeze out} at low temperature. (See Fig.~\ref{Fig:DU}A.)
Degrees of freedom can also become irrelevant at high temperature. For instance, the position degrees of freedom of a gas  do not contribute to the heat capacity  \cite{Reif1967}. We see an analogous high-temperature freeze-out mechanism in the context of inference.

In this section, we investigate the phenomenology of the learning capacity in a series of simple examples. In each case, we first compute $\overline{F}$, then we compute the learning capacity (as defined in Tab.~\ref{Table:stat}). In most of our examples, both computations can be performed analytically.
\begin{itemize}
\item In Sec.~\ref{sec:lc-norm}, we analyze the finite-sample-size behavior of the learning capacity in the context of the normal model with unknown mean and variance. The dependence of the log-likelihood on the variance is anharmonic and therefore there are interesting finite-sample-size corrections to equipartition.
\item In Sec.~\ref{sec:discrete}, we analyze a quantum-like freeze-out phenomenon for a model on a discrete parameter manifold.
\item In Sec.~\ref{sec:Stoich1}, we analyze a problem where a discrete parameter manifold arise naturally: the stoichiometry  of a Poisson processes.
\item In Sec.~\ref{sec:expmix}, we analyze a singular model, the exponential mixture model, which has previously been identified as \textit{sloppy}.  We show that the learning capacity is smaller than predicted by equipartition for parameter values in the vicinity of the singularity.
\item In Sec.~\ref{sec:uniC}, we analyze the learning capacity in a non-regular but analytically-tractable model.   In this example the learning capacity is larger than predicted by equipartition.
\end{itemize}

\begin{figure}
  \centering
    \includegraphics[width=.5\textwidth]{HeatCapCDuo}
    \caption{ \textbf{Panel A: Learning capacity at finite sample size.} At large sample size, equipartition predicts the learning capacity of all models. At sample size $N=1$ the learning capacity diverges for models with unknown variance since the mean and variance cannot be simultaneously estimated from a single measurement. \textbf{Panel B: Learning capacity on a discrete manifold.} The learning capacity of a normal model with an unknown D-dimensional mean $\inliner{\vec{\mu}\in {\mathbb Z}^D}$ and variance $\sigma^2 = 15$. For statistical uncertainty $\delta \mu \gg 1$, the learning capacity is predicted by equipartition since the discrete nature of the parameter manifold cannot be statistically resolved. For $\delta \mu \ll 1$, there is no statistical uncertainty in the parameter value (due to the discreteness of $\mu$) and the degrees of freedom freeze out, giving a learning capacity of zero.
      \label{Fig:heatcapC}}
\end{figure}

\subsubsection{Normal model with an unknown mean }
In many regular statistical models at finite sample size, it is the model structure and not the data that constrain the parameter values. In these cases, structurally constrained parameters will not contribute to the learning capacity. A simple and exactly tractable example of this phenomenon has already been discussed: the normal model with unknown mean and an informative prior.

\medskip
\noindent
\textit{Model:} We define a normal model on a $D$-dimensional observation space with unknown mean and unknown variance $\sigma^2$. The likelihood function is:
\begin{equation}
q(\vec{x}|{\bm \theta}) \equiv (2\pi \sigma^2)^{-D/2}\ \exp [-\textstyle\frac{1}{2\sigma^2}(\vec{x}-\vec{\mu}\,)^2], \label{eqn:normalmodel}
\end{equation}
with ${\bm \theta} \equiv (\vec{\mu})$ and informative prior:
\begin{equation}
\varpi({\bm \theta}) \equiv (2\pi \sigma_\varpi^2)^{-D/2}\ \exp [-\textstyle\frac{1}{2\sigma^2_\varpi}(\vec{\mu}-\vec{\mu}_\varpi\,)^2], \label{eqn:normalmodel}
\end{equation}
We now consider the informative limit where the critical sample size $N_0$ (Eqn.~\ref{eqn:critss}) is finite.

\medskip
\noindent
\textit{Analysis:} The learning capacity can be computed analytically:
\begin{eqnarray}
\overline{C} = \textstyle\frac{K}{2(1+N_0/N)^2}.
\end{eqnarray}
At large sample size, the learning capacity is equal to the equipartition expression (Eqn.~\ref{eqn:equipart}). At small sample size (high temperature), the prior determines the parametrization, and therefore the parameter does not contribute to the learning capacity and $\inliner{\overline{C}\rightarrow 0}$.  (See Fig.~\ref{Fig:DU}B.) This situation is roughly analogous to the heat capacity of a gas. In the solid phase, the position degrees of freedom contribute to the heat capacity in the canonical way whereas in the gas phase, the walls of the box confine the atoms, the energy does not depend on the position degrees of freedom, and these variables no longer contribute to the heat capacity.

\subsubsection{Normal model with an unknown mean and variance}

\label{sec:lc-norm}
The learning capacity $\overline{C}$  typically deviates from  equipartition-like behavior in the small sample-size (high temperature) limit
in analogy to
anharmonic effects in the heat capacity of metals near the melting point \cite{wallace2002statistical,fultz2010vibrational}. To demonstrate this phenomenon, we analyze the normal model with unknown mean and variance.

\medskip
\noindent
\textit{Model:} We define a normal model on a $D$ dimensional observations space with unknown mean and unknown variance $\sigma^2$. The likelihood function is given by Eqn.~\ref{eqn:normalmodel}
with ${\bm \theta} \equiv (\vec{\mu},\sigma)$ and impose an improper Jeffreys prior $\varpi = \sigma^{-D-1}$. Although, the dependence of the information is harmonic in $\mu$, it is non-harmonic in variance $\sigma^2$.

\medskip
\noindent
\textit{Analysis:} The learning capacity can be computed analytically:
\begin{eqnarray}
\overline{C}&=&\textstyle\frac{D}{2}\left[ 1-Dn^2\psi^{(1)}(\textstyle\frac{D(n-1)}{2})+ ...\right. \\
& &\left. +\textstyle\frac{Dn^2}{2}\psi^{(1)}(\textstyle\frac{Dn}{2})-\textstyle\frac{D^2n^3}{4}\psi^{(2)}(\textstyle\frac{D(n-1)}{2})\right],
\end{eqnarray}
in terms of the polygamma functions $\psi$. The learning capacity is plotted as a function of sample size in Fig.~\ref{Fig:heatcapC}A. The learning capacity diverges at sample size $N=1$ since the mean and variance cannot be estimated from a single measurement and the divergence of the learning capacity signals an infinite predictive loss.
 The learning capacity of the normal model with known mean and variance is representative of the behavior of many models: Typically, the learning capacity can show significant differences with the equipartition limit at very small sample size but rapidly converges to the equipartition value as the sample size grows.

\subsubsection{Normal model with discrete mean.}
\label{sec:discrete}

An important exception to the generic behavior described in the previous paragraph occurs when the \textit{parameter manifold is discrete rather than continuous}. In exact analogy to the freeze-out phenomenon in quantum statistical mechanics, as the sample size increases (and the temperature decreases) the discrete nature of the parameter manifold (energy levels) becomes the dominant structure in the analysis and the system condenses into a single distribution (ground state). We will  consider a contrived but analytically tractable example: a normal model with unknown discrete mean.

\medskip
\noindent
\textit{Model:} The likelihood for the $D$-dimensional normal model is defined in Eqn.~\ref{eqn:normalmodel}. The parameters now include only the the mean: ${\bm \theta} = (\vec{\mu})$, with the mean constrained to have an integer values: $\vec{\mu}\in \mathbb{Z}^D$. We assume a flat improper Jeffreys prior: $\varpi = \sigma^{-D}$.

\medskip
\noindent
\textit{Analysis:} The learning capacity can be computed analytically and the expression is given in the Appendix Sec.~\ref{App:discra}. The learning capacity is plotted in Fig.~\ref{Fig:heatcapC}B. To discuss the phenomenology, it is useful to define a frequentist \textit{statistical resolution} with respect to parameter coordinate $\theta^i$:
\begin{equation}
\delta \theta^i(N) \equiv N^{-\frac{1}{2}}\sqrt{[\bm I^{-1}]^{ii}}, \label{eqn:resolution}
\end{equation}
in terms of the Fisher information matrix $\bm I$ (Eqn.~\ref{fisherMat}), which is a naturally covariant symmetric tensor on the continuous parameter manifold. $\delta \theta^i(N)$ is the width of the posterior in the large-sample-size limit. For the normal model, $\delta \mu = \sigma/\sqrt{N}$. For a regular model with discrete parameters in the large sample size limit, the learning capacity is:
\begin{equation}
\overline{C} = \begin{cases}
\textstyle\frac{1}{2}K, & \Delta \theta^i \ll \delta \theta^i\ \text{for all }i \\
0, & \Delta \theta^i \gg \delta \theta^i\ \text{for all }i
\end{cases} \label{eqn:lcdiscrete}
\end{equation}
where $\Delta \theta^i$ is the lattice spacing for parameter coordinate $\theta^i$. The physical interpretation is clear: At large sample size, the system condenses into a single state.  Therefore the corresponding degrees of freedom freeze out, and no longer contribute to the learning capacity. At small sample size, the discrete nature of the parameter manifold cannot be resolved, and the parameter manifold is effectively continuous. The learning capacity therefore assumes the equipartition value (provided that the sample size is large enough such that the information is effectively harmonic, as discussed in Sec.~\ref{sec:lc-norm}).

\subsubsection{Stoichiometry of a Poissn process}

\label{sec:Stoich1}

In the previous example, we considered a somewhat contrived example where the mean of a normal process was discrete, but many important statistical problems are naturally defined on discrete parameter manifolds (\textit{e.g.~}the dimension of a chi-square distribution, the shape of gamma distribution, \textit{etc}). For an explicit example, we will analyze a problem that arises in the context of our experimental work, the analysis of protein stoichiometry.  The molecular complex is known to be a multimer: a complex consisting of a well-defined number of identical protein subunits. In our experiments, we measure stoichiometry by the fluorescence intensity of the protein labels.

\medskip
\noindent
\textit{Model:} The number of photons emitted per fluorophore is well modeled by a Poisson process. We will assume that the average intensity per fluorophore is known (\text{i.e.} the rate $\lambda$ is known).
The stoichiometry, or number of fluorophores, is $m \in \mathbb{N}$, an unknown natural number. Let $x_i$ be the binary observation of a photon ($x=1$) or no photon ($x=0$)  in a short interval length $\delta t$:
\begin{equation}
q(x|{\bm \theta} ) \equiv (m\,\lambda \, \delta t)^x(1-m\,\lambda\, \delta t)^{1-x},
\end{equation}
where we will work in the limit as $\delta t\rightarrow 0$ where the parameter is ${\bm \theta} = (m)$. We apply a flat prior $\varpi_m= 1$.

\begin{figure}
  \centering
    \includegraphics[width=.5\textwidth]{2-posthc_paw2}
    \caption{{\bf Panel A: Posterior for low-temperature freeze-out.} Low-temperature freeze-out occurs when there is no statistical ambiguity in the parameter value. For long interval durations ($\lambda t=500$) the posterior only weights a single parameter value ($m=6$) whereas the manifold is effectively continuous for intermediate interval durations ($\lambda t=10$) and multiple parameter values as weighted.
    {\bf Panel B: Learning capacity for low-temperature freeze-out.}
    For long interval durations ($\lambda t=500$), the stoichiometry $m$ is frozen-out and therefore the learning capacity is zero. For intermediate interval durations ($\lambda t=10$), equipartition applies and the learning capacity is one-half. At short intervals, the large-sample-size limit assumption is violated and the learning capacity diverges from the limiting value.
      \label{Fig:posteriorlc}}
\end{figure}

\medskip
\noindent
\textit{Analysis:} The learning capacity can be computed analytically, as shown in Appendix Sec.~\ref{App:poiss}. The limiting cases can be understood intuitively in terms of the results in the previous section (Eqn.~\ref{eqn:lcdiscrete}). The lattice spacing is $\Delta m = 1$ and the statistical resolution is $\delta m = \sqrt{m/\lambda t}$ and the limiting learning capacity is:
\begin{equation}
\overline{C} = \begin{cases}
\textstyle\frac{1}{2}, & \Delta m \ll \delta m\ \\
0, & \Delta m \gg \delta m
\end{cases} \label{eqn:lcdiscrete}
\end{equation}
where the sample size dependence is represented as an interval duration: $t= N\, \delta t$. For $\Delta m \ll \delta m$, the stoichiometry $m$ appears continuous since the posterior on $m$ spans multiple values of $m$ (Fig.~\ref{Fig:posteriorlc}A) and therefore the learning capacity is predicted by equipartition (Fig.~\ref{Fig:posteriorlc}B). For  $\Delta m \gg \delta m$, there is no statistical ambiguity in the parameter value $m$ (Fig.~\ref{Fig:posteriorlc}A), the parameter freezes out, leading to zero learning capacity (Fig.~\ref{Fig:posteriorlc}B). The plot of the learning capacity (Fig.~\ref{Fig:posteriorlc}B) shows one additional important feature that has already been discussed: At the smallest interval lengths $t$, the large sample-size limit assumed in equipartition fails, leading to a second deviation from equipartition.

\subsubsection{Exponential mixture models.}

\label{sec:expmix}

The previous two examples demonstrated the quantum-like freeze-out that results from discrete parameter manifolds. Our next example illustrates another small sample size (\textit{higher-temperature}) freeze-out phenomenon analogous to the loss of heat capacity when a solid sublimes to form a gas. Here, this phenomenon arises as the result of model singularity: A zero mode appears in the Fisher information matrix corresponding to one (or more) parameter coordinates becoming unidentifiable (Eqn.~\ref{eqn:unident}). To explore this phenomenon, we analyze the exponential mixture model which has previously been identified as \textit{sloppy model} by Transtrum, Machta and coworkers using a criterion defined by the distribution of the eigenvalues of the Fisher information matrix \cite{Machta:2013hl}.

\medskip
\noindent
\textit{Model:} Consider a model for the lifetime of a mixed population consisting of several different chemical species $I$ with different transition rates. Both the transition rates ($k_I$) and the relative abundance of the species ($p_I$) are unknown. For an $m$ species model, the likelihood function for the lifetime $t$ is:
\begin{eqnarray}
q(t|{\bm \theta}) &\equiv& \sum_{I=1}^mp_I\, k_I\, {\rm e}^{-k_It},
\end{eqnarray}
with parameters:
\begin{eqnarray}
{\bm \theta} &\equiv& \left( \begin{array}{ccc} p_1 & ... & p_m \\ k_1 & ... & k_m \end{array} \right),
\end{eqnarray}
subject to the constraint: $\inliner{\sum_I p_I = 1}$ and we apply improper prior $\varpi({\bm\theta}) = 1$. The exponential mixture model is singular since parameter $k_I$ is unidentifiable for $p_I=0$ and $p_1$ is unidentifiable for $k_1=k_2$. (See Eqn.~\ref{eqn:unident}.) For simplicity, we analyze the smallest model with a singularity ($m=2$) to facilitate the numerical Bayesian marginalization.

\medskip
\noindent
\textit{Analysis:} We compute the learning capacity at two locations in parameter manifold, at the singularity (${\bm \theta}_S$) and far from it (${\bm \theta}_R$):
\begin{equation}
{\bm \theta}_S = \left( \begin{array}{cc} 1 & 0 \\ 1 & 10 \end{array} \right) \ \ \ \ \ \ {\text{and}} \ \ \ \ \ {\bm \theta}_R = \left( \begin{array}{cc} {\textstyle\frac{1}{2}} & {\textstyle\frac{1}{2}} \\ 1 & 10 \end{array} \right).
\end{equation}
The learning capacity is computed numerically for $N=100$ observations with distribution $\inliner{T^N\sim q(\cdot|{\bm \theta})}$:
\begin{equation}
\overline{C}({\bm \theta}) = \begin{cases}
0.61, & {\bm \theta}= {\bm \theta}_S \\
1.5, & {\bm \theta}= {\bm \theta}_R
\end{cases}.
\label{eqn:Llc}
\end{equation}
Far from the singularity (${\bm \theta}_R$), the equipartition theorem predicts the learning capacity ($\dim/2$) whereas close to the singularity (${\bm \theta}_S$), where the model is effectively described by only a single parameter ($k_1$), the learning capacity reflects this smaller effective model dimension. As expected, the exponential mixture model is predictively sloppy in the vicinity of the singular point, but not elsewhere.

We expect that the behavior of the exponential mixture model is representative of many machine learning and systems biology problems. In these problems, the effective dimension $K_{\rm eff}$ may be very much smaller than the true dimension of model. In practice, these models do not require exact  structural unidentifiability (Eqn.~\ref{eqn:unident}) to show a reduced learning capacity. For instance, the reduced complexity is not only at the singularity but in the vicinity of the singularity as well. For models with structural unidentifiability, this vicinity-of-singularity region will shrink with sample size. For models without structural unidentifiability (Eqn.~\ref{eqn:unident}) that are regular everywhere, equipartition will hold at sufficiently large sample size.

\subsubsection{Uniform distribution with unknown upper limit.}

\label{sec:uniC}

In the previous example, the non-regular model showed reduced learning capacity at the singularity but non-regular models can also have increased learning capacity as well. To illustrate this phenomenon, consider a continuous version of the German Tank problem, estimation of the support of a uniform distribution \cite{goodman1954some}.

\medskip
\noindent
\textit{Model:} Consider a uniform distribution $X\sim{\cal U}(0,L)$ with unknown end point $L\ge 0$.  The likelihood function is:
\begin{equation}
q(x|{\bm \theta} ) = \begin{cases} L^{-1}, & 0\le x \le L \\0, & \text{otherwise}  \end{cases},
\end{equation}
with parameter $\bm \theta \equiv (L)$ and improper prior $\varpi(L) = L^{-1}$.

\medskip
\noindent
\textit{Analysis:} In this model, neither the first nor second derivative of the cross entropy $H(L;L_0)$ exist at the true parameter $L_0$ and therefore the model is not regular. It is straightforward to compute the learning capacity:
\begin{equation}
\overline{C}=1,
\end{equation}
 corresponding to an effective dimension of two, even though the parameter manifold is one dimensional. This result is exact and independent of sample size $N$. (See Appendix Sec.~\ref{sec:appuni}.)

\subsection{Generalized principle of indifference}

The proposed generalized principle of indifference has properties that rectify significant shortcomings with other approaches to prior selection. We showcase these properties with the analysis of a series of examples.
\begin{itemize}
\item In Sec.~\ref{sec:regmodels}, we compute the GPI prior for regular models in the large-sample-size limit. This analysis reveals a connection between the GPI prior and the Jeffreys prior.

\item In Sec.~\ref{sec:exact}, we demonstrate an exact computation of the GPI prior for a number of non-harmonic models.

\item In Sec.~\ref{sec:nonExact}, we demonstrate an exact computation of the GPI prior for a non-regular model.

\item In Sec.~\ref{sec:discretew}, we analyze the limiting behavior of the GPI prior on a discrete parameter manifold. GPI unifies two conflicting approaches, and interpolates between respective limits as a function of sample size.

\item In Sec.~\ref{sec:poisson2}, we analyze a problem where the GPI prior cannot be computed exactly: the analysis of stoichiometry in a Poisson process.

\end{itemize}

\subsubsection{Approximate GPI prior for regular models}
\label{sec:regmodels}

We will first explore the properties of the generalized principle of indifference by computing the GPI prior in the large-sample-size limit of a regular model.
To define the GPI prior, it is first useful to define the scaled-Jeffreys prior:
\begin{equation}
\rho({\bm \theta};N) \equiv \left(\textstyle\frac{N}{2\pi}\right)^{K/2} I^{1/2}, \label{eqn:scaledJeff}
\end{equation}
where $I$ is the determinant of the Fisher information matrix defined for a single sample (Eqn.~\ref{fisherMat}), $K$ is the dimension of the continuous parameter manifold $\Theta$. The prior $\rho$ is a density on parameter manifold with the qualitative meaning of the inverse volume of indistinguishable models  at sample size $N$. The GPI prior is
\begin{equation}
w({\bm \theta};N) \approx \rho\ e^{-K}, \label{eqn:w}
\end{equation}
where $K=\dim \Theta$, as shown in Appendix Sec.~\ref{sec:jeffreys}.

In the large-sample-size limit, the parameter dependence of the GPI prior is identical to the Jeffreys prior, which has enjoyed a long and successful history \cite{Kass1996}. The Jeffreys prior was initially proposed because it was reparametrization invariant \cite{Jeffreys1946}. More recently the same prior has been motivated by numerous other arguments (\textit{e.g.}~\cite{Bernardo1999,Balasubramanian1997}). From the perspective of parameter inference, in the large-sample-size limit of a regular model, the GPI approach simply recapitulates a widely-applied method rather than generating novel algorithm.

\subsubsection{Exact GPI prior for symmetric models}

\label{sec:exact}

For simple models, symmetry and dimensional analysis often imply that $w$ must still be proportional to the Jeffreys prior even at small sample size. We compute the exact GPI prior analytically for the normal model with unknown mean and variance and the exponential model in the Appendix Sec.~\ref{sec:appExact}. Both these models have a log-likelihood that is anharmonic in the parameters and therefore are expected to have non-trivial high-temperature behavior.
The calculation reveals that the asymptotic form of the GPI prior (Eqn.~\ref{eqn:w}) closely approximates the exact prior. In many models it is convenient to define the finite sample-size correction as an effective complexity $\cal K$ that replaces the model dimension $K$ in Eqn.~\ref{eqn:w}:
\begin{equation}
w({\bm \theta};N) \approx \rho\ e^{-{\cal K}}, \label{eqn:calK}
\end{equation}
$\cal K$ is plotted as a function of sample size ($N$) for a number of different models in Fig.~\ref{Fig:effectComp}. On an empirical basis, it is clear that Eqn.~\ref{eqn:w} is typically an excellent approximation for $w$ even small to intermediate sample sizes for many models.

\subsubsection{Exact GPI prior for a non-regular model}

\label{sec:nonExact}
\begin{figure}
  \centering
    \includegraphics[width=.5\textwidth]{effectiveComp}
    \caption{{\bf Effective complexity of models at finite sample size.} We computed the exact GPI prior for a series of models of different dimension. At large sample size, the dimension determines the effective complexity: ${\cal K} = \dim \Theta / 2$. At finite sample size there are significant corrections. The effective complexity divergences for the normal model (dashed curves) with unknown mean and variance at $N=1$.
      \label{Fig:effectComp}}
\end{figure}

In this section we explore another key motivation to the GPI approach: the analysis of non-regular models. We return to the example of the uniform distribution $X\sim{\cal U}(0,L)$ with unknown end point $L\ge0$.  In this case, the Fisher information matrix (Eqn.~\ref{fisherMat}) is not defined and so the Jeffreys prior approximation (Eqn.~\ref{eqn:w}) cannot be applied. The definition of $w$ does not depend on assuming a regular model and it is still straight forward to compute $w$ (Appendix Sec.~\ref{sec:appuni}):
\begin{equation}
w(L;N) = \textstyle\frac{N}{L}\exp\left[-1-N\log (1+N^{-1})\right].
\end{equation}
which has a scaling of sample size $N$ corresponding to an effective dimension of two, even though the parameter manifold is one dimensional.

\subsubsection{GPI prior for discrete parameter manifolds}
\label{sec:discretew}

For discrete parameter manifolds two competing and well-established methods exist for choosing a prior:
(i) A literal interpretation of the principle of indifference would seem to imply that all parameter values are given equal weight. (ii) Alternatively, we can consider the continuous parameter limit where the prior can be chosen to give consistent results with the Jeffreys prior. Both approaches have desirable properties in different analytical contexts \cite{Berger2012}. GPI provides an elegant resolution to this conflict:
 When the discrete nature of the parameter manifold can be statistically resolved, the GPI prior prior assigns equal weight to all discrete parameter values (i), whereas, if the discreteness of the space cannot be statistically resolved, the large $N$ limit gives rise to a Jeffreys prior (ii):\begin{equation}
w = \begin{cases}
 \rho\ e^{-K} \prod_i \Delta \theta^i, & \Delta \theta^i \ll \delta \theta^i \\
1, & \Delta \theta^i \gg \delta \theta^i
\end{cases}\label{eqn:discretew}
\end{equation}
where $\Delta \theta^i$ is the lattice spacing and the statistical resolution $\delta \theta^i$ is defined in Eqn.~\ref{eqn:resolution}. The GPI prior for a normal model with a discrete mean can be computed exactly and is described in Appendix Sec.~\ref{App:discra}.

\subsubsection{Stoichiometry of a Poissn process (revisited)}

\label{sec:poisson2}

In each example discussed so far, it is possible to compute the GPI prior exactly. In most applications this approach is \textit{not} tractable. Our analysis of regular models suggests that Eqn.~\ref{eqn:w} is often an excellent approximation.
If this does not suffice, a recursive algorithm (Appendix, Sec.~\ref{sec:recursiveAlgorithm}) is a practical refinement to Eqn.~\ref{eqn:w}.
As an example of this  approach, we return the analysis of the stoichiometry of a Poissn process. (See Sec.~\ref{sec:Stoich1}.) In this case it is clear from the learning capacity (Eqn.~\ref{eqn:lcdiscrete}) that  $w$ must interpolate between the discrete and continuous limits of Eqn.~\ref{eqn:discretew} as a function of the stoichiometry $m$. We  compute $w$ numerically using the recursive algorithm (Appendix Sec.~\ref{App:poiss}).

We plot the $w$ (GPI) and flat (PI) priors as a function of the stoichiometry $m$ in Fig.~\ref{Fig:wflatprior}A and the Gibbs entropy of the $w$ and flat priors in Fig.~\ref{Fig:wflatprior}B. The traditional interpretation of the principle of indifference leads to a constant prior and non-constant Gibbs entropy. In contrast, GPI results in a non-constant prior and constant Gibbs entropy.
As is seen in Fig.~\ref{Fig:wflatprior}B, the Gibbs entropy of the flat prior rises for large stoichiometry because models with stoichiometry $m$ and $m+1$ cannot be distinguished for the dimensionless interval length $\lambda t$ and therefore the flat prior \textit{over weights} these indistinguishable models by counting them independently. The GPI prior compensates by weighting these large stoichiometry distributions less, resulting in constant Gibbs entropy. As the dimensionless interval (\textit{i.e.}~sample size) increases, these distributions become increasingly distinguishable and the GPI increases the weight of these models. For small stoichiometry where the models can be statistically distinguished, the improper-flat prior and $w$ are identical (Fig.~\ref{Fig:wflatprior}A).

\begin{figure}
  \centering
    \includegraphics[width=.5\textwidth]{entropiesandpriorPAW2}
      \caption{ {\bf Complementary views on indifference:}  A flat prior and the GPI prior offer two different ways to define the principle of indifference.  {\bf Panel A: The GPI prior depends on parameter and sample-size.} The flat prior is constant with respect to changes in source number $m$, and sample size, while the  GPI prior (for Poisson stoichiometry problem) changes with the parameter $m$, and responds to sample size. {\bf Panel B: The GPI prior has (nearly) constant entropy.} The average entropy under the GPI prior is almost flat and zero everywhere, but the entropy of the flat prior is {\em not constant}. Some models are entropically favored under the flat prior in violation of the generalized principle of indifference.
      \label{Fig:wflatprior}}
\end{figure}

\begin{table
\setlength{\tabcolsep}{1em}
{\renewcommand{\arraystretch}{1.5
\Scale[0.95]{
  \begin{tabular}{|l|c|c|}
\hline
Model & Parameter support & Generative parameters  \\
      & ${\bm \theta}$   &  ${\bm \theta}_0$ \\
\hline
\hline
${\cal N}$ &  & $\mu_0 = 5$, $\sigma_0 = 1$  \\
${\cal N}(\mu)$ & $\mu\in {\mathbb R}$  & $\mu_0 = 6$, $\sigma_0 = 1$  \\
${\cal N}(\mu,\sigma)$ & $\mu\in {\mathbb R}$, $\sigma\in {\mathbb R}_+$ & $\mu_0 = 5$, $\sigma_0 = 0.75$ \\
${\rm Exp}(\lambda)$ & $\lambda\in {\mathbb R}_+$ & $\lambda_0 = 2$ \\
${\cal U}(L)$ & $L\in {\mathbb R}_+$ & $L_0=10$ \\
\hline
\end{tabular} }
\caption{{\bf Models for inference on simulated data.}  Five data sets were generated, one for each model, using the generative parameters: $\inliner{X^N\sim q(\cdot|{\bm \theta}_0)}$. Inference was performed on the simulated data using the GPI prior $w$. \label{tab:models}}}
\end{table}

\begin{figure*}
  \centering

    \includegraphics[width=.9\textwidth]{posteriorModel3}
      \caption{ \textbf{Bayesian inference on model identity.} The posterior of model identity ($y$ axis) was computed for datasets generated by each model ($x$ axis).  \textbf{Panel A: GPI prior.} For the simulated datasets, the generative model had the highest posterior probability as expected.
       \textbf{Panel B: Normalized objective prior.} The non-compactness of the parameter manifolds implies automatic rejection of all the higher-dimensional models. In this case, since the model ${\cal N}$ is parameter-free, it has posterior probability of 1 for all datasets, regardless of the fit. \textbf{Panel C: Revised informative prior.} To avoid this undesirable anomaly, we tune the prior parameter support to result in a reasonable posterior model probability. (See Tab.~\ref{tab:modelsR}.) Inference is no longer objective, as the posterior probabilities depend on how this tuning is performed. One representative  plot of posterior probabilities to shown. In general, inference cannot be expect to identify the generative model unless the KL divergence is so large as to make the prior irrelevant.
      \label{Fig:inference}}
\end{figure*}

\subsection{Inference}

\label{sec:inferenceapp}

To demonstrate that the GPI prior automatically leads to non-anomalous inference (\textit{i.e.}~free from infinite normalization factors) and is also free from \textit{ad hoc} parameters, we analyze simulated datasets for parameters defined on non-compact manifolds.
We consider five competing models: three realizations of the normal model: known mean and variance, unknown mean and known variance, and unknown mean and variance and we also consider and exponential model with unknown rate and the uniform distribution model with unknown end-point. (See Tab.~\ref{tab:models}.) We will generate datasets from all fives models and then perform inference on the model parameters and identity for each dataset. We purposefully choose the true parameter values and sample size ($N=20$) such that cross entropies are not so large as to make prior choice irrelevant. (See Tab.~\ref{tab:models}.)

\subsubsection{GPI approach}

We have computed the GPI prior for each of the proposed models. Inference on parameter values follows the standard Bayesian framework using the GPI prior $w$.
The GPI prior includes the model prior (Appendix Sec.~\ref{sec:BayesReint}) and therefore the posterior probability of model $I$ is:
\begin{equation}
\varpi(I|x^N) = Z_I/\sum_JZ_J,
\end{equation}
where the model index $J$ runs over the five competing models. The model posteriors for the five sets of simulated data for a sample size of $N = 20$ are shown in Fig.~\ref{Fig:inference}A. The results show a number of important characteristics of the GPI prior: (i) There is an unambiguous  Bayesian procedure for computing inference on both parameters and models. (ii) Inference on both parameters and models leads to non-anomalous results in which the generative distribution has non-zero posterior probability. (iii) For the normal models, the higher-dimensional models have lower posterior probability for the data generated by model ${\cal N}$, even though the generative distribution is realizable in ${\cal N}(\mu)$ and ${\cal N}(\mu,\sigma)$. This shows that the GPI prior contains an endogenous model selection mechanism favoring model parsimony, and we will discuss this in detail in Sec.~\ref{Sec:ModelSelection}.

\subsubsection{A canonical uninformative Bayesian approach}
For contrast, we briefly describe a canonical Bayesian approach to this analysis.
We attempt to use the Jeffreys prior for each model.
A problem immediately presents itself in the context of the Uniform model where the Jeffreys prior is undefined.
We must therefore deviate from our protocol and apply some other prior. We set a flat prior on the parameter $L$ motivated by the principle of indifference.  The priors for the four models with parameters cannot be normalized due to their non-compact parameter manifolds. Parameter posteriors for each model can still be computed using an improper prior and the results are identical to the GPI approach, except for the uniform model where no Jeffreys prior exists. If the Bayesian approach is interpreted literally, the model posterior for the parameter-free normal model is one, regardless of which distribution was used to generate the data, due to the prior impropriety of the other models. (See Fig.~\ref{Fig:inference}B.)
This is an undesirable outcome and this phenomenon is discussed in more detail in Sec.~\ref{sec:lindbart}.

A number of \textit{ad hoc} modifications to the proposed procedure are now possible to avoid this outcome.
(i) After having seen the data, a Bayesian will often reconsider the prior and localize it around the values favored by the data. (See Fig.~\ref{Fig:inference}C.) This approach is formalized in variational or empirical Bayesian methods. In this case, the prior is no longer determined \textit{a priori} and this double-use of data can lead to difficulties due to the potential for overfitting.
(ii) A more rigorous approach is to sub-divide the data: One subset is used to train the prior to make it informative and the second set is used to do inference using the canonical procedure. There are two important disadvantages to this technique: An \textit{ad hoc} decisions must then be made about the size of the data partitions. (This approach is essentially equivalent to eliciting a prior for the parameters from an expert.) A second disadvantage is that some information is lost from inference on the model identity from the data in the prior training subset.
(iii) Alternatively, one could use a non-cannocical approach for inference on the model identity, like the use of the pseudo-Bayes factor \cite{Gelman2014,gelfand1994bayesian,Spiegelhalter2002,Vehtari2012,BURNHAM2004}. As we shall discuss below in Sec.~\ref{Sec:ModelSelection}, this approach is consistent with the GPI approach in the large-sample-size limit.

In summary, the GPI approach results in an unambiguous protocol for selecting the prior and then performing inference whereas the canonical Bayesian approach requires \textit{ad hoc} modifications to lead to acceptable results.

\section{Discussion}

\begin{figure}
  \centering
    \includegraphics[width=.48\textwidth]{inferenceGeomShort2_B}
      \caption{{\bf Panel A: Sloppiness is determined by parameter manifold geometry and posterior width.} Parameters are defined on a compact manifold ${\bm \Theta}$. In sloppy regions of parameter manifold, the parameters are model-structure dominated (red posterior) whereas in regular regions of parameter manifold parameters are data dominated (green posterior).  From the perspective of the learning capacity, the model is effectively one dimensional in proximity to the red posterior and two dimensional in proximity to the green posterior.
      {\bf Panel B: Generalized principle of indifference.}  The posterior distribution $\inliner{\varpi({\bm \theta}_0|X^N)}$  is shown schematically for two different sample sizes $N$. The resolution increases with sample size  as the posterior shrinks.  In the GPI  prior (Eqn.~\ref{eqn:gpoi}), all parameter values consistent with the posterior are assigned  unit prior weight collectively.
\label{Fig:GPOI}}
\end{figure}

\subsection{Learning capacity}

One valuable feature of the proposed correspondence is the potential to gain new insights into statistical phenomenology using  physical insights into the thermodynamic properties of physical systems.
Artificial Neural Networks (ANN) and systems-biology models are two examples of systems with a large number of poorly-specified parameters that none-the-less prove qualitatively predictive.  This phenomena has been discovered empirically and has been termed \textit{model sloppiness} \cite{Machta:2013hl,Transtrum:2015ud}.  These models often have a logarithmic distribution of Fisher information matrix eigenvalues and this characteristic has been used as a definition of sloppiness  \cite{Machta:2013hl}. But, this definition is unsatisfactory since it is not reparametrization invariant. It is easy to construct counterexamples for this definition: For instance, in a $K$-dimensional normal model where the variance for each dimension is logarithmically distributed, the Fisher information eigenvalues are likewise logarithmically distributed, but the model none-the-less behaves like a normal regular model from the standpoint of prediction and statistical analyses.

The correspondence suggests a definition directly written in terms of the predictive performance of the model and the equipartition theorem. We propose that  \textit{predictive sloppiness} be defined as models that have a smaller learning capacity than estimated from the model dimension:
\begin{equation}
\overline{C} < \textstyle \frac{1}{2} \dim  {\Theta}.
\end{equation}
This definition (i) would exclude all regular models in the large sample-size limit, (ii) is reparametrization invariant and (iii) can be generalized to other non-Bayesian frameworks by expressing the learning capacity in terms of the predictive performance.

The sloppiness phenomenon, or freeze out, is the result of the model parameters being determined by model structure rather than the data. To understand the role of model structure in the freeze-out phenomenon, it is useful to write a qualitative expression for the free energy $F$. We project the parameters into a regular sector ${\bm \theta}_R$ dimension $K_R$ and a singular sector ${\bm \theta}_S$ dimension $K_S$ and assume the improper prior $\varpi=1$. We can then write:
\begin{equation}
\Scale[.9]{NF \approx NH_0 - K_R\,\log\, (N I_R)^{-\textstyle\frac{1}{2}} - K_S\,\log\, v_S+...,}\label{eqn:NF}
\end{equation}
where $I_R$ is the $K_R$-root of the determinant Fisher information matrix (Eqn.~\ref{fisherMat}) projected onto the regular sector.
The regular sector give rises to an $N$-dependent $K_R$-dimensional parameter volume  $\inliner{V_R\approx (NI_R)^{-K_R/2}}$ whereas the singular sector gives rise to a $N$-independent $K_S$-dimensional parameter volume $\inliner{V_S=v_S^{K_S}}$. The resultant learning capacity is sloppy:
\begin{equation}
\overline{C}=\textstyle\frac{1}{2}K_R\ < \dim {\Theta},
\end{equation}
with only the regular parameter coordinates contributing.
This scenario is drawn schematically in Fig.~\ref{Fig:GPOI}A.
The posteriors for two parameter values are shown on the parameter manifold: At a regular point (green), $K_S = 0$ and $K_R = 2$ and all parameter coordinates are regular and data dominated. At the sloppy point (red), the manifold is not rigorously singular but $K_S$ is effectively $1$ since the manifold constraints determine the parameter value in the vertical coordinate direction.

In summary, it is parameters inference dominated by the model structure rather than data that in each case  give rise to the anomalously small learning capacity and the qualitative phenomenon of anomalously predictive models.

The learning capacity has the potential to offer new insights into the mechanism of learning in more complex systems, including ANNs. Our preliminary investigations suggest some training algorithms may map to physical systems with well-understood thermodynamic properties.  The detailed physical understanding of the complex  phenomenology of physical systems, including phase transitions, renormalization, \textit{etc.},  have great promise for increasing our understanding of the fundamental  mechanisms of learning.

\subsection{The generalized principle of indifference}

We argue that a natural approach to objective Bayesian inference is to choose a  prior  such that the number of indistinguishable distributions is one for all parameter values. This generalized principle of indifference is simply written $S = 0$. (See Eqn.~\ref{eqn:gpoi}.) Schematically, this procedure assigns equal prior weighting to all models that can be distinguished at finite sample size $N$. As the sample size increases, the prior must be modified to accommodate the increased resolution, \textit{i.e.}~$(\delta \theta^i)^{-1}$, due to shrinking of the posterior support.

It is important to stress that GPI gives rise to a \textit{sample-size-dependent prior} and therefore this inference is \textit{not} Bayesian in a classical sense: (i) It violates Lindley's dictum: \textit{today's posterior is  tomorrow's prior}. (ii) Furthermore, the evidence and prior are no longer interpretable as probabilities but rather statistical weightings. On-the-other-hand, the method codes parameter uncertainty in terms of a posterior probability distribution and facilitates Bayesian parameter and model averaging. Therefore, we would argue the approach maintains  all of the  attractive features of the Bayesian framework while avoiding problematic aspects.  At small sample size or in singular models, the GPI prior must be computed explicitly.  (See Fig.~\ref{Fig:GPOI}B.) For a regular model in the large-sample-size limit, no calculation is required and GPI  prior is equal to the scaled-Jeffreys prior (Eqns.~\ref{eqn:scaledJeff}.)

\subsubsection{Model selection}
\label{Sec:ModelSelection}
The normalization of the GPI prior has  significant consequences for inference on model identity (\textit{i.e.}~model selection). Returning to the regular model, it is straight forward to apply the Laplace approximation to compute the minus-log evidence using the GPI prior:
\begin{equation}
-\log Z(x^N;w) \approx -\log q(x^N|\hat{\bm \theta}) + K, \label{eqn:mle}
\end{equation}
where $K=\dim \Theta$. The scaled-Jeffreys prior cancels the Occam factor from the integration. The two remaining contributions each have clear qualitative interpretations: the MLE estimate of the information ($-\log q$) and a penalty for model complexity  ($K$).  Eqn.~\ref{eqn:mle} is already well known as the Akaike Information Criterion (AIC)\footnote{where AIC is defined in nats (rather than the more common demi-nat expression which is twice Eqn.~\ref{eqn:mle})}:
\begin{equation}
-\log Z \approx {\rm AIC}(x^N). \label{eqn:aic}
\end{equation}
Information-based inference is performed by selecting the model which minimizes AIC, maximizing the estimated predictive performance. The reason why AIC and GPI Bayesian inference are equivalent is most easily understood by rewriting Eqn.~\ref{eqn:gpoi}
\begin{equation}
-N\overline{F} \approx -N\overline{U}, \label{eqn:LSSL}
\end{equation}
which in statistical language corresponds to using a prior that makes the log partition function (LHS) an unbiased estimator of the log predictive performance (RHS).
Since the Akaike Information Criterion (AIC) is an unbiased estimator of RHS at large sample size $N$, the generalized principle of indifference encodes an AIC-like model selection \cite{Stone1977} and an information-based (AIC) realization of Occam's razor: \textit{parsimony increases predictivity} \cite{BurnhamBook}.
The log-predictive performance (RHS Eqn.~\ref{eqn:LSSL}) has been advocated in the context of Bayesian model selection through the use of pseudo-Bayes factors by Gelman and coworkers \cite{Gelman2014,gelfand1994bayesian,Spiegelhalter2002,Vehtari2012,BURNHAM2004}.

\subsubsection{Lindley-Bartlett paradox}
\label{sec:lindbart}
In contrast to the unambiguous inference generated by the GPI approach, the canonical uninformative Bayesian approach leads to a number of difficulties and ambiguities, as we discovered in Sec.~\ref{sec:inferenceapp}.
These difficulties arise due to the normalization of the prior.
We can compute the total number of distinguishable distributions defined by the GPI prior in model $I$ at sample size $N$ by integrating the GPI prior  over the parameter manifold:
\begin{equation}
M_I(N) \equiv \int_{\Theta}\!\! {\rm d} {\bm \theta}\ w({\bm \theta};N). \label{eqn:defN}
\end{equation}
For non-compact manifolds, $M_I(N)$ may diverge, but this is of no significance. A more detailed discussion of the Bayesian meaning of this procedure is provided in the Appendix Sec.~\ref{sec:BayesReint}.

To explore the significance of this normalization, we will compute the minus-log evidence of a regular model in the large-sample-size limit with a
normalized Jeffreys prior. As described above, this prior is equivalent to $w$ normalized by the distribution number (Eqn.~\ref{eqn:defN}):
\begin{equation}
\varpi_J({\bm \theta}) =  {M}^{-1}w({\bm \theta}).
\end{equation}
Using the Laplace approximation, it is straightforward to compute the minus-log-evidence:
\begin{align}
\inliner{-\log Z(x^N;\varpi_J) = -\log q(x^N|\hat{\bm \theta}) + \log M+{\cal O}(N^0),}\label{eqn:evidenceOB}
\end{align}
where where the first term (order $N$) has an interpretation of goodness-of-fit, the second term (order $\log N$) is the minus-log Occam factor or log-number of distinguishable distributions (Eqn.~\ref{eqn:defN}).
The corresponding GPI expression is Eqn.~\ref{eqn:mle}.

The difficulty with using evidence in Eqn.~\ref{eqn:evidenceOB} as a measure of statistical support is clearest in the context of a simple example: the normal model. Consider the analysis of a normal model with $N$ observations $x^N$ and known variance $\sigma^2$. Consider two competing models:  mean is zero (the null hypothesis) and  $\mu \ne 0$ (the alternative hypothesis).
We must define an acceptable range of values for $\mu$. Assume a flat proper prior  on an interval length $L$. The log number of distinguishable distributions is approximately:
\begin{align}
\log M = \log \textstyle\frac{L}{\delta \mu}\left(\textstyle\frac{1}{2\pi}\right)^{\frac{1}{2}}+...
\end{align}
where the error in the mean is $\delta \mu \equiv \sigma/N^{\frac{1}{2}}$.
The condition that the evidence is greater for the $\mu \ne 0$ model than for the $\mu=0$ model is:
\begin{equation}
\hat{\mu} \ge \delta \mu\begin{cases} 2^{\frac{1}{2}}, &\text{GPI} \\ (2 \log M)^{\frac{1}{2}}, &\text{normalized}\end{cases}.
\end{equation}
The expression for  GPI is independent of the interval length $L$ whereas the normalized expression still retains a dependence on $L$. In fact, in the limit that the prior is uninformative ($L\rightarrow\infty$), no finite observed mean is large enough to support the $\mu \ne 0$ model for model selection with a normalized prior. We have described just such an example in Sec.~\ref{sec:inferenceapp}. This automatic rejection of high-dimensional models in the context of uninformative priors is called the Lindley-Bartlett paradox \cite{Lindley1957,Bartlett1957}). The use of the GPI prior circumvents this anomalous result.

\subsubsection{Posterior impropriety}

\label{sec:improp}

The use of GPI prior often, but not always, gives non-zero evidence for all models under consideration.
One such exception is shown in Fig.~\ref{Fig:effectComp} which reveals that the normal model with unknown mean and variance has a divergent effective complexity at a sample size of $N=1$. The effect of this divergence is to give these models zero statistical weight. Although this may initially appear problematic, it is an important feature of the generalized principle of indifference. A mean and variance cannot be estimated from a single observation and as a result the model parameter posterior would be improper and the predictive loss would be infinite. Therefore the generalized principle of indifference automatically gives this model zero statistical weight ($Z=0$). The inability of other approaches to automatically handle posterior impropriety is recognized as a significant shortcoming  \cite{Kass1996}.

\subsection{Comparison with existing approaches and novel features}
The generalized principle of indifference subsumes a patchwork of conflicting methods for prior and model selection, resolving many conflicting approaches and generating a single,  generally-applicable and self-consistent framework.
The GPI approach subsumes the following approaches: (i) For discrete parameter manifolds in the large sample size limit, the GPI gives equal weight to all mutually exclusive models, consistent with the original formulation of the principle of indifference by Bayes and Laplace  \cite{Laplace1812,Keynes1921}. (See Eqn.~\ref{eqn:discretew}.) (ii) In the large-sample-size limit, GPI generates a GPI prior proportional to the well-known Jeffreys prior. In this sense, the approach is closely related to the reference prior approach of Bernardo and Berger \cite{BernardoBerger1991,Bernardo1999}. (iii) With respect to model selection (inference on model identity), the GPI evidence behaves like pseudo-Bayes factors (or AIC) and therefore circumvents the Lindley-Bartlett paradox. (See Sec.~\ref{Sec:ModelSelection}.) To date, the pseudo-Bayes approach has always been un-Bayesian in the sense that
the pseudo-Bayes method consists of the \textit{ad hoc} combination of a canonical Bayesian prior for inference on parameters but a cross-validation-based weighting for inference on models. The GPI provides a self-consistent equivalent approach to inference on both parameters and models

The GPI addresses a number of problems with existing approaches to objective Bayesian inference.
(iv) \textit{Lindley-Bartlett paradox:} As already discussed above,  an important shortcoming with existing objective Bayesian approaches relates to the compactness of the parameter manifold and the automatic rejection of higher-dimensional models in model selection (the Bartlett-Lindley paradox \cite{Lindley1957,Bartlett1957}).  More generally, the evidence of the canonical objective Bayesian approach depends on \textit{ad hoc} modeling decisions, like the range of allowed parameter values.
The GPI-Bayes evidence circumvents these anomalies by generating a consistent distribution  density $w$ over competing models. As a result the GPI evidence is independent of \textit{ad hoc} modeling decisions. (See Sec.~\ref{sec:lindbart}.) (v) \textit{Unification of statistical paradigms:} The absence of the Lindley-Bartlett paradox implies coherent inference between paradigms \cite{Cousins2014,LaMont2017b} and therefore the generalized principle of indifference naturally unifies objective Bayesian inference  with information-based inference.
(vi) \textit{Prior and posterior impropriety:} Another important flaw identified in other objective Bayesian approaches is the inability to handle impropriety. In many cases where parameters are defined on non-compact manifolds, the prior (and sometimes the posterior) cannot be normalized. The redefinition of the prior as a density of models introduces a well-defined and consistent method for defining prior normalization, regardless of the global structure of the manifold. Furthermore, the approach automatically assigns zero statistical weight to models that suffer from posterior impropriety. (See Sec.~\ref{sec:lindbart}.) (vii) \textit{Discrete parameter manifolds:} The GPI-Bayes approach also unifies two well-established approaches to defining objective prior on discrete manifolds: equal weight versus Jeffreys prior. GPI Bayes interpolates between these two limits as a function of sample size. (See Eqn.~\ref{eqn:discretew}.) (viii) \textit{Singularity and sloppiness:} Finally, the GPI-Bayes approach does not assume model regularity. It treats singularity and the sloppiness phenomenon in a natural way. (See Sec.~\ref{sec:exact}.)

\subsection{Conclusion}

Nature reveals an elegant formulation of statistics in the thermal properties of physical systems. Measurements of the heat capacity, compressibility or susceptibility reveal unambiguously how Nature enumerates states and defines entropy.
These physical insights provide clues to the definition of novel statistical quantities and the resolutions of ambiguities in the formulation of objective Bayesian statistics.
We have refined a previously proposed correspondence between the Bayesian marginal likelihood and the  partition function of statistical physics.
We demonstrate a novel and substantive mapping between the average energy, heat capacity, entropy and other statistical quantities.
The newly-defined learning capacity is a natural quantity for characterizing and understanding  learning algorithms and generates new insight into the Akaike Information Criterion (AIC) and model sloppiness through a correspondence with the equipartition theorem and the freeze-out phenomenon, respectively.
Finally, we use the Gibbs entropy to define a generalized principle of indifference and an objective Bayesian weighting prior with the property that all distributions have equal prior weight.
This approach subsumes many seemingly inconsistent and disparate methods into a single, coherent statistical approach.

\medskip
\noindent
\textbf{Acknowledgements:} PAW and CHL acknowledge helpful discussions with M.~Linden, J.~Kinney, D.~Mayo, C.~Heilig, M.~Abbott, B.~Machta and M.~Transtrum. This work was support by NSF grants NSF-PHY-084845 and NSF-MCB-1151043-CAREER.

\newpage
\onecolumngrid

redpajama_set_name: RedPajamaArXiv


Joe Alpar. Visiting faculty, Bennington College, Vermont.
Ozan Aksoy. Adjunct Professor, New York University. Postdoctoral Fellow, Middle East and Middle Eastern American Center, CUNY. The music and multiple identities of Kurdish Alevis from Turkey in Germany (2014).
Stephen Amico. Assistant Professor of Music and Media Studies, University of Amsterdam. Blue notes: Gay men and popular music in contemporary urban Russia (2007).
Ryan Bazinet. Adjunct Professor, John Jay College of Criminal Justice, CUNY. Two sides to a drum: duality in Trinidad Orisha music and culture (2013).
Katharine Cartwright. Associate Professor of Music, Northwest Vista College, TX. Performing Vocalist/Flutist. Quotation and Reference in Jazz performance: Ella Fitzgerald's "St. Louis Blues," 1957-1979 (1998).
Alessandra Ciucci. Assistant Professor of Ethnomusicology, Columbia University. Poems of honor, voices of shame: The 'aita and the Moroccan shikhat (2008).
Carl Clements. Visiting Assistant Professor of Music, Hampshire College, MA. Performing Saxophonist/Flutist. Composer. Pannalal Ghosh and the Bānsurī in the Twentieth Century (2010).
David Demotta. Adjunct Assistant Professor, Hunter College, CUNY. Performing Pianist. Composer. The Contributions of Earl 'Bud' Powell to the Modern Jazz Style (2015).
Noe Dinnerstein. Adjunct Professor, John Jay College of Criminal Justice, CUNY. Ladakhi Folksongs: A cultural, musical, and literary study (2013).
Margaret Farrell. Adjunct Assistant Professor, Manhattan College. Guest Lecturer, Manhattan School of Music. Aspects of adaptation in the Egyptian singing film (2012).
David Garcia. Associate Professor, University of North Carolina at Chapel Hill. Arsenio Rodriguez: A back Cuban musician in the dance music milieus of Havana, New York City, and Los Angeles (2003).
Ju-Yong Ha. Cultural Liason for the Arts, The Ministry of Culture, Sports and Tourish, Republic of Korea. Executive Director, Korean Traditional Performing Arts Association, NY. The grand and majestic: Ujo in P'ansori and Sanjo tradition (2010).
Mark Howell. Director, Department of Archives & History, Winterville Mounds Park and Museum, MS. An ethnoarchaeomusicological investigation of highland Guatemalan Maya dance-plays (2004).
Benjamin Lapidus. Associate Professor of Music, John Jay College of Criminal Justice, CUNY. Performing Guitarist. An examination of the Changui genre of Guantanamo, Cuba (2002).
Janice Mahinka. Assistant professor at Harford College. The Musicality of Salsa Dancers: An Ethnographic Study.
Noriko Manabe. Associate Professor of Music Studies, Temple University, PA. Research Associate, School of Oriental and African Studies (SOAS), University of London. Western Music in Japan: The evolution of styles in children's songs, hip-hop, and other genres (2009).
Joseph Palackal. Indic musicologist, singer, composer. Syriac chant traditions in south India (2005).
David Pier. Assistant Professor, University of North Carolina at Chapel Hill. Performing Pianist. The Senator National Cultural Extravaganza of Uganda: A branded African traditional music competition (2009).
David Racanelli. Assistant Professor of Music, Dowling College, NY. Diasporic jeliya in New York: A study of Mande griot repertoire and performance practice (2010).
Catherine Ragland. Associate Professor of Ethnomusicology, University of North Texas. Ni aqui ni alla (neither here nor there): Musica nortena and the Mexican working-class diaspora (2005).
Evan Rapport. Assistant Professor of Ethnomusicology & Chair of Contemporary Music, The New School, NY. The musical repertoire of Bukharian Jews in Queens, New York (2006).
Patrick Rivers. Assistant Professor, University of New Haven, CT. The mad science of hip-hop: History, technology, and poetics of hip-hop's music, 1975-1991 (2014).
Alexander Stewart. Associate Professor & Jazz Studies Coordinator, University of Vermont. Forensic musicologist. Composition and performance in contemporary New York City big bands, 1989-1999 (2000).
Angelina Tallaj-Garcia. Visiting Instructor of Music, Franklin & Marshall College, PA. Performing Blackness, Resisting Whiteness: Negotiating Racial Identity through Music in the Dominican Republic (2015).
Jadranka Vazanova. Senior Editor, Repertoire International de Litterature Musicale (RILM). Svadobne noty: Ceremonial wedding tunes in the context of Slovak traditional culture (2008).
Elizabeth Wollman. Associate Professor, Baruch College, CUNY. The aesthetic development of the rock musical on the New York stage (2002).
Thomas Zlabinger. Assistant Professor of Music, York College, CUNY. Performing Bassist. Free from jazz: The jazz and improvised music scene in Vienna after Ossiach, 1971-2011 (2013).


Willem Dafoe's inspired and heartfelt performance as Vincent van Gogh raises "At Eternity's Gate" far above standard biopics. Along with the direction of painter-filmmaker Julian Schnabel, Dafoe's work offers an affecting vision of what van Gogh's deeply troubled and otherworldly inner life might have been like.
The film is a dreamlike depiction of the last days of the artist's short life (he died at 37). Van Gogh's musings, often spoken over a dark screen, and scenes of his solitary work alternate with sequences of his interactions with the few others he felt close to, mostly his brother Theo and Paul Gauguin.
The movie begins with van Gogh's assertion, never to be realized, that "I just want to be one of them." He may be referring to the platoons of painters and sculptors who are making Paris the center of the art world. Or he may simply mean anyone who is blessedly free of the kind of demons that haunt his soul.
Van Gogh is attracted to what he sees as the rebellious spirit of Gauguin (Oscar Isaac), who openly disdains the Paris art scene, an attitude that we finally come to see as the loud talk of an ambitious man. The two discuss the role of nature in painting and the genuine source of inspiration, and Gauguin is not reticent about criticizing van Gogh's technique of layering on the paint until his canvases seem more like sculptures.
Given the men's differences, a parting of the ways becomes inevitable, and the scene in which Gauguin informs his friend that he will be leaving is among the movie's most wrenching. Van Gogh howls like an abandoned child — he is truly crushed. His one steady relationship is with brother Theo (Rupert Friend), an art dealer who supports him financially and emotionally.
The brothers' mutual affection is deeply moving, particularly in a scene when Theo visits Vincent in a hospital. Theo crawls into his bed, and Vincent clings to him like an ailing child to its mother. It's a lifeline for the artist, whose isolation and peculiarities earned him the ire of his neighbors in Arles. For as much as van Gogh is seen here as the holy innocent, he is also capable of explosive rages. Only in painting can he escape his torments.
Van Gogh's obvious mental turmoil, including the famous incident in which he cut off part of his ear, eventually lands him in an asylum in Saint-Remy. In one of the film's most affecting scenes, van Gogh talks to a priest (Mads Mikkelsen) who sees in his paintings only signs of madness. Here is where van Gogh best articulates what his art means to him and the uniqueness of his vision.
Schnabel's subjective camerawork, in which shots are jittery or partly out of focus, is a distraction. I am also uneasy with Schnabel's decision to endorse the theory that van Gogh's death was not a suicide; this idea remains controversial.
Don't be put off by these quibbles. Schnabel is a capable filmmaker, as anyone who has seen "The Diving Bell and the Butterfly" can attest. He clearly has been profoundly moved by the life and work of his fellow artist, and the movie feels like something extremely important to him.
Perhaps his best decision was hiring Dafoe (who despite being much older bears a striking resemblance here to his character), with whom he worked before in a film about a very different kind of artist, "Basquiat." In "At Eternity's Gate," Dafoe often works in silence but tells us everything we need to know with his face and eyes.
Small wonder that his performance won the best actor award at this year's Venice Film Festival.


text:
B3 COLLABORATES WITH OpERA ON ECO INVERTEBASE FOR BIOSAFETY RESEARCH
Home > News > B3 COLLABORATES WITH OpERA ON ECO INVERTEBASE FOR BIOSAFETY RESEARCH
A unique and comprehensive database called Eco Invertebase is helping New Zealand and overseas scientists to test the impact of various stressors such as biological control agents on native and valued introduced invertebrates. The database is housed on the B3 website (http://b3nz.org/) and was developed by scientists in Plant & Food Research in B3 and the Optimising Environmental Risk Assessment (OpERA) programmes.
The Eco Invertebase holds information on New Zealand invertebrate taxa, including data describing their biology, ecology, distribution and reproduction. The data entered into the database are limited to published data to ensure its integrity and accuracy. This information feeds into a model called PRONTI (Priority Ranking of Non-Target Invertebrates), which was developed by the OpERA group as a method of more consistently and objectively deciding which non-target invertebrate species should be tested in a laboratory as part of a risk assessment to determine the potential effects of genetically modified (GM) plants before they are released into the environment.
PRONTI works by ranking organisms in the Eco Invertebase by their degree of risk from the stressor, and hence their priority for testing. The higher the ranking given to an invertebrate, the greater its priority for testing. The model can specifically look at organisms that occur within the environment into which the GM organism will be introduced.
The work is aimed at helping the Environmental Protection Authority (EPA), which receives applications for the introduction of new organisms into New Zealand, including GM organisms. The benefit of this system is that it gives scientists and regulators a more objective and consistent way to rank the importance of testing various organisms.
Dr Jacqui Todd of Plant & Food Research in Auckland was one of the key scientists who helped develop the PRONTI model and Eco Invertebase, and through her PhD research, she investigated how PRONTI could be used to prioritise invertebrates for testing the effects of proposed biological control agents. Using the existing Eco Invertebase, Todd modified PRONTI so that it could be used to assess the risks posed by a biological control agent to each invertebrate in the database and to rank them for testing with the agent.
One of the key advantages of the Eco Invertebase is that the huge amount of data on individual species can be used with different stressors, providing flexibility of use for the database. Only the details of the stressor and the likely interaction with each species need to be changed in order to run the PRONTI model for a new scenario. The database currently includes invertebrates from diverse ecosystems including forestry, horticulture, and pasture.
Interest in the database has been widespread, with collaborators as far away as Canada showing interest in how it could be applied in other contexts overseas. For example, Dr Peter Mason of Agriculture and Agrifood Canada is testing the database and PRONTI model using North American biological control agents.
Being housed on the B3 website means that the Eco Invertebase is now easily accessible for simultaneous data entry by multiple people across several organisations. The goal is to make the database available to the rest of the scientific community within 3-4 years, once the group has fully tested the model.
The EPA are supportive of the concept of this work and Kate Bromfield, Senior Advisor, New Organisms, said, "The EPA supports any research that provides us with consistent decision-making tools, particularly when evaluating the off-target effects of biological control agents."
For further information contact:Jacqui.Todd@plantandfood.co.nz

redpajama_set_name: RedPajamaCommonCrawl


A platter of lo mein covered with chicken fingers and smothered with shrimp, beef, roast pork and mixed vegetables.
Crispy delicious duck marinated, prepared in traditional way, served with crepes, cucumber, scallions and hoisin sauce.
With 5 pancakes. Spicy if requested.
Meat may be omitted by request.
Served with fine egg noogle. All times steamed, spices optional. No MSG, salt, corn starch, oil or sugar.


Charles Dudley Warner (1829–1900) was an American essayist, novelist, and friend of Mark Twain, with whom he co-authored the novel The Gilded Age: A Tale of Today. Warner was born of Puritan descent in Plainfield, Massachusetts. From the ages of six to fourteen he lived in Charlemont, Massachusetts, the scene of the experiences pictured in his study of childhood, Being a Boy. He then moved to Cazenovia, New York, and in 1851 graduated from Hamilton College, Clinton, NY.
No man but feels more of a man in the world if he have but a bit of ground that he can call his own. However small it is on the surface, it is four thousand miles deep; and that is a very handsome property.
Barring some piece of luck I have seen but few men get rich rapidly except by means that would make them writhe to have known in public.
Lettuce is like conversation: it must be fresh and crisp, and so sparkling that you scarcely notice the bitter in it.
The most popular persons are those who take the world as it is, who find the least fault.
The wise man does not permit himself to set up even in his own mind any comparisons of his friends. His friendship is capable of going to extremes with many people, evoked as it is by many qualities.
Hoeing in the garden on a bright, soft May day, when you are not obligated to, is nearly equal to the delight of going trouting.
the most satisfactory thing a man can do.
It is only fools who keep straining at high C all their lives.
If there was any petting to be done…he chose to do it. Often he would sit looking at me, and then, moved by a delicate affection, come and pull at my coat and sleeve until he could touch my face with his nose, and then go away contented.
The love of dirt is among the earliest of passions, as it is the latest. Mud-pies gratify one of our first and best instincts. So long as we are dirty, we are pure. Fondness for theground comes back to a man after he has run the round of pleasure and business, eaten dirt, and sown wild oats, drifted about the world, and taken the wind of all its moods. The love of digging in the ground (or of looking on while he pays another to dig is as sure to come back to him, as he is sure, at last, to go under the ground, and stay there.
To poke a wood fire is more solid enjoyment than almost anything else in the world.
Memory has the singular characteristic of recalling in a friend absent, as in a journey long past, only that which is agreeable.
Did a woman ever love who would not give all the years of tasteless serenity for one year, for one month, for one day of uncalculating delirium of love poured out upon the man who returned it.
Perhaps nobody ever accomplishes all that he feels lies in him to do but nearly every one who tries his power touches the walls of his being occasionally, and learns about how far to attempt to spring.


text:
The Royal Thai Embassy witnessed groundbreaking ceremony of an investment by Banpu, a leading Thai energy company, in Pennsylvania
On January 15, 2019, Deputy Chief of Mission and Minister Boosara Kanchanalai visited Tuckhannock in northern Pennsylvania to witness groundbreaking of Banpu's BKV Operating LLC.
Minister Boosara congratulated Banpu executives on another step forward and rapid expansion in their oil and gas business in the United States. She expressed her appreciation for the support from the local government and local community in Tuckhannock. She underlined that this investment will further contribute to job creation and local economy of the area, and represented another showcase of close friendship between Thailand and the United States.
Executives from Banpu included Mr. Chanin Vongkusolkit, Chairman, Mrs. Somruedee Somphong, CEO, Mr. Anon Sirisaengtaksin, Director, and Mr. Thiti Mekavichai, CEO of Banpu North America Corporation as well as Mr. Chris Kalnin, CEO of Kalnin Ventures. In congratulating the company, Minister Boosara was joined by Mr. Nipon Petchpornprapas, Consul-General of Thailand to New York.
Banpu is one of the success stories of Thai private sector. The company has grown from a local coal producer in Thailand to a multinational company with innovation, diversifying its original portfolio to renewable energy as well as oil and gas businesses in 10 countries including the United States.
The Royal Thai Embassy in Washington is fully supportive of efforts to further promote the mutually beneficial economic relations between Thailand and the United States through two-way investment.


Here you will find our technical articles. We'll be adding more as needed, particularly as subjects come up in the forum. The articles stress the basic things a novice user needs to know about Windows and computers so that you have a strong foundation of understanding upon which to build more detailed knowledge.
Are You Ready for a New Computer?
What is fragmentation and what do I need to do about it?
These articles are designed to give the novice user a working vocabulary along with the knowledge to use some of these mysterious words.
In no way are any of these articles designed to answer specific computer issues. Please go to the forums to get answers to specific problems.
It has taken a lot of work to update this site to make it look more professional and modern. We've also changed a lot of things technically, behind the scenes. Many articles required extensive rewriting, as things have changed since they were originally written, and with the advent of Microsoft's newest operating system, Windows 10. In the interest of getting the new stuff up, many of the old articles are not yet ready. We'll be getting those back up as they are done.


Follow this guide to create your own bottled gardens with simple tools and accesible materials! This pamphlet includes a picture how to along with a brief history on terrariums and care tips. It's great for anyone that wishes to engage more with horticulture or loves plants.
Softcover, staple-bound, 20 pages, 4 inches x 7 inches, English, first edition, Hato Press, 2017.