text:
\section{Introduction}

The calculation of accurate potential energies of molecules and materials at affordable cost is at the heart of computational chemistry.
While state-of-the-art \textit{ab initio} electronic structure theories can yield highly accurate results, they are computationally too expensive for routine applications.
Density functional theory (DFT) is computationally cheaper and has thus enjoyed widespread applicability.
However, DFT is hindered by a lack of systematic improvability and from an uncertain quality for many applications.

In recent years, a variety of machine learning approaches has emerged which promise to mitigate the cost of highly accurate electronic structure methods while preserving accuracy. \cite{bartok_gaussian_2010, rupp_fast_2012,hansen_assessment_2013,ramakrishnan_big_2015, behler_perspective_2016, brockherde_bypassing_2017, schutt_schnet_2017, schutt_quantum-chemical_2017,  smith_ani-1_2017, mcgibbon_improving_2017, collins_constant_2018, fujikake_gaussian_2018, lubbers_hierarchical_2018,  nguyen_comparison_2018, welborn_transferability_2018, wu_moleculenet_2018, yao_tensormol-01_2018, cheng_universal_2019, cheng_regression_2019, christensen_operators_2019, grisafi_transferable_2019, smith_approaching_2019, unke_physnet_2019,
fabrizio_machine_2020, chen_ground_2020, christensen_fchl_2020, dick_machine_2020, liu_transferable_2020,  qiao_orbnet_2020,manzhos_machine_2020}
While these machine learning methods share similar goals, they differ in the representation of the molecules and in the machine learning methodology itself.
Here, we will focus on the molecular-orbital-based machine learning (MOB-ML) approach. \cite{welborn_transferability_2018, cheng_universal_2019, cheng_regression_2019}
The defining feature of MOB-ML is its framing of learning highly accurate correlation energies as learning a sum of orbital pair correlation energies. These orbital pair correlation energies can be individually  regressed with respect to a feature vector representing the interaction of the molecular orbital pairs.
Without approximation, it can be shown that such pair correlation energies add up to the correct total correlation energy for single-reference wave function methods.
Phrasing the learning problem in this manner has the  advantage that a given pair correlation energy, and, hence, a given feature vector, is independent of molecular size (after a certain size threshold has been reached) because of the inherent spatial locality of dynamic electron correlation.
Consequently, operating in such an orbital pair interaction framework converts the general extrapolation task of training on small molecules and predicting on large molecule into an interpolation task of training on orbital pairs in a small molecule and predicting on the same orbital pairs in a large molecule.

In this work, we address challenges introduced by operating in a vectorized molecular orbital pair interaction framework (Section~\ref{sec:theory}). We  show how changes to the feature design affect the performance and transferability of MOB-ML models within the same molecular family (Section~\ref{subsec:sec1}) and across molecular families (Sections~\ref{subsec:sec2}-\ref{subsec:sec3}). We probe these effects on relative- and total-energy predictions for organic and transition-metal containing molecules, and we investigate the applicability of MOB-ML to transition-state structures and non-covalent interactions.

\section{Theory}
\label{sec:theory}

MOB-ML predicts correlation energies based on information from the molecular orbitals. \cite{welborn_transferability_2018, cheng_universal_2019, cheng_regression_2019}
The correlation energy $E^\text{corr}$ in the current study is defined as the difference between the true total electronic energy and the Hartree--Fock (HF) energy for a given basis set. Without approximation, the correlation energy is expressed as a sum over correlation energy contributions from pairs of occupied orbitals $i$ and $j$, \cite{nesbet_brueckners_1958}
\begin{equation}
     E^\text{corr} = \sum_{ij} \epsilon_{ij}.
\end{equation}
Electronic structure theories offer different ways of approximating these pair correlation energies.
For example, the second-order M{\o}ller-Plesset perturbation theory  (MP2) correlation energy is \cite{moller_note_1934}
\begin{equation}
     \epsilon_{ij}^\text{MP2} = \sum_{ab} \frac{\left<ia||jb\right>^2}{F_{aa}+F_{bb} - F_{ii} - F_{jj}},
\end{equation}
where $a,b$ denote virtual orbitals, $F$ the Fock matrix in the molecular orbital basis, and $\left<ia||jb\right>$ the anti-symmetrized exchange integral.
We denote a general repulsion integral over the spatial coordinates $\mathbf{x}_1, \mathbf{x}_2$ of molecular orbitals $p,q,m,n$ following the chemist's notation as
\begin{equation}
\begin{split}
[\resizebox{!}{\CapLen}{$\kappa$}^{pq}]_{mn} &= \left< pq | mn \right> \\&= \int d \mathbf{x}_1 d \mathbf{x}_2 p(\mathbf{x}_1)^* q(\mathbf{x}_1) \frac{1}{|\mathbf{x}_1 - \mathbf{x}_2|} m(\mathbf{x}_2)^* n(\mathbf{x}_2).
\end{split}
\end{equation}
The evaluation of correlation energies with post-HF methods like MP2 or coupled-cluster theory (including CCSD(T)) involves  computations that  exceed the cost of HF theory by orders of magnitude.
By contrast, MOB-ML predicts the correlation energy at negligible cost by machine-learning the map
\begin{equation}
    \epsilon_{ij} \approx \epsilon^\text{ML}(\mathbf{f}_{ij}),
\end{equation}
where $\mathbf{f}_{ij}$ denotes the feature vector into which information on the molecular orbitals is compiled.

Following our previous work,\cite{cheng_universal_2019} we define a canonical order of the orbitals $i$ and $j$ by rotating them into gerade and ungerade combinations (see Eq.~(7) in Ref.~\onlinecite{cheng_universal_2019}), creating the rotated orbitals $\tilde{i}$ and $\tilde{j}$.
The feature vector $\mathbf{f}_{ij}$ assembles information on the molecular orbital interactions: (i)  Orbital energies of the valence-occupied and valence-virtual orbitals $F_{pp}$, (ii)
mean-field interaction energy of valence-occupied and valence-occupied orbitals and of valence-virtual and valence-virtual orbitals $F_{pq}$,
(iii) Coulomb interaction of valence-occupied and valence-occupied orbitals, of valence-occupied and valence-virtual orbitals, and valence-virtual and valence-virtual orbitals $[\resizebox{!}{\CapLen}{$\kappa$}^{pp}]_{qq}$, and
(iv) exchange interaction of valence-occupied and valence-occupied orbitals, of valence-occupied and valence-virtual orbitals, and valence-virtual and valence-virtual orbitals $[\resizebox{!}{\CapLen}{$\kappa$}^{pq}]_{pq}$.
We note that all of these pieces of information enter either the MP2 or the MP3 correlation energy expressions, which helps to motivate their value within our machine learning framework.
We remove  repetitive information from the feature vector and  separate the learning problem into the cases where (i) $i\ne j$ where we employ the feature vector as defined in Eq.~(\ref{eq:off-diag_f}) and (ii) $i=j$ where we employ the feature vector as defined in Eq.~(\ref{eq:diag_f}),
\onecolumngrid
\begin{equation}
\label{eq:off-diag_f}
\begin{split}
    \mathbf{f}_{ij} =& \{ \{F_{\tilde{i}\tilde{i}}, F_{\tilde{i}\tilde{j}}, F_{\tilde{j}\tilde{j}}\}, \{F_{\tilde{i}k}\}, \{F_{\tilde{j}k}\}, \{F_{ab}\}, \\
    &   \{[\resizebox{!}{\CapLen}{$\kappa$}^{\tilde{i}\tilde{i}}]_{\tilde{i}\tilde{i}}, [\resizebox{!}{\CapLen}{$\kappa$}^{\tilde{i}\tilde{i}}]_{\tilde{j}\tilde{j}}, [\resizebox{!}{\CapLen}{$\kappa$}^{\tilde{j}\tilde{j}}]_{\tilde{j}\tilde{j}}\}, \{[\resizebox{!}{\CapLen}{$\kappa$}^{\tilde{i}\tilde{i}}]_{kk}\}, \{[\resizebox{!}{\CapLen}{$\kappa$}^{\tilde{j}\tilde{j}}]_{kk}\}, \{[\resizebox{!}{\CapLen}{$\kappa$}^{\tilde{i}\tilde{i}}]_{aa}\}, \{[\resizebox{!}{\CapLen}{$\kappa$}^{\tilde{j}\tilde{j}}]_{aa}\}, \{[\resizebox{!}{\CapLen}{$\kappa$}^{aa}]_{bb}\} , \\
    &  \{[\resizebox{!}{\CapLen}{$\kappa$}^{\tilde{i}\tilde{j}}]_{\tilde{i}\tilde{j}}\}, \{[\resizebox{!}{\CapLen}{$\kappa$}^{\tilde{i}k}]_{\tilde{i}k}\}, \{[\resizebox{!}{\CapLen}{$\kappa$}^{\tilde{j}k}]_{\tilde{j}k}\}, \{[\resizebox{!}{\CapLen}{$\kappa$}^{\tilde{i}a}]_{\tilde{i}a}\}, \{[\resizebox{!}{\CapLen}{$\kappa$}^{\tilde{j}a}]_{\tilde{j}a}\}, \{[\resizebox{!}{\CapLen}{$\kappa$}^{ab}]_{ab}\} \}, \\
\end{split}
\end{equation}{}
\begin{equation}
\label{eq:diag_f}
\begin{split}
    \mathbf{f}_{i} =& \{ F_{ii}, \{F_{ik}\}, \{F_{ab}\},  [\resizebox{!}{\CapLen}{$\kappa$}^{ii}]_{ii}, \{[\resizebox{!}{\CapLen}{$\kappa$}^{ii}]_{kk}\},  \{[\resizebox{!}{\CapLen}{$\kappa$}^{ii}]_{aa}\}, \{[\resizebox{!}{\CapLen}{$\kappa$}^{aa}]_{bb}\},  \{[\resizebox{!}{\CapLen}{$\kappa$}^{ik}]_{ik}\}, \{[\resizebox{!}{\CapLen}{$\kappa$}^{ia}]_{ia}\}, \{[\resizebox{!}{\CapLen}{$\kappa$}^{ab}]_{ab}\} \}. \\
\end{split}
\end{equation}{}
\twocolumngrid
\noindent
Here, the index $k$ denotes an occupied orbital other than $i$ and $j$.
For blocks in the feature vector that include more than one element, we specify a canonical order of the feature vector elements.
In our previous work,\cite{cheng_universal_2019} this order was given by the sum of the Euclidean distances between the centroids of orbital $\tilde{i}$ and $p$ and between the centroids of orbital $\tilde{j}$ and $p$.
In the current work, we introduce a different strategy to sort the feature vector elements (Section~\ref{sec:sorting}), we modify the protocol with which we obtain the feature vector elements associated with $\tilde{i}, \tilde{j}$ (Section~\ref{sec:invariance}), and we revise our feature vector elements to ensure size consistency (Section~\ref{sec:sizeconsistency}).
We provide a conceptual description of the changes to the feature set below and we give the full definition of the feature vector elements and the criteria according to which the feature elements are ordered in Tables~S3--S6 in the Supporting information.

\subsection{Defining importance of feature vector elements}
\label{sec:sorting}
Careful ordering of the elements of the feature vector blocks in necessary in the current work  because Gaussian process regression (GPR) is  sensitive to permutation of the feature vector elements.
Furthermore, the application of a Gaussian process requires that the feature vectors be of fixed length. \cite{rasmussen_gaussian_2006}

Given the near-sighted nature of dynamical electron correlation, it is expected that only a limited number of orbital-pair interactions are important to predict the pair correlation energy with MOB-ML.
To construct the fixed-length feature vector, a cutoff criterion must be introduced.\cite{welborn_transferability_2018}
For some feature vector elements, a robust definition of importance is straight-forward.
The spatial distance between the orbital centroids $i$ and $a$ is, for example, a reliable proxy for the importance of the feature vector elements $\{[\resizebox{!}{\CapLen}{$\kappa$}^{ii}]_{aa}\}$ of the feature vector $ \mathbf{f}_{i}$.
However, the definition of importance is less straightforward for feature vector elements that involve more than two indices.
The most prominent example is the $\{[\resizebox{!}{\CapLen}{$\kappa$}^{ab}]_{ab}\}$ feature vector block of $\mathbf{f}_{ij}$, which contains the exchange integrals between the valence-virtual orbitals $a$ and $b$ and which should be sorted with respect to the importance of these integrals for the prediction of the pair correlation energy $\epsilon_{ij}$.
It is non-trivial to define a spatial metric which defines the importance of the feature vector elements $\{[\resizebox{!}{\CapLen}{$\kappa$}^{ab}]_{ab}\}$ to predict the pair correlation energy $\epsilon_{ij}$; instead, we employ the the MP3 approximation for the pair correlation energy,
\begin{equation}
\label{eq:mp3}
\begin{split}
    \epsilon_{ij}^\text{MP3} &= \frac{1}{8} \sum_{abcd}
    \left(t_{ij}^{ab}\right)^* \left<ab||cd\right> t_{ij}^{cd} +
    \frac{1}{8} \sum_{klab}
     \left(t_{ij}^{ab}\right)^* \left<kl||ij\right> t_{kl}^{ab}
    \\&-
    \sum_{kabc}
     \left(t_{ij}^{ab}\right)^* \left<kb||ic\right> t_{kj}^{ac}, \\
\end{split}
\end{equation}
where $t_{ij}^{ab}$ denotes the T-amplitude.
Although we operate in a local molecular orbital basis,  the canonical formulae are used to define the importance criterion; if we consider orbital localization as a perturbation (as in Kapuy--M{\o}ller--Plesset theory \cite{kapuy_application_1983}), the canonical expression is the leading order term.
The term we seek to attach an importance to, $\{[\resizebox{!}{\CapLen}{$\kappa$}^{ab}]_{ab}\}$, appears in the first term of  Eq.~(\ref{eq:mp3}) and all integrals necessary to compute this term are readily available as (a combination of) other feature elements, i.e., we do not incur any additional significant computational cost to obtain the importance of the feature vector elements.

The way in which we determine the importance of the $\{[\resizebox{!}{\CapLen}{$\kappa$}^{ab}]_{ab}\}$ elements here is an example of a more general strategy that we employ, in which the importance is assigned according to the lowest-order perturbation theory in which the features first appear in.
Similar considerations have to be made for each feature vector block, all of which are specified in detail in Tables~S3 and S4 in the Supporting Information.

\subsection{Orbital-index permutation invariance}
\label{sec:invariance}

The Fock, Coulomb, and exchange matrix elements that comprise MOB features are naturally invariant to rotation and translation of the molecule.
However, some care is needed to ensure that these invariances are not lost in the construction of symmetrized MOB features.
In particular, rotating the valence-occupied orbitals into gerade and  ungerade combinations leads to an orbital-index permutation variance for energetically degenerate orbitals $i,j$ because the sign of the feature vector elements $M_{\tilde{j}p}$,
\begin{equation}
 M_{\tilde{j}p} = \frac{1}{\sqrt{2}} M_{ip} - \frac{1}{\sqrt{2}} M_{jp},
\end{equation}
depends on the arbitrary assignment of the indices $i$ and $j$.
To rectify this issue, we include the absolute value of the generic feature vector element $M$ in the feature vector instead of the signed value,
\begin{equation}
 M_{\tilde{j}p} = \left| \frac{1}{\sqrt{2}} M_{ip} - \frac{1}{\sqrt{2}} M_{jp} \right|,
\end{equation}
where $M_{\tilde{j}p}$ may be $F_{\tilde{j}p}$, $[\resizebox{!}{\CapLen}{$\kappa$}^{\tilde{j}\tilde{j}}]_{pp}$, or $[\resizebox{!}{\CapLen}{$\kappa$}^{\tilde{j}p}]_{\tilde{j}p}$.
The corresponding equation,
\begin{equation}
 M_{\tilde{j}p} = \frac{1}{\sqrt{2}} M_{ip} + \frac{1}{\sqrt{2}} M_{jp},
\end{equation}
is already orbital-index permutation invariant because we chose $M_{pq}$ ($p\ne q$) to be positive. \cite{cheng_universal_2019}

\subsection{Size consistency}
\label{sec:sizeconsistency}

Size consistency is the formal property by which the energy of two isolated molecules equals the sum of their dimer upon infinite separation.\cite{bartlett_many-body_1981, SzaboSizeConsistent}
In the context of MOB-ML, satisfaction of this property requires that the contributions from the diagonal feature vectors are not affected by distant, non-interacting molecules and that
\begin{equation}
    \epsilon^{ML}(\mathbf{f}_{ij})=0 \text{ for } r_{ij} = \infty
\end{equation}
for contributions from the off-diagonal feature vectors.
To ensure that MOB-ML exhibits size-consistency without the need for explicit training on the dimeric species, the following modifications to the feature vectors are made.

\paragraph{Diagonal feature vector.}
The feature vector as defined in Eq.~(\ref{eq:diag_f}) contains three blocks whose elements are independent of orbital $i$, $\{F_{ab}\}$, $\{[\resizebox{!}{\CapLen}{$\kappa$}^{aa}]_{bb}\}$, and $\{[\resizebox{!}{\CapLen}{$\kappa$}^{ab}]_{ab}\}$. The magnitude of these feature vector elements does not decay with an increasing distance between orbital $i$ localized on molecule $I$ and an orbital (for example, $a$) localized on molecule $J$.
To address this issue, we multiply these feature vector elements by their estimated importance (see Section~\ref{sec:sorting}) so that they decay smoothly to zero.
The other feature vector elements decay to zero when the involved orbitals are non-interacting albeit at different rates; we take the cube of feature vector elements of the type $\{[\resizebox{!}{\CapLen}{$\kappa$}^{pp}]_{qq}\}$ to achieve a similar decay rate for all feature vector elements in the short- to medium-range which facilitates machine learning.

\paragraph{Off-diagonal feature vector.}
We modify the off-diagonal feature vector such that $\mathbf{f}_{ij}=\mathbf{0} \text{ for }r_{ij} = \infty$ by first applying the newly introduced  changes  for $\mathbf{f}_{i}$ also for $\mathbf{f}_{ij}$.
Further action is needed for the off-diagonal case because many feature vector elements do not decay to zero when the distance between $i$ and $j$ is large due to rotation of the orbitals into a gerade and an ungerade combination, e.g., $F_{\tilde{i}k}=  \left| \frac{1}{\sqrt{2}} F_{ik} + \frac{1}{\sqrt{2}} F_{jk} \right| = \left| \frac{1}{\sqrt{2}} F_{ik} \right| \text{for } r_{ij}=\infty, r_{jk}=\infty$.
As a remedy, we apply a damping function of the form $ \frac{1}{1+\frac{1}{6}(r_{ij}/r_0)^6}$ to each feature vector element.
The form of this damping function is inspired by the semi-classical limit of the MP2 expression as it is also used for semi-classical dispersion corrections. \cite{grimme_dispersion_2016}
The damping radius, $r_0$, needs to be sufficiently large as to not interfere with machine learning at small $r_{ij}$.
If a damping radius close to zero would be chosen, all off-diagonal feature vectors would be zero which nullifies the information content; however,  the damping radius $r_0$ also should not be too large as size-consistency has to be fully learned until the off-diagonal feature vector is fully damped to zero. Therefore, we employ a damping radius in the intermediate-distance regime and we empirically found $r_0 = 5.0$~Bohr to work well.

Lastly, we enforce that $\epsilon^{ML}(\mathbf{0})=0$.
The MOB features are engineered to respect this limit and would, for example, in a linear regression with a zero intercept trivially predict a zero-valued pair correlation energy without any additional training.
However, the Gaussian process regression we apply in this work does not trivially yield a zero-valued pair correlation energy for a zero-valued feature vector.
In the case that a training set does not include examples of zero-valued feature vectors, we need to include zero-valued feature vectors and zero-valued pair correlation energies in training to ensure that $\epsilon^{ML}(\mathbf{0})=0$.
For no model trained in the current study were more than 5\% zero-valued feature vectors included.

The resulting MOB-ML model leads to size consistent energy predictions to the degree to which the underlying MO generation is.
It is not required that the dimer is explicitly part of training the MOB-ML model to obtain this result.
The detailed definition of each feature vector block is summarized in Tables~S5 and S6. We apply the feature set defined in Tables~S5 and S6 consistently in this work.

\section{Computational details}

We present results for five different data sets:
(i) a series of alkane molecules,
(ii) the potential energy surface of the malonaldehyde molecule,
(iii) a thermalized version of the QM7b and the GDB13 data set (i.e., QM7b-T and GDB13-T), \cite{cheng_thermalized_2019}
(iv) a set of backbone-backbone interactions (BBI) \cite{burns_biofragment_2017}, and
(v) a thermalized version of a subset of mononuclear, octahedral transition metal complexes put forward by Kulik and co-workers \cite{nandy_strategies_2018}.
We refer to the Supporting Information Section II for a description how the structures were obtained or generated.
All generated structures are available in Ref.~\onlinecite{caltech_data}.

The features for all structures were generated with the \textsc{entos qcore} \cite{manby_entos_2019} package.
The feature generation is based on a HF calculation applying a cc-pVTZ\cite{dunning_gaussian_1989} basis for the elements H, C, N, O, S, and Cl. We apply a def2-TZVP basis set \cite{weigend_balanced_2005} for all transition metals.
The HF calculations were accelerated with density fitting for which we applied the corresponding cc-pVTZ-JKFIT\cite{weigend_fully_2002} and def2-TZVP-JKFIT \cite{weigend_accurate_2006} density fitting bases.
Subsequently, we localized the valence-occupied and the valence-virtual molecular orbitals with the Boys--Foster localization scheme \cite{boys_construction_1960,foster_canonical_1960} or with the intrinsic bond orbital (IBO) localization scheme \cite{knizia_intrinsic_2013}.
We implemented a scheme to localize the valence-virtual orbitals with respect to the Boys--Foster function (for details on this implementation, see Section II in the Supplementary Information).
We applied the Boys--Foster localization scheme for the data sets (i), (iii), (iv), and (v) for valence-occupied and valence-virtual molecular orbitals.
IBO localization for valence-occupied and valence-virtual molecular orbitals led to better results for data set (ii).

The resulting orbitals are imported into the Molpro 2018.0 \cite{werner_molpro_2018,werner_molpro_2020} package via the \texttt{matrop} functionality to generate the non-canonical MP2 \cite{werner_fast_2003} or CCSD(T) \cite{scuseria_comparison_1990, hampel_local_1996, schutz_local_2000} pair correlation energies  with the same orbitals we applied for the feature generation.
These calculations are accelerated with the resolution of the identity approximation.
The frozen-core approximation is invoked for all correlated calculations.

We follow the machine learning protocol outlined in   previous work \cite{cheng_universal_2019} to train the MOB-ML models.
In a first step, we perform MOB feature selection by evaluating the mean decrease of accuracy in a random forest regression in the \textsc{Scikit-learn} v0.22.0 package \cite{pedregosa_scikit-learn_2011}.
We then regress the diagonal and off-diagonal pair correlation energies separately with respect to the selected features in the \textsc{GPy} 1.9.6 software package. \cite{gpy_gpy_2012}
We employ the Mat\'ern 5/2 kernel with white noise regularization\cite{rasmussen_gaussian_2006}.
We minimize the negative log marginal likelihood objective with respect to the kernel hyperparameters with a scaled conjugate gradient scheme for 100 steps and then apply the BFGS algorithm until full convergence.
As indicted in the results, both random-sampling and active-learning strategies were employed for the selection of molecules in the training data sets.
In the active-learning strategy, we use a previously trained MOB-ML model to evaluate the Gaussian process variance for each molecule,
and then include the points with the highest variance in the training data set, as outlined in Ref.~\onlinecite{shapeev_active_2020}.
To estimate the Gaussian process variance for each molecule, it was assumed the variances per molecular orbital pair are mutually independent.

\section{Results}

\subsection{Transferability within a molecular family}
\label{subsec:sec1}

We first examine the effect of the feature vector generation strategy on the transferability of MOB-ML models within a molecular family.
To this end, we revisit our alkane data set \cite{cheng_universal_2019} which contains 1000 ethane and 1000 propane geometries as well as 100 butane and 100 isobutane geometries.
We perform the transferability test outlined in Ref.~\onlinecite{cheng_universal_2019}, i.e., training a MOB-ML model on correlation energies for 50 randomly chosen ethane geometries and 20 randomly chosen propane geometries to predict the correlation energies for the 100 butane and 100 isobutane geometries (see Figure~\ref{figure:figure1}).
\begin{figure}[htbp]
\includegraphics[width=\columnwidth]{figures/figure1.png}
\caption{
Errors in the predicted correlation energies with respect to the CCSD(T) reference values for butane and isobutane. The bar attached to each prediction error indicates the associated Gaussian process variance. The MOB-ML model used for these predictions was trained on 50 ethane and 20 propane molecules. The gray shaded area corresponds to the region where the error is smaller than chemical accuracy (1~kcal/mol).
}
\label{figure:figure1}
\end{figure}
This transferability test was repeated with 10000 different training data sets (each consisting of data for 50 ethane molecules and 20 propane molecules) to assess the training set dependence of the MOB-ML models.
As suggested in Ref.~\onlinecite{chen_ground_2020}, we consider various performance metrics to assess the prediction accuracy of the MOB-ML models: (i) the mean error (ME, Eq.~(S3)), (ii) the mean absolute error (MAE, Eq.~(S4)), (iii) the maximum absolute error (MaxAE, Eq.~(S5)), and (iv) the mean absolute relative error (MARE, Eq~(S6)) which applies a global shift setting the mean error to zero.
We report the minimum, peak, and maximum encountered MAREs in Table~\ref{tab:table1} alongside literature values obtained in our previous work \cite{cheng_universal_2019}, by Dick \textit{et al.}, \cite{dick_machine_2020} and by Chen \textit{et al}. \cite{chen_ground_2020} The MEs, MAEs, and MaxAEs are reported in Figure~S1.
\begin{center}
\begin{table}[htbp]
    \centering
    \begin{tabular}{p{1.6cm}p{2.15cm}p{0.6cm}p{0.6cm}p{0.6cm}p{0.6cm}p{0.6cm}p{0.6cm}}
         \hline \hline
         Method & Feature set & \multicolumn{6}{c}{MARE} \\
                &             & \multicolumn{3}{c}{Butane} & \multicolumn{3}{c}{Isobutane}\\
                & & min & peak & max & min & peak & max  \\
         \hline
         NeuralXC\cite{dick_machine_2020} & --- & & 0.15 & & & 0.14 & \\
         DeepHF\cite{chen_ground_2020} & --- & 0.06& 0.11 & 0.43 & 0.07 & 0.13 & 0.53 \\
         \hline
         MOB-ML & Ref.~\onlinecite{cheng_universal_2019} & & 0.20 & & & 0.21 & \\
         & \textbf{this work } & \textbf{0.06} & \textbf{0.11} & \textbf{0.19 } & \textbf{0.06} & \textbf{0.10} & \textbf{0.19}\\
         \hline \hline
    \end{tabular}
    \caption{Comparison of the minimum, peak, and maximum  mean absolute error after global shift (MARE) in~kcal/mol for the prediction of CCSD(T) correlation energies for butane and isobutane obtained with different methods.}
    \label{tab:table1}
\end{table}
\end{center}
In general, MOB-ML as well as NeuralXC\cite{dick_machine_2020} and DeepHF\cite{chen_ground_2020} produce MAREs well below chemical accuracy for correlation energies of butane and isobutane when trained on correlation energies of ethane and propane.
Updating the feature vector generation strategy for MOB-ML results in the best peak MAREs for butane as well as for isobutane which are 0.11~kcal/mol and 0.10~kcal/mol, respectively.
As in our previous work, \cite{cheng_universal_2019} we note that the total correlation energy predictions may be shifted  with respect to the reference data so that the MEs for MOB-ML range from $-0.92$ to 2.70~kcal/mol for butane and from $-0.18$ to 1.02~kcal/mol for isobutane (see also Figure~S1).
This shift is strongly training-set dependent, which was also observed for results obtained with DeepHF \cite{chen_ground_2020}.

The results highlight that this is an extrapolative transferability test.
A considerable advantage of applying GPR in practice is that each prediction is accompanied by a Gaussian process variance which, in this case, indicates that we are in an extrapolative regime (see Figure~\ref{figure:figure1}).
Extrapolations might be associated with quality degradation which we see, most prominently, for the mean error in butane.
By contrast, other machine learning approaches like neural networks are less clear in terms of whether the predictions are in an interpolative or extrapolative regime.\cite{hirschfeld_uncertainty_2020}
By including the butane molecule with the largest variance in the training set (which then consists of 50 ethane, 20 propane, and 1 butane geometries) we  reduce the ME from 0.78 to 0.25, MAE from 0.78 to 0.26, MaxAE from 1.11 to 0.51, and the MARE from 0.11 to 0.09~kcal/mol for butane (see Figure~S2).
These results directly illustrate that MOB-ML can be systematically improved by including training data that is more similar to the test data; the improved confidence of the prediction is then also directly reflected in the associated Gaussian process variances.

As a second example, we examine the transferability of a MOB-ML model trained within a basin of a potential energy surface to the transition-state region of the same potential energy surface.
We chose malonaldehyde for this case study as it has also been explored in previous machine learning studies \cite{brockherde_bypassing_2017}.
We train a MOB-ML model on 50 thermalized malonaldehyde structures which all have the property that $d$(O$^1$--H) + $d$(O$^2$--H) > 0.4~\AA\ (where $d$ denotes the distance between the two nuclei) which ensures that we are sampling from the basins.
We then apply this trained model to predict the correlation energies for a potential energy surface mapping out the hydrogen transfer between the two oxygen atoms (see Figure~\ref{figure:figure2}).
MOB-ML produces an accurate potential energy surface for the hydrogen transfer in malonaldehyde only from information on the basins (compare left and middle left panel of Figure~\ref{figure:figure2}).
The highest encountered errors on the minimum potential energy path are smaller than 1.0~kcal/mol.
Unsurprisingly, the predicted minimum energy structure ($d$(O$^1$--H) = 1.00~\AA, $d$(O$^2$--H) = 1.63~\AA) is very similar to the reference minimum energy structure ($d$(O$^1$--H) = 1.00~\AA, $d$(O$^2$--H) = 1.64~\AA).
Strikingly, the predicted energy of 2.65~kcal/mol at the saddle point at $d$(O$^1$--H) = $d$(O$^2$--H) = 1.22~\AA\ differs from the reference energy by only 0.35~kcal/mol, although the MOB-ML model was not trained on any transition-state like structures.
The highest errors are encountered in the high-energy regime and this region is also associated with the highest Gaussian process variance indicating low confidence in the predictions (compare middle right and right panel of Figure~\ref{figure:figure2}).
The Gaussian process variance reflects the range of structures the MOB-ML model has been trained in and highlights again that we did not include transition-state-like structures in the training.

\onecolumngrid
\begin{center}
\begin{figure}[htbp]
\includegraphics[width=\textwidth]{figures/figure2.png}
\caption{
Relative energies obtained with MP2/cc-pVTZ (left panel), relative energies predicted with MOB-ML (middle left panel), the difference between the MOB-ML prediction and the reference data (middle right panel), and the Gaussian process variance (right panel) for the proton transfer in malonaldehyde as a function of the distance of the proton from the two oxygen atoms.
}
\label{figure:figure2}
\end{figure}
\end{center}
\twocolumngrid

\subsection{Transferability across organic chemistry space}
\label{subsec:sec2}

The Chemical Space Project\cite{reymond_chemical_2015} computationally enumerated all possible organic molecules up to a certain number of atoms, resulting in the GDB databases.\cite{blum_970_2009}
In this work, we examine thermalized subsets \cite{cheng_universal_2019} of the GDB13 data set \cite{blum_970_2009} to investigate the transferability of MOB-ML models across organic chemistry space.
The application of thermalized sets of molecules has the advantage that we can study the transferability of our models for chemical and conformational degrees of freedom at the same time.
To test the transferability of MOB-ML across chemical space, we train our models on a thermalized set of seven and fewer heavy-atom molecules (also known as QM7b-T \cite{cheng_universal_2019}) and then we test the prediction accuracy on a QM7b-T test set and on a thermalized set of molecules with thirteen heavy atoms (GDB13-T \cite{cheng_universal_2019}; see also Section V in the Supporting Information), as also outlined in our previous work. \cite{cheng_universal_2019, cheng_regression_2019}

We first investigate the effect of changing the feature vector generation protocol on the QM7b-T$\rightarrow$QM7b-T prediction task (see Figure~\ref{figure:figure3}).
\begin{figure}[htbp]
\includegraphics[width=\columnwidth]{figures/figure3.png}
\caption{
Comparison of the prediction mean absolute errors of total correlation energies for QM7b-T test molecules as a function of the number of QM7b-T molecules chosen for training for different machine learning models: MOB-ML as outlined in Ref.~\onlinecite{cheng_universal_2019} (orange circles), MOB-ML as outlined in this work with random sampling (green circles), and MOB-ML as outlined in this work with active sampling. The green shaded area corresponds to the 90\% confidence interval for the predictions obtained from 50 random samples of the training data.
}
\label{figure:figure3}
\end{figure}
In Ref.~\onlinecite{cheng_universal_2019}, we found that training on about 180 structures is necessary to achieve a model with an MAE below 1~kcal/mol.
The FHCL method yields an MAE below 1~kcal/mol when training on about 800 structures \cite{christensen_fchl_2020} and the DeepHF method already exhibits an MAE below 1~kcal/mol when training on their smallest chosen training set which consists of 300 structures (MAE=0.79~kcal/mol). \cite{chen_ground_2020}
The refinements in the current work reduce the number of required training structures to reach chemical accuracy to about 100 structures when sampling randomly.
This number is, however, strongly training set dependent.
We can remove the training-set dependence by switching to an active learning strategy where we can achieve an MAE below 1~kcal/mol reliably with about 70 structures.
In general, the MAE obtained with the active learning strategy is comparable to the smallest MAEs obtained with random sampling strategies.
This has the advantage that a small number of reference data can be generated in a targeted manner.

In general, our aim is to obtain a machine learning model which reliably predicts broad swathes of chemical space.
For an ML model to be of practical use, it has to be able to describe out-of-set molecules of different sizes to a similar accuracy when accuracy is measured size-intensively. \cite{SzaboSizeConsistent}
We probe the ability of MOB-ML to describe out-of-set molecules with a different number of electron pairs by applying a model trained on correlation energies for QM7b-T molecules to predict correlation energies for GDB13-T.
We collect the best results published for this transfer test in the literature in Figure~\ref{figure:figure4}.
\begin{figure}[htbp]
\includegraphics[width=\columnwidth]{figures/figure4.png}
\caption{
Comparison of the prediction mean absolute errors of total correlation energies for GDB13-T molecules as a function of the number of QM7b-T molecules chosen for model training for different machine learning models: MOB-ML as outlined in this work with random sampling (green circles),
MOB-ML with a single GPR\cite{cheng_universal_2019} (orange circles), MOB-ML with RCR/GPR \cite{cheng_regression_2019} (brown circles), DeepHF\cite{chen_ground_2020} (red squares), FHCL18\cite{christensen_fchl_2020} (purple squares).
The green shaded area corresponds to the 90\% confidence interval for the predictions obtained from 50 random samples of the training data.
}
\label{figure:figure4}
\end{figure}
Our previous best single GPR model achieved an MAE of 2.27~kcal/mol when trained on 220 randomly chosen structures. \cite{cheng_universal_2019}
The modifications in the current work now yield a single GPR model which achieves an MAE of 1.47--1.62~kcal/mol for GDB13-T when trained on 220 randomly chosen QM7b-T structures.
Strikingly, MOB-ML outperforms machine learning models trained on thousands of molecules like our RCR/GPR model and FHCL18 \cite{christensen_fchl_2020}. The current MOB-ML results are of an accuracy that is similar to the best reported results from DeepHF (an MAE of 1.49 kcal/mol);\cite{chen_ground_2020}
however, MOB-ML only needs to be trained on about 3\% of the molecules in the QM7b data set while DeepHF is trained on 42\% to obtain comparable results (MAE of 1.52~kcal/mol for 3000 training structures). The best reported result for DeepHF (MAE of 1.49 kcal/mol) was obtained by training on 97\% of the molecules of the QM7b data set.
We attribute the excellent transferability of MOB-ML to the fact that it focuses on the prediction of orbital-pair contributions,
thereby reframing an extrapolation problem into an interpolation problem when training machine learning models on small molecules and testing them on large molecules.
The pair correlation energies predicted for QM7b-T and for GDB13-T span a very similar range (0 to $-20$~kcal/mol), and they are predicted with a similar Gaussian process variance (see Figure~S5) which we would expect in an interpolation task.
The final errors for GDB13-T are larger than for QM7b-T, because the total correlation energy is size-extensive; however, the size-intensive error per electron pair spans a comparable range for QM7b-T and for GDB13-T (see Figure~S4).
This presents a significant advantage of MOB-ML over machine learning models which rely on a whole-molecule representation and creates the opportunity to study molecules of a size that are beyond the reach of accurate correlated wave function methods.

Most studies in computational chemistry require accurate relative energies rather than accurate total energies.
Therefore, we also assess the errors in the relative energies for the sets of conformers for each molecule in the QM7b-T and in the GDB13-T data sets obtained with MOB-ML with respect to the reference energies (see Figure~\ref{figure:figure5}).
\begin{figure}[htbp]
\includegraphics[width=\columnwidth]{figures/figure5.png}
\caption{
Prediction mean absolute errors for relative correlation energies  as a function of the number of QM7b-T molecules chosen for model training for QM7b-T (blue circles) and for GDB13-T (orange crosses).
The blue and orange shaded areas correspond to the 90\% confidence interval for the predictions obtained from 50 random samples of the training data.
The gray shaded area corresponds to the region where the error is smaller than chemical accuracy (1~kcal/mol).
}
\label{figure:figure5}
\end{figure}
We emphasize that MOB-ML is not explicitly trained to predict conformer energies, and we include at most one conformer for each molecule in the training set.
Nevertheless, MOB-ML produces on average chemically accurate relative conformer energies for QM7b-T when trained on correlation energies for only 30 randomly chosen molecules (or 0.4\% of the molecules) in the QM7b set.
We obtain chemically accurate relative energies for the GDB13-T data set when training on about 100 QM7b-T molecules.
The prediction accuracy improves steadily when training on more QM7b-T molecules reaching a mean MAE of 0.43~kcal/mol for the relative energies of the rest of the QM7b-T set and of 0.77~kcal/mol for the GDB13-T set.

We now present the first reported test of MOB-ML for non-covalent interactions in large molecules.
To this end, we examine the backbone-backbone interaction (BBI) data set \cite{burns_biofragment_2017} which was designed to benchmark methods for the prediction of interaction energies encountered within protein fragments.
Using the implementation of MOB-ML described here and using only 20 randomly selected QM7b-T molecules for training, the method achieves a mean absolute error of 0.98~kcal/mol for the BBI data set (see Figure~\ref{figure:figure6}).
\begin{figure}[htbp]
\includegraphics[width=\columnwidth]{figures/figure6.png}
\caption{
Top panel: Errors in predictions were made with a MOB-ML model trained on 20 randomly selected QM7b-T molecules with FS~3 with respect to reference MP2/cc-pVTZ interaction energies for the BBI data set.
Bottom panel:
Errors in predictions were made with a MOB-ML model trained on 20 randomly selected QM7b-T molecules and augmented with the 2 BBI data points with the largest variance (orange circles) with respect to reference MP2/cc-pVTZ interaction energies.
The bar attached to each prediction error indicates the associated Gaussian process variance.
The gray shaded area corresponds to the region where the error is smaller than chemical accuracy (1~kcal/mol).
}
\label{figure:figure6}
\end{figure}
However, these predictions are uncertain as indicated by the large Gaussian process variances associated with these data points which strongly suggested that we are now, as expected, in an extrapolative regime.
We further improve the predictive capability of MOB-ML by augmenting the MOB-ML model with  data from the BBI set.
Specifically, we can draw on an active learning strategy and consecutively include data points until all uncertainties are below 1~kcal/mol which in this case corresponds to only two data points.
This reduces the MAE to 0.28~kcal/mol for the remaining 98 data points in the BBI set.
Including more reference data points would further improve the performance for this specific data set. However, this is not the focus of this work.
Instead, we simply emphasize that MOB-ML is a clearly extensible strategy to accurately predict energies for large molecules and non-covalent intermolecular interactions while providing a useful estimation of confidence.

\subsection{Transition-metal complexes}
\label{subsec:sec3}

We finally present the first application of MOB-ML to transition-metal complexes.
To this end, we train a MOB-ML model on a thermalized subset of mononuclear, octahedral transition-metal complexes  introduced by Kulik and co-workers \cite{nandy_strategies_2018} which we denote as TM-T.
The chosen closed-shell transition-metal complexes feature different transition metals (Fe, Co, Ni) and ligands.
The ligands span the spectrochemical series from weak-field (e.g., thiocyanate) over to strong-field (e.g., carbonyl) ligands.
We see in Figure~\ref{figure:figure7} that the learning behaviour between TM-T and QM7b-T is similar when the error is measured per valence-occupied orbital.
\begin{figure}[h]
\includegraphics[width=\columnwidth]{figures/figure7.png}
\caption{
 Learning curve for the prediction of MP2 correlation energies per valence-occupied orbital for transition metal complexes (TM-T) and for QM7b-T as a function of the number of structures the MOB-ML model was trained on.
}
\label{figure:figure7}
\end{figure}
These results demonstrate that MOB-ML formalism can be straightforwardly applied outside of the organic chemistry universe without additional modifications.
It is particularly notable that the learning efficiency for TM-T is comparable to that for QM7b-T, as seen in the relatively simple organic molecules in QM7b-T (Fig.~\ref{figure:figure7}).
We note that whereas MP2 theory is not expected to be fully quantitative for transition metal complexes, \cite{weymuth_new_2014, husch_calculation_2018}
it provides a demonstration of the learning efficiency of MOB-ML for transition-metal complexes in the current example;
and as previously demonstrated, MOB-ML learns other correlated wave function methods with similar efficiency.\cite{welborn_transferability_2018, cheng_universal_2019}

\section{Conclusions}

Molecular-orbital-based machine learning (MOB-ML) provides a general framework to learn correlation energies at the cost of  molecular orbital generation.
In this work, we  demonstrate that preservation of physical symmetries and constraints leads to machine-learning methods with greater learning efficiency and transferability.
Exploiting physical principles like size consistency and energy invariances  not only leads to a conceptually more satisfying method, but it also leads to substantial improvements in prediction errors for different data sets covering total and relative energies for thermally accessible organic and transition-metal containing molecules, non-covalent interactions, and transition-state energies.
With the modifications presented in the current work,
MOB-ML is shown to be highly data efficient, which is important due to the high computational cost of generating  reference correlation energies.
Only 1\% of the QM7b-T data set (containing organic molecules with seven and fewer heavy atoms) needs to be drawn on to train a MOB-ML model which produces on average chemically accurate total energies for the remaining 99\% of the data set.
Without ever being trained to predict relative energies, MOB-ML provides chemically accurate relative energies for QM7b-T when training on only 0.4\% of the QM7b-T molecules.
Furthermore, we have demonstrated that MOB-ML is not restricted to the organic chemistry space and that we are able to apply our framework out-of-the box to describe a diverse set transition-metal complexes when training on correlation energies for tens of molecules.

Beyond data efficiency, MOB-ML models are are shown to be very transferable across chemical space.
We demonstrate this transferability by training a MOB-ML model on QM7b-T and predicting energies for a set of molecules with thirteen heavy atoms (GDB13-T).
We obtain the best result for GDB13-T reported to date despite only training on 3\% of QM7b-T.
The successful transferability of MOB-ML  is shown to result from its recasting of a typical extrapolation task (i.e., larger molecules) into an interpolation task (i.e., by predicting on the basis of size-intensive orbital-pair contributions).
Even when MOB-ML enters an extrapolative regime as identified by a large Gaussian process variance, accurate results can be obtained; for example, we predict the transition-state energy for the proton transfer in malonaldehyde and interaction energies in the protein backbone-backbone interaction data set to chemical accuracy without training on transition-state-like data or non-covalent interactions, respectively.
In this case, the uncertainty estimates also offer a clear avenue for active learning strategies which can further improve the model performance.
Active learning offers an attractive way to reduce the number of expensive reference calculations further by picking the most informative molecules to be included in the training set.
This provides a general recipe how to evolve a MOB-ML model to describe new regions of chemical space with minimal effort.

Future work will focus on the expansion of MOB-ML to cover more of chemical space.
Specifically, particular areas of focus include open-shell systems and electronically excited states.
Physical insight from exact conditions in electronic structure theory \cite{bartlett_power_2017} will continue to guide the development of the method, with the aim of providing a machine-learning approach for energies and properties of arbitrary molecules with controlled error.

\begin{acknowledgments}
This work is supported in part by the U.S. Army Research Laboratory (W911NF-12-2-0023), the U.S. Department of Energy (DE-SC0019390), the Caltech DeLogi Fund, and the Camille and Henry Dreyfus Foundation (Award ML-20-196).
T.H. acknowledges funding through an Early Post-Doc Mobility Fellowship  by  the  Swiss  National  Science Foundation (Award P2EZP2\_184234).
S.J.R.L. thanks the Molecular Software Sciences Institute (MolSSI) for a MolSSI investment fellowship.
Computational resources were provided by the National Energy Research Scientific
Computing Center (NERSC), a DOE Office of Science User Facility
supported by the DOE Office of Science under contract DE-AC02-05CH11231.
\end{acknowledgments}

\section*{Supporting Information}
Details on feature generation for all data sets used in this work, definition of error metrics, expanded results for the alkane transferability test, expanded results for the transferability within the organic chemistry space.
Features and labels for all data sets used in this work.

\section*{Data Availability Statement}
The data that supports the findings of this study are available within the article and its supplementary material.
Additional data that support the findings of this study are openly available in Caltech Data Repository.\cite{caltech_data}

redpajama_set_name: RedPajamaArXiv


text:
Study finds taxpayers subsidize big-box stores at cost of $150 million per year in Minnesota
Black Friday protests last year drew more than 200 people to the Midway Walmart in St. Paul.
Retail workers and activists who join the protest outside Walmart in St. Paul on Black Friday will be armed with a scathing new report shedding light on the public cost of poverty-level wages paid by Walmart, Target and other big-box department stores.
"At What Cost," released Nov. 12 by the union-backed Minnesotans for a Fair Economy, reveals how Minnesota taxpayers, by picking up the cost of basic necessities thousands of retail workers can't afford, are subsidizing big-box stores' low wages.
"We estimate that over 20,000 big-box department store workers in Minnesota are enrolled in some form of public assistance program, either for themselves or family members, at a cost of over $150 million a year to taxpayers," the report says.
[Click here to RSVP for Black Friday protests in the Twin Cities on Facebook!]
Not surprisingly, the majority of retail workers accessing safety-net programs for themselves or their families come from Walmart and Target. The report estimates 5,850 Target workers in Minnesota access public assistance at an annual cost to taxpayers of $44.7 million, while 5,311 Walmart employees statewide seek about $40.6 million in public assistance annually.
That these two retailers are the worst offenders is hardly surprising, considering the wages paid in their stores.
The average hourly wage of a Walmart sales associate, according to the report, is $8.81, while the average pay for Target's frontline staff ranges from $8.10 per hour for cashiers to $8.34 per hour for "team members" on the sales floor.
A full-time worker earning those wages wouldn't sniff the federal poverty line, which explains why so many turn to food stamps and other safety-net programs to support themselves and their families. What's worse, over half of retail workers are "involuntarily part-time," making it even harder to make ends meet without public assistance.
The public cost of health care for big-box store workers, meanwhile, is particularly taxing. Roughly half of Walmart associates and a third of Target "team members" are enrolled in the companies' health insurance plans. Workers "say that the health plans are too expensive or inaccessible," according to the report.
The report provides the first local look at findings introduced by Democrats on the U.S. House Committee on Education and the Workforce in May.
Analyzing Medicaid data, the committee report found that a single 300-employee Walmart Supercenter in Wisconsin cost taxpayers between $900,000 and $1.75 million per year – or roughly $5,815 per employee.
At the conclusion of "At What Cost," the authors offer three suggestions for reducing the strain on public assistance created by big-box retailers, particularly Target and Walmart.
Big-box retailers should pay a living wage of $14.41 per hour, state lawmakers in Minnesota should raise the minimum wage to $9.50 per hour and, the report says, retailers should "stop interfering with their employees' rights to speak out for better jobs."
"Big-box retail employers could listen to their employees' calls for change, rather than retaliate against them for speaking out."
Filed Under: Politics Tagged With: big box stores, Black Friday, economy, low wages, minimum wage, Minnesotans for a Fair Economy, protest, public assistance, retail, Target, union busting, Walmart
« Josh Wise: Trade Promotion Authority puts U.S. on fast track to more job losses
Labor Board: Walmart violated workers' rights »
Walmart workers go on strike in Brooklyn Center, kick off week of action for better wages – Union Advocate says:
[…] recent report by Minnesotans for a Fair Economy estimated the cost to taxpayers of providing public assistance to […]

redpajama_set_name: RedPajamaCommonCrawl


text:
Kelencontent sells "Mantracker" format into Australia, NZ
ActiveTV Australia (The Amazing Race Australia) has picked up the format rights to the Canadian reality adventure series, Mantracker, for the two territories.
ActiveTV Australia has picked up the format rights to the Canadian reality adventure series, Mantracker, for Australia and New Zealand.
The local version, from the producers of The Amazing Race Australia, will have a "Down Under" flavor, with an Australian survivalist, indigenous trackers and Australian Outback locations. There is currently no broadcaster attached to the series.
Mantracker is in its seventh season on OLN in Canada and now airs on Science Channel in the U.S. The adventure series sees a Canadian cowboy tracker, aka Mantracker, and his local guide use tracking know-how and instincts to find their prey – two people who have a compass, map and their wits who must cross 40 kilometers of wilderness to reach a final destination without being caught. Mantracker was created by Ihor Macijiwsky, executive producer and founder of Bonterra Productions, producers of the original Mantracker series.
Amanda Peppard, VP Development, activeTV, said: "We'll cast our 'prey' the same way we cast for The Amazing Race – couples from a diverse range of backgrounds and skills sets all with unique relationships, which will add to the drama. Once we drop them into the Australian outback or rainforests, remote beaches or mountain ranges, it'll be anybody's guess as to who will be victorious."
Tanya Kelen, president of kelencontent, added: "Mantracker features many of the elements that activeTV has become renowned for following the success of The Amazing Race formats they have produced – each episode is based on a race; the casting of the contestants is key; and it will feature incredible challenges performed in one of the world's toughest and remote terrains."
ActiveTV Australia, kelencontent, Mantracker, OLN
Buck Productions bringing saw dogs and dark deals to OLN
BBC, OLN, C5 get "Wild" with Dominic Monaghan
Wildscreen '12: Monaghan gets "Wild"
Peacock Alley bringing "Illusions Of Grandeur" to MIPCOM


Perform Advanced diagnostics including programming. adjustments, activations, and service light reset.
Complete diagnostic kit for Fiat Type 250/290 Ducato Vans and variants to 2019. Uses the power of your Windows PC to offer near dealer level diagnostics on Fiat Ducato models including Motorhomes.
This kit now includes the Genuine OBDlink SX version 2.0. There will be NO connection issues, and by using a genuine OBDLink tool you can rest assured the interface will not cause any issues with your vehicle unlike some Chinese cloned ELM interfaces.
Why use a genuine OBDlink SX interface?
The OBDlink SX is a high quality interface that is fully compatible with all versions of the MultiECUScan software can reliably handle the Hi speed Can Messages on the later model Ducato, it is able to handle proxy alignment and other programming that require large data transfer that Chinese clone ELM327 could struggle with and possibly fail causing major ECU problems that may require module replacement.
There will be no connection issues, this interface fully supports all protocols used by Fiat, some Chinese clones do not support all the latest protocols and will not communicate reliably with the new Models.
THE LIST OF VEHICLES BELOW USE LSCAN, PLEASE FOLLOW THE PROMPTS SHOWN IN THE SOFTWARE IN REGARD TO THE PROPER CABLE/ADAPTER SELECTION.
Sample of coverage available with this kit shown below For full coverage or to confirm if your vehicle is covered contact us on 02891 310043.


text:
SOW 78 – Digital Media with Naresh Vissa
Naresh Vissa is a Digital Media Sales Consultant, having done work for clients such as the Hard Assets Alliance, The Wall Street Shuffle, and The Institute for Energy Research. He is also the former Director of The Stansberry Radio Network and former Director of Media Strategy for Stansberry & Associates Investment Research, which is one of the…
How to Write an Irresistible Headline
How important are headlines? Well, if you're reading this, then the one up there did its job, didn't it? Here's a powerful statistic from Copyblogger.com. Though 8 out of 10 people will read a headline, only 2 of those 10 go on to read the rest of the copy, no matter how scintillating it might…
SOW 77 – Inside the Secret Life of a Super Affiliate with Zac Johnson
Jason Hartman welcomes Zac Johnson, a widely respected leader in the blogging and online marketing community. Through Zac's popular blogs, bloggingtips.com andzacjohnson.com, he has helped thousands of readers grow their brands and make money online. Zac shares his success story, including tips for gathering leads, the step-by-step affiliate marketing process, becoming a merchant, finding offers…
SOW 76 – Combining CRM and Social Media with Jon Ferrara
Jon Ferrara is the Founder & CEO of Nimble. Nimble is revolutionizing social media and CRM through social selling. Ferrara explains how he is powering it. Ferrara also discusses why he is doing everything in his power to help start-ups. Jon has over 20 years of experience in Customer Relationship Management (CRM) and Sales Force…
SOW 75: Maximizing Technology For Marketing with Sierra Modro CEO of Leap to Greatness
There is hardware and software technology available for every business need under the sun, with more being developed every day, yet many people who purchase these tools rarely use them to their full potential or the tools become dormant and replaced by the next new thing on the market. Jason Hartman is joined by self-proclaimed…
SOW 74: Outside-the-Box Marketing with Michael Houlihan & Bonnie Harvey Authors of 'The Barefoot Spirit'
Jason Hartman invites Michael Houlihan and Bonnie Harvey, co-founders of Barefoot Wine, to discuss "out-of-the-box" marketing principles that led to Michael's and Bonnie's highly successful wine business. Though his business card reads, "Head Stomper," Michael officially served as president and CEO for 19 years. She was Vice President and "The Original Foot" for 19 years. There she had a wide variety…
SOW 73: Easy Effortless & Enjoyable Marketing with David Newman Author of 'Do it! Marketing'
Marketing techniques have evolved exponentially with the introduction of the internet and social media apps, and to stand out from the competition, it requires strong thought leadership. Jason Hartman is joined by marketing expert, David Newman, to talk about the four levels of marketing. David explains what 80% of business owners miss that causes marketing overwhelm and sales burnout,…
3 Proven Techniques for Podcasting Profits
No matter how much you like jabbering into a microphone, creating podcast after podcast, there comes a time when the thought comes to mind, "I'd really like to get paid for all this time and effort." The bad news is you need an audience before you're going to notice any sort of serious cash flow.…
Memphis – Quiet Hotbed for Cash Flow Real Estate
When the conversation turns to hot areas for real estate investment, Memphis is not the first city that springs to mind, but this laid-back mid-south metro area is certainly not playing the blues when it comes to low price, high profit opportunities for potential income property investors. And the time has never been better to…
Searching for the Perfect Podcasting Microphone
It wasn't so long ago that the podcasting world was new and shiny and could do no wrong. Audio quality was an afterthought. Heck, that thirty-four cent microphone built into your keyboard worked just fine. While the background noise made it seem like a passing stranger was being mauled by a grizzly bear, listeners could…


In an age where the concept of ceremony has been lost, I work closely with a couple to create a personal and meaningful experience that they can share with those they are closest to.
I truly believe that a wedding ceremony can be a unifying and transformative experience for two people. When planning a wedding, it's tempting for couples to gloss over the ceremony and focus their energy on the food, drinks, the dress, the music – which are all important. However, it can mean you easily lose sight of why you are getting married. The ceremony process that I walk couples through has the ability to bring you back to what matters and gives you time to reflect on why you are getting married.
After 10 years as a social worker, I bring a unique insight into the role of a celebrant, understanding people and the complexity of our relationships and interactions. I have a love of human connection, and invest time in to understanding who you are and what makes you choose each other. I will work closely with you to create a ceremony that is warm, authentic, and genuine reflection of you and your relationship. Your guests will feel involved and engaged, and leave the ceremony knowing that they were a part of something special.
To hold space for a couple as they make this commitment to each other for life is an honour and a privilege and it's not lost on me that I am incredibly lucky to be a part of that.


Winter is arrived. It was snowing outside when we woke up on Saturday.
Here came my nightmare. I hate winter.
I wanna lock myself in our apartment and never leave out of the door.
We were at the party. I don't know how many people were there.
But it was a lot of adults. I felt ashamed that I couldn't behave on Saturday.
with Robin, he always denied to listen and acted like he never heard what I said.
Specially when people mention if we're engaged yet or when we will have a kid?
Of course we've been talking about this all the time. That's why Robin always get sick.
We both agreed in 5 years we will have kid but before having kids, we will travel to many places.
We both have plans what we want to do before make a big step.
Robin wants to get his master degree and back pack with his friends.
And they plan to go away for 6 months.
For me my plan is study but I don't know which way should I go? Right now I need a job.
We want to have better future together, cause people around us made mistakes in the past.
That's why we learned not to do the same.
What we are trying to do now is not only for us, is for our kid too.
Sometimes what we do in the present effects the future.
Specially thanks to her for party!
And thank you so much for food and cakes! Both Johanna and Diana!


Is your Business or School looking for new talent? The 2018 Annual Meeting of the Decision Sciences Institute attracts some of the brightest minds from around the world, and is an excellent venue to interview candidates.
DSI Placement Services will be hosting a private interview room during the Annual Conference in Chicago. This room will provide the ability to schedule and conduct face to face interviews in a quiet secluded area.
Includes one table and three chairs.
NOTE: DSI does not approve of interviews taking place in areas other than the designated interview room. Hotel rooms are inappropriate settings for conducting interviews.
To reserve a table, DOWNLOAD the form here.
Payments for interview tables can be mailed to the DSI Home Office directly or processed over the phone with a credit card.
For more information or to reserve a space contact the Decision Sciences Institute home office at info@decisionsciences.org or call 713-743-4815.


text:
Songs produced behind bars earn convict street cred in Burkina Faso
16 July 2019 | 10:44 am
In a time of talent shows, the Ouagadougou prison and correctional facility known as MOCO is home to Burkina Faso's latest musical find - Roland Tapsoba, AKA 'Rolbi'.
Burkina Faso: Dozens dead after bus hits IED
The explosion hit a convoy carrying supplies in the north of the country, an area where Islamist militants have escalated attacks in recent years.
Soldiers take over the streets of Burkina Faso capital
Soldiers have been seen along the main avenue in Ouagadougou leading to the presidential palace and national television station.
Jihadi violence in Burkina Faso has Ghana concerned
Conflict in Burkina Faso has displaced more than 2 million people, with many of them now fleeing south into Ghana as jihadi attacks continue. DW's Maxwell Suuk visited a camp hosting refugees from Burkina Faso close to the border.
Burkina Faso junta leader agrees to quit after mutiny
Paul-Henri Sandaogo Damiba offered his resignation, two days after junior military officers rebelled against his rule. The West African country is experiencing a coup within a coup after January's military takeover.
Burkina Faso: New coup leader blames violence on 'counteroffensive'
The leader of Friday's coup has accused the military man he deposed of plotting a counter-offensive with French aid. Violence has persisted in the capital with gunshots heard.
Ousted Burkina Faso leader Damiba in Togo after coup: government
Togo's government has confirmed reports that Burkina Faso's ousted coup leader Paul-Henri Damiba fled there following a mutiny of junior officers. Meanwhile, ECOWAS envoys were being sent to Ouagadougou.
Burkina Faso's new military government meets Ecowas bloc in Ouagadougou
Days after the latest coup, Burkina Faso's new military leaders finally meet with a delegation from West African regional bloc ECOWAS at the airport in Ouagadougou, after anti-ECOWAS protesters gathered outside the designated conference venue.
Burkina Faso coup: videos out of context claim to show French intervention
The whole region knows the drill by now. Dateline Ouagadougou. If you are watching images of pro-junta demonstrators protesting the visit of West African mediators looking for rule of law guarantees from their new interlocutors after a coup, don't reach for the remote control. It is not a rerun.
Children displaced by insecurity in Burkina Faso fall out of education
One week after another coup d'état, our correspondents take stock of years of political instability and terrorism in Burkina Faso. International organisations say there are now two million internally displaced people inside the country, and many of them are children. Also in this edition: Lesotho goes to the polls.
Evin prison incident 'is part of what is going on in the streets'
The fire at Tehran's Evin prison over the weekend killed eight people, authorities said. Details over the blaze were scarce as officials sought to distance the incident from nationwide protests.
Evin prison incident 'is part of what is going on in the streets'.