text:
These are not great times for the battle against climate change. With America officially withdrawing from Paris Agreement, the global alliance lost one of its major champions. While this year's World Environment Day (celebrated each year on 5 June) was rather muted, the determination to continue with the momentum remains steadfast. What's important is that governments, academia, and industry need to work together to make a difference. It is naïve to believe that only industry and enterprises contribute to climate change. There is a great need for scientists to harness those elements of technology that have the potential to hasten our understanding not just of the drivers of climate change but also help illustrate relevant solutions so that requisite action can be taken
The Data Revolution
It's well known that while big data has had a transformative effect on academic research as well as business applications, its application in climate change has till now remained nascent. The good news is that a global data revolution is unfolding and accelerating decision-making around climate change. Using big data and analytics solutions, scientists are now analysing a wider framework of data including, for the first time, privacy-protected digital data — such as mobile data or credit/debit card transactions, for example — to get important insights into human consumption patterns that in turn can be correlated to climate risk. The resultant data streams are, therefore, providing an unprecedented opportunity to scientists, academia, and private enterprise to catalyse climate innovation and influence decision-making for the public good.
Public-Private Partnerships and Data Philanthropy
A promising initiative related to this is the United Nations Global Pulse programme called 'Data for Climate Change'. The UN Global Pulse is an agency that partners with data-rich organizations along with other UN agencies, governments and "problem owners" to grapple with challenges that could benefit from new insights that help discover, build, and test high-potential applications of this big data. The focus is on sectors such as food security, agriculture, employment, infectious disease, urbanization, and disaster response and others. Climate change is a new addition to this list. The 'Data for Climate Action' initiative launched in March this year is an 'open innovation challenge' to scientists and researchers from across the world to harness data science and big data from the private sector to fight climate change. The challenge aims to leverage private big data to identify revolutionary new approaches to 'climate mitigation and adaptation'.
This initiative stands apart due to two reasons. First is that the global challenge has attracted companies from across industries and countries to participate through acts of data philanthropy. Secondly, the data being generated is not as much around climate data as it is about human behaviour and its effect on climate change. The challenge offers researchers an opportunity to gain unprecedented access to national, regional, and global datasets — anonymized and aggregated to protect privacy — and robust tools to support their research. The results of this is expected to be made public later this year.
Addressing Global Warming Cheaply and Effectively
The issue from a policy and implementation perspective is to reduce the effects of global warming in a manner that is both cost- and means-effective. Scientists and administrators do not have the time or the resources to try out different approaches and then decide the best one. Big data, advanced analytic techniques, and algorithms are playing an important role in enabling this. For example, specific to a country or region, data can show where flooding is most likely to occur or the areas that are most prone to drought or other natural calamities and the timeliness of this information is a critical factor. This can enable the local government to make accurate resource allocation, thereby minimising wastage and over-spills. Agriculture is another area that has a significant cause-and-effect relationship with global warming. It stands to reason that everything we do directly correlates to furthering environmental impact with catastrophic social and economic effects. An analysis of global social media data on the issue of global warming and climate change is also proving very effective in terms of highlighting the various elements that country populations are sensitive to — whether it's energy, climate, the state of the oceans, agriculture, forests and natural resources, etc. This provides an important public perspective to policymakers and academicians.
While traditional sources of climate data help describe how and to what extent the climate is changing, they do not always illustrate the solutions that are likely to be most effective in reducing emissions and helping build community resilience. New sources of big data from different industries and geographies, can be applied to construct a more complete picture, thereby significantly enhancing the understanding of the deeply interrelated relationships between human action and climate change. Combine this with increasing amounts of computing resources available at a much lower cost than ever before plus the ability to process them on distributed platforms in the cloud and the potential to exploit data and analytics in an important area like climate change is more real than ever before. With these new adaptative or mitigation interventions, innovations can be developed and will lead to, it is hoped, the beginning of a new and enabling ecosystem.
Click here to learn more about Teradata's involvement in environmental sustainability and efficiency.
Rajesh Shewani
Rajesh Shewani heads Technology and Solutions at Teradata India. He comes with close to two decades of experience in the areas of Data Management, Advanced Analytics, Solution Architecture, Enterprise data warehousing and Business Intelligence to name a few.
At Teradata he is responsible for leading a team of experienced Business Analytics consultants, data scientists and solution architects focusing on Teradata Data Analytics solutions. Rajesh has worked in various areas of information management across different industries and functional domains, advising customers on devising corporate performance management and business analytics strategies that will help them achieve differential advantage.
Prior to Teradata, Rajesh was with IBM for over 14 years and performed various roles in technology leadership, architecture and consulting. In his most recent role at IBM he was Country Manager for technical sales for IBM Business Analytics portfolio that included Cognos, SPSS, OpenPages and Algorithmics, leading a team of analytics functional and technical sales architects. Rajesh is certified in technologies such as Cognos BI & FPM, SOA, WebSphere Application Server and DB2.
He holds a Masters in Business Marketing & Information Management from Narsee Monjee Institute of Management Studies.
View all posts by Rajesh Shewani


Need a cute way to introduce fragmentation? We're currently on Week 7, which deals with animal reproduction. Mary knows what live birth and eggs are, but we haven't gotten into fragmentation yet. So, after doing some digging, I found this cute video of the Octonauts that demonstrates fragmentation (specifically at the 4:25 mark)!
And, I just had to throw in these next few pictures of my girls at the Tennessee Aquarium over the weekend. They were just too sweet!
I hope you're all having a blessed week!


text:
text/xml; subtype=gml/3.1.1
service-view (4)
NERC_DDC (4)
Digital maps (4)
British Geological Survey (4)
BGS Services (4)
GeoSciML (4)
application/vnd.ogc.gml (2)
image/gif (4)
image/png (4)
image/png; mode=24bit (4)
image/svg+xml (4)
image/tiff (4)
image/vnd.wap.wbmp (4)
text/html (2)
text/plain (2)
text/xml (2)
text/xml; subtype=gml/3.1.1 (4)
Not planned (1)
BGS Bedrock and Superficial geology (OGC WxS)
This service is the BGS contribution to the OneGeology project, providing UK onshore geological data at 1:625 000 scale and UK continental shelf seabed sediment data at 1:1 million scale. This map data is collected as part of an ongoing BGS project: Digital Geological Map of Great Britain (DiGMapGB). MapServer software is used to provide these OGC WMS and WFS services.
BGS GeoSure (OGC WxS)
The 1:50 000 scale GeoSure (natural ground stability) data covering the map sheets for Loughborough and Kilmarnock are available in this OGC WMS service for personal, non-commercial use only. This service is a contribution to the OneGeology-Europe initiative. For information about more of the British Geological Survey's maps and other digital products please visit http://www.bgs.ac.uk/products. MapServer software is used to provide this OGC WMS service.
BGS bedrock geology (OGC WxS)
Data from the DiGMap covering the whole of the United Kingdom at a scale of 1:625 000 is available in this OGC WMS service for personal, non-commercial use only. The service is a contribution to the OneGeology-Europe initiative. The layers can be displayed either by age or by lithology. For more information about the digital maps available from the British Geological Survey, please visit https://www.bgs.ac.uk/geological-data/.
BGS Surface geology (OGC WxS)
Data from the DiGMap covering the whole of the United Kingdom at a scale of 1:625 000 is available in this OGC WMS service for personal, non-commercial use only. The service is a contribution to the OneGeology-Europe initiative. The layers can be displayed either by age or by lithology. For more information about the digital maps available from the British Geological Survey, please visit https://www.bgs.ac.uk/datasets/bgs-geology/

redpajama_set_name: RedPajamaCommonCrawl


text:
Adam & The Ants (Stuart Leslie Goddard / Adam and The Ants)
New Age / Alternative / Pop
Alternative Pop/Rock / Post-Punk / New Wave / Pop-Rock
URL's:
www.facebook.com/pages/Adam-Ant/69349615331
www.Adam-Ant.com
www.Adam-Ant.net
www.discogs.com/artist/10380-Adam-Ant
Discography (total 6 Albums )
Albums | Compilations, Remixes
2013 Adam Ant Is the BlueBlack Hussar in Marrying the Gunner's Daughter 17 01:08:46
2002 Manners & Physique 10 00:44:42
2000 Strip 10 00:39:07
2000 Wonderful 11 00:46:45
1982 Friend Or Foe 12 00:39:35
1980 Kings Of The Wild Frontier 13 00:46:56
To download entire album:
Mark the checkbox of the album you want to buy. To Select/Unselect all albums available on this page simply click В«CheckВ» above the checkboxes column.
Click the Add to Cart button beneath to add the album to your shopping cart


text:
\section{More Details On Softmax Function~\eqref{eq:softmax_log_likelihood}}
\subsection{Relationship to Logistic Regression with $ \pm1 $-labels}
\label{sec:logistics_pm1}
Sometimes, in the literature, for the two-class classification problem, instead of $ \{0,1\} $ the labels are marked as $ \pm1 $. In this case, the corresponding logistic regression is written as
\begin{align*}
&F(\xx)  = \sum_{i=1}^{n} \log \left( 1+e^{-b_{i} \xx^{T} \aa_{i} } \right).
\end{align*}
In this case, we have
\begin{align*}
&F(\xx) = \sum_{i=1}^{n} \log \left( e^{\frac{- \xx^{T} \aa_{i}}{2}} + e^{\frac{\xx^{T} \aa_{i}}{2}} \right) - \frac{b_{i} \xx^{T} \aa_{i}}{2} \\
&= \sum_{i=1}^{n} \log \left(e^{\frac{- \xx^{T} \aa_{i}}{2}} \left( 1 + e^{\xx^{T} \aa_{i}} \right) \right) - \frac{b_{i} \xx^{T} \aa_{i}}{2} \\
&= \sum_{i=1}^{n} \log \left( 1 + e^{\xx^{T} \aa_{i}} \right) - \frac{(1+b_{i}) \xx^{T} \aa_{i}}{2} \\
&= \sum_{i=1}^{n} \log \left( 1 + e^{\xx^{T} \aa_{i}} \right) - \tilde{b}_{i} \xx^{T} \aa_{i},
\end{align*}
where $ \tilde{b}_{i} \in \{0,1\} $. Hence this formulation co-incides with~\eqref{eq:softmax_log_likelihood}.

\subsubsection{Softmax Multi-Class problem is (strictly) convex}
\label{sec:softmax_convex}
Consider the data matrix $X \in \mathbb{R}^{n \times d}$ where each row, $\aa_{i}^{T}$, is a row vector corresponding to the $i^{th}$ data point. The Hessian matrix can be written as
\begin{align*}
\nabla^{2} \mathcal{L} = \mathbf{X}^{T} \mathbf{W} \mathbf{X},
\end{align*}
where
\begin{align*}
\mathbf{X} &= \begin{bmatrix}
X & 0  & \ldots & 0 \\
0& X  & \ldots & 0 \\
\vdots & & \ddots & \vdots \\
0& 0  & \ldots & X
\end{bmatrix}_{(n\times (C-1)) \times (d \times (C-1))}, \\
\mathbf{W} &= \begin{bmatrix}
W_{1,1} & W_{1,2}  & \ldots & W_{1,C-1} \\
W_{2,1}& W_{2,2}  & \ldots & W_{2,C-1} \\
\vdots & & \ddots & \vdots \\
W_{C-1,1}& W_{C-1,2}  & \ldots & W_{C-1,C-1}
\end{bmatrix},
\end{align*}
and each $W_{c,c}$ and $W_{c,b}$ is a $n \times n$ diagonal matrix corresponding to~\eqref{eq:softmax_hessian_wc_wc} and~\eqref{eq:softmax_hessian_wc_wb}, respectively.
Note that since
\begin{align*}
&\left(\frac{e^{\lin{\aa_{i}, \xx_{c}}}}{1+\sum_{c' = 1}^{C-1} e^{\lin{\aa_{i}, \xx_{c'}}}} - \frac{e^{2\lin{\aa_{i}, \xx_{c}}}}{\left(1+\sum_{c' = 1}^{C-1} e^{\lin{\aa_{i}, \xx_{c'}}}\right)^{2}} \right) - \\
&\sum_{\substack{b = 1 \\ b \neq c}}^{C-1} \frac{e^{\lin{\aa_{i}, \xx_{\hat{c}} + \xx_{c}}}}{\left(1+\sum_{c' = 1}^{C-1} e^{\lin{\aa_{i}, \xx_{c'}}}\right)^{2}} \\
& = \left(\frac{e^{\lin{\aa_{i}, \xx_{c}}}}{1+\sum_{c' = 1}^{C-1} e^{\lin{\aa_{i}, \xx_{c'}}}} - \frac{e^{2\lin{\aa_{i}, \xx_{c}}}}{\left(1+\sum_{c' = 1}^{C-1} e^{\lin{\aa_{i}, \xx_{c'}}}\right)^{2}} \right) \\
&  \quad \quad  - \frac{e^{\lin{\aa_{i}, \xx_{c}}}}{1+\sum_{c' = 1}^{C-1} e^{\lin{\aa_{i}, \xx_{c'}}}} \left( \sum_{\substack{b = 1 \\ b \neq c}}^{C-1}  \frac{e^{\lin{\aa_{i}, \xx_{\hat{c}} }}}{1+\sum_{c' = 1}^{C-1} e^{\lin{\aa_{i}, \xx_{c'}}}} \right) \\
& = \left(\frac{e^{\lin{\aa_{i}, \xx_{c}}}}{1+\sum_{c' = 1}^{C-1} e^{\lin{\aa_{i}, \xx_{c'}}}} - \frac{e^{2\lin{\aa_{i}, \xx_{c}}}}{\left(1+\sum_{c' = 1}^{C-1} e^{\lin{\aa_{i}, \xx_{c'}}}\right)^{2}} \right) \\
&  \quad  \quad - \frac{e^{\lin{\aa_{i}, \xx_{c}}}}{1+\sum_{c' = 1}^{C-1} e^{\lin{\aa_{i}, \xx_{c'}}}} \left( 1-\frac{1 + e^{\lin{\aa_{i},\xx_{c}}}}{1+\sum_{c' = 1}^{C-1} e^{\lin{\aa_{i}, \xx_{c'}}}}  \right) \\
& = \frac{e^{\lin{\aa_{i}, \xx_{c}}}}{\left(1+\sum_{c' = 1}^{C-1} e^{\lin{\aa_{i}, \xx_{c'}}}\right)^{2}} > 0,
\end{align*}
the matrix $\mathbf{W}$ is strictly diagonally dominant, and hence it is symmetric positive definite. So the problem is convex (in fact it is strictly-convex if the data matrix $ \vec{A} $ is full column rank).

\section{Tensorflow's Performance Comparison on Various Compute Platforms}
\label{sec:platform-cpu-gpu}
\input ./tex/page_cpu_gpu_comparison.tex
\section{Conclusions and Future Directions}
\label{sec:conclusions}

We have developed a novel distributed Inexact Newton method based on a global consensus ADMM formulation.
We compare our method with
state-of-the-art optimization methods and show that our method has much lower distributed overhead, achieves
superior generalization errors, and has significantly lower epoch-times on standard benchmarks.
We have also shown that our method can handle large datasets, while delivering sub-second epoch times -- establishing desirable scalability characteristics of our method.  Our results  establish Inexact Newton-ADMM as the new benchmark for
performance of distributed optimization techniques.

\section{Experimental Evaluation}
\label{sec:expt}
In this section, we evaluate the performance of Newton-ADMM as compared with several state-of-the-art alternatives.
\begin{table}[h]
\small
\centering
\caption{Description of the datasets.}
\label{t1}
\begin{tabular}{|c|c|c|c|c|}
\hline
Classes&Dataset& Train Size & Test Size & Dims \\ \hline
2                 & HIGGS    & 10,000,000       & 1,000,000   & 28 \\
 10                & MNIST    & 60,000          & 10,000     & 784                \\
                10                & CIFAR-10 & 50,000          & 10,000     & 3,072               \\
                20                & E18      & 1,300,128         & 6,000      & 279,998             \\ \hline
\end{tabular}
\end{table}
\paragraph{\textbf{Experimental Setup and Data:}} All algorithms are implemented in PyTorch/0.3.0.post4 with Message Passing Interface (MPI) backend. We test performance of the methods on two hardware platforms. The first platform is a server with
 384 Intel Xeon Platinum 8168 processors and 8 Tesla P100 GPU cards. The second platform is a CentOS 7 cluster with 15 nodes with 100 Gbps Infiniband interconnect. Each node has 96GB RAM, two 12-Core Intel Xeon Gold processors, and 3 Tesla P100 GPU cards. We validate our proposed method using real-world datasets, described in Table~\ref{t1}, and compare with state-of-the-art first-order and second-order optimizers. These datasets are chosen to cover a wide range of problem characteristics (problem-conditioning, features, problem-size). MNIST is a widely used dataset for validation -- it is relatively well-conditioned. CIFAR-10 is 3.9x larger than MNIST and is relatively ill-conditioned. HIGGS\cite{baldi2014searching} is a low-dimensional dataset, however it is the largest (in terms of problem size). This dataset is easy to solve for our method, but is harder for first-order variants because of high communication overhead. The largest data set, E18  \footnote{    \href{https://support.10xgenomics.com/single-cell-gene-expression/datasets}{ E18 data set source.}     } , in terms of dimension and number of samples, is used to highlight  the scalability of our proposed method.

\paragraph{\textbf{Newton-type method as a highly efficient subproblem solver for ADMM.}}
\label{LBFGS-ADMM}

We establish our (GPU-accelerated) Newton-type optimizer as a highly efficient inner solver for ADMM by comparing its performance against an ADMM-L-BFGS solver.  The per-iteration computation cost and memory footprint of L-BFGS is lower than our Newton-type method because of the rank-2 approximation of the Hessian matrix. This, however, comes at the cost of a worse convergence rate for L-BFGS. While Newton-type methods compute matrix vector products with the full Hessian, we use a Conjugate Gradient method with early stopping to solve the linear system, $ \HH \xx = -\bgg $. In our experiments we use no more than 10 CG iterations and a tolerance level of $ 10^{-3} $. The resulting \textit{Inexact} Newton-type method is GPU-accelerated,
with an efficient implementation of Hessian-vector product. We show that, in practice, ADMM method suitably aided by efficient implementation of Newton-type subproblem solvers, yields significantly better results compared to the state-of-the-art. Furthermore, the use of true Hessian in our inexact solver, a second-order method, makes it resilient to problem ill-conditioning and immune to hyper-parameter tuning. These results are shown in Figure~\ref{fig:ADMM_BFGS}. We clearly notice that the performance gap between L-BFGS and Inexact Newton-type method becomes larger when number of compute nodes is increased. The only exception is on HIGGS dataset. This is because the dimension of the HIGGS datasets is only 28 and it is a binary classification problem. Consequently, the dimension of Hessian is significantly lower than among all the datasets. In this case, L-BFGS solver yields similar results compared to our Inexact Newton method. Most importantly, we note the following key results: (i) Inexact Newton yields performance improvements from 0 (MNIST) to 550\% (HIGGS and CIFAR-10) over L-BFGS on a single node; (ii) when using 8 compute nodes, the performance of L-BFGS-ADMM {\em never} catches up with that of Newton-ADMM (in terms of training objective function) in three of four benchmarks (MNIST, CIFAR-10, and E18), conclusively establishing the superiority of our proposed method over L-BFGS-ADMM.

\begin{figure*}[h]
\hspace*{-2.5cm}
    \begin{subfigure}[t]{0.5\textwidth}
        \includegraphics[width=170mm]{./plot/ADMM_SGD_CPU_GPU.pdf}
    \end{subfigure}%
\caption{Training objective function and test accuracy as functions of time for Newton-ADMM and synchronous SGD, both with GPU enabled and GPU disabled, with 4 workers. Overall, Newton-ADMM favors GPUs, enjoys minimal communication overhead, and enjoys faster convergence compared to synchronous SGD. }
\label{fig:ADMM_SGD_Compare_1e-5}
\end{figure*}

\paragraph{\textbf{Comparison with Distributed First-order Methods.}} \label{sec:ADMM_vs_SGD}
 While the per-iteration cost of first-order methods is relatively low, they require larger numbers of iterations, increasing associated communication overhead, and CPU-GPU transactions, if GPUs are used (Please see detailed discussion in section \ref{GPU-utils}). In this experiment, we demonstrate that these drawbacks of first order methods are significant, both with the GPUs enabled and GPUs disabled. The results are shown in Figure \ref{fig:ADMM_SGD_Compare_1e-5}. Specifically, we note that GPU-accelerated Newton-ADMM method with minimal communication overhead yields significantly better results -- over an order of magnitude faster in most cases, when compared to synchronous SGD.

We present the ratio of CPU time to GPU time for Newton-ADMM and SGD in Table \ref{CPU-GPU-ratio}. We observe that for both Newton-ADMM and SGD, the CPU-GPU time ratio is proportional to the dimensions of datasets. For example, on the dataset with the lowest dimension (HIGGS), the CPU-GPU time ratio is the least for both  Newton-ADMM and SGD, whereas on the dataset with the highest dimension (E18), the CPU-GPU time ratios are the highest for both Newton-ADMM and SGD. In all cases, the use of GPUs results in highest speedup for Newton-ADMM. The gain in GPU utilization is compromised by large number of CPU-GPU memory transfers for SGD. As a result, SGD shows meaningful GPU acceleration only for the E18 dataset.

\begin{table}[h]
\caption{GPU Speedup for Newton-ADMM and SGD.}
\small
\label{CPU-GPU-ratio}
\centering
\begin{tabular}{|c|c|c|}
\hline
\textbf{\begin{tabular}[c]{@{}c@{}}CPU/GPU \\ Time Ratio\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Newton-ADMM\end{tabular}} & \textbf{SGD} \\ \hline
\textbf{MNIST}                                                         & 44.7345904                                                     & 0.47896507   \\ \hline
\textbf{CIFAR-10}                                                      & 112.670178                                                     & 0.8212862    \\ \hline
\textbf{HIGGS}                                                         & 11.842679                                                      & 0.26789652   \\ \hline
\textbf{E18}                                                           & 154.425688                                                     & 1.54673642   \\ \hline
\end{tabular}
\end{table}

Second, we observe that Newton-ADMM has much lower communication cost, compared to SGD. This can be observed from the Figure \ref{fig:ADMM_SGD_Compare_1e-5}. In all cases, SGD takes longer than Newton-ADMM with GPUs enabled. This is mainly because SGD requires a large number of gradient communications across nodes. As a result, we observe that Newton-ADMM is 4.9x, 6.3x, 22.6x, and 17.8x, times faster than SGD on MNIST, CIFAR-10, HIGGS, and E18 datasets, respectively.

Finally, we conclude that Newton-ADMM has superior convergence properties compared to SGD. This is demonstrated in Figure \ref{fig:ADMM_SGD_Compare_1e-5} for the HIGGS dataset. We observe that Newton-ADMM converges to low objective values in just few iterations. On the other hand, the objective value, even at 100-th epoch for SGD, is still higher than Newton-ADMM.

\paragraph{\textbf{Comparison with Distributed Second-order Methods.}}
We compare Newton-ADMM against DANE \cite{daneshmand2016dynaNewton}, AIDE \cite{reddi2016aide}, and GIANT \cite{wang2017giant}, which have been shown in recent results to perform well.
In each iteration, DANE~\cite{daneshmand2016dynaNewton} requires an exact solution of its corresponding subproblem at each node. This constraint is relaxed in an inexact version of DANE, called InexactDANE \cite{reddi2016aide}, which uses SVRG \cite{johnson2013accelerating} to approximately solve the subproblems. Another version of DANE, called Accelerated Inexact
DanE (AIDE), proposes techniques for accelerating convergence, while still using InexactDANE to solve individual subproblems \cite{reddi2016aide}. However, using SVRG to solve subproblems is computationally inefficient due to its double loop formulation, with the outer loop requiring full gradient recalculation and several stochastic gradient calculations in inner loop.

\begin{figure}[h]
\centering
    \begin{subfigure}[t]{0.5\textwidth}
       \hspace*{-1.5cm} \includegraphics[width=85mm]{./plot/DANE_0_00001.png}
    \end{subfigure}%
\caption{Training objective function and test accuracy comparison over time for Newton-ADMM, GIANT, InexactDANE, and AIDE on MNIST dataset with $\lambda=10^{-5}$. We run both Newton-ADMM and GIANT for 100 epochs. Since the computation times per epoch for InexactDANE and AIDE are high, we only run 10 epochs for these methods. We present details of hyperparameter settings in \ref{hyparameters}.}
\label{fig:DANE_weak_0_00001}
\end{figure}

Figure \ref{fig:DANE_weak_0_00001} shows the comparison between these methods on the MNIST dataset. Although InexactDANE and AIDE start at lower objective function values, the average epoch time compared to Newton-ADMM and GIANT is orders of magnitude higher (\textit{order of 1000x}). For instance, to reach an objective function value less than 0.25 on the MNIST dataset, Newton-ADMM takes only 2.4 seconds, whereas InexactDANE consumes \textit{an hour and a half}. Since InexactDANE and AIDE are significantly slower than Newton-ADMM and GIANT (on other datasets as well -- for which we do not show results here), we restrict our discussion of results on performance and scalability to Newton-ADMM and GIANT in the rest of this section.

\paragraph{\textbf{Scalability of Newton-ADMM.}} Figure \ref{fig:avg_t} presents strong- and weak-scaling results for Newton-ADMM and GIANT. In strong-scaling experiments, we keep the number of training samples constant, while increasing the number of workers, and for weak-scaling, the number of the training samples per node is kept constant. For strong-scaling, as number of workers increases, average epoch time for both Newton-ADMM and GIANT decreases. For all the datasets, as the number of workers is doubled, the average epoch time halved for both methods. For weak scaling, as the number of workers doubles, the average epoch time nearly remains constant for both methods. Both Newton-ADMM and GIANT use CG to compute Newton directions. However, compared to GIANT, Newton-ADMM has lower epoch times for the following reasons: first, to guarantee global convergence on non-quadratic problems, GIANT uses a globalization strategy based on line search. For this, the $i$-th worker computes the local objective function values $f_{\mathcal{D}_i}(\xx_{i}+\alpha\pp)$ for all $\alpha$'s in a pre-defined set of step-sizes $S=\{2^0,2^{-1},...,2^{-k}\}$, where $k$ is the maximum number of line search iterations. Thus, for each epoch, all workers need to compute a fixed number of objective function values. In contrast, Newton-ADMM performs line search only locally, allowing each worker to terminate line search before reaching the maximum number of line search iterations, and hence reducing the overhead of redundant computations. Second, Newton-ADMM only requires one round of messages per iteration, whereas GIANT needs three. Our experiments are performed on a Gigabit-interconnet cluster, where communication fabric is highly optimized. However, in environments with lower bandwidth and higher latency, we expect Newton-ADMM to perform significantly better compared to GIANT.

\begin{figure*}[h]

    \begin{subfigure}[t]{0.5\textwidth}
    \centering
    \hspace*{-1.0cm}
    \hspace*{1.0cm}
        \includegraphics[width=120mm]{./plot/Legend_avg_t.pdf}
    \end{subfigure}\\
    \begin{subfigure}[t]{0.5\textwidth}
    \hspace*{1.0cm}
    \includegraphics[width=100mm]{./plot/strong_s_t.pdf}
    \end{subfigure}\\
    \begin{subfigure}[t]{0.5\textwidth}
    \hspace*{1.0cm}
    \includegraphics[width=100mm]{./plot/weak_s_t.pdf}
    \end{subfigure}%
\caption{Avg. Epoch Time for Strong and Weak Scaling for Newton-ADMM and GIANT.}
\label{fig:avg_t}
\end{figure*}

We now compare the convergence of Newton-ADMM with GIANT in a distributed setting. Instead of comparing the test accuracy or objective value over time, we compare how close the objective value obtained from the solver is to the optimal objective value. Specifically, define $\theta = ({F(\xx^{k})-F(\xx^{*})})/{F(\xx^{*})}$, we measure $\theta$ as a function of time. (Here, $F(.)$ denotes the objective function, $\xx^{k}$ is the approximate solution obtained by the solver at the k-th iterate, and the ``optimal'' solution vector $\xx^{*}$ is obtained by running Newton's method on a single node to high precision).  Figure \ref{fig:ADMM_GIANT_t} shows $\theta$, in log scale, as a function of time for MNIST, CIFAR-10, and HIGGS using 8 compute nodes. From Figure \ref{fig:ADMM_GIANT_t}, we observe that, given the same amount of time, Newton-ADMM can reach lower $\theta$ in each case.
 We also measure the number of epochs taken by the solver to reach $\theta < 0.05$. Table \ref{t2} shows the number of epochs for Newton-ADMM and GIANT to reach $\theta < 0.05$ on 8 nodes.
 \begin{table}[h]
\caption{Performance comparison of Newton-ADMM and GIANT -- we present the number of epochs for a solver to reach $\theta <0.05$. The speedup ratio is defined as the fraction of time taken by GIANT to achieve a specified value of $\theta$ to the corresponding time taken by Newton-ADMM on the same hardware platform.}
\centering
\small
\label{t2}
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{}         & \textbf{\begin{tabular}[c]{@{}c@{}}NT-ADMM\\ Epochs\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}GIANT\\ Epochs\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Speedup \\ \end{tabular}} \\ \hline
\textbf{MNIST}    & 252                                                                         & 1086                                                                   & 5.15                                                               \\ \hline
\textbf{CIFAR-10} & 1204                                                                         & 3215                                                                   & 11.14                                                              \\ \hline
\textbf{HIGGS}    & 1                                                                            & 1                                                                      & 1.35                                                               \\ \hline
\end{tabular}
\end{table}

\begin{figure*}[t]
   \centering
    \begin{subfigure}[h]{0.5\textwidth}
        \hspace*{-3.5cm}
        \includegraphics[width=110mm]{./plot/ADMM_GIANT.pdf}
        \caption{}
        \label{fig:ADMM_GIANT_t}
    \end{subfigure}%
    \hfill
    \begin{subfigure}[h]{0.45\textwidth}
        \hspace*{-0.2cm}
        \includegraphics[width=90mm]{./plot/ADMM_GIANT_E18.pdf}
        \caption{}
        \label{fig:ADMM_GIANT_E18}
    \end{subfigure}%
    \caption{Convergence and scalability performance of Newton-ADMM. Figure \ref{fig:ADMM_GIANT_t} shows $\log_{10}(\theta)$ as a function of time for Newton-ADMM and GIANT on MNIST, CIFAR-10, and HIGGS datasets. Newton-ADMM can reach lower $\theta$, given the same amount of time, compared to GIANT. Note that for the HIGGS dataset, both methods can reach low $\theta$ soon. Figure \ref{fig:ADMM_GIANT_E18} shows training objective function and test accuracy as function of time for Newton-ADMM and GIANT on E18 dataset using 32 nodes. We note that GIANT lingers at higher objective values in the initial iterations, while Newton-ADMM drops to lower objective values rapidly.}
    \label{fig:simulation}
\end{figure*}

From Table \ref{t2}, we can see that Newton-ADMM converges to optimal solution significantly faster than GIANT.  Specifically, to reach $\theta \leq 0.05$, for the MNIST dataset, Newton-ADMM takes 252 epochs while GIANT takes 1086 epochs. For the CIFAR-10 dataset, Newton-ADMM takes 1204 epochs while GIANT takes 3215 epochs. The speed up ratio on MNIST and CIFAR-10 is 5.15 and 11.14, respectively. Both Newton-ADMM and GIANT behave well on HIGGS. It only takes 1 epoch for both solvers to reach $\theta \leq 0.05$. We note that the superior performance of these methods on HIGGS does not carry over to first-order methods.

 Finally, we stress that Newton-ADMM scales well on large datasets in large-scale distributed environments. From Figure \ref{fig:ADMM_GIANT_E18}, we note that Newton-ADMM takes significantly smaller amount of time to achieve lower objective values and higher test accuracy on E18 running on 32 compute nodes.
 The large dimensionality of E18 (280K) highlights the memory- and compute-efficient formulation of our Hessian-vector products and subproblem solves on GPUs (please see section ~\ref{sec:MVP} for full GPU utilization characteristics of our solvers) -- the average epoch time for the E18 dataset on 32 nodes is only 1.98 seconds!

 \section{Conclusions and Future Directions}
\label{sec:conclusions}

We have developed a novel distributed Inexact Newton method based on a global consensus ADMM formulation.
We compare our method with
state-of-the-art optimization methods and show that our method has much lower distributed computing overhead, achieves
superior generalization errors, and has significantly lower epoch-times on standard benchmarks.
We have also shown that our method can handle large datasets, while delivering sub-second epoch times -- establishing desirable scalability characteristics of our method.  Our results  establish Inexact Newton-ADMM as the new benchmark for
performance of distributed optimization techniques.

\section{Introduction}
Estimating parameters of a model from a given dataset is a critical component of a wide variety of machine learning (ML) applications.
The parameter estimation problem often translates to one of finding a minima of a suitably formulated objective function. The key challenges in modern ``big-data'' problems relate to very large numbers of model parameters (which translates to high dimensional optimization problems), large training sets, and learning models with low generalization errors. Recognizing the importance of the problem, a significant amount of research effort has been invested into addressing these challenges.

The most commonly used optimization technique in machine learning is gradient descent and its stochastic variant, stochastic gradient descent (SGD).
Gradient descent algorithms, which solely rely on gradient information, are often referred to as first-order methods. Recent results \cite{kylasa2018gpu,berahas2017investigation,roosta2019sub} have shown that the use of curvature information in the form of Hessian, or approximations thereof, can lead to significant improvements in terms of performance as manifest in their convergence rate, time, and quality of solutions.

A key challenge in optimization for machine learning problems is the large, often, distributed nature of the training dataset. It may be infeasible to collect the entire training set at a single node and process it serially because of resource constraints (the training set may be too large for a single node, or that the associated data transfer overhead may be large), privacy  (data may be constrained to specific locations), or the need for reducing optimization time. In each of these cases, there is a need for optimization methods that are suitably adapted to parallel and distributed computing environments.

Distributed optimization solvers adopt one of two strategies -- (i) executing each operation in conventional solvers (e.g., SGD or (quasi) Newton) in a distributed environment \cite{chen2016revisiting,goyal2017accurate,jin2016scale,dean2012large,wang2017giant,crane2019dingo,daneshmand2016dynaNewton,reddi2016aide,zhang2015disco}; or (ii) executing an ensemble of local optimization procedures that operate on their own data, with a coordinating procedure that harmonizes the models over iterations \cite{xu2017adaptiveb,xu2017adaptivea}.  The trade-offs between these two methods are relatively well understood in the context of existing solvers -- namely that the communication overhead of methods in the first class is higher, whereas, the convergence rate of the second class of methods is compromised. For this reason, methods in the first class are generally preferred in tightly coupled data-center type environments, whereas methods in the latter class are preferred for wide area deployments.

Alternating Direction Method of Multipliers (ADMM), is a well known method in
distributed optimization for solving consensus problems \cite{boyd2011distributed}.
To achieve superior convergence and efficient solution of the corresponding sub-problems, the choices of penalty parameters associated with global consensus and inner (node-local) sub-problem solver are critical. In particular, the quality of inner sub-problem solutions dictates the accuracy of the descent direction computed by ADMM. In this paper, we present a novel solver that uses the Spectral Penalty Selection (SPS) method in ADMM \cite{xu2017adaptiveb} for setting the penalty parameters and a variant of Newton's method as sub-problem solver.
Our choices are motivated by the observation that first-order solvers are known to suffer from slow convergence rates, and are notoriously sensitive to problem ill-conditioning and the choice of hyper-parameters. In contrast, Newton-type methods are less sensitive to such adversarial effects. However, this feature comes with increased per-iteration computation cost. In our solution, we leverage lower iteration counts to minimize communication cost and efficient GPU implementations to address increased computational cost.

\paragraph{\textbf{Contributions:}}
Our contributions in this paper can be summarized as follows:
\textit{
\begin{itemize}
\item We propose a novel distributed,
GPU-accelerated Newton-type method based on an ADMM framework that has low communication overhead, good per-iteration compute characteristics through effective use of GPU resources, superior convergence properties, and minimal resource overhead.
\item Using a range of real-world datasets (both sparse and dense), we demonstrate that our proposed method yields significantly better results compared to a variety of state-of-the-art distributed optimization methods.
\item Our pyTorch implementation is publicly available\footnote{https://github.com/fang150/Newton\_ADMM}. Our solver can be readily used for practical applications by data scientists and can be easily integrated with other well-known tools like Tensoflow.
\end{itemize}}
\section{Problem Formulation and Algorithm Details}
\label{sec:theory}

In this section, we describe the optimization problem formulation, and present our proposed Newton-ADMM optimizer.

\paragraph{\textbf{Notation}}
We use bold lowercase letters to denote vectors, e.g., $\vv$, and bold upper case letters to denote matrices, e.g.,
$ \vec{V} $. $\nabla f(\xx)$ and $\nabla^{2} f(\xx)$ represent the gradient and the Hessian of function $f$ at $\xx$,
respectively. The superscript, e.g., $\xx^{(k)}$, denotes iteration count, and the subscript, e.g., $ \xx_{i} $,
denotes the \textit{local}-value of the vector $ \xx $ at the $ i^{th} $ compute node in a distributed setting.
$\mathcal{D}$ denotes the input dataset, and its cardinality is denoted by $|\mathcal{D}|$.
Function $ F_i(\xx) $ represents the objective
function, $ F(\xx) $, evaluated at point $ \xx $ using $ i^{th}-$ observation. Function $ F_{\mathcal{D}}(\xx) $
represents the objective function evaluated on the entire dataset $\mathcal{D}$.

\subsection{Problem Formulation} \label{sec:multi_class}

Consider a finite sum optimization problem of the form:
\begin{align}
\label{eq:obj}
\min_{\xx \in \mathbb{R}^d} F(\xx) \triangleq \sum_{i=1}^n f_i(\xx) + g(\xx),
\end{align}
where each $f_{i}(\xx)$ is a smooth convex function and $ g(\xx) $ is a (strongly) convex and smooth regularizer. In ML applications, $f_{i}(\xx)$ can be viewed as loss (or misfit) corresponding to the $i^{th}$ observation (or measurement)\cite{tibshirani1996regression, friedman2001elements,bottou2016optimization, sra2012optimization}. In our study, we choose multi-class classification using soft-max and cross-entropy loss function, as an important instance of finite sum minimization problem. Consider a $p$
dimensional feature vector $\aa$, with corresponding labels $b$, drawn from $C$ classes. In such a classifier, the probability that $\aa$ belongs to a
class $c \in \{1,2,\ldots,C\}$ is given by:
\begin{align*}
\Pr\left(b = c \mid \aa,\xx_{1},\ldots, \xx_{C}\right) = \frac{e^{\lin{\aa,\xx_{c}}}}{\sum_{c' = 1}^{C} e^{\lin{\aa, \xx_{c'}}}},
\end{align*}
where $ \xx_{c} \in \mathbb{R}^{p}$ is the weight vector corresponding to class $ c $.
Recall that there are only $ C-1 $ degrees of freedom, since probabilities must
sum to one. Consequently, for training data
$\{\aa_{i},b_{i}\}_{i=1}^{n} \subset \mathbb{R}^{p} \times \{1,2,\ldots,C\}$,
the cross-entropy loss function for $ \xx = [\xx_{1}; \xx_{2}; \ldots; \xx_{C-1}] \in \mathbb{R}^{(C-1)p} $
can be written as:
\begin{align}
\label{eq:softmax_log_likelihood}
&F(\xx) \triangleq F(\xx_{1},\xx_{2},\ldots,\xx_{C-1}) \nonumber \\
= &\sum_{i=1}^{n} \left(\log \left(1+\sum_{c' = 1}^{C-1} e^{\lin{\aa_{i}, \xx_{c'}}}\right) - \sum_{c = 1}^{C-1}\mathbf{1}(b_{i} = c) \lin{\aa_{i},\xx_{c}}\right).
\end{align}
Note that  $ d = (C-1) p $.  After the training phase, a new data instance $\aa$ is classified as:
\begin{align*}
b = \arg \max &\left\{\left\{\frac{e^{\lin{\aa,\xx_{c}}}}{\sum_{c' = 1}^{C-1} e^{\lin{\aa, \xx_{c'}}}}\right\}_{c=1}^{C-1}, 1- \frac{e^{\lin{\aa,\xx_{1}}}}{\sum_{c' = 1}^{C} e^{\lin{\aa, \xx_{c'}}}}  \right\}.
\end{align*}

\subsection{Numerical Stability}
\label{sec:numerical_stability}

To avoid over-flow in the evaluation of exponential functions
in~\eqref{eq:softmax_log_likelihood}, we use the ``Log-Sum-Exp''
trick~\cite{murphy2012machine}. Specifically, for each data point $\aa_{i}$, we first find
the maximum value among $\lin{\aa_{i},\xx_{c}}, \; c=1,\ldots, C-1$. Define:
\begin{align}
\label{eq:max_x}
M(\aa) = \max \Big\{ 0, \lin{\aa,\xx_{1}}, \lin{\aa,\xx_{2}}, \ldots, \lin{\aa,\xx_{C-1}} \Big\},
\end{align}
and
\begin{align}
\label{eq:log_sum_trick}
\alpha(\aa) \mathrel{\mathop:}= e^{-M(\aa)}+\sum_{c' = 1}^{C-1} e^{\lin{\aa, \xx_{c'}} - M(\aa)}.
\end{align}

Note that $M(\aa) \geq0, \alpha(\aa) \geq 1$. Now, we have:
\begin{align*}
1+\sum_{c' = 1}^{C-1} e^{\lin{\aa_{i}, \xx_{c'}}}
& = e^{M(\aa_{i})} \alpha(\aa_{i}).
\end{align*}
For computing~\eqref{eq:softmax_log_likelihood}, we use:
\begin{align*}
& \log \left(1+\sum_{c' = 1}^{C-1} e^{\lin{\aa_{i}, \xx_{c'}}}\right) \\
&= M(\aa_{i}) + \log \left(e^{-M(\aa_{i})}+\sum_{c' = 1}^{C-1} e^{\lin{\aa_{i}, \xx_{c'}} - M(\aa_{i})}\right) \\
&= M(\aa_{i}) + \log \big( \alpha(\aa_{i}) \big).
\end{align*}
Note that in all these computations, we are guaranteed to have all the exponents appearing
in all the exponential functions to be negative, hence avoiding numerical over-flow.

\subsection{ADMM Framework}

Let $ \mathcal{N} $ denote the number of nodes (compute elements) in the distributed environment.
Assume that the input dataset $ \mathcal{D} $ is split among the $ \mathcal{N} $ nodes as $ \mathcal{D} = \mathcal{D}_{1} \cup \mathcal{D}_{2} \ldots \cup \mathcal{D}_{\mathcal{N}} $.
Using this notation, \eqref{eq:obj} can be written as:
\begin{align}
\label{eq:dist-obj}
\min & \sum_{i = 1}^{\mathcal{N}}  \sum_{j \in \mathcal{D}_i} f_j( \xx_i ) + g( \zz )   \\
& \textrm{s.t.} \quad  \xx_i - \zz = 0 , \quad  i = 1, \ldots, \mathcal{N}, \nonumber
\end{align}
where $ \zz $ represents a global variable enforcing consensus among $\xx_{i}$'s at all the nodes.
In other words, the constraint enforces a consensus among the nodes so that all the local variables,
$ \xx_i $, agree with global variable $\zz$. The formulation \eqref{eq:dist-obj} is often referred to as a \textit{global consensus} problem.  ADMM is based on an augmented Lagrangian framework; it solves the global consensus problem by alternating iterations on primal/ dual variables. In doing so, it inherits the benefits of the decomposability of dual ascent and the superior convergence properties of the method
of multipliers.

ADMM methods introduce a penalty parameter $\rho$, which is the weight on the measure of \textit{disagreement} between $ \xx_{i} $'s and
global consensus variable, $ \zz $. The most common adaptive penalty parameter selection is
Residual Balancing \cite{boyd2011distributed}, which tries to balance the
dual norm and residual norm of ADMM.
Recent empirical results using SPS \cite{xu2017adaptiveb}, which is based on the estimation of the local curvature of subproblem on each node, yields significant improvement in the efficiency of ADMM. Using the SPS strategy for penalty parameter selection, ADMM iterates can be written as follows:
{\small
\begin{subequations}
\begin{align}
\xx_{i}^{k+1} &= \argmin_{\xx_i} f_i(\xx_i) + \frac{\rho_i^{k}}{2} || \zz^{k} - \xx_i + \frac{\yy_{i}^{k}}{\rho_i^{k}} ||_{2}^{2}, \label{eq:x-update} \\
\zz^{k+1} &= \argmin_{\zz} g(\zz) + \sum_{i=1}^{\mathcal{N}}  \frac{\rho_i^{k}}{2} || \zz -\xx_{i}^{k+1} +\frac{\yy_{i}^{k}}{\rho_i^{k}}  ||_{2}^{2}, \label{eq:z-update} \\
\yy_{i}^{k+1} &= \yy_{i}^{k} + \rho_i^{k} (\zz^{k+1}- \xx_i^{k+1} ). \label{eq:y-update}
\end{align}
\end{subequations}
}%
With $\ell_{2}-$regularization, i.e., $ g(\xx) = \lambda \|\xx\|^{2}/2 $, \eqref{eq:z-update} has a closed-form solution given by
\begin{align}
\zz^{k+1}(\lambda + \sum_{i=1}^{\mathcal{N}}\rho_i^{k}) = \sum_{i=1}^{\mathcal{N}} \bigl[\rho_i^{k} \xx_i^{k+1}- \yy_i^{k}  \bigr],
\end{align}
where $\lambda$ is the regularization parameter.

Algorithm \ref{alg:dist-Newton} presents our proposed method incorporating the above formulation of ADMM.

\begin{algorithm} [!htbp]
	\caption{ADMM method (outer solver)}
	\label{alg:dist-Newton}
	\SetAlgoLined
	\SetKwInOut{Input}{Input}
	\SetKwInOut{Parameter}{Parameters}
	\Input{$\xx^{(0)}$ (initial iterate),  $\mathcal{N}$ (no. of nodes) }
	\Parameter{ $ \beta $, $ \lambda $ and $ \theta < 1$}
	Initialize $\zz^{0}$ to 0 \label{dn:line:1} \\
    Initialize $\yy_i^{0}$ to 0 on all nodes.\label{dn:line:2} \\
	\ForEach{$k = 0,1,2,\ldots$}{
	 Perform Algorithm \ref{alg:standalone-Newton} with, $\xx_i^{k}$, $\yy_i^{k}$, and $\zz^{k}$ on all nodes \\
	 Collect all local $\xx_i^{k+1} $ \\
		 Evaluate $\zz^{k+1}$ and $\yy_i^{k+1}$ using  \eqref{eq:z-update} and  \eqref{eq:y-update}. \\
		\lnl{dn-7} Distribute $\zz^{k+1}$ and $\yy_i^{k+1}$ to all nodes. \\
		 Locally, on each node, compute spectral step sizes and penalty parameters as in \cite{xu2017adaptiveb}
	}
\end{algorithm}

Steps  \ref{dn:line:1}-\ref{dn:line:2} initialize the multipliers, $\yy$, and consensus vectors, $\zz$, to  zeros.
In each iteration, Single Node Newton method, Algorithm \ref{alg:standalone-Newton}, is run with
local $\xx_{i}$, $\yy_{i}$, and global $\zz$ vectors. Upon termination of
Algorithm \ref{alg:standalone-Newton} at all nodes, resulting local Newton directions, $\xx_{i}^{k}$,
are gathered at the master node, which generates the next iterates for vectors $\yy$ and $\zz$ using spectral
step sizes described in \cite{xu2017adaptiveb}.
These steps are repeated until convergence.

\begin{remark}
Note that in each ADMM iteration only \emph{one} round of
communication is required (a ``gather'' and a ``scatter'' operation),
which can be executed in $\mathcal{O}(\log (\mathcal{N}))$ time. Further, the application of the GPU-accelerated inexact Newton-CG Algorithm \ref{alg:standalone-Newton} at each node significantly speeds-up the local computation per epoch. The combined effect of these algorithmic choices contribute to the high overall efficiency of the proposed Newton-ADMM Algorithm \ref{alg:dist-Newton}, when applied to large datasets.
\end{remark}

\subsubsection{ADMM Residuals and Stopping Criteria }
The consensus problem \eqref{eq:dist-obj} can be solved by combining ADMM subproblems \eqref{eq:x-update}, \eqref{eq:y-update}, and \eqref{eq:z-update}. To monitor the convergence of ADMM, we can check the norm of primal and dual residuals, $\rr^k$ and $\dd^k$, which are defined as follows:
\begin{align}
\rr^{k} = \begin{bmatrix}
           \rr_{1}^k \\
           \vdots \\
          \rr_{\mathcal{N}}^k
         \end{bmatrix}, \dd^{k} = \begin{bmatrix}
           \dd_{1}^k \\
           \vdots \\
          \dd_{\mathcal{N}}^k
         \end{bmatrix}
\end{align}
where $\forall i \in \{1,2,\dots,\mathcal{N}\}$,
\begin{align}
\rr_{i}^k = \zz^k - \xx_{i}^k, \dd_{i}^k =- \rho_i^{k}(\zz^{k}-\zz^{k-1})
\end{align}
 As $k\rightarrow\infty$,  $\zz^k \rightarrow\zz^*$ and  $\forall i, \xx_{i}^{k}\rightarrow\zz^*$. Therefore, the norm of primal and dual residuals, $|| \rr^k ||$ and $|| \dd^k ||$, converge to zero. In practice, we do not need the solution to high precision, thus ADMM can be terminated as $|| \rr_{i}^k || \leq \epsilon^{pri}$ and $|| \dd_{i}^k || \leq \epsilon^{dual}$ . Here, $\epsilon^{pri}$ and $\epsilon^{dual}$ can be chosen as:
\begin{flalign}
&\epsilon^{pri}=\sqrt{\mathcal{N}}\epsilon^{abs}+\epsilon^{rel}\max\{\sum_{i=1}^{\mathcal{N}}||\xx_{i}^k||^2,\mathcal{N}||\zz^{k}||^2\}\\
&\epsilon^{dual}=\sqrt{d}\epsilon^{abs}+\epsilon^{rel}\max\{\sum_{i=1}^{\mathcal{N}}||\yy_{i}^k||^2\}
\end{flalign}
The choice of absolute tolerance $\epsilon^{abs}$ depends on the chosen problem and the choice of relative tolerance $\epsilon^{rel}$ for the stopping criteria is, in practice, set to $10^{-3}$ or $10^{-4}$.

\subsection{Inexact Newton-CG Solver}
The optimization problem \eqref{eq:obj} is decomposed by ADMM framework into sub-problems \ref{eq:x-update}, \ref{eq:y-update}, and \ref{eq:z-update}. Among these sub-problems, only \ref{eq:x-update} does not have closed form solution. Thus, it is critical to find an iterative method that can produce high quality solutions with low computation cost. To this end, we develop an \textit{inexact Newton-CG solver} for  solving sub-problem \ref{eq:x-update}. Let the objective in Equation \ref{eq:x-update} be $\hat{f}(\xx)$, in each iteration; the gradient and Hessian are given by:\\
\begin{subequations}\label{eq:Newton_G_H}

\begin{equation}
\bgg(\xx) \triangleq  \sum_{j \in \mathcal{D}} \nabla \hat{f}_{j}(\xx) ,\label{Newton_G}
\end{equation}

\begin{equation}
\HH(\xx) \triangleq  \sum_{j \in \mathcal{D}} \nabla^{2} \hat{f}_{j}(\xx) . \label{Newton_H}
\end{equation}

\end{subequations}
At each iterate $ \xx^{(k)} $, using the corresponding Hessian, $\HH(\xx^{(k)})$,
and the gradient, $\bgg(\xx^{(k)})$, we consider \emph{inexact} Newton-type iterations of the form:
\begin{subequations}
\begin{align}
\label{eq:update}
\xx^{(k+1)} = \xx^{(k)} + \alpha_{k} \pp_{k},
\end{align}
where $ \pp_{k} $ is a search direction satisfying:
\begin{align}
\label{eq:inexact}
\| \HH(\xx^{(k)})\pp_{k} + \bgg(\xx^{(k)})\| \leq \theta \|\bgg(\xx^{(k)})\|,
\end{align}
for some inexactness tolerance $ 0 < \theta < 1 $ and $\alpha_{k}$ is the largest $\alpha \leq 1$ such that
\begin{align}
F(\xx^{(k)} + \alpha \pp_{k}) \leq F(\xx^{(k)}) + \alpha \beta \pp_{k}^{T} \bgg(\xx^{(k)}),
\label{eq:armijo}
\end{align}
for some $\beta \in (0,1)$.
\label{eq:inexact_Newton_itrs}
\end{subequations}

Requirement \eqref{eq:armijo} is often referred to as Armijo-type
line-search \cite{nocedal2006numerical}. To compute the step-size, $\alpha$ in eq.  \eqref{eq:armijo},
~we use a \textit{backtracking} line search, as shown in algorithm \ref{alg:linesearch}.
This function takes parameters, $\alpha $ as initial step-size, which in our case is always set to 1, $ \beta < 1 $, $\pp$ is the Newton-direction,
and the gradient vector is $\bgg$. The loop at line \ref{ls:line:3} is repeated until desired reduction
is achieved along the Newton-direction, $\pp$, by successively decreasing the step-size by a factor $ \gamma < 1$.

\begin{algorithm} [!htbp]
\caption{Line Search}
\label{alg:linesearch}
	\SetAlgoLined
	\SetKwInOut{Input}{Input}
	\SetKwInOut{Parameter}{Parameters}

	\Input{  \\
		$ \xx $ - Current point \\
	         $ \pp $ - Newton's direction \\
			 $F(.) $ - Function pointer\\
			 $\bgg(\xx)$ - Gradient vector
			 }
	\Parameter{\\
			$\alpha$ - Initial step size \\
			$0< \beta < 1$ - Sufficient descent constant\\
			$0< \gamma < 1$ - Back-tracking parameter \\
			$ i_{\max} $ - Maximum line search iterations
			}
	$\alpha = 1$ \\
    $ i = 0 $ \\
	\While{ $ F ( \xx + \alpha  \pp) > F ( \xx ) + \alpha \beta  \pp^{T} \bgg(\xx) $ }{ \label{ls:line:3}
		\If{ $ i > i_{\max} $}{
			\lnl{ls:5} break \\
		}
		$ i = i + 1 $\\
		$ \alpha \leftarrow \gamma \alpha $	}
\label{alg-ls}
\end{algorithm}

Condition \eqref{eq:inexact} is the
$\theta$-relative error approximation of the exact solution to the linear system:
\begin{align}
\label{eq:exact}
\HH(\xx^{(k)}) \pp_{k} &= -\bgg(\xx^{(k)}),
\end{align}
Note that in (strictly)
convex settings, where the Hessian matrix is symmetric positive definite (SPD),
conjugate gradient (CG) with early stopping can be used to obtain an approximate solution
to \eqref{eq:exact} satisfying \eqref{eq:inexact}. In \cite{roosta2019sub}, it has been shown that a mild value for $ \theta $, in the order of inverse of \emph{square-root of the condition number}, is sufficient to ensure that the convergence properties of the exact Newton's method are preserved. As a result, for ill-conditioned problems, an approximate solution to \eqref{eq:exact} using CG yields good performance, comparable to an exact update (see examples in Section \ref{sec:expt}). Putting all of these together, we obtain Algorithm \ref{alg:standalone-Newton}, which
is known to be globally linearly convergent, with problem-independent local convergence rate \cite{roosta2019sub}.

\begin{algorithm} [!htbp]
	\caption{Inexact Newton-type Method}
	\label{alg:standalone-Newton}
	\SetAlgoLined
	\SetKwInOut{Input}{Input}
	\SetKwInOut{Parameter}{Parameters}
	\Input{$\xx^{(0)}$}
	\Parameter{$0 < \beta, \theta < 1$}
	\ForEach{$k = 0,1,2,\ldots$}{
		Form $\bgg(\xx^{(k)})$ and $\HH(\xx^{(k)})$ as in \eqref{eq:Newton_G_H} \\
		\If{$\|\bgg(\xx^{(k)})\| < \epsilon$}{
			STOP
		}
		Update $\xx^{(k+1)}$ as in \eqref{eq:inexact_Newton_itrs}\\
	}
\end{algorithm}

\subsection{GPU Utilization and Communication Overhead}
\label{GPU-utils}
 Our proposed methods use Inexact Newton-type iterates, with a linear-quadratic convergence rate for strongly convex sub-problems \eqref{eq:x-update}. Besides theoretical performance guarantees, our Newton-type method has practical advantages over first order methods. In practice, mini-batch stochastic gradient descent is widely used over full-batch gradient descent and other methods. However, this method often requires a large number of epochs to achieve good generalization errors. Furthermore, the mini-batch update scheme results in significantly lower GPU occupancy (idle GPU cores because of smaller batch sizes). The number of CPU-GPU memory transfers per epoch for mini-batch SGD is $\frac{n}{m}$, where $n$ is the size of dataset, and $m$ is the size of mini-batch. Usually, $n >> 1$  and $m$ is typically between a hundred and a thousand. In contrast, Newton's method utilizes the complete dataset for computing Newton direction. Therefore, there is only one CPU-GPU memory transfer for computing Newton direction, which greatly increases utilization of the GPU for reasonably sized datasets. With a judicious mix of statistical methods and carefully formulated Hessian-vector operations, as discussed in Section \ref{sec:MVP}, we are able to transform this computation-heavy operation into an efficient and highly scalable GPU-accelerated operation with low memory overhead. For this reason, Newton's method, in general, is more amenable to high GPU utilization, due to higher concurrency and lower CPU-GPU data transfer cost, compared to SGD. Furthermore, in distributed implementations, Synchronous SGD induces a communication overhead of $\frac{nd\log(\mathcal{N})}{m\mathcal{N}}$, while ADMM only requires one round of communication. In solving high dimensional problems, where $d >> 1$, this overhead is significant when network bandwidth is limited. The gains from using Newton method and ADMM, along with low communication and high GPU utilization, as compared to first order methods, are demonstrated in our empirical results.

\subsection{Hessian Vector Product}
\label{sec:MVP}

Given a vector $\vv \in \mathbb{R}^{d}$, we
can compute the Hessian-vector product on the GPU without explicitly forming the Hessian.
For notational simplicity, define
\begin{align*}
h(\aa,\xx) \mathrel{\mathop:}= \frac{e^{\lin{\aa, \xx}-M(\xx)}}{\alpha(\aa)},
\end{align*}
where $M(\xx)$ and $ \alpha(\xx) $ were defined in eqs.~\eqref{eq:max_x} and~\eqref{eq:log_sum_trick}, respectively.
Now using matrices
\begin{align*}
\stepcounter{equation}\tag{\theequation}\label{eq:matrix-a}
\vec{V} = \begin{bmatrix}
\lin{\aa_{1},\vv_{1}} & \lin{\aa_{1},\vv_{2}} & \dots  & \lin{\aa_{1},\vv_{C-1}} \\
\lin{\aa_{2},\vv_{1}} & \lin{\aa_{2},\vv_{2}} & \dots  & \lin{\aa_{2},\vv_{C-1}} \\
\vdots & \vdots & \ddots & \vdots \\
\lin{\aa_{n},\vv_{1}} & \lin{\aa_{n},\vv_{2}} & \dots  & \lin{\aa_{n},\vv_{(C-1)}}
\end{bmatrix}_{n \times (C-1)},
\end{align*}
and
\begin{align*}
\stepcounter{equation}\tag{\theequation}\label{eq:matrix-b}
\W = \begin{bmatrix}
h(\aa_{1},\xx_{1}) & h(\aa_{1},\xx_{2}) & \dots  & h(\aa_{1},\xx_{C-1}) \\
h(\aa_{2},\xx_{1}) & h(\aa_{2},\xx_{2}) & \dots  & h(\aa_{2},\xx_{C-1}) \\
\vdots & \vdots & \ddots & \vdots \\
h(\aa_{n},\xx_{1}) & h(\aa_{n},\xx_{2}) & \dots  & h(\aa_{n},\xx_{C-1}) \\
\end{bmatrix}_{n \times (C-1)},
\end{align*}
we compute
\begin{align*}
\stepcounter{equation}\tag{\theequation}\label{eq:matrix-c}
\vec{U} = \vec{V} \odot \W - \W \odot \Big( \big(\left(\vec{V} \odot \W\right) \ee \big) \ee^{T} \Big),
\end{align*}
to get
\begin{align*}
\stepcounter{equation}\tag{\theequation}\label{eq:hessianvec}
\H\vv = \text{vec} \left( \vec{A}^{T} \vec{U} \right),
\end{align*}
where $ \vv = [\vv_{1};\vv_{2};\ldots;\vv_{C-1}] \in \mathbb{R}^{d}$, $ \vv_{i} \in \mathbb{R}^{p}, i=1,2,\ldots,C-1 $, $\ee \in \mathbb{R}^{C-1}$ is a vector of all $1$'s, and each row of the matrix
$ \vec{A} \in \mathbb{R}^{n \times p}$ is a row vector corresponding to the $i^{th}$ data point,
i.e, $\vec{A}^{T} = \begin{bmatrix} \aa_{1}, \aa_{2}, \ldots, \aa_{n} \end{bmatrix}$.

Note that the memory overhead of our GPU-accelerated Newton-type method is determined by the dimensions of the matrices
$ \vec{U} $, $ \vec{V} $ and $ \W $, which are determined by the local dataset size and number of classes in multi-class classification
problem at hand. With reasonably sized GPU clusters this memory footprint can be easily managed for large datasets.
This enables Newton-type method to scale to large problems inaccessible to traditional second-order methods.

\begin{figure*}[ht]
\hspace*{-2.5cm}
    \begin{subfigure}[t]{0.5\textwidth}
        \includegraphics[width=180mm]{./plot/ADMM_BFGS.pdf}
    \end{subfigure}%
\caption{Training Objective function comparison over time for different choice of inner-solve for ADMM. For the inner solver, we compare the performance of Inexact Newton solver with L-BFGS (with history size 25, 50, 100). The step size of Inexact Newton method is chosen by linesearch following Armijo rule, whereas the step size of L-BFGS is chosen by linesearch satisfying Strong Wolfe condition. We can see that the per-iteration computation cost of L-BFGS is lower than Inexact Newton with the exception on HIGGS dataset. This is because L-BFGS is sensitive to the scale of step size so that more iterations of Strong Wolfe linesearch procedure are required to satisfy the curvature condition. In general, we observe that L-BFGS performs well on binary class problems, while the performance degrades on multiclass problems, when the number of compute nodes increases. }
\label{fig:ADMM_BFGS}
\end{figure*}

\section{Related Research}
\label{sec:related-work}
First-order methods \cite{beck2017first,bubeck2015convex} -- gradient descent and its variants are commonly used in ML applications. This is mainly because these methods are simple to implement and have low per-iteration costs. However,  it is known that these methods often take a large number of iterations to achieve reasonable generalization, primarily due to their sensitivity to problem ill-conditioning.
Second-order methods make use of curvature information, in the form the Hessian matrix, and as a result are more robust to problem ill-conditioning \cite{roosta2019sub}, and hyper-parameter tuning \cite{berahas2017investigation,xuNonconvexEmpirical2017}. However, they can have higher memory and computation requirements due to the application of the Hessian matrix. In this context, quasi-Newton methods \cite{nocedal2006numerical} can be used to approximate the Hessian by using the history of gradients.
However, a history of gradients must be stored in order to approximate the Hessian matrix, and extra computation cost is incurred to satisfy the strong Wolfe condition. In addition, these methods are observed to be unstable when used on mini-batches \cite{kylasa2018gpu}.

Several distributed solvers have been developed recently \cite{chen2016revisiting,goyal2017accurate,jin2016scale,dean2012large,wang2017giant,crane2019dingo,daneshmand2016dynaNewton,reddi2016aide,zhang2015disco}. Among these, \cite{chen2016revisiting,goyal2017accurate,jin2016scale,dean2012large} are classified as first-order methods. Although they incur low computational overhead, they have higher communication costs due to a large number of messages exchanged per mini-batch and high total iteration counts. Second-order variants \cite{wang2017giant,crane2019dingo,daneshmand2016dynaNewton,reddi2016aide,zhang2015disco} are designed to improve convergence rate, as well as to reduce  communication costs. DANE \cite{daneshmand2016dynaNewton}, and the accelerated scheme AIDE \cite{reddi2016aide} use SVRG \cite{johnson2013accelerating} as the subproblem solver to approximate the Newton direction. These methods are often sensitive to the fine-tuning of SVRG. DiSCO \cite{zhang2015disco} uses distributed preconditioned conjugate gradient (PCG) to approximate the Newton direction. The number of communications across nodes per PCG call is proportional to the number of PCG iterations. In contrast to DiSCO, GIANT \cite{wang2017giant} executes CG at each node and approximates the Newton direction by averaging the solution from each CG call. Empirical results have shown that GIANT outperforms DANE, AIDE, and DiSCO. The solver of Dunner et al. \cite{dunner2018distributed} is shown to outperform GIANT, however, it is constrained to sparse datasets. A recently developed variant, DINGO \cite{crane2019dingo}, can be applied to a class of non-convex functions, namely invex \cite{craven1981invex}, which includes convexity as a special sub-class. However, in the absence of invexity, the method can converge to undesirable stationary points.

A popular choice in distributed settings is ADMM \cite{boyd2011distributed}, which combines dual ascent method and the method of multipliers. ADMM only requires one round of communication per iteration. However, ADMM's performance is greatly affected by the selection of the penalty parameter \cite{xu2017adaptiveb,xu2017adaptivea} as well as the choice of local subproblem solvers.
\section{Appendix}
In the appendix, we provide a detailed description of  parameter settings in all experiments.

\subsection{Algorithms Parameter Settings}
We generated all the experiment results using the following settings:
\label{hyparameters}
\begin{itemize}
	\item Synchronous SGD : we tune the step size from $10^{-4}$ to $10^4$ and select the best result to report.
	\item Newton-ADMM : We used 10 CG iterations along with $10^{-4}$ CG tolerance to compute Newton direction at each compute node. The step size was chosen by line search with 10 iterations.
	\item GIANT : The configurations for CG and linesearch are the same as the configurations use in Newton-ADMM.
	\item Inexact DANE : we use learning rate $\eta=1.0$ and regularization term $\mu=0.0$ for solving subproblems as prescribed in \cite{daneshmand2016dynaNewton}. We set SVRG iterations to 100 and update frequency as $2n$, where $n$ is the number of local sample points. We run SVRG step size from the set $10^{-4}$ to $10^{4}$ in increments of 10 and select the best value to report.
	\item AIDE : The configurations for SVRG is the same as the configurations used in Inexact DANE. As to the additional hyper-parameter introduced in AIDE, $\tau$, we also run $\tau$ from the set $10^{-4}$ to $10^{4}$ and select the best to report.
	\item Regularization parameter $\lambda$ : we used $\lambda = 10^{-5}$ for all the experiments.
\end{itemize}

redpajama_set_name: RedPajamaArXiv


The Dupray ONE™ steam cleaner is the most efficient non-invasive method for unclogging food and washroom drains. Visit ONE™ steam cleaner to learn more about using the power of steam to unclog any blocked drain. Using Dupray's unique Plunger tool, which creates a tight seal around the drain gap, dry superheated steam is introduced into the pipe to loosen the material at the blockage point. Whether it be a residential kitchen drain, a shower drain or a bathtub drain, the Dupray ONE™ is always up to the task.


Sorry if similar subject already posted.
What did I missed ? How can I have my Share folder available ?
You have to enable web access from the service option.
The WebAccess service is Enabled in the section Service Setup.
Now go to www.buffalonas.comand enter the buffalonas.com username you specified. If this works correctly you will be asked for a username and password, enter the username "test" and the password "test" (no quotes)This should then log you onto the test folder and if you have put any data on it then the files will be displayed.
That's what I've already done except that I don't want to use BuffaloNAS.com. My IAP has set a unique IP address for my connection and gave me a named address. Port 9000 is routed to the NAS server IP address. When selecting my named address (or my IP address), a Buffalo login screen appears but no folder tree. Just a short menu on the left (with Home, User Management and Logout) and corresponding info on the right... The final address displayed is like "myname.xxxxxxx.fr/cgi-bin/top.cgi". Something's missing?
I found the reason. When not using BuffaloNAS.com, you should end your web address with ":9000" to connect to the Web interface.


text:
Q: Rewrite sql to support joins I have the following query which in oracle databases should work but current wordpress instalation uses a mysql database. Could you please help me out to rewrite the next query so it will work on mysql ?
$wpdb->get_results("SELECT a.ID, a.post_title, DAYOFMONTH(b.meta_value) as dom "
."FROM $wpdb->posts a, $wpdb->postmeta b "
."WHERE b.meta_value >= '{$thisyear}-{$thismonth}-01 00:00:00' "
."AND b.meta_value <= '{$thisyear}-{$thismonth}-{$last_day} 23:59:59' "
."AND a.post_type = 'post' AND a.post_status = 'publish' AND a.ID = b.postid AND   b.meta_key='Event Date'"
);

A: There isn't an issue with your query in MySQL from what I can tell.  Both of the following statements produce the same results (although I prefer to use the JOIN).  Here is a simplified version:
SELECT  *
FROM table1 t
    JOIN table2 t2 on t.id = t2.id
WHERE t.dt >= '2011-01-02 00:00:00';

SELECT  *
FROM table1 t, table2 t2
WHERE t.dt >= '2011-01-02 00:00:00'
   AND t.id = t2.id;

SQL Fiddle Demo

A: You may have a problem with the date format of the constant.  I would suggest converting the date into the same format using to_char().  I'm thinking something along the lines of:
SELECT a.ID, a.post_title, to_char(b.meta_value, 'DD') as dom "
."FROM $wpdb->posts a, $wpdb->postmeta b "
."WHERE to_char(b.meta_value, 'YYYY-MM-DD HH:MI:SS') >= '{$thisyear}-{$thismonth}-01 00:00:00' "
."AND to_char(b.meta_value, 'YYYY-MM-DD HH:MI:SS') <= '{$thisyear}-{$thismonth}-{$last_day} 23:59:59' "
."AND a.post_type = 'post' AND a.post_status = 'publish' AND a.ID = b.postid AND   b.meta_key='Event Date'

Or, more simply:
SELECT a.ID, a.post_title, to_char(b.meta_value, 'DD') as dom "
."FROM $wpdb->posts a, $wpdb->postmeta b "
."WHERE to_char(b.meta_value, 'YYYY-MM') = '{$thisyear}-{$thismonth}' "
."AND a.post_type = 'post' AND a.post_status = 'publish' AND a.ID = b.postid AND   b.meta_key='Event Date'

This assumes that meta-value is stored as a date.  If not, you have to deal with conversion of values into dates.  I'm making this assumption because you are using a function DAYOFMONTH.

redpajama_set_name: RedPajamaStackExchange


text:
Q: Http post over a captcha protected jsp page using curl Hello Im trying to make an Http post over a jsp page using curl. but it's not working.
this is the php code I'm trying.
<?php

if(isset($_POST['submit'])){
$captcha = $_POST['answer'];

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"http://example.com/post_comments.jsp");
curl_setopt($ch, CURLOPT_REFERER, 'http://www.example.com');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,
"post_id=123&body=blabla&title=blabla&name=blablabla8&captcha=$captcha");

curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$result= curl_exec ($ch);
curl_close ($ch);

echo $result;

}else {

?>

<form action="<?php echo $_SERVER['PHP_SELF']; ?>" method="post">

 <input name="answer" />

<input type="submit" name="submit" value="submit">

</form>

<?php
}

?>

Anybody please guide to correct it.
thanks in advance

A: Web pages are made to be navigated by humans, thats why you might miss many things... Like cookies or session data. Your best bet is to make a valid request to that page and copy all the requests made. You can do it in firefox/firebug for example, or wireshark.
In short: you are probably missing cookies, emulate them


Or you didn't get in the lottery. Or you didn't train in time for the race thanks to the flu.
Here are a few other races going on around the region you can still get in on, before the weather warms up too much. By June, I have a feeling we'll all be looking for the nearest air conditioned building, and having to hit the pavement at 5 a.m. just to escape the humidity.
The Hills Are Alive 5K & 10K. March 23 at 9 a.m. Fort Dupont Park. $40-$45, with proceeds going to fund a six-month childhood obesity program sponsored by DC Running Clubs. Noted as the toughest 5K & 10K in Washington DC. More information and registration here.
National Police Week 5K. May 11 at 9 a.m. 4th & F Streets, NW. $35, with proceeds supporting Officer Down Memorial Page and Concerns of Police Survivors. More information and registration here.
Susan G. Komen Global Race for the Cure 5K Run/Walk. May 11 at 8 .m. National Mall. $40-$45 for individual registration. More information and registration available here.
Bowie Annual Spring 5K/2 Mile Walk. March 23 at 8.m. Bowie Town Center. $25, with proceeds going to the Bowie Food Pantry. More Information and registration here.
Rocky on the Run 5K Run/Walk. March 23, 8:30 a.m. Lyndwood Square Shopping Center in Elkridge. $25 for adults, $15 for children. No race day registration. Proceeds will go to the Rockburn Elementary School PTA. More information and registration here.
Warrior Dash. May 18 at 9 .m. Mechanicsville. Mud obstacle run. More information and registration here.
Run for Rights 5K. March 30 at 8 a.m. Bon Air Park and Bluemont Park in Arlington. $25 after March 18, proceeds supporting Amnesty International Mid-Atlantic Region. More information here.
Crystal Run 5K Fridays. April 5, 12, 19, 26 at 6:30 p.m. Crystal City, Arlington. $20 per race, or $60 for all four runs. More information and registration here.
Charlottesville Marathon/Half Marathon/8K. April 6, and noted as "the most scenic marathon course in the East" by Running Times Magazine. More information and registration here.
2013 Parkway Classic 10 Mile & 5K. April 21 at 8 a.m. Old Town Alexandria. More information and registration here.
Virgina Beach Rock 'n Roll Half Marathon – September 1, 2013. More information and registration here.
Ragnar Relay – October 4-5, 2013. More information here.
Army Ten Miler – October 20, 2013. More information and registration here.
Trying to decide if you should head to the Inauguration?


text:
Romone McCrae (born 25 August 1991 in England) is a footballer who last played for Conference South team Sutton United.

Career
He made his professional debut on 5 April 2010 in a 2–2 draw against Barnsley for Peterborough United. After joining Conference National side Histon on a season long loan in August 2010, he scored on his full league debut against Barrow on the opening day of the 2010–11 season. McCrae went on to make a further 17 appearances in the league.

McCrae was released by Peterborough in the summer of 2011, and was signed by London-based Sutton United. Manager Paul Doswell said "He's a young lad who is not the finished product, but has promise and potential and the sort of player I like getting my teeth into to develop." McCrae scored his first goal for Sutton in his second appearance for the club, in a friendly against Kingstonian.

References

External links
 Romone McCrae profile at theposh.com

 Romone McCrae profile at histonfc.co.uk

1991 births
Living people
English footballers
Crawley Town F.C. players
Peterborough United F.C. players
English Football League players
National League (English football) players
Sutton United F.C. players
Association football midfielders

redpajama_set_name: RedPajamaWikipedia