text:
\section{Introduction}
\label{sec:introduction}

The training of deep neural networks is resource intensive and time-consuming.
With the expansion of data and model scale, it may take a few days or weeks to train a deep model by using mini-batch SGD
on a single machine/GPU. To accelerate the training process, distributed optimization provides an effective tool for deep net training by allocating the computation to multiple computing devices (CPUs or GPUs).

When variants of mini-batch SGD are applied to a distributed system, communication between computing devices will be incurred to keep the same convergence rate as mini-batch SGD.
As a matter of fact, the extra communication cost in a distributed system is the main factor which prevents a distributed optimization algorithm from achieving the linear time speedup, although the computation load is the same as its single machine version.
In addition, the communication cost, which is often linearly proportional to the number of workers, can be extremely expensive when the number of workers is huge.
Therefore, it is critical to reduce the communication bottleneck to make better use of the hardware resources.

Given that the total amount of communication bits equals the number of communications multiplied by the number of bits per communication, several works are proposed to accelerate training by reducing the communication frequency~\citep{zhou2017convergence,yu2018parallel,stich2018local} or communication bits~\citep{lin2017deep,stich2018sparsified,wen2017terngrad,alistarh2017qsgd}.
However, even when the communication frequency or the number of bits per communication is reduced, hardware resources are not fully exploited in traditional synchronous distributed algorithms because of the following two reasons: (1) only partial resources can be used when workers are communicating with each other and (2) the computation and the communication are interdependent in each iteration.
Specifically, the computation of the $t$-th step relies on the communication of the $(t-1)$-th step and the communication in the $t$-th step follows the computation of the $t$-th step.
To address that, another line of research aims to run computation and communication in parallel by using stale gradients~\citep{li2018pipe}, but the communication complexity is still high as all gradients need to be communicated.

To tackle this dilemma, we propose computation and communication decoupled stochastic gradient descent (CoCoD-SGD) for distributed training.
Instead of waiting for the completion of communication as in traditional synchronous SGD, workers in CoCoD-SGD continue to calculate stochastic gradients and update models locally after the communication started.
After the communication finishes, each worker updates its local model with the result of communication and
the difference of the local models.
As a result, CoCoD-SGD can make full use of resources by running computation and communication in parallel.
In the meantime, workers in CoCoD-SGD communicate with each other periodically instead of in each iteration.
As a benefit, the communication complexity of CoCoD-SGD is lower. By running computation and communication in parallel while reducing the communication complexity, faster distributed training is achieved.
In addition, CoCoD-SGD is suitable for both homogeneous and heterogeneous environments.
Further, it is theoretically justified with a linear iteration speedup with respect to the total computation capability of hardware devices.

Contributions of this paper are summarized as follows:
\begin{itemize}
  \setlength{\itemsep}{0pt}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
\item We propose CoCoD-SGD, a computation and communication decoupled distributed algorithm, to make full use of hardware resources and reduce the communication complexity.
\item From the theoretical perspective, we prove that CoCoD-SGD has a linear iteration speedup with respect to the number of workers in a homogeneous environment. Besides, it has a theoretically justified linear iteration speedup with respect to the total computation capability in a heterogeneous environment.
\item When training deep neural networks with multiple GPUs, CoCoD-SGD achieves a better time speedup comparing with existing distributed algorithms.
\item In both homogeneous and heterogeneous environments, experimental results verify the effectiveness of the proposed algorithm.
\end{itemize}

\section{Related work}
\label{sec:related-work}

A conventional framework for distributed training is the centralized parameter server architecture~\citep{li2014communication}, which is supported by most existing systems such as Tensorflow~\citep{abadi2016tensorflow}, Pytorch~\citep{paszke2017automatic} and Mxnet~\citep{chen2015mxnet}.
In each iteration, the parameter server, which holds the global model, needs to communicate with $O(N)$ workers. This becomes a bottleneck which slows down the convergence when $N$ is large.
Therefore, the decentralized ring architecture using the Ring-AllReduce algorithm
became popular in recent years. In Ring-AllReduce, each worker only transports $O(1)$ gradients to its neighbors to get the average model over all workers.

The Ring-AllReduce algorithm can be directly combined with synchronous stochastic gradient descent (S-SGD), which has a theoretically justified convergence rate $O(\frac{1}{\sqrt{TM}})$ for both general convex~\citep{dekel2012optimal} and non-convex problems~\citep{ghadimi2013stochastic}, where $T$ is the number of iterations and $M$ is the mini-batch size.
Such rate shows its linear iteration speedup with respect to the number of workers since increasing the number of workers is equivalent to increasing the mini-batch size.
Although only $O(1)$ gradients are needed to be communicated per worker in each iteration for Ring-AllReduce, $O(N)$ handshakes are needed for each worker.
Therefore, the communication cost grows as the number of the workers increases.

There have been many attempts to reduce the communication bottleneck of S-SGD while maintaining its linear iteration speedup property. Among them,
Local-SGD is a variant of S-SGD with low communication frequency, in which workers update their models locally and communicate with each other every $k$ iterations.
It has been proved to have linear iteration speedup for both strongly convex~\citep{stich2018local} and non-convex~\citep{zhou2017convergence,yu2018parallel} problems.
QSGD~\citep{alistarh2017qsgd} and TernGrad~\citep{wen2017terngrad} compress the gradients
from 32-bit float to lower bit representations.
Sparse SGD~\citep{aji2017sparse} is another method that communicates part of the gradients in each iteration and is proved to have the same convergence rate as SGD~\citep{stich2018sparsified,alistarh2018convergence}.
Although the communication complexity is reduced in the above methods, the hardware resources are not fully utilized because only a part of them is used for communication.

To fully utilize hardware resources, asynchronous stochastic gradient descent (A-SGD), a distributed variant of SGD based on the parameter server architecture, is proposed.
After one communication started, a worker uses a stale model to compute the next stochastic gradient without waiting for the completion of the communication.
A linear iteration speedup is proved for both convex~\citep{agarwal2011distributed} and non-convex~\citep{lian2015asynchronous} problems.
However, stale gradients may slow down the training and make it converge to a poor solution especially when the number of workers is large, which implies a large delay~\citep{chen2016revisiting}.
As a remedy, Pipe-SGD~\citep{li2018pipe} is proposed to integrate the advantages of A-SGD and Ring-AllReduce. Specifically, it employs Ring-AllReduce to get the average gradient and updates the model with a stale gradient.
Pipe-SGD can control the delay as a constant because of the efficiency of the Ring-AllReduce algorithm.
Nevertheless, one disadvantage of Pipe-SGD is that its communication complexity is $O(T)$, which is higher than Local-SGD and QSGD.
In comparison, the proposed algorithm which makes full use of hardware resources and has
a lower communication complexity, achieves the advantages of both Pipe-SGD and Local-SGD.

\section{Algorithm}

In this section, we introduce the CoCoD-SGD algorithm with the following three techniques: (1) computation and communication decoupling, (2) periodically communicating and (3) proportionally sampling.

\subsection{Preliminary}
\subsubsection{Problem Definition}
We focus on data-parallel distributed training, in which
each worker can access only part of the data.
We use $\mathcal{D}$ to denote the full training dataset and $\mathcal{D}_i$ to denote the local dataset stored in the $i$-th worker.
We have $\mathcal{D} = \mathcal{D}_1 \cup \cdots \cup \mathcal{D}_N$ and $\mathcal{D}_i \cap \mathcal{D}_j = \varnothing, \forall i \neq j$.
The objective can be written as
\begin{equation}\label{basic_object}
\min_{x \in \mathbb{R}^d} f(x) := \frac{1}{|\mathcal{D}|} \sum_{\xi \in \mathcal{D}} f(x, \xi) = \sum_{i=1}^{N} p_i f_i(x),
\end{equation}
where $f_i(x) := \frac{1}{|\mathcal{D}_i|} \sum_{\xi_i \in \mathcal{D}_i} f(x, \xi_i)$ is the local loss of the $i$-th worker and $p_i$'s define the partition of data among all workers.
Specifically, $p_i$ is proportional to size of the local dataset on the $i$-th worker: $p_i = \frac{|\mathcal{D}_i|}{|\mathcal{D}|}$ and we have $\sum_{i=1}^{N} p_i = 1$.
We use $\mathcal{P}$ to denote the distribution of $p_i$'s.

\subsubsection{Notations}
\begin{itemize}
\item $\| \cdot \|$ indicates the $\ell_2$ norm of a vector.
\item $f^*$ represents the optimal value of (\ref{basic_object}).
\item $\mathbb{E}$ indicates a full expectation with respect to all the randomness, which includes the random indexes sampled to calculate stochastic gradients in all iterations.
\item $\mathcal{C}_i$ represents the computation capability, which indicates the computing speed, of the $i$-th worker.
\end{itemize}

\subsection{Computation and Communication Decoupled Stochastic Gradient Descent}
The complete procedure of CoCoD-SGD is summarized in Algorithm~\ref{CoCoD-SGD}.
On one hand, the goal of  CoCoD-SGD is two-fold: (1) running computation and communication in parallel and (2) reducing the communication complexity. To achieve that, we propose the following two techniques:
\begin{itemize}

\item \textbf{Computation and communication decoupling:}
Different from Pipe-SGD which uses stale gradients to decouple the dependency of computation and communication, CoCoD-SGD continues to update models in workers locally after the communication starts.
If one communication starts after the $t$-th update, workers need to communicate with each other to get the average model $\hat{x}_t$.
After the communication starts, the $i$-th worker continues to update its local model $x_{t}^i$ by mini-batch SGD (line 4-5).
As a result, computation and communication can be executed simultaneously.

\item \textbf{Periodically communicating:} In practice, when the number of workers is large, the communication time may exceed the time of one update, which will cause idle time of the computing devices. Therefore, we let workers keep on updating the local models $k$ times instead of waiting for the completion of the communication after only one local update (line 5). In this way, the communication complexity can be also reduced.
\end{itemize}

\begin{algorithm}[!tb]
\caption{CoCoD-SGD}
\label{CoCoD-SGD}
\hspace*{0.02in}{\bf Input:}
The number of workers $N$, the number of iterations $T$, communication period $k$, mini-batch sizes $M_1,\cdots, M_N$ and initial point $\hat{x}_{0} \in \mathbb{R}^{d}$.

\hspace*{0.02in}{\bf Initialize:}
$t=0, x_{i}^{0} = \hat{x}_0, i=1, 2, \cdots, N$.

\begin{algorithmic}[1]
\WHILE{$t < T$}
  \STATE \underline{Worker $W_i$ does}:
  \STATE Run \textbf{Step \uppercase\expandafter{\romannumeral1} } and \textbf{Step \uppercase\expandafter{\romannumeral2} } in parallel.
  \STATE \textbf{Step \uppercase\expandafter{\romannumeral1} }: Store $x_t^i$ in the memory and communicate with other workers to get the weighted mean of all local models: $\hat{x}_t = \sum_{i=1}^N \frac{M_i}{\sum_{j=1}^{N} M_j} x_t^i$.
  \STATE \textbf{Step \uppercase\expandafter{\romannumeral2} }:
  \begin{enumerate}
    \setlength{\itemindent}{0.5em}
    \item[] \textbf{for} $\tau = t$ to $t+k-1$ \textbf{do}
    \item[] \ \ \ \ \ Compute a mini-batch stochastic gradient $G_{\tau}^i = \frac{1}{M_i} \sum_{j=1}^{M_i} \nabla f_i(x_{\tau}^i, \xi_{\tau}^{i,j}), \xi_{\tau}^{i,j} \in \mathcal{D}_i$, and update
    \item[] \  locally: $x_{\tau + 1}^i = x_{\tau}^i - \gamma G_{\tau}^i$
    \item[] \textbf{end for}
  \end{enumerate}
  \STATE $x_{t+k}^i = \hat{x}_t + (x_{t+k}^i  - x_t^i)$
  \STATE $t = t+k $
\ENDWHILE
\end{algorithmic}
\end{algorithm}

On the other hand, CoCoD-SGD is designed to be suitable for heterogeneous distributed system, where workers may have different computation capabilities.
Intuitively, when all workers use the same batch size to compute stochastic gradients, the faster workers need to wait for the slower ones after finishing their updates.
Therefore, we use proportionally sampling in CoCoD-SGD:
\begin{itemize}
  \item \textbf{Proportionally sampling:} Workers in CoCoD-SGD use different mini-batch sizes $M_i$'s to compute stochastic gradients.
   The batch sizes are proportional to the computation capabilities of workers, i.e., $\frac{M_i}{M_j} = \frac{\mathcal{C}_i}{\mathcal{C}_j}$.
   Under this setting, all workers will finish $k$ updates at the same time.
   Meanwhile, to let all workers finish one epoch simultaneously, we proportionally divide the dataset among workers, i.e., $\frac{|\mathcal{D}_i|}{|\mathcal{D}_j|} = \frac{\mathcal{C}_i}{\mathcal{C}_j}$.
   In addition, we define the average model $\hat{x}_t$ as the weighted mean of all local models: $\hat{x}_t = \sum_{i=1}^N \frac{M_i}{\sum_{j=1}^{N} M_j} x_t^i$ (line 4).
\end{itemize}

Besides, different from QSGD~\citep{alistarh2017qsgd} and Sparse-SGD~\citep{stich2018sparsified}, which are based on the parameter server architecture, CoCoD-SGD can communicate with the Ring-AllReduce algorithm by the definition of $\hat{x}^t$.

After finishing local iterations and the communication, the $i$-th worker updates its local model with the average model and
the difference of the local models
by $x_{t+k}^i = \hat{x}_t + (x_{t+k}^i  - x_t^i)$ (line 6).

\section{Theoretical Analysi}

In this section, we provide the theoretical analysis for CoCoD-SGD and show that CoCoD-SGD has the same convergence rate as S-SGD. In addition, we show that CoCoD-SGD has lower communication complexity and better time speedup.
Due to the space limit, all proofs are deferred to the supplemental material.
In the subsequence analysis, we will use the following definitions.

\begin{definition} We denote the total number of iterations and the total time used to converge when using $N$ workers as $T_N$ and $\mathcal{T}_N$, respectively. Then the
iteration speedup (IS) and the time
speedup (TS) are respectively defined as
  \begin{equation}\label{def_is}
  {\rm IS}^N = \frac{T_1}{T_N},
  \end{equation}
  \begin{equation}\label{def_ts}
  {\rm TS}^N = \frac{\mathcal{T}_1}{\mathcal{T}_N}.
  \end{equation}
\end{definition}

\subsection{Main Results}
Before establishing our main results, we introduce the following assumptions, all of which are commonly used in the analysis of distributed algorithms~\citep{lian2015asynchronous,aji2017sparse,yu2018parallel}.
\begin{assumption} \label{assumptions} ~~
\begin{itemize}
\item[(1)] \textbf{Lipschitz gradient}: All local functions $f_{i}$'s have $L$-Lipschitz gradients
      \begin{equation}
      \| \nabla f_i(x) - \nabla f_i(y) \| \leq L \| x - y \|, \forall i , \forall x, y \in \mathbb{R}^d.
      \end{equation}
\item[(2)] \textbf{Unbiased estimation}:
      \begin{eqnarray}
      \mathbb{E}_{\xi \sim \mathcal{D}_i} \nabla f_{i} (x, \xi) &=& \nabla f_{i}(x), \forall i,
      \\
      \mathbb{E}_{i \sim \mathcal{P}} \mathbb{E}_{\xi \sim \mathcal{D}_i} \nabla f_{i} (x, \xi) &=& \nabla f(x).
      \end{eqnarray}
\item[(3)] \textbf{Bounded variance}: There exist constants $\sigma$ and $\zeta$ such that
      \begin{eqnarray}
      \mathbb{E}_{\xi \sim \mathcal{D}_i} \| \nabla f_{i} (x, \xi) - \nabla f_{i} (x)\|^2 \leq \sigma^2, ~~\forall x \in \mathbb{R}^d, \forall i,
      \\
      \mathbb{E}_{i \sim \mathcal{P}} \| \nabla f_i(x) - \nabla f(x) \|^2 \leq \zeta^2, ~~\forall x \in \mathbb{R}^d.~~~~~~
      \end{eqnarray}
\item[(4)] \textbf{Dependence of random variables}: $\xi_{t}^{i,j}$'s are independent random variables, where $t \in \{0, 1, \cdots, T-1\}$, $i \in \{1,2, \cdots, N\}$, and $j \in \{1,2, \cdots, M_i\}$.
\end{itemize}
\end{assumption}

To evaluate the convergence rate, the metric in nonconvex optimization is to bound the weighted average of the  $\ell_2$ norm of all gradients~\citep{ghadimi2013stochastic,lian2015asynchronous,yu2018parallel}.
\begin{theorem} \label{general_theorem}
Under Assumption~\ref{assumptions}, if the learning rate satisfies $\gamma \leq \frac{1}{L}$, we have the following convergence result for Algorithm~\ref{CoCoD-SGD}:
{\small
\begin{eqnarray} \label{general_threorem_result}
\frac{1}{T} \sum_{t=0}^{T-1} D_1 \mathbb{E} \| \nabla f(\hat{x}_t) \|^2
\leq\frac{ 2(f(\hat{x}_0) - f^*)}{T\gamma} +  D_2 (\frac{N\sigma^2}{\sum_{i=1}^{N} M_i} + 2k\zeta^2) + \frac{ \gamma L \sigma^2}{\sum_{i=1}^{N} M_i},
\end{eqnarray}
}
where $k$ is the communication period and
\begin{equation}
D_1 = 1 - 2kD_2, ~~~~D_2 = \frac{8 \gamma^2 L^2 k}{1 - 16 \gamma^2 k^2 L^2}.
\end{equation}
\end{theorem}

Choosing the learning rate $\gamma$ appropriately,
we have the following corollary.
\begin{corollary} \label{general_corollary}
Under Assumption~\ref{assumptions},
when the learning rate is set as $\gamma = \frac{1}{\sigma \sqrt{\frac{T}{\sum_{i=1}^{N} M_i}}}$ and the total number of iterations satisfies
{\small
\begin{eqnarray}\label{convergence_condition}
T \geq \max \Bigg \{ \frac{L^2(\sum_{i=1}^{N} M_i)}{\sigma^2}, \frac{48(\sum_{i=1}^{N} M_i)L^2k^2}{\sigma^2},
\frac{144(\sum_{i=1}^{N} M_i)^3}{\sigma^6}L^2k^2 \left( \frac{N\sigma^2}{\sum_{i=1}^{N} M_i} + 2k\zeta^2 \right)^2 \Bigg\},
\end{eqnarray}
}
we have the following convergence result for Algorithm~\ref{CoCoD-SGD}:
\begin{equation}\label{convergence_rate}
\frac{1}{T} \sum_{t=0}^{T-1} \mathbb{E} \| \nabla f(\hat{x}_t) \|^2 \leq \frac{4\sigma(f(\hat{x}_0) - f^* + L)}{\sqrt{T\sum_{i=1}^{N} M_i}}.
\end{equation}
\end{corollary}

Corollary~\ref{general_corollary} indicates that the convergence rate of the weighted average model is $O\left(1/\sqrt{\sum_{i=1}^{N} M_i T} \right)$, which is consistent with S-SGD~\citep{ghadimi2013stochastic}.
Next, we establish the linear iteration speedup in both homogeneous and heterogeneous environments, and show the communication complexity of CoCoD-SGD.

\begin{remark}
 {\rm(linear iteration speedup in the homogeneous environment)}. For CoCoD-SGD in a homogeneous environment, all workers use the same mini-batch size: $M_1 = M_2 = \cdots = M_N = M$. According to (\ref{convergence_rate}), CoCoD-SGD converges at the rate $O\left(1/\sqrt{NMT}\right)$. Consequently, to achieve the $\epsilon$-approximation solution, $O(1/ (NM\epsilon^2)$ iterations are needed, which means CoCoD-SGD has a linear iteration speedup with respect to the number of workers according to the definition of IS in (\ref{def_is}).
\end{remark}

\begin{remark}
{\rm (linear iteration speedup in the heterogeneous environment)}.
With Proportionally Sampling, we have $M_i / M_j = \mathcal{C}_i / \mathcal{C}_j$.
According to (\ref{convergence_rate}), to achieve the $\epsilon$-approximation solution, the number of iterations required is $O\left(1/(\sum_{i=1}^{N} M_i \cdot \epsilon^2)\right)$, that is $O\left( \mathcal{C}_1 / \left(\sum_{i=1}^{N} \mathcal{C}_i \cdot M_1 \epsilon^2 \right) \right)$, which means CoCoD-SGD has a linear iteration speedup with respect to the total computation capability of all workers according to (\ref{def_is}).
\end{remark}

\begin{remark} \label{remark_3}
{\rm (communication complexity)}.
From (\ref{convergence_condition}), we have $T \geq \max \{ O(N), O(N k^2), O(N^3 k^4 ) \}$. Therefore, when the communication period is bounded by  $O\left(T^{\frac{1}{4}}/N^{\frac{3}{4}} \right)$, the convergence rate in Corollary~\ref{general_corollary} is achievable.
As a result, the total communication complexity of CoCoD-SGD is $O\left(T/\left(T^{\frac{1}{4}}/N^{\frac{3}{4}} \right)\right)$, that is $O\left(T^{\frac{3}{4}} N^{\frac{3}{4}}\right)$.
\end{remark}

\begin{remark}\label{remark_4}
{\rm (choice of learning rate)}. When we run CoCoD-SGD for a fixed number of epochs, $\sum_{i=1}^N M_i T$ will be a constant. The learning rate suggested in Corollary~\ref{general_corollary} can be written as $\gamma = \frac{\sum_{i=1}^N M_i}{\sigma \sqrt{\sum_{i=1}^N M_i T}}$. Thus, for CoCoD-SGD with $N$ workers, the learning rate should be set as $\gamma_N = N \cdot \gamma_1$ in the homogeneous environment and $\gamma_N = \sum_{i=1}^{N} \mathcal{C}_i / \mathcal{C}_1 \cdot \gamma_1$ in the heterogeneous environment.
\end{remark}

\begin{figure}[t]
\centering
\includegraphics[scale=0.36]{./Pictures/Compare.pdf}
\caption{Comparison of S-SGD, Pipe-SGD, Local-SGD and CoCoD-SGD. A green block indicates computation which can make full use of the hardware resources, and a yellow block corresponds to communication which only uses $a\%$ of the resources. S-SGD and Local-SGD run computation and communication sequentially, while Pipe-SGD and CoCoD-SGD run them in parallel. Both Local-SGD and CoCoD-SGD communicate every $k$ iterations.}
\label{fig:baseline_comparison}
\end{figure}

\subsection{Time Speedup Analysis}
Next, we compare and analyse the time speedup of CoCoD-SGD and other distributed algorithms when they are applied in practice.

When multiple nodes with GPUs are used to train a deep neural network,
GPUDirect communication,
where all computation threads and communication threads are executed in GPUs, is widely adopted~\citep{paszke2017automatic,chen2015mxnet,sergeev2018horovod} and verified to be efficient~\citep{potluri2013efficient}.
The computation threads can make full use of GPU resources especially when the size of mini-batch is large, while the communication threads can only use partial of the GPU resources, which is assumed to be $a\%$ for ease of analysis.
We further denote the time of one computation as $\mathcal{T}_{\rm comp}$ and the time of communication when using $N$ workers as $\mathcal{T}_{\rm comm}^N$.

A comparison of the running processes of CoCoD-SGD and other standard distributed algorithms is shown in Figure~\ref{fig:baseline_comparison}.
All algorithms have provable linear iteration speedup.
Thus, the time speedup equals $(N \cdot \mathcal{T}_1^{\rm av}) / \mathcal{T}_N^{\rm av}$ according to (\ref{def_ts}), where $\mathcal{T}_N^{\rm av}$ is denoted as the average time required to finish one iteration among $N$ workers.
For S-SGD, computation and communication are sequential.
Therefore, its time speedup is
\begin{equation}\label{TS_S-SGD}
{\rm TS}_{\rm S\textrm{-}SGD}^N = \frac{N \cdot \mathcal{T}_{\rm comp}}{\mathcal{T}_{\rm comp} + \mathcal{T}_{\rm comm}^N}.
\end{equation}
On the other hand, Pipe-SGD decouples the dependence of computation and communication by using a stale gradient, but the communication complexity is still $O(T)$ as it needs to communicate all gradients.
Accordingly, its time speedup is
\begin{equation}\label{TS_Pipe-SGD}
{\rm TS}_{\rm Pipe\textrm{-}SGD}^N = \frac{N \cdot \mathcal{T}_{\rm comp}}{\mathcal{T}_{\rm comp} + \mathcal{T}_{\rm comm}^N \cdot a\%}.
\end{equation}
According to Corollary~1 in \citep{yu2018parallel} and Remark~\ref{remark_3}, Local-SGD and CoCoD-SGD have the same convergence rate as S-SGD when the communication period is $O(T^{\frac{1}{4}} / N^{\frac{3}{4}})$.
The difference between Local-SGD and CoCoD-SGD is that CoCoD-SGD runs computation and communication in parallel while Local-SGD runs them sequentially. As a result, their time speedups are
\begin{eqnarray}\label{TS_Local-SGD}
{\rm TS}_{\rm Local\textrm{-}SGD}^N = \frac{N \cdot \mathcal{T}_{\rm comp}}{\mathcal{T}_{\rm comp} + \mathcal{T}_{\rm comm}^N  / k },
\end{eqnarray}
and
\begin{eqnarray}\label{TS_CoCoD-SGD}
{\rm TS}_{\rm CoCoD\textrm{-}SGD}^N  = \frac{N \cdot \mathcal{T}_{\rm comp}}{\mathcal{T}_{\rm comp} + (\mathcal{T}_{\rm comm}^N \cdot a\%) / k },
\end{eqnarray}
respectively, where $k = O (T^{\frac{1}{4}} / N^{\frac{3}{4}})$. As (\ref{TS_CoCoD-SGD}) is bigger than (\ref{TS_S-SGD}), (\ref{TS_Pipe-SGD}), and (\ref{TS_Local-SGD}), we can verify that CoCoD-SGD achieves the best time speedup.

\section{Experiments}
In this section, we validate the performance of CoCoD-SGD in both homogeneous and heterogeneous environments.

\subsection{Experimental Settings}
\paragraph{Hardware.}
We evaluate CoCoD-SGD on a cluster where each node has 3 Nvidia Geforce GTX 1080Ti GPUs, 2 Xeon(R) E5-2620 cores and 64 GB memory. The cluster has 6 nodes, which are connected with a 56Gbps InfiniBand network.
Each GPU is viewed as one worker in our experiments.

\paragraph{Software.}
We use Pytorch~0.4.1~\citep{paszke2017automatic} to implement the algorithms in our experiments, and use Horovod~0.15.2~\citep{sergeev2018horovod}, OpenMPI~3.1.2\footnote[1]{https://openmp.org}, and NCCL~2.3.7\footnote[4]{N Luehr. Fast multi-gpu collectives with nccl, 2016.}
to conduct the GPUDirect communication with the Ring-AllReduce algorithm.

\paragraph{Datasets.}
We use two datasets for image classification.
\begin{itemize}
  \item CIFAR10~\citep{krizhevsky2009learning}: it consists of a training set of 50, 000 images from 10 classes, and a test set of 10, 000 images.
  \item CIFAR100~\citep{krizhevsky2009learning}: it is similar to CIFAR10 but has 100 classes.
\end{itemize}

\paragraph{Tasks.} We train ResNet18~\citep{he2016deep} and VGG16~\citep{simonyan2014very} on the two datasets.

\paragraph{Baselines.} We compare CoCoD-SGD with S-SGD, Pipe-SGD~\citep{li2018pipe} and Local-SGD~\citep{stich2018local}.
All of them support Ring-AllReduce communication.

\begin{table}[!b]
\centering
\setlength{\tabcolsep}{1.5mm}{
\begin{tabular}{|l|l|l|l|l|}
\hline
& \multicolumn{2}{|c|}{CIFAR10} & \multicolumn{2}{|c|}{CIFAR100} \\
\hline
& ResNet18 & VGG16 & ResNet18 & VGG16 \\
\hline
S-SGD & 94.00\% & 93.25\% & 75.08\% & 70.81\% \\
\hline
Pipe-SGD & 93.94\% & 93.09\% & 75.13\% & 70.59\%    \\
\hline
Local-SGD & 94.35\% & 93.30\% & 75.42\% & 71.13\%    \\
\hline
CoCoD-SGD & 94.41\% & 93.38\% & 75.67\% & 72.24\%    \\
\hline
\end{tabular}}
\caption{Final best test accuracy for all tasks in a homogeneous environment. 16 workers in total.}
\label{final_accuracy}
\end{table}

\begin{figure*}[t]
\centering

\subfigure[{\tiny CIFAR10-ResNet-Epoch-Loss}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar10-resnet-epoch-loss.png}
\end{minipage}
}
\subfigure[{\tiny CIFAR10-VGG-Epoch-Loss}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar10-vgg-epoch-loss.png}
\end{minipage}
}
\subfigure[{\tiny CIFAR100-ResNet-Epoch-Loss}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar100-resnet-epoch-loss.png}
\end{minipage}
}
\subfigure[{\tiny CIFAR100-VGG-Epoch-Loss}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar100-vgg-epoch-loss.png}
\end{minipage}
}
\caption{Training loss for ResNet18 and VGG16 on CIFAR10 and CIFAR100 w.r.t epochs in a homogeneous environment. All algorithms have a similar convergence rate.}
\label{Compare_epoch}
\end{figure*}

\begin{figure*}[t]
\centering
\subfigure[{\tiny CIFAR10-ResNet-Time-Accuracy}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar10-resnet-time-accu.png}
\end{minipage}
}
\subfigure[{\tiny CIFAR10-VGG-Time-Accuracy}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar10-vgg-time-accu.png}
\end{minipage}
}
\subfigure[{\tiny CIFAR100-ResNet-Time-Accuracy}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar100-resnet-time-accu.png}
\end{minipage}
}
\subfigure[{\tiny CIFAR100-VGG-Time-Accuracy}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar100-vgg-time-accu.png}
\end{minipage}
}
\caption{Test accuracy for ResNet18 and VGG16 on CIFAR10 and CIFAR100 w.r.t time in a homogeneous environment. CoCoD-SGD achieves the fastest convergence and the results are consistent with the analysis in Section 4.2.}
\label{Compare_time}
\end{figure*}

\begin{figure*}[!t]
\centering
\subfigure[{\tiny CIFAR10-ResNet-Speedup}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar10-resnet-speedup.png}
\end{minipage}
}
\subfigure[{\tiny CIFAR10-VGG-Speedup}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar10-vgg-speedup.png}
\end{minipage}
}
\subfigure[{\tiny CIFAR100-ResNet-Speedup}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar100-resnet-speedup.png}
\end{minipage}
}
\subfigure[{\tiny CIFAR100-VGG-Speedup}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar100-vgg-speedup.png}
\end{minipage}
}
\caption{The comparison of time speedup for ResNet18 and VGG16 on CIFAR10 and CIFAR100 in a homogeneous environment. CoCoD-SGD achieves the best time speedup.}
\label{Compare_speedup}
\end{figure*}

\paragraph{Hyper-parameters.} We use the following hyper-parameters.
\begin{itemize}
  \item Basic batch size: 32 for both ResNet18 and VGG16.
  \item Basic learning rate: For both networks we start the learning rate from 0.01 and decay it by a factor of 10 at the beginning of the 81st epoch.
  \item Momentum: 0.9.
  \item Weight decay: $10^{-4}$.
  \item Communication period and gradient staleness: Since the variance of stochastic gradients is higher in the beginning, we set the communication period to 1 for the first 10 epochs and 5 for
  the subsequential epochs.
  The staleness of gradients in Pipe-SGD is set to 1 as suggested in \citep{li2018pipe}.
\end{itemize}

\subsection{Homogeneous Environment}
In a homogeneous environment, all workers have the same computation speed.
So for $N$ workers, we set $\gamma_N = N \gamma$ as suggested in Remark~\ref{remark_4}.
And the learning rate warm-up scheme proposed in~\citep{goyal2017accurate} is adopted.
\paragraph{Comparison of convergence rate.}
Figure~\ref{Compare_epoch} shows the training loss with regard to epochs of ResNet18 and VGG16 on 16 GPUs.
All
algorithms have similar convergence speed, which validates the theoretical results claimed in Section 4.1.
\paragraph{Comparison of convergence speed.}
Figure~\ref{Compare_time} shows the test accuracy regarding time on 16 GPUs and Table~\ref{final_accuracy} shows the best test accuracies of all algorithms on the two datasets.
We evaluate the accuracy on the test set during training, but we only accumulate the time used for training.
As shown in Figure~\ref{Compare_time}, CoCoD-SGD achieves almost $2\times$ and $2.5\times$ speedup against S-SGD for ResNet18 and VGG16 respectively.
Although Pipe-SGD can run computation and communication in parallel, it is still slower than Local-SGD and CoCoD-SGD since its communication complexity is higher.
CoCoD-SGD converges faster than others since it not only runs computation and communication simultaneously but also has a lower communication complexity.
The results verify our theoretical results claimed in Section 4.2.
In the meanwhile, we can observe from Table~\ref{final_accuracy} that CoCoD-SGD does not sacrifice the test accuracy
on both datasets
and may get better results than S-SGD.
\paragraph{Comparison of time speedup.}
Figure~\ref{Compare_speedup} shows the time speedup for ResNet18 and VGG16 when the number of workers increases from 1 to 16.
We run the experiments for 120 epochs on 1 GPU and multiple GPUs.
The speedup for ResNet18 is better due to its smaller model size.
On both tasks, CoCoD-SGD achieves the fastest convergence and the best time speedup, which validates our time speedup analysis.

\begin{figure*}[!b]
\centering
\subfigure[{\tiny Heter-CIFAR10-ResNet-Epoch-Loss}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar10-resnet-epoch-loss-heter.png}
\end{minipage}
}
\subfigure[{\tiny Heter-CIFAR10-VGG-Epoch-Loss}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar10-vgg-epoch-loss-heter.png}
\end{minipage}
}
\subfigure[{\tiny Heter-CIFAR100-ResNet-Epoch-Loss}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar100-resnet-epoch-loss-heter.png}
\end{minipage}
}
\subfigure[{\tiny Heter-CIFAR100-VGG-Epoch-Loss}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar100-vgg-epoch-loss-heter.png}
\end{minipage}
}
\caption{Training loss for ResNet18 and VGG16 on CIFAR10 and CIFAR100 w.r.t epochs in a heterogeneous environment. All algorithms have a similar convergence rate.}
\label{Compare_epoch_heter}
\end{figure*}

\begin{figure*}[!b]
\centering
\subfigure[{\tiny Heter-CIFAR10-ResNet-Time-Accuracy}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar10-resnet-time-accu-heter.png}
\end{minipage}
}
\subfigure[{\tiny Heter-CIFAR10-VGG-Time-Accuracy}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar10-vgg-time-accu-heter.png}
\end{minipage}
}
\subfigure[{\tiny Heter-CIFAR100-ResNet-Time-Accuracy}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar100-resnet-time-accu-heter.png}
\end{minipage}
}
\subfigure[{\tiny Heter-CIFAR100-VGG-Time-Accuracy}]{
\begin{minipage}[t]{0.225\linewidth}
\centering
\includegraphics[width=1.1\textwidth] {./Pictures/cifar100-vgg-time-accu-heter.png}
\end{minipage}
}
\caption{Test accuracy for ResNet18 and VGG16 on CIFAR10 and CIFAR100 w.r.t time in a heterogeneous environment. CoCoD-SGD converges fastest and does not sacrifice the test accuracy.}
\label{Compare_time2_heter}
\end{figure*}

\subsection{Heterogeneous Environment}
To simulate a heterogeneous environment, we use 16 workers in our experiments and reduce the computation speed of 8 workers by half.
For S-SGD and Pipe-SGD, \emph{Proportionally Sampling} proposed in Section 3.2 can be also applied in the heterogeneous environment, which is
equivalent to increasing the mini-batch size.
And we denote corresponding algorithms as S-SGD+PS and Pipe-SGD+PS respectively.
When one algorithm employs \emph{Proportionally Sampling}, the batch size is set to 32 for the slower workers and 64 for the faster workers.
In the heterogeneous environment with 16 workers, we set $\gamma_{16}^{\rm ps} = 0.01 * (8 + 8 * 2)$ for algorithms with \emph{Proportionally Sampling} and $\gamma_{16} = 0.01 * 16$ for others as suggested in Remark~\ref{remark_4}.
In addition, the asynchronous version of Local-SGD (AsynLocal-SGD), which is proposed for heterogeneous environments in~\citep{yu2018parallel}, is included
in comparison with CoCoD-SGD
in our experiments.
\paragraph{Comparison of convergence rate.}
Figure~\ref{Compare_epoch_heter} presents the training loss regarding epochs of ResNet18 and VGG16 on 16 GPUs, which exhibits similar convergence rate for all the algorithms in comparison. Besides, we can see from Table~\ref{final_accuracy_heter} that CoCoD-SGD does not lose the test accuracy.
\paragraph{Comparison of convergence speed.}
Figure~\ref{Compare_time2_heter} shows the curves of the test accuracy regarding time.
As we can see, CoCoD-SGD converges fastest and achieves almost 2.5$\times$ and 3$\times$ speedup against S-SGD for ResNet18 and VGG16 respectively.
Comparing with the results in
Figure~\ref{Compare_time},
 S-SGD converges slower due to the heterogeneous computation, while CoCoD-SGD is robust.
When equipped with \emph{Proportionally Sampling}, S-SGD and Pipe-SGD converge faster since the hardware utilization
is improved.

\begin{table}
\centering
\setlength{\tabcolsep}{1.25mm}{
\begin{tabular}{|l|l|l|l|l|}
\hline
& \multicolumn{2}{|c|}{CIFAR10} & \multicolumn{2}{|c|}{CIFAR100} \\
\hline
& ResNet18 & VGG16 & ResNet18 & VGG16 \\
\hline
S-SGD & 94.18\% & 93.09\% & 74.30\% & 70.51\% \\
\hline
S-SGD+PS & 93.73\% & 92.96\% & 74.16\% & 70.32\% \\
\hline
Pipe-SGD & 94.17\% & 92.36\% & 73.95\% & 70.26\%    \\
\hline
Pipe-SGD+PS & 94.10\% & 92.71\% & 73.84\% & 70.15\%    \\
\hline
AsynLocal-SGD & 94.47\% & 93.24\% & 75.21\% & 71.80\%    \\
\hline
CoCoD-SGD & 94.33\% & 92.69\% & 75.74\% & 71.33\%    \\
\hline
\end{tabular}}
\caption{Final best test accuracy for all tasks in a heterogeneous environment. 16 workers in total.}
\label{final_accuracy_heter}
\end{table}

\section{Conclusion}
In this paper, we propose a computation and communication decoupled stochastic gradient descent (CoCoD-SGD) for distributed optimization.
In comparison with existing distributed algorithms, CoCoD-SGD can run computation and communication simultaneously to make full use of hardware resources and has lower communication complexity.
CoCoD-SGD is also theoretically justified to have the same convergence rate as S-SGD and obtains linear iteration speedup in both homogeneous and heterogeneous environments.
In addition, CoCoD-SGD achieves faster distributed training with superior time speedup when comparing with others.
Experimental results demonstrate the efficiency of the proposed algorithm.

\section*{Acknowledgments}
This research was supported by the National Natural Science Foundation of China (No. 61673364, No. 91746301) and the Fundamental Research Funds for the Central Universities (WK2150110008).
We also gratefully acknowledge the support of Cheng Li and Youhui Bai from USTC for providing the experimental environment.

\bibliographystyle{named}

redpajama_set_name: RedPajamaArXiv


The loving Heart is a testament of what I have seen in the world between 1999 until 2010. During intense film assignments of four to six weeks I mostly stayed with poor people in developing countries. I experienced their hospitality, saw their vitality, hope and love, but I was also deeply touched by big differences and injustice.
In Ecuador I saw a child dying at home in his bed. His parents didn't have money for surgery, whereas a bit further in the same country an American multinational is taking oil out of the jungle. The wealth is there, but how is it divided?
We are ourselves the society. We can ourselves change this situation. I believe in a world without poverty. I believe in creating communion through film. The moving picture creates consciousness and connection. With The Loving Heart I want to open eyes and hearts.
Do you also think this is important? Then join in! In unity lies our true happiness. Together we can shape the world as we like. A world without poverty is really possible.


We're starting off the new week with an excellent roundup of the best daily deals you'll find on Monday. Highlights from the list include a rare opportunity to get a 4K Fire TV with HDR for the price of a Fire TV Stick, discounts on all of the most popular Philips Hue smart bulbs, a little gadget that cooks perfect eggs every time for just $19.99, a rare discount on Apple's beloved AirPods, $10 off the Echo Dot, $50 off the best wireless noise cancelling headphones Bose has ever made, the best-selling windproof umbrella on all of Amazon for just $21, discounts on awesome Nerf guns, $25 off the only tool set you'll ever need, and so much more. See all of today's best bargains below.


text:
using namespace std;
using namespace boost;
using boost::graph::distributed::mpi_process_group;

// Type printing utils
#define DECLARE_TYPE_NAME(x) template<> const char *type_name<x>::name = #x;
#define GET_TYPE_NAME(x) (type_name<typeof(x)>::name)
template <typename T> class type_name
{
public:
    static const char *name;
};

// MPI utils
template<typename T>
T average(T result)
{
    T sum = 0;
    mpi::communicator world;
    mpi::reduce(world, result, sum, std::plus<T>(), 0);
    return sum / world.size();
}

template<typename T>
std::vector<T> gather_all(T result)
{
    std::vector<T> results;
    mpi::communicator world;
    mpi::gather(world, result, results, 0);
    return results;
}

void log_all_results_values(std::ostream& out, const char *algorithm, const char *graph_type, benchmark::event_list &events, bool root)
{
    unsigned long long microseconds = events.average_time().total_microseconds();
    std::vector<unsigned long long> all_results = gather_all(microseconds);

    if (root) {
        size_t j = 0;

        for (std::vector<unsigned long long>::iterator i = all_results.begin(); i != all_results.end(); ++i, ++j) {
            boost::posix_time::time_duration time = boost::posix_time::microseconds(*i);
            out << algorithm << " " << graph_type << " " << j << " " << time << " " << *i / 1000000.0  << endl;
        }
    }
}

void log_all_results_header(std::ostream &out, bool root)
{
    if (root) {
        out << "# Wall clock time for each process" << endl;
        out << "# Algorithm GraphType Process AverageTimeString AverageTimeSeconds" << endl;
    }
}

void log_average_values(std::ostream& out, const char *algorithm, const char *graph_type, benchmark::event_list &events, bool root)
{
    unsigned long long microseconds = events.average_time().total_microseconds();
    boost::posix_time::time_duration avg = boost::posix_time::microseconds(average(microseconds));

    if (root) {
        mpi::communicator world;
        out << algorithm << " " << graph_type << " " << world.size() << " " << avg << " " << avg.total_microseconds() / 1000000.0  << endl;
    }
}

void log_average_header(std::ostream& out, bool root)
{
    if (root) {
        out << "# Average wall clock time" << endl;
        out << "# Algorithm GraphType Processes AverageTimeString AverageTimeSeconds" << endl;
    }
}

void write_average_log(const char *algorithm, const char *graph_type, const size_t nodes, const size_t edges, benchmark::event_list &events, bool root, bool declare = true)
{
    std::ostringstream filename;
    filename << algorithm << "-" << graph_type << "-n" << nodes << "-e" << edges << "-Average.log"; // Why not sprintf? I hate buffer overflows.
    std::ostream *of;
    mpi::communicator world;

    if (world.size() == 1) {
        if (root) {
            of = new std::ofstream(filename.str().c_str());
        } else {
            of = &std::cout;
        }

        log_average_header(*of, root);
    } else {
        if (root) {
            of = new std::ofstream(filename.str().c_str(), ios_base::app);
        } else {
            of = &std::cout;
        }
    }

    log_average_values(*of, algorithm, graph_type, events, root);

    if (root && declare) {
        std::cout << "Wrote " << filename.str() << std::endl;
        delete of;
    }
}

void write_all_results_log(const char *algorithm, const char *graph_type, const size_t nodes, const size_t edges, benchmark::event_list &events, bool root, bool declare = true)
{
    mpi::communicator world;
    std::ostringstream filename;
    filename << algorithm << "-" << graph_type << "-n" << nodes << "-e" << edges << "-" << world.size() << "Processes.log";
    std::ostream *of;

    if (root) {
        of = new std::ofstream(filename.str().c_str());
    } else {
        of = &std::cout;
    }

    log_all_results_header(*of, root);
    log_all_results_values(*of, algorithm, graph_type, events, root);

    if (root && declare) {
        std::cout << "Wrote " << filename.str() << std::endl;
        delete of;
    }
}

redpajama_set_name: RedPajamaGithub


text:
'Prioritise doctors' welfare to curb brain drain'
Dayo Ojerinde
The Provost of the College of Medicine, University of Lagos, Prof. Afolabi Lesi, has urged the federal and state governments to improve on the welfare of doctors to curb brain drain in the Nigerian medical profession.
Lesi said this in Lagos on Thursday at the inauguration of a newly rehabilitated road on the college premises, financed by the Chapel of Healing Cross, Lagos.
He said, "The government must improve the conditions of service such that people don't want to leave. The issue of emigration is not affecting only the doctors, it has to do with all health workers.
"We can never have perfect condition, it doesn't exist anywhere in the world; but we must have a society where professionals are treated fairly. A society where they can have a life and practice the profession the way it ought to be practised. If that is done, the desire to emigrate will be less. We will never eliminate it."
According to NOIPolls, Nigeria needs no fewer than 303,000 medical doctors and at least 10,605 new doctors annually to cover the gaping physician-patient ratio.
Statistics revealed that the country has about 72,000 medical doctors registered with the Medical and Dental Council of Nigeria, with only approximately 35,000 practising in Nigeria.
The provost commended the church for rehabilitating the bad road, stating that it would be beneficial to both students and staff of the college.
"This project is something that both the management and students of the institution directly benefit from. We are very grateful for the Chapel's mandate to impact their environment," he said.
Why do we wake up with a stuffy nose?
Hallmark Health Services appoints Omoyele, Modibbo as directors
Address issues behind brain drain, ex-commissioner tells govt
5,722 Nigerian specialists, 1,540 others practising medicine in UK – GMC
Our stool: The mirror of our health
We can make Nigeria open defecation-free by 2025—Osinbajo

redpajama_set_name: RedPajamaCommonCrawl


Source: http://risoforening.no License: All Rights Reserved.
The new logotype for the Norwegian Riso Association depicts the Norwegian national bird, Fossekallen (the white throated dipper), inside the wheel of the riso squirrel. The name, as well as their lastest newsletter, is typeset in Frode Helland's Karelia, released by Monokrom Skriftforlag in 2018.


Trade barriers and poor infrastructure are preventing sugar producers in sub-Saharan Africa from accessing under-supplied regions on the continent as an imminent end to import quotas in the European Union compels them to find new markets.
A preferential-access deal with the EU for African, Caribbean and Pacific sugar producers ends in September 2017, potentially depriving the farmers further access to a duty-free market. Exports to the EU account for a fifth of the sub-Saharan region's current annual output of about 7.5 million metric tons, according to Cooperatieve Rabobank UA.
While sub-Saharan Africa consumes more sugar than it produces, growers may struggle to plug this shortfall because insufficient infrastructure makes deliveries between regions difficult and import duties lift the cost of sales, said Lindsay Jolly, a senior economist at the International Sugar Organization.
Sub-Saharan Africa will consume 10.2 million tons of the sweetener in 2016, creating supply shortfall of about 2.4 million tons in the region, according to the International Sugar Organization. Sales to the EU account for the vast majority of exports from Mauritius and Mozambique, and about half of those from Swaziland, Gareth Forber, head of sugar research at LMC International Ltd., said at the conference.
Sub-Saharan Africa will consume 10.2 million tons of the sweetener in 2016, creating supply shortfall of about 2.4 million tons in the region, according to the International Sugar Organization.
While EU sugar production is expected to increase when the reform measures take effect next year, African producers may still sell to some markets in the region, especially in southern Europe, Jack Marrian, head of south and east African sugar trading at ED&F Man Holdings, said in Maputo. Deliveries from Africa to this region are in many instances cheaper than shipments from northern Europe, which accounts for the bulk of the continent's production, Marrian said. Many European producers make sugar from beets planted on local soil.
Current projects may lift sub-Saharan Africa's sugar output to 10 million tons over the next six years, Schers said. While global consumption is growing at an annual rate of 1.5 percent to 2 percent, the continent's demand may exceed 12.2 million tons, about 22 percent more than current consumption, by 2020 and create more opportunities for growers, he said.
Despite increasing demand, some producers may shelve new projects in regions where oversupply exists, said Larry Riddle, a commercial director at Illovo Sugar Ltd. in South Africa.
Growers may invest in new industries that will create demand for their produce, such as ethanol production, Jose Orive, executive director of the London-based International Sugar Organization, said at the conference. Countries such as Mauritius have started investing in refining infrastructure to diversify away from raw-sugar exports, Schers said.
African growers can find new buyers on the continent if they displace some 1.5 million tons to 2.5 million tons of Brazilian, Indian and Thai sugar that are shipped to eastern and southern Africa annually, LMC International's Forber said.


Interested in Channelside Lofts for sale in Tampa FL? Definitely one of the hottest places to call home if you're after more of a modern industrial-type living space, the Channelside Lofts building won't leave you disappointed. Available units are usually pretty reasonable, and the building itself is also positioned in a very walkable location that's close to nightlife, shopping, dining, and entertainment. For more information about lofts for sale at Channelside Lofts, call or text us today!
Although originally built back in 2006 and acting as one of the more recently developed loft buildings in Tampa, the Channelside Lofts feature some of the more authentic loft-style living spaces you'll find in town. Floor plans inside lofts for sale at the Channelside Lofts are open and spacious and the modern industrial vibe inside should certainly be desirable for anyone who's looking for that ultimate urban living space. Look for most lofts at Channelside Lofts in Tampa to feature polished concrete flooring, high ceilings and multiple levels, and plenty of modern track lighting and exposed duct work as well.
Love what you see from lofts for sale at Channelside Lofts in Tampa? Give us a call or shoot us a text to schedule a private showing for an available loft residence you see listed on the MLS above.


text:
Manrique de Jesús Quesada Sandoval (Heredia; 12 de noviembre de 1938) es un exfutbolista costarricense que se desempeñaba como centrocampista.

Trayectoria
Debutó en la Primera División de Costa Rica el 3 de abril de 1957 con el Herediano, club que estuvo hasta su retiro en 1967.

Selección nacional
Con la selección de Costa Rica debutó en 1959 y registró tres goles en ocho partidos.

Palmarés

Títulos nacionales

Títulos internacionales

Referencias

Futbolistas de Costa Rica
Futbolistas de la selección de fútbol de Costa Rica en los años 1950
Futbolistas de la selección de fútbol de Costa Rica en los años 1960
Futbolistas del Club Sport Herediano en los años 1950
Futbolistas del Club Sport Herediano en los años 1960
One Club Man
Centrocampistas de fútbol

redpajama_set_name: RedPajamaWikipedia


text:
Q: Problem on removing the class of jquery script after the content is loaded I have the following code that gets the value of a drop down list and displays the value.php and valueb.php files in different divs. I already added a loading class, but I can't remove it when the page is loaded.
I tried .removeClass('loading') before the end of statement with no result (when i add the removeclass, the pic is not loading).
The follow displays the loading pic onChange and stays showing. Thank you.
    $('#result').empty().addClass('loading').load(val + '.php');

A: Try
 $('#firstresult').empty().addClass('loading').load(val + '.php', function(){$('#firstresult').removeClass('loading'); });

redpajama_set_name: RedPajamaStackExchange


Please read this Subscription Agreement carefully before attempting to subscribe to the subscription materials and information available on the web site (the "Subscription Service") operated by ClearView Financial Media Limited ("ClearView Financial Media Limited", "we" or "us"). In particular, we draw your attention to clauses 10 (Applicability of Subscription Materials) and 15 (Liability).
By clicking on the "I Accept" button at the bottom of these terms and conditions and subscribing to the Subscription Service or by completing the Sales Order form you agree to be legally bound by these terms and conditions as they may be modified and posted on our web site from time to time.
If you do not wish to be bound by these terms and conditions then you may not subscribe to the Subscription Service.
On receipt of your application for a trial subscription, we shall, subject to clause 6.3, provide you with such user name(s) and password(s) as are required by you to give access to the Subscription Service to you and/or any employees identified by you on your application for a trial subscription (such individuals in receipt of a user name being referred to in these terms and conditions as "Users").
You shall keep and, where relevant, shall procure that your employees keep confidential the user name(s) and password(s) provided by us to you for access to the Subscription Service. You must notify us immediately of any unauthorised use of them or any other breach of security regarding our web site that comes to your attention.
You shall be entirely responsible for all access to the Subscription Service where access is facilitated by the user name(s) and password(s) provided to you whether such access is with or without your consent.
For the fourteen (14) day period following receipt by you of your user name(s) and password(s), you shall be entitled to access the Subscription Service on a free trial basis (the "Trial Period") and access shall be subject to these terms and conditions.
On expiry of the Trial Period, continued access to the Subscription Service shall be subject to completion of the full subscription and payment of the fee by you in accordance with clause 6 (Payment).
We grant you a non-exclusive, non-transferable licence to use the Subscription Service for: (a) the Trial Period; and(b) subject to clause 4.2, following expiry of the Trial Period for 1 year, and for yearly periods afterwards, on the payment of an annual renewal fee.
This licence to use the materials and information made available through the Subscription Service including without limitation any news items published in the site (the "Materials") is granted on the terms and conditions of this Subscription Agreement.
In relation to the Subscription Service: (a) you and each User may: i. display the Materials electronically on a single computer; ii. download and store one copy of the Materials in machine readable form; and iii. print one copy of the Materials; iv. in relation to any part of the Subscription Service which offers the "email a friend" function, use such function in respect of no more than 50 articles in each year, for their own individual use and for the purposes of any matter or transaction on which they are advising provided that you shall not be entitled to reproduce the Materials on any electronic or print media (including without limitation on any intranet or extranet) other than as permitted under this clause. (b) you and any User may not: v. download, store, reproduce, transmit, display (including without limitation display on any intranet or extranet site), copy, distribute or use Materials other than in accordance with clause 5.3(a)above; vi. sub-license, rent, lease, transfer or attempt to assign the rights in the Materials to any other person; vii. alter or remove any copyright notices or other notices indicating the proprietary ownership by us or any third party of any intellectual property in the Materials; viii. use the Materials in any manner except as expressly permitted, or transfer or export the Materials or any copies into any country, other than in compliance with applicable laws; ix. in relation to any part of the Subscription Service which offers the "email a friend" function, use such function in respect of no more than 50 articles in each year, for their own individual use and for the purposes of any matter or transaction on which they are advising provided that you shall not be entitled to reproduce the Materials on any electronic or print media (including without limitation on any intranet or extranet) other than as permitted under this clause. x. reproduce, modify, or in any way commercially exploit any of the Materials; xi. allow any person in respect of whom a user name and password has not been allocated by us to use the Materials; xii. make the Materials available on a network; xiii. create a database in electronic or structured form by systematically downloading and storing all or any of the Materials.
ClearView Financial Media Limited reserves the right to monitor the usage of the "email a friend" function and to contact you if we consider that you are using this function unreasonably. In the event that any usage of this function continues in breach of clause 5.3(a)(iv) following notification under this clause, we reserve the right to terminate your use of the Subscription Service in accordance with clause 16.1.
To access the Subscription Service on expiry of the Trial Period you will need to contact a ClearView Financial Media Limited sales or customer service representative (contact details are on ClearView Financial Media Limited).
Details of our prices for full subscription to the Subscription Service, and the procedures for payment are available from ClearView Financial Media Limited sales and customer care representatives. You may pay by credit card, cheque or bank transfer within 30 days of receiving a valid invoice. The method of payment shall be indicated by you at the time of your full subscription request made in accordance with clause 6.1 and the payment will cover 1 year's access to the Subscription Service unless otherwise agreed. The price of any full subscription is the price in force at the date and time of your order. We may change the price of any full subscription before you place a full subscription request. If payment is not received within 30 days of receiving a valid invoice, ClearView Financial Media Limited reserves the right to restrict access to the Subscription Service and/or charge interest under the Late Payment of Commercial Debts (Interest) Act 1998.
We are entitled to refuse any trial or full subscription request placed by you. If your full subscription request is accepted, we will confirm acceptance to you by subscription electronic means ("Confirmation") to the email address you have given us on registration or on ordering. The Subscription Service will be made available to you via email on Confirmation. On expiry of the Trial Period, making the Subscription Service available to you on Confirmation constitutes performance of our services and, other than in accordance with clause 7.2, you will have no right to cancel the contract once we have done so.
You undertake that all details you provide to us for the purpose of subscribing to the Subscription Service for the Trial Period or otherwise will be correct, including without limitation and where relevant, that the credit card which you use is your own and that there are sufficient funds or credit facilities to cover the cost of any subscription. We reserve the right to obtain validation of your credit card details before providing you with access to the Subscription Service.
We warrant that we will provide the Subscription Service with reasonable skill and care.
If you wish to cancel your full subscription to any part of the Subscription Service within 7 working days of Confirmation for any reason (including if you simply change your mind), you may do so if you send us a notice in writing or other durable medium (including email), but only if you have not yet accessed the Materials following expiry of the Trial Period. In these circumstances, you may claim a refund from us, which will be paid as soon as possible but in any event within 30 days.
If you have any complaints, or if you wish to exercise your right of cancellation referred to in clause 7.2 you may send your complaint or notice of cancellation to us via email at clientsupport@ClearViewPublishing.com or by post to Head of Customer Services, ClearView Financial Media Limited, 19 Heathman's Road, London, SW6 4TJ. If you wish, you may use the form of cancellation set out at the end of these terms and conditions.
By accepting these terms and conditions, you agree and, where relevant, shall procure that your employees agree to the processing and disclosure of the Personal Information for the Purposes. You also agree and, where relevant, shall procure that your employees also agree that the Purposes may be amended to include other uses or disclosures of Personal Information following notification by means of a notice on our web site, which should be checked regularly. If you or, where relevant, your employees would like to review or modify any part of the Personal Information which we hold relating to you or your employees then an email should be sent to clientsupport@ClearViewPublishing.com.
Where Materials published in the Subscription Service are supplied by third parties, you understand that we do not control or endorse their contents in any way. All Materials which are offered or provided by third parties, are published in good faith but we do not (to the extent permitted by applicable law) accept responsibility for the accuracy or otherwise of those Materials (on or off-line) and the use of those Materials. You assume total responsibility and risk for your use of the Materials and the Subscription Service.
We have used our best endeavours to ensure that all Materials comply with UK law but we make no representations that the Materials and the Subscription Service are appropriate or available for use in the United Kingdom . We also make no representations that the Materials and the Subscription Service are appropriate or available for use in locations outside the United Kingdom.
We make no warranties, express or implied that making the Materials and the Subscription Service available in any particular jurisdiction outside the UK is permitted under any applicable non-UK laws or regulations. Accordingly, if making the Subscription Service, the Materials or any part available in your jurisdiction or to you (by reason of nationality, residence or otherwise) is prohibited, the Subscription Service or the relevant Materials are not offered for subscription by you. You accept that if you are resident outside the UK, you must satisfy yourself that you are lawfully able to subscribe to the Subscription Service. We accept no liability, to the extent permitted by applicable law, for any costs, losses or damages resulting from or related to the access or attempted access of the Subscription Service by persons in jurisdictions outside the UK or who are nominees of or trustees for citizens, residents or nationals of other countries.
The contents of the Subscription Service are protected by international copyright laws and other intellectual property rights. The owner of these rights is ClearView Financial Media Limited or other third party licensors. All product and company names and logos contained within our web site or the Subscription Service are the trade marks, service marks or trading names of their respective owners, including us.
The Materials may not address your particular requirements and are not intended to constitute a definitive or complete statement of the law on any subject and are not intended to constitute legal or investment advice in any specific situation. You should take your own advice and make your own decisions without placing any reliance on any of our Materials.
We exclude all express or implied terms, conditions, warranties, representations or endorsements whatsoever with regard to the Materials, the Subscription Service, our web site or any information or service provided through our web site not contained in this Subscription Agreement. We will do our best to ensure that all Materials are accurate, but please note that all Materials and information contained in the Subscription Service are provided on an "as is" basis.
It is a feature of the ClearView Financial Media Limited web site that much of the Materials are contributed to us by many different third parties and we do not control those parties or their information and have no responsibility for any information they contribute. Opinions expressed within the Materials are those of the authors and do not represent our opinions.
We do our best to ensure that the Materials are accurate. However, we make no representations that the Material is accurate and up to date or complete and accept no liability for any loss or damage caused by inaccurate information. The Materials do not constitute any form of advice or recommendation by ClearView Financial Media Limited and are not intended to be relied upon in making (or refraining from making) any specific investment, or other, decisions. If you find any inaccurate information on the web site let us know and we will correct it, where we agree, as soon as practicable.
In relation to the provision of the Subscription Service, we accept no liability for any indirect or consequential loss or damage, or for any loss of data, profit, revenue or business (whether direct or indirect), however caused, even if foreseeable. In circumstances where you suffer loss or damage arising out of or in connection with the viewing, use or performance of the Subscription Service or the Materials, we accept no liability for this loss or damage (except where we have been negligent) whether due to inaccuracy, error, omission or any other cause and whether on the part of ClearView Financial Media Limited or our servants or agents.
If we are liable to you for any reason, our liability will be limited to the amount paid by you for the Subscription Service in the year in which such liability arose. This limit does not apply to any liability we may have for death or personal injury resulting from our negligence.
You are responsible for ensuring that your computer system meets all relevant technical specifications necessary to use the Subscription Service and is compatible with our web site. You also understand that we cannot and do not guarantee or warrant that any material available for downloading from the Subscription Service will be free from infection, viruses and/or other code that has contaminating or destructive properties. You are responsible for implementing sufficient procedures and virus checks (including anti-virus and other security checks) to satisfy your particular requirements for the accuracy of data input and output.
The limitations and exclusions in this clause 15 do not affect your non-excludable statutory rights which shall only apply to the extent permitted by applicable law.
We may terminate or suspend your use of the Subscription Service at any time if you are found in material breach or we have reasonable grounds to believe that you are in material breach of any of these terms, including without limitation the prohibition on sharing or disclosure of user name(s) and password(s) set out in clause 3 (Access to the Subscription Service) and the reproduction of Materials set out in clause 5.4. and such breach, if capable of remedy, is not amended within 30 days of notification of such material breach.
If you are notified of termination of your rights of use, you must immediately destroy all of your copies of the Materials. In these circumstances you will not be entitled to any refund. The Client may continue to use and store any licensed materials to the extent necessary for the Client to comply with any statutory or regulatory requirements.
Other than in accordance with our rights set out in clause 16.1, in the event that under clause 8 (Modifications to the Subscription Service) we discontinue, suspend or terminate your access to all of the Subscription Service, we shall refund to you the pro rata share of the annual fee paid by you which relates to the unexpired part of the year in relation to which such annual fee was paid.
We may alter these terms and conditions from time to time and post the new version on our web site, following which all use of the Subscription Service will be governed by that version.
These terms and conditions and your use of our web site are governed by English law and you submit to the non-exclusive jurisdiction of the English courts.
ClearView Financial Media Limited will not be held liable for any failure to perform any obligation to the other due to causes beyond our respective reasonable control.
All notices shall be given: · to us via email at clientsupport@ClearViewPublishing.com; or by post to Head of Customer Services, ClearView Financial Media Limited, 19 Heathman's Road, London, SW6 4TJ; or · to you at either the e-mail or postal address you provide during any application process unless otherwise notified to us. Notice will be deemed received when an email is received in full (or else on the next business day if it is received on a weekend or a public holiday in the place of receipt) or 3 days after the date of posting.
Please click on the "I Accept" button to confirm your acceptance of these terms and conditions. Note: Completion of the Sales Order also serves as confirmation of your acceptance of these terms and conditions.


text:
Q: How do I get my API client to post to my company page? We created an app on LinkedIn for our business page through my profile (as an admin for the page). We have the client ID, client secret and all other relevant information for posting. We are able to post to my page by php api client, but not the company page.
We are using the following API client to post content to our business page on LinkedIn: https://github.com/zoonman/linkedin-api-php-client
We set the following scopes:
'r_liteprofile','w_member_social', 'w_share','rw_organization_admin',

We retrieve the profile with this: $profile = $client->get('me');
We are trying to get the company's information with this:
$companyInfo = $client->get('organizationAcls',['q' => 'roleAssignee']);

This results in the following error: Not enough permissions to access: GET-roleAssignee /organizationAcls
Please advise.


I pray this letter finds you and yours well. This month, I want to write about a most significant topic, to those whom I believe love the Lord and His Church. I believe a reformation is occurring and in these days, I want us to prayerfully ask together, "What is the Church?" You will probably agree with me, this is no small issue!
I literally "grew up in church" – for a while, we actually lived in an apartment located in the church building. I was born in a hospital operated by a church denomination. I attended two colleges and a seminary, all operated by a church denomination. It would be at least ungrateful of me to attack churches. That is certainly not my purpose.
Having said that, when God called me to the ministry, I was most uncomfortable with the idea of pastoring a church. Though I did become a pastor, I have always struggled – as have many other pastors_with how Christians understand Church. We all bear responsibility for that understanding. Now, almost 50 years after responding to the call of God to be a minister, I believe that the concept of "Church" is in the midst of a very significant reformation.
I have witnessed great revivals including the Pentecostal and Charismatic ones, and others as well. I have seen visitations of God to local assemblies, including one that I pastored for 14 years. Years after those visitations, I have noticed the lack of long-term effects. My conclusion from these observations, as well as the conclusions gathered from biblical studies, is that we are thinking of the Church in the wrong way. If my observation sounds arrogant, please forgive me.
Recently, I was asked to meet with a group of elders, who also happen to be long-term friends with each other and with me. We spent several days discussing the future of that particular local church. Like many other churches, it seems stuck in a particular orbit. As we talked, I offered two metaphors for church life, which I will offer to you: Circle or Vine?
First let's look at a circle. By definition it is inanimate – not living. It is a single round closed line with a center. Theoretically, every part of the circle is equally distant or related to the center. It could be, by definition, an assembly around a person. It could be an association of friends with a common interest. It could be an orbit around some center of gravity. A circle includes everything within it and excludes everything outside of it. But most sadly of all, a circle is a line that ends where it begins.
Now let's look at the second figure or metaphor: the vine. A vine is a living plant that has significant roots that draw life. It grows outward along the ground, but opportunistically clasps fixed objects in its path to grow upward. It produces long stems which produce joints and other stems. And in the case of Jesus' vine (see John 15), it bears fruit.
It should be noted that the vine analogy is used to describe both Israel and the Church (see Psalm 80:8; Isaiah 5:1; Jeremiah 2:21; John 15:1-8, 16). The people of God are never referred to as a circle. But have local churches today become more like a circle than a vine? I think so.
We see the church as something to go to, not from. The church circle is something we join – not each other. Church circles are often built around a particular doctrine or method, or even less worthy, a particular convenience. Often church life becomes an orbit of activity. Our time and resources are consumed by the circle. Then our personal circle becomes a matter of including or excluding based upon who is in or out of the church circle.
A worse part about a circle mentality is that it all too often ends where it begins. That is, even after all of our best efforts at preaching, teaching, and worshipping; we only maintain what we started.
I could go on and discuss how the center of the circle has often become a matter of entertainment instead of a matter of mission, but let's move on to our second figure, the vine.
While the circle is inanimate, the vine is a living plant. The circle is a closed figure, but the vine is an outward moving one. Which figure did Jesus use to describe the Church for which He died?
Jesus was rooted in the "soil" of eternity and the Father's will.
Jesus was relational and discipling in His church planting.
Jesus was continually outward in His mission.
Jesus was interested in all of life and not just the synagogue.
Jesus met people where they were, doing what they did.
Jesus built effective "joints" with those to whom He imparted life and truth.
Jesus' intent was to teach all nations through replicated leadership.
Jesus measured success personally not corporately, and expected real disciples to produce fruit.
The vine analogy does not stop at any one place, but gives life to a universal mission that is accomplished by Spirit-filled and Spirit-led individuals.
The vine church is rooted in eternal life and eternal truth. It is not a mushroom. Jesus Himself is that root and vine-trunk. The vine church is Jesus dependent, life producing, and outwardly focused. The vine church is extended by branches that are truly joined to one another (see Ephesians 4:16).
The vine church is not highly centralized; it is not personality or program driven. It is driven by the life and character of Jesus (His DNA). The vine church covers the ground, goes thru, over, and around, as each new branch reaches out. It is opportunistic, clasping onto what lies before it.
The vine church not only relates to its life and itself; it relates to the world before it – business, labor, professions – wherever it finds itself. It is not programmed from without, but from the life within. It is never confounded by obstacles; it climbs them and goes on. It does not say, "come to us;" it says, "we are coming to you." It is not defensive; it goes on the offense and moves forward. And even when saints give their lives to extend the vine, the martyr's blood becomes Kingdom seed.
The blood of Jesus was not given to nourish a circle; it was given to nourish the branches that would produce – some 30, some 60, and some 100 fold. In the vine analogy, dead or non-productive branches were pruned by the Father, gathered, and burned. Every parable indicates the Lord's great displeasure at those who draw life but fail to produce fruit (a frightening thought).
When Jesus commanded us to be fruitful, was He simply speaking of us having the "fruit of the Spirit," which Paul later wrote about in Galatians 5? I don't believe so. He was talking about reproductivity – spawning new branches and bearing reproductive fruit that reached the world.
I suppose that the vine resides in most "church circles"…that is, there are believers joined together in Christ to bear fruit. But in most cases, the vine is much smaller than the circle. What the New Testament calls for is a vine that not only grows inside the circle, but advances outside – far outside! There should be more life produced in the world than in the circle – that's where the fruit is. No, I don't mean "evangelism" as it is often understood. I mean fruit bearing on a personal level – not merely corporate, local church growth.
One might appropriately ask, "Charles, what is your purpose here?" It is not to be a mere critic – that's too easy and irresponsible. I love the Church and for nearly 50 years have given my life to it. My purpose is to offer a different view of what the Church ought to be – a view that releases life to the world.
Good churches are warehouses of Divine truth and life. But we are not growing beyond our walls. Significantly, in many cases, we are not truly reaching the younger generation – one which is very relational. They just don't see the Church as being relevant to their lives because Christian individuals are not able to relate to them. It is not the programs that we fail to offer, it's the people that we fail to offer. Our people are not "programmed" to relate the love of Christ to "strangers".
What I hope to offer is a different philosophy, mindset, or blueprint for Church life. It is personal versus corporate; relational versus mechanical; outward versus inward; and above all, Christ-centered versus personality driven. Like the vine, we should grow one person at a time – that's not slow growth if all the branches "get the picture". That's compound interest.
One thing is for sure; the true vine will not end up where it started. It started in Bethlehem, Nazareth, and Jerusalem. It will end up covering the islands of the sea and every nation, kindred, and tribe. We will not do that going in circles!


As a response to recent changes in the political landscape, the Observational Practices Lab, initiated the research project OBJECT AMERICA to explore the idea of "America" through everyday objects. We invited Ellen Lupton, Senior Curator at the Cooper Hewitt, Smithsonian Design Museum, to choose an object she believed would represent "America" into the future. Thirteen researchers from very different disciplinary backgrounds, from climate science to poetry, investigated this object. The observational methods that emerge throughout the duration of this project are documented and disseminated publicly in order to inspire new ways of seeing and contribute to the conversation about who and what is "America". This site is an in-process archive of these public forums to date.
What role will observation play in the future? How might it expose, elucidate—or speak back to—systems of power and politics? The Observational Practices Lab is planning three more phases of OBJECT AMERICA in 18/19, 19/20 and 20/21. Our aim is to make the unseen visible, and speculate about the future of America, through three new "American" objects housed in different types of US collections. During this era of the Trump administration, we seek to create a platform for open exchange, exploring other ways of seeing, in an environment of playful curiosity, intelligence, and mutual respect.
We would like to thank The New School Provost's Office, and our colleagues and staff in Art, Media and Technology, and Art History & Design Theory, for their support. This research was supported by the Innovations in Education Fund Grant, the New School, and Cross-School Funding, Parsons, The New School. We would like to thank Cooper Hewitt, Smithsonian Design Museum, in New York for hosting our initial briefing session with Ellen Lupton. We also gratefully acknowledge our interdisciplinary panel of researchers and our students for their generous participation in helping us launch OBJECT AMERICA.


The Aston Martin Vantage V12 Zagato is set to go into limited production as a means of marking the 100th anniversary of Zagato Milano.
See the difference between the Porsche 911 GT3RS and the regular GT3 in this series of quarter-mile drag races.
Can a McLaren 600LT keep up with the mightiest Porsche of them all, the 911 GT2 RS?
It's Ferrari versus McLaren in a half-mile drag race in Florida.
Is a Tesla Model 3 capable of keeping up with a Dodge Challenger SRT Hellcat?
This sleeper Cadillac absolutely surprises us all and manages to defeat a brand new Dodge Charger SRT.
We finally have a better look at the new Koenigsegg Jesko in some very artistic photos taken in the Swiss Alps.
General Motors has finally admitted that the Chevrolet Corvette will drop in the form of a mid-sized engine 2020 C8 - and it's coming this July, per Car And Driver.
In this match of supercar heavyweights, the Lamborghini Aventador SVJ takes on a McLaren 720S in a quarter-mile drag race.
We know how fast the Dodge Demon is in the quarter-mile, but what if it went a little further?


text:
Seznam senatorjev 30. parlamenta Kraljevine Italije je urejen po letu imenovanja.

1939

Michele Adinolfi
Cesare Agostini
Adriano Albertini
Luigi Aldrovandi Marescotti
Roberto Alessandri
Giulio Alessandrini
Pompeo Aloisi
Giacomo Appiotti
Luigi Arborio Mella di Sant'Elia
Mario Arlotta
Fortunato Tommaso Arnoni
Giuseppe Mario Asinari Rossillon
Federico Baistrocchi
Federigo Michele Barbi
Lorenzo Bardelli
Alfonso Bartoli
Domenico Bartolini
Antonio Baslini
Giuseppe Bastianelli
Ettore Bastico
Alberto Beneduce
Arturo Beretta
Temistocle Bernardi
Romeo Bernotti
Mario Betti
Giuseppe Bianchini
Valentino Bobbio
Ambrogio Bollati
Ugo Bono
Giuseppe Boriani
Febo Borromeo d'Adda
Orazio Botturini
Umberto Bucci
Luigi Burgo
Michele Cagnetta
Pio Calletti
Inigo Campioni
Camillo Cantarano
Giuseppe Cantù
Aristide Carapelle
Giuseppe Cardinali
Pericle Cardinali
Giuseppe Carlo Catalano
Ugo Cei
Giuseppe Celi
Angelo Chiarini
Ugo Ciancarelli
Alessandro Ciano
Ettore Cipolla
Ambrogio Clerici
Iginio Coffari
Gaetano Mario Columba
Alessandro Contini Bonacossi
Felice Coralli
Giovanni Corsi
Gaetano Cosentino
Luigi D'Amato
Michele D'Aquino
Fidenzio Dall'Ora
Nicola Giuseppe Dallorso
Vincenzo De Feo
Stefano De Ruggiero
Brizio De Sanctis
Nicodemo Del Vasto
Ugolino Della Gherardesca
Michele Delle Donne
Salvatore Denti Amaro di Pirajno
Alfredo Dentice di Frasso
Francesco Dentice di Accadia
Giovanni Battista Dho
Luigi Di Lella
Gerardo Di Martino
Beniamino Donzelli
Aurelio Drago
Giorgio Emo di Capodilista
Giuseppe Fagiolari
Francesco Falcetti
Ferdinando Farina
Cristoforo Ferrari
Felice Ferrari Pallavicino
Giacomo Ferretti
Nicola Festa
Attilio Ugo Fioretti
Luigi Maria Foschini
Melchiade Gabba
Achille Gaggia
Silvo Gai
Bindo Galli
Fausto Gambardella
Gerolamo Gaslini
Giuseppe Gavazzi
Cesare Genovesi
Giuseppe Gentile
Alberto Geremicca
Adolfo Giaquinto
Cesare Giovara
Antonio Gismondi
Ferdinando Giuseppe Giuli Rosselmini Gualandi
Arturo Giuliano
Francesco Goggia
Pier Gabriele Goidanich
Camillo Grossi
Agostino Guerresi
Francesco Guidi
Guido Guidotti
Amedeo Guillet
Giulio Ingianni
Giuseppe Innocenti
Oreste Jacobini
Guido Larcher
Ettore Leopardi
Arturo Liotta
Umberto Locatelli
Rodolfo Loffredo
Vincenzo Lombard
Luigi Lombardi
Giuseppe Maladra
Bruto Mancini
Maurizio Maraviglia
Annibale Marinelli de Marco
Adriano Marinetti
Alessandro Mariotti
Gjon Marka Gjoni
Giovanni Marro
Giuseppe Marzano
Giovanni Masnata
Andrea Matarazzo
Oreste Mattirolo
Paolo Medelaghi
Davide Mele
Mustafà Merlika Kruja
Gian Carlo Messa
Ottorino Mezzetti
Filippo Mezzi
Cesare Micheli
Domenico Milani
Marcello Minale
Luigi Miraglia
Riccardo Moizio
Giulio Cesare Montagna
Raffaele Montuori
Giuseppe Morelli
Manlio Morgagni
Maurizio Mario Moris
Teodoro Morisani
Riccardo Moroder
Francesco Saverio Mosso
Riccardo Motta
Guglielmo Nasi
Giorgio Nobili
Enea Noseda
Paolo Orano
Giovanni Oriolo
Leopoldo Parodi Delfino
Guido Pasolini dall'Onda
Giovanni Penna
Alberto Pepere
Giovanni Perez
Amedeo Perna
Arnaldo Pedretti
Michele Arcangelo Petrone
Carlo Pignatti Morano di Custoza
Vladimiro Pini
Alessandro Poss
Roberto Pucci
Angelo Pugnani
Francesco Quarta
Carlo Alberto Quilico
Eolo Rebua
Arturo Riccardi
Aurelio Ricchetti
Giuliano Ricci Lotteringi del Riccio
Umberto Ricci
Giulio Ronga
Francesco Rossi
Giovanni Sabini
Guido Sagramoso
Edoardo Salazar
Pasquale Salvatore Samperi
Ruggero Santini
Alessandro Saporiti
Enrico Scodnik
Carlo Senni
Arrigo Serpieri
Carlo Sigismondi
Umberto Silvagni
Emanuele Silvestri
Ugo Sirovich
Arrigo Solmi
Umberto Somma
Edoardo Spasiano
Luigi Martino Spolverini
Raimondo Targetti
Federico Tesio
Carlo Torlonia
Emanuele Trigona
Luigi Trivelli
Angelo Tua
Francesco Tullio
Vangjel Turtulli
Giuseppe Tusini
Arturo Vacca Maggiolini
Giulio Valli
Ercole Varzi
Shevket Verlaci
Guido Viale
Edoardo Vicario
Gaetano Vinci
Umberto Zamboni

1943

Emilio Arlotti
Guido Asinari di San Marzano
Giovanni Battista Bibolini
Antonio Bifani
Piero Bolzon
Rodolfo Borghese
Alessandro Brizi
Alfredo Bruchi
Giuseppe Bruni
Vincenzo Buronzo
Alberto Calza Bini
Massimo Capialbi
Luigi Capri Cruciani
Livio Ciardi
Carlo Costamagna
Edmondo Del Bufalo
Guido Donegani
Vittorio Umberto Fantucci
Neri Farina Cini
Alessandro Chigi
Renato Macarini Carmignani
Giuseppe Mazzini
Giovanni Milani
Eugenio Morelli
Francesco Paoloni
Ludovico Pellizzari
Antonio Pesenti
Mario Racheli
Edoardo Rotigliano
Luigi Russo
Francesco Sacco
Antonio Trapani Lombardo

Glej tudi
 seznam senatorjev parlamenta Kraljevine Italije

030


First of all, we want to thank you for taking the time to explore our website, and venture onto our blog. The concept of this blog is to introduce (and invite) ladies all around the world to unleash their inner bridal excitement. We plan on bringing you everything from our favorite wedding planners, to a pre-wedding detox guide that will get you in the best shape of your life.
If you're not familiar with Kelly Faetanini yet, she's gained a massive following as one of the prior designers at Priscilla of Boston. At a young age, Kelly knew she wanted to escape her small Ohio town to make it as a successful designer/fashionista in the Big Apple. After feeling comfortable enough around bridal dresses on all scales, she decided to venture out on her own and take the risk of beginning her own company. Welcome to the KF brand.
Fast forward several years, a marriage, and a Manhattan office, Kelly continues to give women all around the globe the chance to look and feel beautiful without having an overindulgent budget.


This group of drugs includes anastrozole (Arimidex), letrozole (Femara), and exemestane (Aromasin). They work by blocking an enzyme (aromatase) in fat tissue that converts male hormones from the adrenal glands into estrogen. Aromatase inhibitors are taken daily as pills. They have been very effective in treating breast cancer in women, but they have not been well-studied in men. Still, some doctors use them to treat advanced breast cancer in men, often combined with a luteinizing releasing hormone analog to turn off hormone production by the testicles.
These drugs are generally used if tamoxifen stops working. They are also sometimes used as the first line of hormone therapy instead of tamoxifen, but this has been linked to poorer outcomes. The main side effects of these drugs are thinning of the bones and joint and muscle pain.


Two old warhorses, Jesse Ryder and Bronson Meehan, have become the architects in helping their Napier Technical Old Boys book flights to Auckland to defend their crown as the best premier men's club in New Zealand.
Player/coach Ryder and former Pay Excellence Hawke's Bay representative Meehan showed little mercy to oppositions in the Central Districts region qualifying finals in Palmerston North with centuries a piece yesterday and handsome contributions today.
"The boys are pretty ecstatic, watching the reaction because I wasn't on when we were won because I popped my calf again, so it was pretty pleasing to watch that sort of pride in the team but also the number of supporters we had here too was really astonishing," said Meehan of the Liam Rukuwai-led red machine tonight.
The Innovative Electrical NTOB side had to toil against hosts Palmerston North Marist CC in the final for a 44-run victory today after a walk across Fitzherbert Park yesterday with a 171-run result.
Ryder, at No 4, scored 77 runs from 86 balls while Meehan added 38 runs to the collective after No 2 Christian Leopard posted 42 runs for what appeared to be a modest 229/7 total today.
The former Black Caps batsman scored eight fours and a six at a strike rate of 89.6.
"Jesse's obviously a freak and he's incredible at what he does but, I think, just relying on your experiences is important because we've all played for our Tech side and at the higher levels so you can draw on that to get you through those rough patches," said of the 34-year-old linchpin from Napier who lost his CD contract this summer.
"That's what Jess and I did because we knew if we set it up for the last 10 overs our batting is very destructive so we didn't need to go too early and just leave it to the end to smash 100 runs just like that," he said.
The 29-year-old, who is a teacher at Maraekakaho School and whose charges had high expectations of him this weekend because he coaches the year 7-8 mixed gender cricket team, gave a shout out.
"They all knew I was going to play down here this weekend and we have a big cricket sign in school so they'll be keen so I'll surely be sharing my successes tomorrow with them which will be exciting," said Meehan who is an ardent Tech man who seldom ever isn't available when the premier men send out an SOS.
As clichéd as it sounded, the Texans contributed collectively today and Meehan had suspected the total was going to be a tall order for Marist considering the champion Napier side's bowling attack.
"We don't rely on one person to pick so many wickets so if a guy doesn't turn up he's not going to bowl and he'll be taken off."
Opening bowler Connor Morris was Marist's pick with 3-39 from nine overs, including two maidens. Hugh Symes took 2-45 and unwanted opening seamer Navin Patel took 1-44 but it was Arana Noema-Barnett, whose Stags brother Kieran plays for NTOB, was the most frugal with 1-28 from 10, including a maiden.
In their run chase, Marist had several 20-something starts, including those from brothers Floyd na Nagara and Whetu na Nagara as well as Noema-Barnett, but no batsmen carried on to forge any meaningful partnerships to make NTOB sweat.
Right-arm medium/fast Todd Watson took 3-45 while Stags allrounder Leopard and Bay rep left-arm spinner Jayden Lennox collected two each while Meehan and Kieran Noema-Barnett claimed one each.
Meehan said the scoreboard reflected a one-sided battle yesterday in the walloping of Ballance Agri-Nutrients New Plymouth Old Boys CC but it wasn't a cake walk by any yardstick.
"The conditions were a lot different today," he said. The wicket was slow and sticky and held up the ball.
Yesterday, Ryder and Meehan had put on a don't-argue fend with their willows.
The pair helped post a daunting 307/4 in their allotted 50 overs after the opposition won the toss and chose to shine the ball.
Ryder carved up 134 runs from 121 balls, including 19 boundaries and two sixes, at No 4 while Meehan, at No 5, forged a formidable partnership with an unbeaten 111 from 155 deliveries, including nine fours and five sixes.
Right-arm medium pacer Kyle Pillay was the pick of New Plymouth OB bowlers, claiming 3-50 from his 10 overs, including two maidens.
In reply, New Plymouth OB could muster 136 before they were skittled in 31.2 overs.
Opening batsman Jackson Braddock-Pajo provided some hope with 38 runs but only No 7 Ross Bryans came to the party with an unbeaten 40 runs.
The Texans got the biggest traction from Watson, at first change, who finished with 3-22 from six overs while Lennox, at second change, also chimed in with 3-21 from 5.2 overs, including two maidens.
Leopard and Rukuwai claimed two wickets each.


text:
Hazard Chase at the BBC Proms
Photo: BBC Proms
Prom 19: Saturday 29th July (12.00)
Grant Llewellyn conducts the BBC National Orchestra of Wales in a RELAXED PROM
Prom 23: Tuesday 1st August (19.30)
Callum Thorpe sings bass in Handel's ISRAEL IN EGYPT with the Orchestra and Choir of the Age of Enlightenment conducted by William Christie
Proms at Southwark Cathedral: Saturday 12th August (15.00)
Adrian Thompson appears with the BBC Singers and the Nash Ensemble in Judith Weir's new commission IN THE LAND OF UZ
Prom 44: Thursday 17th August (22.15)
Bang on a Can All-Stars with the BBC Proms Youth Ensemble
Prom 46: Saturday 19th August (19.30)
Peter Hoare (Klaus the Jester) joins the solo line-up for Schoenberg's GURRELIEDER with the London Symphony Orchestra & Chorus conducted by Sir Simon Rattle
Prom 59: Monday 28th August (19.00)
Anna Stéphany sings Sesto in LA CLEMENZA DI TITO with Glyndebourne Festival Opera conducted by Robin Ticciati


May 2008: A honey blonde BRAVO.
This entry was posted in News and tagged Archtop guitars, Benedetto Bravo model, Benedetto Guitars, Benedetto honey blonde archtops, Bravo model, jazz guitars. Bookmark the permalink.