text:
\section{Introduction}
\IEEEPARstart{D}{istributed} optimization problem is to minimize a finite sum
of functions over a network made up of multi agents, where each agent holds the local objective function and can communicate with its neighbors.  Distributed optimization problem has many applications  in large-scale machine learning \cite{boyd2011}, wireless networks \cite{naghshineh1996distributed}, parameter estimation \cite{Towfic2015}, to name a few. Over the last decades, numerous algorithms for  distributed optimization problem have been developed, such as (sub)gradient method \cite{pu2021sharp,bianchi2013performance,Mor2012dsa,omidvar2020hybridorder,swenson2020distributed,fallah2019robust,de2017variance}, dual averaging method \cite{Tsi2012dual,DC2012}, primal dual method \cite{Lei2018,yi2020primaldual}, gradient push method \cite{Nedic2016sGpush,pmlr-v97-assran19a,qureshi2020pushsaga}, gradient tracking method \cite{pu2020distributed,Xin2012improv,Xin_2020}.  We refer  to the survey \cite{YANG2019278} for the new development on distributed optimization.

Recently,  Xin and Khan \cite{Xin2018linear} and  Pu et al. \cite{pu2018push}  propose  the $\mathcal{AB}$/push-pull method, which is suitable for  the  cases that the communication network is directed.  Compared with distributed gradient tracking method \cite{Nedic2017achive},  $\mathcal{AB}$/push-pull method  utilizes a  row  stochastic  matrix  for mixing the  decision variables and a  column  stochastic  matrix  for tracking the average gradients.
Since  $\mathcal{AB}$/push-pull method may unify much of the existing decentralized first-order methods  based on gradient tracking technique \cite{RX2020}, many works have been devoted to the development of $\mathcal{AB}$/push-pull method. To communicate more efficiently, Pu \cite{pu2020robust} proposes a robust push-pull method which introduces a new iteration scheme and utilizes the difference between successive iterates to track the average gradient; Xiong et al. \cite{xiong2021quantized} propose a novel quantized distributed gradient tracking method, which can be rewritten as an inexact version of $\mathcal{AB}$/push-pull method. However,  the  iterates of quantized distributed gradient tracking method
can converge to the optimal solution rather than  the neighborhood of the optimal solution   \cite{pu2020robust}.
Saadatniaki, Xin and Khan \cite{saadatniaki2018optimization} propose a variant of $\mathcal{AB}$/push-pull method with time-varying weight matrices and show that the proposed method converges linearly to the optimal solution when each local objective is smooth and the global objective is strongly-convex. Accelerate techniques have also been incorporated into  $\mathcal{AB}$/push-pull method, such as, Xin and Khan \cite{Xin2020heavy} employ the heavy-ball method to accelerate the  $\mathcal{AB}$/push-pull method  where the R-linear rate for strongly convex smooth objective has been obtained;  Xin, Jakoveti and Khan \cite{RX2019Nestrov} combine  nesterov gradient method with $\mathcal{AB}$/push-pull method and show that the new method achieves robust numerical performance  for both convex and strongly-convex objectives. Moreover, the extended versions of $\mathcal{AB}$/push-pull method have been used to solve practical problem, such as, resource allocation \cite{zhang2020dual} and the distributed multi-cluster game \cite{zimmermann2021gradienttracking}.

More recently,    stochastic gradient version of $\mathcal{AB}$/push-pull method ($\mathcal{S}$-$\mathcal{AB}$) has been proposed \cite{RX2019,pu2020distributed}.
Xin et al. \cite{RX2019} focus on the case that the step-size is constant and show that  the iterates of  $\mathcal{S}$-$\mathcal{AB}$ converge linearly  to  a  neighborhood  of  the  global minimizer when agents' cost functions are smooth and
strongly-convex. Pu and Nedi\'{c} \cite{pu2020distributed} study the convergence properties of distributed stochastic gradient tracking method (DSGT) which can be regarded as a special $\mathcal{S}$-$\mathcal{AB}$ with doubly stochastic weight matrix.  Motivated by \cite{RX2019,pu2020distributed}, this paper is devoted to developing $\mathcal{S}$-$\mathcal{AB}$ by studying the convergence properties  when the step-size is  diminishing and the weight matrices are not doubly stochastic. We focus on the convergence of
the iterate of $\mathcal{S}$-$\mathcal{AB}$ in the mean square sense and its asymptotic normality.

Indeed,  the convergence of stochastic approximation (SA) based methods for distributed  stochastic optimization problem with diminishing step-size have been well studied \cite{bianchi2013performance,Lei2018,pu2020distributed,Nedic2016sGpush}, where most of the results are based on the fact that the  stochastic gradient noise  is  martingale difference  sequence.
We analyze the convergence of $\mathcal{S}$-$\mathcal{AB}$ by following the  standard steps of establishing convergence of distributed stochastic gradient descent method \cite{bianchi2013performance}, that is,  study    the agreement  and stability of  $\mathcal{S}$-$\mathcal{AB}$ first and then  analyze the convergence by the strongly convexity of the objective function.   However,
	for the stochastic gradient tracking technique based distributed  methods, agents' accumulated stochastic gradient noises during gradient tracking steps
form autoregressive moving average (ARMA) processes, which means that the weighted average of the accumulated noise among agents forms a martingale difference sequence only if  the weight matrices are doubly stochastic \cite{pu2020distributed}.
This motivates us to employ   the    intrinsic structure of  the accumulated noise among agents to extend the   nonlinear  inequalities  system  in  \cite[Lemma 3]{bianchi2013performance},   which has been used to study the agreement and stability of distributed stochastic gradient descent method.
The  nonlinear  inequalities  system
plays  a key role   in studying the stability and agreement of $\mathcal{S}$-$\mathcal{AB}$.
We present that the iterates of $\mathcal{S}$-$\mathcal{AB}$ converge to the solution
with the rate  $\mathcal{O} (1/k)$ (in the mean square sense),  which arrives at the optimal rate of vanilla stochastic gradient decent method with diminishing step-size.

We present that Polyak-Ruppert averaged $\mathcal{S}$-$\mathcal{AB}$ converges in distribution to a
normal random vector for any agent
by combining the convergence rate of $\mathcal{S}$-$\mathcal{AB}$ and  results on asymptotic normality of
SA based algorithm \cite[Theorem 3.4.2]{chen2006stochastic}.
The asymptotic normality  of   stochastic approximation based algorithms can be traced back to 1950s \cite{chung1954stochastic,fabian1968asymptotic}.
For the asymptotic normality on distributed stochastic optimization problem, we refer \cite{bianchi2013performance,Mor2012dsa,Lei2018} for unconstrained problems and \cite{Sahu2016,zhao2020asymptotic} for constrained problems.
Complement to the works mentioned above, the new result does not need  the communication network to be undirected or weight matrix to be doubly
stochastic. Moreover,  the asymptotic normality of Polyak-Ruppert averaged $\mathcal{S}$-$\mathcal{AB}$ has been used to  construct confidence regions of optimal solutions.

The rest of this paper is organized as follows. Section 2 introduces the distributed stochastic optimization problem model, the $\mathcal{S}$-$\mathcal{AB}$ method and presents some standard assumptions in distributed stochastic optimization problem. Section 3 focuses on the convergence rate and asymptotic normality of the $\mathcal{S}$-$\mathcal{AB}$ method.  At last,
numerical results are presented in Section 4
to validate the theoretic results.

Throughout this paper, we use the following notation. $\mathbb{R}^d$ denotes  the d-dimension Euclidean space endowed with norm $\|x\|=\sqrt{\langle x,x\rangle}$.  Denote $\mathbf{1}:=(1~1\dots1)$, $\mathbf{0}\define(0~0\dots0)^\intercal$ and whose dimension will be clear in the context.
$\mathbf{I}_{d}\in\mathbb{R}^{d\times d}$ stands for the identity matrix.
$\mathbf{A}\otimes \mathbf{B}$ denotes the Kronecker product of matrix $\mathbf{A}$ and $\mathbf{B}$.
For a sequence of random vectors $\{\xi_k\}$ and a random vector $\xi$, $\xi_k\stackrel{d}{\rightarrow} \xi$ represents the convergence in distribution, \Cov($\xi$) denotes the covariance matrix of random vector $\xi$. $N\left(\mu,\Sigma\right)$ is the normal distribution with mean $\mu$ and covariance matrix $\Sigma$. For any sequences $\{a_k\}$ and $\{b_k\}$ of positive number,  $a_k=\mathcal{O}(b_k)$ if there exists $c>0$ such that $a_k\le c b_k$.

\section{Problem formulation and $\mathcal{S}$-$\mathcal{AB}$ method}\label{mod-ass}

In this paper, we consider the following distributed stochastic optimization  (DSO) problem
\begin{equation}
\begin{aligned}\label{problem model}
&\min_{x\in\mathbb{R}^d} ~f(x)=\sum_{j=1}^nf_j(x),
\end{aligned}
\end{equation}
where $f_j(x):=\mathbb{E}[g_j(x;\zeta_j)]$ denotes the cost  of agent $j$, $g_j(x;\zeta_j):\mathbb{R}^d\times \Omega\rightarrow \mathbb{R}$ is the measure function, $\zeta_j$ is a random variable defined on a probability space $(\Omega,\mathcal{F},\P)$ and $\mathbb{E}[\cdot]$ denotes the expectation with respect to probability $\P$. For DSO (\ref{problem model}),
the communication relationship between agents is characterized by a directed graph, $\mathcal{G}=\left(\mathcal{V},\mathcal{E}\right)$, where $\mathcal{V}=\{1,2,...,n\}$ is the node set with node $i$ representing agent $i$ and $\mathcal{E}\subseteq\mathcal{V}\times\mathcal{V}$ denotes the edge set with edge $(j,i)$ representing agent $i$ can receive information from agent $j$.

The  $\mathcal{S}$-$\mathcal{AB}$ method in \cite{RX2019} reads as follows.
\begin{algorithm}[H]
	\caption{$\mathcal{S}$-$\mathcal{AB}$: At each node $i\in\mathcal{V}=\{1,2,...,n\}$}\label{alg:SAB}
	\textbf{Require:} initial value $x_{i,0}\in\mathbb{R}^{d}$, $y_{i,0}=\nabla g_i(x_{i,0},\zeta_{i,0})$, weight matrices $\mathbf{A}=\{a_{ij}\}$ and $\mathbf{B}=\{b_{ij}\}$
	\begin{itemize}
		\item[1:]\textbf{For} $k=1,2,\cdots$ \textbf{do}
		\item [2:]\textbf{State update:}
		\begin{equation}\label{alg:x}
		x_{i,k+1}=\sum_{j=1}^n a_{ij} x_{j,k}-\alpha_k y_{i,k}.
		\end{equation}
		\item[3:]\textbf{Gradient tracking update:}
		$$
		 y_{i,k+1}=\sum_{j=1}^n b_{ij} y_{j,k}+\nabla g_i(x_{i,k+1},\zeta_{i,k+1})-\nabla g_i(x_{i,k},\zeta_{i,k}),
		$$
		where $\zeta_{i,0},\zeta_{i,1},\cdots$  is independent and identically distributed sample of $\zeta_{i}$.
		\item[4:]\textbf{end for}
	\end{itemize}
\end{algorithm}

Throughout our analysis in the paper,  we make the following two assumptions on the objective function and weight matrices $\A$ and $\Z$.

\begin{ass}[\textbf{objective function}]\label{ass:objective}

	\begin{itemize}
		\item [(i)] $f(x)$ is $\mu$-strongly convex in $x$, that is
		\begin{equation*}
		f(y)\ge f(x)+\langle \nabla f(x),y-x\rangle+\frac{\mu}{2}\|x-y\|^2, \quad \forall x,y\in\mathbb{R}^d.
		\end{equation*}
		\item [(ii)] Let $x^*$ be an optimal solution. There exists a constant $c>0$ such that
	\begin{equation*}
		\|\nabla f(x)-\nabla^2 f(x^*)\left(x-x^*\right)\|\le c\|x-x^*\|^2,\quad \forall x\in\mathbb{R}^n.
		\end{equation*}
		\item [(iii)]For any $i\in\mathcal{V}$, there exist constants $p>2$, $U_1>0$ and a positive random variable $L_i(\zeta_i)$ such that $\mathbb{E}[L_i^p(\zeta_i)]<\infty$ and
		\begin{equation*}
		\begin{aligned}
		&\mathbb{E}\left[\|\nabla g_i(x;\zeta_{i})-\nabla f_i(x)\|^{p}\right]\le U_1^{p/2},\\
		&\|\nabla g_i(x;\zeta_i)-\nabla g_i(y;\zeta_i)\|\le L_i(\zeta_i)\|x-y\|, \quad \forall x,y\in\mathbb{R}^d.
		\end{aligned}
		\end{equation*}
	\end{itemize}

\end{ass}

 Assumption \ref{ass:objective} (i)- (ii) are standard conditions on objective  and have been well used to study the  asymptotic properties of SA based method \cite{duchi2016asymptotic,Polyak1992}.
In Assumption \ref{ass:objective} (iii),  the constant $p=2$ is sufficient to investigate the convergence rate of $\mathcal{S}$-$\mathcal{AB}$ as \cite{RX2019,RX2020},  while   $p>2$ is needed for   the  asymptotic normality of $\mathcal{S}$-$\mathcal{AB}$.

\begin{ass}  [\textbf{weight matrices and networks}]\label{ass:matrix}
	Let $\mathcal{G}_A=\left(\mathcal{V},\mathcal{E}_\A\right)$ and $\mathcal{G}_{B^\intercal}=\left(\mathcal{V},\mathcal{E}_{\Z^\intercal}\right)$ be graphs induced by  matrices $\mathbf{A}$ and $\mathbf{B}^\intercal$ respectively.  Suppose that
	\begin{itemize}
		\item [(i)]The matrix $\mathbf{A}\in\mathbb{R}^{n\times n}$ is nonnegative row stochastic and $\mathbf{B}\in\mathbb{R}^{n\times n}$ is nonnegative column stochastic, i.e., $\mathbf{A}\mathbf{1}=\mathbf{1}$ and $\mathbf{1}^\intercal\mathbf{B}=\mathbf{1}^\intercal$. In addition, the diagonal entries of $\mathbf{A}$ and $\mathbf{B}$ are positive, i.e., $\mathbf{A}_{ii}>0$ and $\mathbf{B}_{ii}>0$ for all $i\in \mathcal{V}$.
		\item[(ii)]  The graphs $\mathcal{G}_A$ and $\mathcal{G}_{B^\intercal}$ each contain at least one spanning tree. Moreover, there exists at least one node that is a root of spanning trees for both $\mathcal{G}_A$ and $\mathcal{G}_{B^\intercal}$, i.e. $\mathcal{R}_A\cap \mathcal{R}_{B^\intercal}\ne\emptyset$, where $\mathcal{R}_A$ ( $\mathcal{R}_{B^\intercal}$) is the set of roots of all possible spanning trees in the graph $\mathcal{G}_A$ ( $\mathcal{G}_{B^\intercal}$).
	\end{itemize}
\end{ass}

Assumption \ref{ass:matrix} is weaker than requiring that both $\mathcal{G}_A$ and $\mathcal{G}_{B^\intercal}$
are strongly connected \cite{pu2020push}.  Under  Assumption \ref{ass:matrix}, the matrix $\mathbf{A}$ has a nonnegative left eigenvector $u^\intercal$ (w.r.t. eigenvalue 1) with $u^\intercal\mathbf{1}=n$, and the matrix $\mathbf{B}$ has a nonnegative right eigenvector $v$ (w.r.t. eigenvalue 1) with $\mathbf{1}^Tv=n$. Moreover, $u^\intercal v>0$.

	For easy of presentation, we rewrite Algorithm \ref{alg:SAB} in a compact form:
	\begin{equation}\label{alg:new form}
	\begin{aligned}
	&x_{k+1}=\tilde{\mathbf{A}}x_k-\alpha_ky_k,\\
	&y_{k+1}=\tilde{\mathbf{B}}y_k+\nabla G_{k+1}-\nabla G_k,
	\end{aligned}
	\end{equation}
	where $\tilde{\mathbf{A}}\define\mathbf{A}\otimes \mathbf{I}_d,~ \tilde{\mathbf{B}}\define\mathbf{B}\otimes \mathbf{I}_d$, the vectors $x_k$, $y_k$ and $\nabla G_k$ concatenate all $x_{i,k}$'s, $y_{i,k}$'s and $\nabla g_i(x_{i,k},\zeta_{i,k})$'s respectively.

\section{The convergence rate and asymptotic normality}
In this section, we study the convergence  rate and asymptotic normality of $\mathcal{S}$-$\mathcal{AB}$ method.
As commented in the introduction, the convergence rate of $\mathcal{S}$-$\mathcal{AB}$ has been discussed in \cite{RX2019,pu2020distributed}. We focus on the rate of $x_{i,k}\to x^*$ rather than  $x_{i,k}$ converging into
a neighbor of $x^*$ \cite{RX2019}. Moreover, different from \cite{pu2020distributed},  the weighted average of the accumulated noises among agents can not form the martingale difference sequence as weight matrices $\mathbf{A}$ and $\mathbf{B}$ are not doubly stochastic.

We first study the    convergence rate  of iterates of $\mathcal{S}$-$\mathcal{AB}$ in the mean square sense. The following technical Lemma is  an extension of   \cite[Lemma 3]{bianchi2013performance}, which plays a key role in the
stability and agreement of  $\mathcal{S}$-$\mathcal{AB}$ method.

\begin{lem}\label{lem:ieq-1}
	Suppose that positive sequences $\{\gamma_{k}\}$,  $\{\rho_{k}\}$,  $\{\phi_{k}\}$ satisfy
	\begin{itemize}
		\item[(i)]$\{\gamma_{k}\}$,  $\{\rho_{k}\}$ are $[0,1]$-valued sequences such
		that $\sum_{k=0}^\infty\gamma_{k}^2<\infty$.
		\item[(ii)]
		\begin{align*}
			&\limsup_{k\rightarrow\infty}\left(\gamma_{k}\sqrt{\phi_k}+\frac{\phi_{k-1}}{\phi_k}\right)<\infty,\quad \sum_{k=0}^\infty\phi_k^{-1}<\infty,\\
			&\liminf_{k\rightarrow\infty}\left(\gamma_{k}\sqrt{\phi_k}\right)^{-1}\left(\frac{\phi_{k-1}}{\phi_k}-\rho_k\right)>0.
			\end{align*}
	\end{itemize}
	If for scalars $M>0$ and $1>\rho>0$, sequences $\{u_k\},~\{v_k\}$ satisfy that for $k\ge k_0$,
	\begin{align}
	\label{form-1}u_{k+1}&\le\rho_ku_k+M\gamma_{k}\sqrt{u_k(1+u_k+v_k)}\\
	&\quad+M\gamma_{k}\left(\gamma_{k}+\sum_{t=k_0}^k\gamma_{t}\rho^{k-t}u_t+\sum_{t=k_0}^k\gamma_{t}\rho^{k-t}v_t\right),\notag\\
	\label{form-2}v_{k+1}&\le v_k+Mu_k+M\gamma_{k}\sqrt{u_k(1+u_k+v_k)}\\
	&+M\gamma_{k}\left(\gamma_{k}+\sum_{t=k_0}^k\gamma_{t}\rho^{k-t}u_t+\sum_{t=k_0}^k\gamma_{t}\rho^{k-t}v_t\right),\notag
	\end{align}
	then
	\begin{equation*}
	\sup_kv_k<\infty,\quad \limsup_{k\rightarrow\infty}\phi_ku_k<\infty.
	\end{equation*}
\end{lem}

With Lemma \ref{lem:ieq-1} at hand, we are  ready for
presenting the stability and agreement of $\mathcal{S}$-$\mathcal{AB}$.
\begin{lem}\label{lem:rate}
	Suppose that Assumptions \ref{ass:objective} and \ref{ass:matrix} hold. Let step-size $\alpha_{k}=a/(k+b)^\alpha$, where $\alpha\in (1/2,1]$, positive scalars $a,b$ satisfy  $\frac{a}{b^\alpha}\le \frac{2n}{u^\intercal vL_0}$, $L_0\define\max_{1\leq i \leq n}\mathbb{E}[L_{i}(\zeta_i)]$, where $u^\intercal$ and $v$ are the left eigenvector of $\mathbf{A}$ and the right eigenvector of $\mathbf{B}$ respectively (see Part \ref{mod-ass} for the definitions). Define average $\bar{x}_k\define \left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)x_k$ and auxiliary sequence
	$\{y_k^{'}\}$ as
	\begin{equation*}
	\begin{aligned}
	&y_{k+1}^{'}=\tilde{\mathbf{B}}y_k^{'}+\nabla F_{k+1}-\nabla F_k,\quad y_{0}^{'}=\nabla F_{0},
	\end{aligned}
	\end{equation*}
	where vector $\nabla F_k$ concatenates all $\nabla f_i(x_{i,k})$'s. Then there exists a positive constant $U_2$ such that
	\begin{equation*}
	\sup_{k}\mathbb{E}\left[\|\bar{x}_{k}-x^*\|^2\right]\le U_2,\quad
	\mathbb{E}\left[\|x_{k}-\mathbf{1}\otimes\bar{x}_{k}\|^2\right]\le U_2\alpha_k^2
	\end{equation*}
	and
	\begin{equation*}
	\mathbb{E}\left[\|y_{k}^{'}-v\otimes\bar{y}_{k}^{'}\|^2\right]\le U_2\alpha_k^2,
	\end{equation*}
	where $\bar{y}_k^{'}\define\left(\frac{\mathbf{1}^\intercal}{n}\otimes\mathbf{I}_{d}\right)y_k^{'}$.
\end{lem}

Before presenting the proof of Lemma \ref{lem:rate}, we recall two matrix norms $\|\cdot\|_\A$ and $\|\cdot\|_\Z$ introduced in \cite[Lemma 4]{pu2020push}, which have the following properties.
\begin{itemize}
	\item [(i)]\cite[Lemma 4]{pu2020push}.
	\begin{align}
	\label{consensus-para}&\tau_\mathbf{A}\define\left\|\tilde{\mathbf{A}}-\frac{\mathbf{1}u^\intercal}{n}\otimes \mathbf{I}_d\right\|_\mathbf{A}< 1,\\ \label{consensus-para-1}&\tau_\mathbf{B}\define\left\|\tilde{\mathbf{B}}-\frac{v\mathbf{1}^\intercal}{n}\otimes \mathbf{I}_d\right\|_\mathbf{B}<1.
	\end{align}
	\item [(ii)]\cite[Lemma 5]{pu2020push}. There exist vector norms $\|\cdot\|_\A$ and $\|\cdot\|_\Z$ such that
	\begin{equation*}
	\|\mathbf{C}x\|_\A\le \|\mathbf{C}\|_\mathbf{A}\|x\|_\mathbf{A},\quad \|\mathbf{C}x\|_\mathbf{B}\le \|\mathbf{C}\|_\mathbf{B}\|x\|_\mathbf{B}
	\end{equation*}
	for any $\mathbf{C}\in\mathbb{R}^{nd\times nd},x\in\mathbb{R}^{nd}$.
	\item [(iii)] \cite[Lemma 6]{pu2020push}. Let $\|\cdot\|_*$ and $\|\cdot\|_{**}$ be any two vector norms of $\|\cdot\|$, $\|\cdot\|_\A$ and $\|\cdot\|_\Z$. There exists a constant $\X>1$ such that
    \begin{equation}\label{norm-bound-1}
	\|x\|_*\le \X \|x\|_{**},\quad x\in\mathbb{R}^{nd}.
	\end{equation}
\end{itemize}

\begin{proof}
	We employ  Lemma 1 to prove Lemma \ref{lem:rate} and  finish the proof by the following two steps: find relationships of $u_k$ and $v_k$ in the forms of (\ref{form-1}) and (\ref{form-2}) first, and then verify conditions (i)-(ii) of Lemma \ref{lem:ieq-1}.

	\textbf{Step 1.}
	Define $\epsilon_{i,k}\define\nabla g_i(x_{i,k};\zeta_{i,k})-\nabla f_i(x_{i,k})$ and sequence $\{\xi_k\}$ as
	\begin{equation}\label{auxi-seq}
	\xi_{k+1}=\tilde{\mathbf{B}}\xi_k+\epsilon_{k+1}-\epsilon_k,\quad \xi_{0}=\epsilon_{0},
	\end{equation}
	where $\epsilon_k$ concatenates all  $\epsilon_{i,k}$'s.
	By the definitions of $\bar{x}_{k}$, $y_k^{'}$ and $\xi_{k+1}$, we have $y_k=y_k^{'}+\xi_{k+1}$ and
	\begin{equation}\label{consensus-new-form}
	\begin{aligned}
	\bar{x}_{k+1}
	&=\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\tilde{\mathbf{A}}x_k-\alpha_k\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)y_k\\
	&=\bar{x}_{k}-\alpha_k\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(y_k^{'}+\xi_k\right)\\
	&=\bar{x}_{k}-\frac{u^\intercal v\alpha_{k}}{n^2}\nabla f(\bar{x}_k)-\frac{u^\intercal v\alpha_{k}}{n}\left(\bar{y}^{'}_k-\frac{1}{n}\nabla f(\bar{x}_k)\right)\\
	&\quad-\alpha_k\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(y_k^{'}-v\otimes\bar{y}^{'}_k\right)-\alpha_{k}\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\xi_k,
	\end{aligned}
	\end{equation}
	where the second equality follows from  $u^\intercal\mathbf{A}=\mathbf{1}$. By the recursion(\ref{consensus-new-form}),
	{\small\begin{equation}\label{eq-auxi-3}
		\begin{aligned}
		&\mathbb{E}\left[\|\bar{x}_{k+1}-x^*\|^2\right]\\
		&=\mathbb{E}\left[\left\|\bar{x}_{k}-x^*-\frac{u^\intercal v\alpha_{k}}{n^2}\nabla f(\bar{x}_k)\right\|^2\right]\\
		&\quad+\alpha_{k}^2\mathbb{E}\left[\left\|\frac{u^\intercal v}{n}\left(\bar{y}^{'}_k-\frac{1}{n}\nabla f(\bar{x}_k)\right)-\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(y_k^{'}-v\otimes\bar{y}^{'}_k\right)\right.\right.\\
		&\quad\left.\left.-\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\xi_k\right\|^2\right]+2\mathbb{E}\left[\left\langle \bar{x}_{k}-x^*-\frac{u^\intercal v\alpha_{k}}{n^2}\nabla f(\bar{x}_k),\right.\right. \\
		&\quad\left.\left.\alpha_{k}\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(y_k^{'}-\frac{v}{n}\otimes \nabla f(\bar{x}_k)+\xi_k\right)\right\rangle\right]\\
		&\le\left(1-\frac{u^\intercal v\mu\alpha_{k}}{n^2}\right)^2\mathbb{E}\left[\left\|\bar{x}_{k}-x^*\right\|^2\right]+3\alpha_{k}^2\mathbb{E}\left[\left(\frac{u^\intercal v}{n}\right)^2\frac{L^2\X^2}{n}\right.\\
		&\quad \left.\left\|x_k-\mathbf{1}\otimes\bar{x}_k\right\|_\mathbf{A}^2+\frac{\|u\|^2\X^2}{n^2}\left\|y_k^{'}-v\otimes\bar{y}^{'}_k\right\|_\mathbf{B}^2\right]+3\alpha_{k}^2\frac{\|u\|^2c_b^2U_1}{n(1-\tau_\Z)^2}\\
		&\quad+2\mathbb{E}\left[\left\langle \bar{x}_{k}-x^*, \alpha_{k}\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(y_k^{'}-\frac{v}{n}\otimes \nabla f(\bar{x}_k)+\xi_k\right)\right\rangle\right]-\\
		&\quad 2\frac{u^\intercal v\alpha_{k}^2}{n^2}\mathbb{E}\left[\left\langle \nabla f(\bar{x}_k), \left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(y_k^{'}-\frac{v}{n}\otimes \nabla f(\bar{x}_k)+\xi_k\right)\right\rangle\right],
		\end{aligned}
		\end{equation}}
	where  $c_b=\max\left\{\X^2,\frac{\left\|\mathbf{B}-\mathbf{I}_{n}\right\|}{\tau_\Z}\X^2\right\}$, $\X$ is defined in (\ref{norm-bound-1}), the inequality follows from the Lipschitz continuity of $\nabla f_i(\cdot)$, (\ref{bound-noi}) in Appendix \ref{apd:proof of Lem-rate} and the fact by \cite[Lemma 10]{qu2017harnessing}\footnote{By the definition of $\alpha_{k}$, $\frac{u^\intercal v\mu\alpha_{k}}{n^2}\le\frac{u^\intercal v\mu a}{n^2b^\alpha}\le \frac{2}{nL_0}$, which satisfies the condition of \cite[Lemma 10]{qu2017harnessing}.} that
	{\small$$\left\|\bar{x}_{k}-x^*-\frac{u^\intercal v\alpha_{k}}{n^2}\nabla f(\bar{x}_k)\right\|^2\le\left(1-\frac{u^\intercal v\mu\alpha_{k}}{n^2}\right)^2\left\|\bar{x}_{k}-x^*\right\|^2.$$}
	  The upper bound of the fourth term on the right hand side of (\ref{eq-auxi-3}) has be obtained in Lemma \ref{lem:noi-bound}  (Appendix \ref{apd:proof of Lem-rate}). By the Lipschitz continuity of $\nabla f_i(\cdot)$, the fifth  term on the right hand side of (\ref{eq-auxi-3})
	{\small\begin{equation}\label{term3}
		\begin{aligned}
		&-2\frac{u^\intercal v\alpha_{k}^2}{n^2}\mathbb{E}\left[\left\langle \nabla f(\bar{x}_k), \left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(y_k^{'}-\frac{v}{n}\otimes \nabla f(\bar{x}_k)+\xi_k\right)\right\rangle\right]\\
		&\le\frac{u^\intercal v\alpha_{k}^2}{n}\mathbb{E}\left[\frac{L^2}{n}\|\bar{x}_k-x^*\|^2+\left(\frac{u^\intercal v}{n}\right)^2\frac{L^2\X^2}{n}\left\|x_k-\mathbf{1}\otimes\bar{x}_k\right\|_\mathbf{A}^2\right]\\
		&+\frac{u^\intercal v\alpha_{k}^2}{n}\mathbb{E}\left[\frac{\|u\|^2\X^2}{n^2}\left\|y_k^{'}-v\otimes\bar{y}^{'}_k\right\|_\mathbf{B}^2\right]+\frac{u^\intercal v\|u\|^2c_b^2U_1}{n^2(1-\tau_\Z)^2}\alpha_{k}^2.
		\end{aligned}
		\end{equation}}
	 Then, there exists a constant $c_0>0$ such that
	\begin{equation}\label{ine-0}
	\begin{aligned}
	&\mathbb{E}\left[\|\bar{x}_{k+1}-x^*\|^2\right]\\
	&\le \left(1-\frac{u^\intercal v\mu\alpha_{k}}{n}\right)^2\mathbb{E}\left[\left\|\bar{x}_{k}-x^*\right\|^2\right]\\
	&\quad+2\alpha_{k}\left(\mathbb{E}\left[\left\|\bar{x}_{k}-x^*\right\|^2\right]\left(2\frac{(u^\intercal vL\X)^2}{n^{3}}\mathbb{E}\left[\left\|x_k-\mathbf{1}\otimes\bar{x}_k\right\|_\A^2\right]\right.\right.\\
	&\quad\left.\left.+2\frac{\|u\|^2\X^2}{n^2}\mathbb{E}\left[\left\|y_k^{'}-v\otimes \bar{y}^{'}_k\right\|_\Z^2\right]\right)\right)^{1/2}+c_0\alpha_{k}^2\\
	&\quad +c_0\alpha_{k}\sum_{t=0}^{k}\alpha_{t}\tau_\Z^{k-t}\bigg(\mathbb{E}\left[\|y_t^{'}-v\otimes \bar{y}^{'}_t\|_\mathbf{B}^2\right]\\
	&\quad+\mathbb{E}\left[\|x_t-\mathbf{1}\otimes\bar{x}_t\|_\mathbf{A}^2\right]+\mathbb{E}\left[\|\bar{x}_t-x^*\|^2\right]\bigg).
	\end{aligned}
	\end{equation}

	Note that for  any random vectors $\theta_1$, $\theta_2$ and positive scalar $\tau$,
	\begin{equation}\label{general-form}
	\begin{aligned}
	\mathbb{E}\left[\left|\left\|\theta_1+\theta_2\right\|\right|^2\right]

	\le (1+\tau)\mathbb{E}\left[\|\theta_1\|^2\right]+\left(1+\frac{1}{\tau}\right)\mathbb{E}\left[\|\theta_2\|^2\right],
	\end{aligned}
	\end{equation}
	where $|\|\cdot\||$ could be $\|\cdot\|_\A$ or $\|\cdot\|_\Z$.
	Choosing
	\begin{align*}
	&\theta_1=\left(\tilde{\mathbf{A}}-\frac{\mathbf{1}u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(x_k-\mathbf{1}\otimes\bar{x}_{k}\right),\\
& \theta_2=-\alpha_{k}\left(\mathbf{I}_{nd}-\frac{\mathbf{1}u^\intercal}{n}\otimes\mathbf{I}_{d}\right)y_k,
	\end{align*}
	we have $x_{k+1}-\mathbf{1}\otimes\bar{x}_{k+1}=\theta_1+\theta_2$

	and
	\begin{equation}\label{x-bar}
	\begin{aligned}
	&\mathbb{E}\left[\|x_{k+1}-\mathbf{1}\otimes\bar{x}_{k+1}\|_\A^2\right]\\
	&\le (1+\tau)\mathbb{E}\left[\left\|\left(\tilde{\mathbf{A}}-\frac{\mathbf{1}u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(x_k-\mathbf{1}\otimes\bar{x}_{k}\right)\right\|_\A^2\right]\\
	&\quad+\left(1+\frac{1}{\tau}\right)\mathbb{E}\left[\left\|\alpha_{k}\left(\mathbf{I}_{nd}-\frac{\mathbf{1}u^\intercal}{n}\otimes\mathbf{I}_{d}\right)y_k\right\|_\A^2\right]\\
	&\le (1+\tau)\tau_\A^2\mathbb{E}\left[\left\|x_k-\mathbf{1}\otimes\bar{x}_{k}\right\|_\A^2\right]\\
	&\quad+\left(1+\frac{1}{\tau}\right)\left\|\mathbf{I}_{n}-\frac{\mathbf{1}u^\intercal}{n}\right\|_\A^2\X^2\mathbb{E}\left[\left\|y_k\right\|^2\right]\\
	&\le\frac{1+\tau_\mathbf{A}^2}{2}\mathbb{E}\left[\left\|x_k-\mathbf{1}\otimes\bar{x}_{k}\right\|_\A^2\right]\\
	&\quad+\alpha_{k}^2\frac{1+\tau_\mathbf{A}^2}{1-\tau_\mathbf{A}^2}\left\|\mathbf{I}_{n}-\frac{\mathbf{1}u^\intercal}{n}\right\|_\A^2\X^2\mathbb{E}\left[\left\|y_k\right\|^2\right],
	\end{aligned}
	\end{equation}
	where $\tau_\A$ is defined in (\ref{consensus-para}), the last inequality follows from the setting $\tau=(1-\tau_\A^2)/(2\tau_\A^2)$. For the term $\mathbb{E}\left[\left\|y_k\right\|^2\right]$,
{\small	\begin{equation}\label{y-error}
	\begin{aligned}
	\mathbb{E}\left[\left\|y_k\right\|^2\right]
	&\le 4 \mathbb{E}\left[\left\|y_k^{'}-v\otimes\bar{y}^{'}_k\right\|^2+\left\|v\otimes\left(\bar{y}^{'}_k-\frac{1}{n}\nabla f(\bar{x}_k)\right)\right\|^2\right.\\
	&\quad\left.+\left\|\frac{v}{n}\otimes \nabla f(\bar{x}_k)\right\|^2+\left\|\xi_k\right\|^2\right]\\
	&\le 4 \mathbb{E}\left[\X^2\left\|y_k^{'}-v\otimes\bar{y}^{'}_k\right\|_\Z^2+\frac{(\|v\|L\X)^2}{n}\left\|x_k-\mathbf{1}\otimes\bar{x}_{k}\right\|_\A^2\right.\\
	&\quad\left.+\frac{\|v\|^2L^2}{n}\left\|\bar{x}_k-x^*\right\|^2\right]+4\frac{c_b^2}{(1-\tau_\Z)^2}nU_1,
	\end{aligned}
	\end{equation}	}
	where the second inequality follows from the fact $\bar{y}^{'}_k=\frac{1}{n}\sum_{i=1}^n\nabla f_i(x_{i,k})$ and the Lipschitz continuity of $\nabla f_i(\cdot)$ and Lemma \ref{lem:noi-bound} in Appendix \ref{apd:proof of Lem-rate}. Substitute (\ref{y-error}) into (\ref{x-bar}),
	\begin{equation}\label{ine-1}
	\begin{aligned}
	&\mathbb{E}\left[\|x_{k+1}-\mathbf{1}\otimes\bar{x}_{k+1}\|_\A^2\right]\\
	&\le\frac{1+\tau_\mathbf{A}^2}{2}\mathbb{E}\left[\left\|x_k-\mathbf{1}\otimes \bar{x}_{k}\right\|_\A^2\right]+c_1\alpha_k^2\frac{c_b^2}{(1-\tau_\Z)^2}nU_1\\
	&+c_1\alpha_{k}^2\mathbb{E}\left[\X^2\left\|y_k^{'}-v\otimes\bar{y}^{'}_k\right\|_\Z^2+\frac{\|v\|^2L^2}{n}\X^2\left\|x_k-\mathbf{1}\otimes \bar{x}_{k}\right\|_\A^2\right.\\
	&\quad\left.+\frac{\|v\|^2L^2}{n}\left\|\bar{x}_k-x^*\right\|^2\right],
	\end{aligned}
	\end{equation}
	where constant $c_1=4\frac{1+\tau_\mathbf{A}^2}{1-\tau_\mathbf{A}^2}\left\|\mathbf{I}_{n}-\frac{\mathbf{1}u^\intercal}{n}\right\|_\A^2\X^2$.

	Choosing
	\begin{align*}
	&\theta_1=\left(\tilde{\mathbf{B}}-\frac{v\mathbf{1}^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(y_{k}^{'}-v\otimes \bar{y}_{k}^{'}\right),\\
	&\theta_2=\left(\mathbf{I}_{nd}-\frac{v\mathbf{1}^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(\nabla F_{k+1}-\nabla F_k\right)
	\end{align*}
	in (\ref{general-form}), by the definitions of $y_{k+1}^{'}$ and $\bar{y}_{k+1}^{'}$, we have $y_{k+1}^{'}-\mathbf{1}\otimes\bar{y}_{k+1}^{'}=\theta_1+\theta_2$ and
\begin{equation}\label{consensus-3}
	\begin{aligned}
	&\mathbb{E}\left[\|y_{k+1}^{'}-v\otimes\bar{y}_{k+1}^{'}\|_\Z^2\right]\\
	&\le(1+\tau)\mathbb{E}\left[\left\|\left(\tilde{\mathbf{B}}-\frac{v\mathbf{1}^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(y_{k}^{'}-v\otimes\bar{y}_{k}^{'}\right)\right\|_\Z^2\right]\\
	&\quad+\left(1+\frac{1}{\tau}\right)\mathbb{E}\left[\left\|\left(\mathbf{I}_{nd}-\frac{v\mathbf{1}^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(\nabla F_{k+1}-\nabla F_k\right)\right\|_\Z^2\right]\\
	&\le
	(1+\tau)\tau_\Z^2\mathbb{E}\left[\left\|y_{k}^{'}-v\otimes \bar{y}_{k}^{'}\right\|_\Z^2\right]\\
	&\quad+\left(1+\frac{1}{\tau}\right)\left\|\mathbf{I}_{nd}-\frac{v\mathbf{1}^\intercal}{n}\otimes\mathbf{I}_{d}\right\|_\Z^2\X^2\mathbb{E}\left[\left\|\nabla F_{k+1}-\nabla F_k\right\|^2\right]\\
	&\le
	\frac{1+\tau_\mathbf{B}^2}{2}\mathbb{E}\left[\left\|y_{k}^{'}-v\otimes \bar{y}_{k}^{'}\right\|_\Z^2\right]\\
	&+\frac{1+\tau_\mathbf{B}^2}{1-\tau_\mathbf{B}^2}\left\|\mathbf{I}_{nd}-\frac{v\mathbf{1}^\intercal}{n}\otimes\mathbf{I}_{d}\right\|_\Z^2\X^2\mathbb{E}\left[\left\|\nabla F_{k+1}-\nabla F_k\right\|^2\right],
	\end{aligned}
	\end{equation}
	where $\tau_\Z$ is defined in (\ref{consensus-para-1}), the last inequality follows from the setting $\tau=(1-\tau_\Z^2)/(2\tau_\Z^2)$. Moreover,
	\begin{equation}\label{F-error}
	\begin{aligned}
	&\mathbb{E}\left[\left\|\nabla F_{k+1}-\nabla F_k\right\|^2\right]\\
	&\le L^2\left\|x_{k+1}-x_{k}\right\|^2\\
	&=L^2 \mathbb{E}\left[\left\|\left(\tilde{\mathbf{A}}-\mathbf{I}_{nd}\right)\left(x_k-\mathbf{1}\otimes \bar{x}_k\right)-\alpha_ky_k\right\|^2\right]\\
	&\le 2L^2 \X^2\mathbb{E}\left[\left\|\tilde{\mathbf{A}}-\mathbf{I}_{nd}\right\|^2 \left\|x_k-\mathbf{1}\otimes \bar{x}_k\right\|_\A^2\right]+2L^2\alpha_k^2\mathbb{E}\left[\left\|y_k\right\|^2\right],
	\end{aligned}
	\end{equation}
	where the first inequality follows from the Lipschitz continuity of $\nabla f_i(\cdot)$ and the equality follows from the fact $\left(\tilde{\mathbf{A}}-\mathbf{I}_{nd}\right)(\mathbf{1}\otimes \bar{x}_k)=\mathbf{0}$ by the row stochasticity of $\mathbf{A}$. Substitute (\ref{y-error}) and (\ref{F-error}) into (\ref{consensus-3}),
	{\small\begin{equation}\label{ine-2}
	\begin{aligned}
	&\mathbb{E}\left[\|y_{k+1}^{'}-v\otimes\bar{y}_{k+1}^{'}\|_\Z^2\right]\\
	&\le \frac{1+\tau_\mathbf{B}^2}{2}\mathbb{E}\left[\left\|y_{k}^{'}-v\otimes \bar{y}_{k}^{'}\right\|_\Z^2\right]\\
	&\quad+c_2\left\|\tilde{\mathbf{A}}-\mathbf{I}_{nd}\right\|^2\X^2\mathbb{E}\left[\left\|x_k-\mathbf{1}\otimes \bar{x}_k\right\|_\A^2\right]+\alpha_k^2\frac{4c_2c_b^2nU_1}{(1-\tau_\Z)^2}\\
	&\quad+4c_2\alpha_{k}^2\mathbb{E}\left[\X^2\left\|y_k^{'}-v\otimes\bar{y}^{'}_k\right\|_\Z^2+\frac{\|v\|^2L^2}{n}\X^2\left\|x_k-\mathbf{1}\otimes \bar{x}_{k}\right\|_\A^2\right.\\
	&\quad\left.+\frac{\|v\|^2L^2}{n}\left\|\bar{x}_k-x^*\right\|^2\right],
	\end{aligned}
	\end{equation}}
	where the constant
	\begin{equation}\label{parameter-2}
	c_2=2L^2\frac{1+\tau_\mathbf{B}^2}{1-\tau_\mathbf{B}^2}\left\|\mathbf{I}_{nd}-\frac{v\mathbf{1}^\intercal}{n}\otimes\mathbf{I}_{d}\right\|_\Z^2\X^2.
	\end{equation}

	Multiplying $c_3=\frac{1-\tau_\mathbf{A}^2}{4c_2\left\|\tilde{\mathbf{A}}-\mathbf{I}_{nd}\right\|^2\X^2}$ on both sides of inequality (\ref{ine-2}),
	{\small\begin{equation*}
	\begin{aligned}
	&c_3\mathbb{E}\left[\|y_{k+1}^{'}-v\bar{y}_{k+1}^{'}\|_\Z^2\right]\\
	&\le \frac{1+\tau_\mathbf{B}^2}{2}c_3\mathbb{E}\left[\left\|y_{k}^{'}-v\otimes \bar{y}_{k}^{'}\right\|_\Z^2\right]+\frac{1-\tau_\mathbf{A}^2}{4}\mathbb{E}\left[\left\|x_k-\mathbf{1}\otimes \bar{x}_k\right\|_\A^2\right]\\
	&\quad+4c_3c_2\alpha_k^2\frac{c_b^2}{(1-\tau_\Z)^2}nU_1+4c_3c_2\alpha_{k}^2\mathbb{E}\left[\X^2\left\|y_k^{'}-v\otimes\bar{y}^{'}_k\right\|_\Z^2\right.\\
	&\quad\left.+\frac{\|v\|^2L^2}{n}\X^2\left\|x_k-\mathbf{1}\otimes \bar{x}_{k}\right\|_\A^2+\frac{\|v\|^2L^2}{n}\left\|\bar{x}_k-x^*\right\|^2\right].
	\end{aligned}
	\end{equation*}}
	Substitute above inequality into (\ref{ine-1}), we have
	{\small\begin{equation}\label{ine-3}
	\begin{aligned}
	&\mathbb{E}\left[\|x_{k+1}-\mathbf{1}\otimes\bar{x}_{k+1}\|_\A^2\right]+c_3\mathbb{E}\left[\|y_{k+1}^{'}-v\bar{y}_{k+1}^{'}\|_\Z^2\right]\\
	&\le \frac{1+\tau_\mathbf{B}^2}{2}c_3\mathbb{E}\left[\left\|y_{k}^{'}-v\otimes \bar{y}_{k}^{'}\right\|_\Z^2\right]+\frac{3+\tau_\mathbf{A}^2}{4}\mathbb{E}\left[\left\|x_k-\mathbf{1}\otimes \bar{x}_k\right\|_\A^2\right]\\
	&\quad+\frac{(4c_3c_2+c_1)c_b^2nU_1}{(1-\tau_\Z)^2}\alpha_k^2+(4c_3c_2+c_1)\mathbb{E}\left[\X^2\left\|y_k^{'}-v\otimes\bar{y}^{'}_k\right\|_\Z^2\right.\\
	&\quad\left.+\frac{\|v\|^2L^2}{n}\X^2\left\|x_k-\mathbf{1}\otimes \bar{x}_{k}\right\|_\A^2+\frac{\|v\|^2L^2}{n}\left\|\bar{x}_k-x^*\right\|^2\right].
	\end{aligned}
	\end{equation}}
	Denote
	\begin{align*}
&	u_k=\mathbb{E}\left[\|x_{k}-\mathbf{1}\otimes\bar{x}_{k}\|^4\right]+c_3\mathbb{E}\left[\|y_{k}^{'}-v\otimes\bar{y}_{k}^{'}\|^2\right],\\
& v_k=\mathbb{E}\left[\|\bar{x}_{k}-x^*\|^2\right],\\
& \rho_k=\max\left\{\frac{1+\tau_\mathbf{B}^2}{2},~\frac{3+\tau_\mathbf{A}^2}{4}\right\},
	\end{align*}
	$\gamma_{k}=\alpha_{k}$ and $\rho=\tau_\Z$. Then by inequalities (\ref{ine-2}) and (\ref{ine-3}),
	\begin{align}
	\label{form-3}u_{k+1}&\le\rho_ku_k+M\gamma_{k}\sqrt{u_k(1+u_k+v_k)}\\
	&\quad+M\gamma_{k}\left(\gamma_{k}+\sum_{t=0}^k\gamma_{t}\rho^{k-t}u_t+\sum_{t=0}^k\gamma_{t}\rho^{k-t}v_t\right),\notag\\
	\label{form-4}v_{k+1}&\le v_k+Mu_k+M\gamma_{k}\sqrt{u_k(1+u_k+v_k)}\\
	&+M\gamma_{k}\left(\gamma_{k}+\sum_{t=0}^k\gamma_{t}\rho^{k-t}u_t+\sum_{t=0}^k\gamma_{t}\rho^{k-t}v_t\right),\notag
	\end{align}
where
\begin{align*}
&M=\max\left\{2\sqrt{2}\frac{u^\intercal vL\X}{n^{1.5}},~2\sqrt{2}\frac{\|u\|\X}{n\sqrt{c_3}},~(4c_3c_2+c_1)\frac{\|v\|^2L^2}{n^2}c_0,\right.\\&\quad\quad\quad\quad\quad\left.c_0,~(4c_3c_2+c_1)\X^2,~\frac{(4c_3c_2+c_1)c_b^2nU_1}{(1-\tau_\Z)^2}\right\}.
\end{align*}
Obviously, (\ref{form-3}) and (\ref{form-4}) fall into the forms of (\ref{form-1}) and (\ref{form-2}).

	\textbf{Step 2.}
	By the facts  $\rho_k=\max\left\{\frac{1+\tau_\mathbf{B}^2}{2},~\frac{3+\tau_\mathbf{A}^2}{4}\right\}$ and $\gamma_k=\alpha_{k}$, there exists a positive integer $k_0$ such that $\gamma_{k}$,  $\rho_{k}$ are $[0,1]$-valued when $k\ge k_0$ (without loss of generality, suppose $k_0=0$). Then condition (i) of Lemma \ref{lem:ieq-1} holds.

	Let  $\phi_k=1/\alpha_{k}^2$. Obviously,
	\begin{align*}
	&\limsup_{k\rightarrow\infty}\left(\gamma_{k}\sqrt{\phi_k}+\frac{\phi_{k-1}}{\phi_k}\right)<\infty,\quad \sum_{k=0}^\infty\phi_k^{-1}<\infty,\\
	&\liminf_{k\rightarrow\infty}\left(\gamma_{k}\sqrt{\phi_k}\right)^{-1}\left(\frac{\phi_{k-1}}{\phi_k}-\rho_k\right)>0,
	\end{align*}
	which implies the condition (ii) of Lemma \ref{lem:ieq-1}. Then by Lemma \ref{lem:ieq-1},   $\sup_{k}\mathbb{E}\left[\|\bar{x}_{k+1}-x^*\|^2\right]<\infty$ and
	\begin{equation*}
	\sup_{k} \frac{1}{\alpha_{k}^2}\left(\mathbb{E}\left[\|x_{k}-\mathbf{1}\otimes\bar{x}_{k}\|^2\right]+c_3\mathbb{E}\left[\|y_{k}^{'}-\mathbf{1}\otimes\bar{y}_{k}^{'}\|^2\right]\right)<\infty.
	\end{equation*}
	The proof is complete.
\end{proof}

The proof of Lemma \ref{lem:rate} follows the steps in the proof of \cite[Lemma 5]{bianchi2013performance}, where the tough work is how to handle the non-martingale-difference sequence $\{\xi_k\}$ in (\ref{consensus-new-form}).  In \cite[Lemma 5]{bianchi2013performance}, $y_k'$  and $\xi_k$ in  (\ref{consensus-new-form}) represent the gradient and  stochastic gradient noise respectively and then $\{\xi_k\}$ is martingale difference sequence. On the other hand, for $\mathcal{S}$-$\mathcal{AB}$,   $\xi_k$ is  the accumulated stochastic gradient noise, which depends on all the past stochastic gradient noise and can not form martingale difference sequence.
\begin{thm}\label{thm:rate}
Suppose that Assumptions \ref{ass:objective} and \ref{ass:matrix} hold. Let step-size $\alpha_{k}=a/(k+b)^\alpha$, where $\alpha\in (1/2,1)$, positive scalars $a,b$ satisfy  $\frac{a}{b^\alpha}\le \frac{2n}{u^\intercal vL_0}$, $L_0\define\max_{1\leq i \leq n}\mathbb{E}[L_{i}(\zeta_i)]$. Then
\begin{equation}\label{rate-1}
\mathbb{E}\left[\|\bar{x}_k-x^*\|^2\right]=\mathcal{O}\left(\alpha_k\right).
\end{equation}
Moreover, if $\alpha_{k}=a/(k+b)$ and positive scalars $a,b$ satisfy  $\frac{n^2}{u^\intercal v\mu }<a\le \frac{2n}{u^\intercal vL_0}b$, $b\ge \frac{nL_0}{2\mu}$,
$$\mathbb{E}\left[\|\bar{x}_k-x^*\|^2\right]=\mathcal{O}\left(\frac{1}{k}\right).$$
\end{thm}
\begin{proof}
Recall formula (\ref{eq-auxi-3}) in the proof of Lemma \ref{lem:rate},
{\small\begin{equation}\label{ieq-0}
	\begin{aligned}
	&\mathbb{E}\left[\|\bar{x}_{k+1}-x^*\|^2\right]\\
	&\le\left(1-\frac{u^\intercal v\mu\alpha_{k}}{n^2}\right)^2\mathbb{E}\left[\left\|\bar{x}_{k}-x^*\right\|^2\right]\\
	&\quad+3\alpha_{k}^2\mathbb{E}\left[\left(\frac{u^\intercal v}{n}\right)^2\frac{L^2\X^2}{n}\left\|x_k-\mathbf{1}\otimes\bar{x}_k\right\|_\mathbf{A}^2\right]\\
	&\quad+3\alpha_{k}^2\mathbb{E}\left[\frac{\|u\|^2\X^2}{n^2}\left\|y_k^{'}-v\otimes\bar{y}^{'}_k\right\|_\mathbf{B}^2\right]+3\alpha_{k}^2\frac{\|u\|^2c_b^2U_1}{n(1-\tau_\Z)^2}\\
	&\quad+2\mathbb{E}\left[\left\langle \bar{x}_{k}-x^*, \alpha_{k}\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(y_k^{'}-\frac{v}{n}\otimes \nabla f(\bar{x}_k)+\xi_k\right)\right\rangle\right]\\
	&\quad+\frac{u^\intercal v\alpha_{k}^2}{n}\mathbb{E}\left[\frac{L^2}{n}\|\bar{x}_k-x^*\|^2+\left(\frac{u^\intercal v}{n}\right)^2\frac{L^2\X^2}{n}\left\|x_k-\mathbf{1}\otimes\bar{x}_k\right\|_\mathbf{A}^2\right.\\
	&\quad\left.+\frac{\|u\|^2\X^2}{n^2}\left\|y_k^{'}-v\otimes\bar{y}^{'}_k\right\|_\mathbf{B}^2\right]+\frac{u^\intercal v\|u\|^2c_b^2U_1}{n^2(1-\tau_\Z)^2}\alpha_{k}^2,
	\end{aligned}
	\end{equation}}
where  $\bar{y}^{'}_k\define1/n\sum_{i=1}^ny_{i,k}^{'},~c_b=\max\left\{\X^2,\frac{\left\|\mathbf{B}-\mathbf{I}_{n}\right\|}{\tau_\Z}\X^2\right\}$, $\X$ is defined in (\ref{norm-bound-1}), the inequality follows from the fact $\bar{y}^{'}_k=\frac{1}{n}\sum_{i=1}^n\nabla f_i(x_{i,k})$ and the Lipschitz continuity of $\nabla f_i(\cdot)$. For the fifth term on the right hand side of (\ref{ieq-0}),
{\small\begin{equation}\label{ieq-1}
	\begin{aligned}
	&2\mathbb{E}\left[\left\langle \bar{x}_{k}-x^*, \alpha_{k}\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(y_k^{'}-\frac{v}{n}\otimes \nabla f(\bar{x}_k)+\xi_k\right)\right\rangle\right]\\
	&\le \left(\frac{u^\intercal vL}{n^{1.5}}+\frac{\|u\|}{n}\right)\tau \alpha_k\mathbb{E}\left[\left\|\bar{x}_{k}-x^*\right\|^2\right]\\
	&\quad+\frac{u^\intercal vL\alpha_k}{n^{1.5}\tau}\mathbb{E}\left[\left\|x_k-\mathbf{1}\otimes\bar{x}_k\right\|^2\right]+\frac{\|u\|\alpha_k}{n\tau}\mathbb{E}\left[\left\|y_k^{'}-v\otimes \bar{y}^{'}_k\right\|^2\right]\\
	&\quad+ \frac{2\|u\|c_b\alpha_{k}}{n(1-\tau_\Z)}\sum_{t=0}^{k-1}\alpha_{t}\tau_\Z^{k-t}\left(\mathbb{E}\left[\X^2\|y_t^{'}-v\otimes \bar{y}^{'}_t\|_\mathbf{B}^2\right]\right.\\
	&\quad\left.+\frac{\|v\|^2L^2\X^2}{n}\mathbb{E}\left[\|x_t-\mathbf{1}\otimes\bar{x}_t\|_\mathbf{A}^2\right]+\frac{\|v\|^2L^2}{n^2}\mathbb{E}\left[\|\bar{x}_t-x^*\|^2\right]\right)\\
	&\quad+\frac{u^\intercal v\|u\|^2c_b^2U_1}{n^2(1-\tau_\Z)^2}\alpha_{k}^2,
	\end{aligned}
	\end{equation}}
where $\tau$ could be any positive number, the inequality follows from the facts $\bar{y}^{'}_k=\frac{1}{n}\sum_{i=1}^n\nabla f_i(x_{i,k})$, $2ab\le\tau a^2+b^2/\tau$, the Lipschitz continuity of $\nabla f_i(\cdot)$ and (\ref{2-momon-2}) in Appendix \ref{apd:proof of Lem-rate}. Substitute (\ref{ieq-1}) into (\ref{ieq-0}), then by Lemma \ref{lem:rate} and Lemma \ref{lem:weighted-seq},
{\small\begin{equation*}
	\begin{aligned}
	&\mathbb{E}\left[\|\bar{x}_{k+1}-x^*\|^2\right]\\
	&\le\left[\left(1-\frac{u^\intercal v\mu\alpha_{k}}{n^2}\right)^2+\left(\frac{u^\intercal vL}{n^{1.5}}+\frac{\|u\|}{n}\right)\tau \alpha_k\right]\mathbb{E}\left[\left\|\bar{x}_{k}-x^*\right\|^2\right]\\
	&\quad+\mathcal{O}\left(\frac{L^2}{n}\frac{\alpha_k^3}{\tau}\right)+\mathcal{O}\left(\frac{L^2}{n(1-\tau_\Z)^2}\alpha_k^2\right)\\
	&=\left(1-\frac{u^\intercal v\mu\alpha_{k}}{n^2}+\left(\frac{u^\intercal v\mu\alpha_{k}}{n^2}\right)^2\right)\mathbb{E}\left[\left\|\bar{x}_{k}-x^*\right\|^2\right]\\
	&\quad+\mathcal{O}\left(\alpha_k^3\right)+\mathcal{O}\left(\frac{L^2}{n(1-\tau_\Z)^2}\alpha_k^2\right)\\
	&\le \left(1-\frac{u^\intercal v\mu\alpha_{k}}{n^2}\right)\mathbb{E}\left[\left\|\bar{x}_{k}-x^*\right\|^2\right]\\
	&\quad+\left(\frac{u^\intercal v\mu}{n^2}\right)^2U_2\alpha_{k}^2+\mathcal{O}\left(\alpha_k^3\right)+\mathcal{O}\left(\frac{L^2}{n(1-\tau_\Z)^2}\alpha_k^2\right),
	\end{aligned}
	\end{equation*}}
where the equality follows from the setting $\tau=\left(\frac{u^\intercal v\mu}{n^2}\right)/\left(\frac{u^\intercal vL}{n^{1.5}}+\frac{\|u\|}{n}\right)$. Then by  \cite[Lemma 5 in Chapter 2]{polyak1987Introduction}, we have
\begin{equation*}
\mathbb{E}\left[\|\bar{x}_{k+1}-x^*\|^2\right]= \mathcal{O}\left(\alpha_k\right).
\end{equation*}
Note that $\lim_{k\rightarrow\infty} \frac{\alpha_{k-1}}{\alpha_{k}}=1$, then (\ref{rate-1}) holds.
If $\alpha_{k}=a/(k+b)$ and positive scalars $a,b$ satisfy that $\frac{n^2}{u^\intercal v\mu }<a\le \frac{2n}{u^\intercal vL_0}b$, $b\ge \frac{nL_0}{2\mu}$, we have $\frac{u^\intercal v\mu a}{n^2}>1$ and then \cite[Lemma 4 in Chapter 2]{polyak1987Introduction} implies
$$\mathbb{E}\left[\|\bar{x}_k-x^*\|^2\right]=\mathcal{O}\left(\frac{1}{k}\right).$$
The proof is complete.
\end{proof}

Combining  with Lemma \ref{lem:rate}, Theorem \ref{thm:rate}  implies that $\mathbb{E}\left[\|x_{i,k}-x^*\|^2\right]=\mathcal{O}\left(\frac{1}{k}\right)$ for any $i\in \mathcal{V}$, which is the optimal rate of  stochastic gradient method \cite{Rakhlin2012making}.
Complement to \cite[Theorem 2]{pu2020distributed}, Theorem \ref{thm:rate} focuses on the undoubly stochastic weight matrices and directed graphs.

In what follows, we show the asymptotic normality of $\mathcal{S}$-$\mathcal{AB}$ algorithm. Some necessary technical results of proof are provided in Appendix \ref{apd:thm-asym-norm}.
\begin{thm}\label{thm:asym norm}
	Suppose that Assumptions \ref{ass:objective} and \ref{ass:matrix} hold. Let step-size $\alpha_{k}=a/(k+b)^\alpha$, where $\alpha\in (1/2,1)$, positive scalars $a,b$ satisfy  $\frac{a}{b^\alpha}\le \frac{2n}{u^\intercal vL_0}$, $L_0\define\max_{1\leq i \leq n}\mathbb{E}[L_{i}(\zeta_i)]$. Then for any $i\in\mathcal{V}$, $x_{i,k}$ satisfies
	\begin{equation}\label{limit distribution}
	\frac{1}{\sqrt{k}}\sum_{t=0}^{k-1}\left(x_{i,t}-x^*\right)\stackrel{d}{\longrightarrow} N\left(\mathbf{0},\mathbf{H}^{-1}\mathbf{S}\mathbf{H}^{-1}\right),
	\end{equation}
	where $\mathbf{H}\define \nabla^2 f(x^*)$ and  $\mathbf{S}\define\Cov(\sum_{j=1}^n\nabla g_i(x^*,\zeta_{j}))$.
\end{thm}
\begin{proof}
	By Lemma \ref{lem:rate},
	\begin{equation*}
	\begin{aligned}
	&\mathbb{E}\left[\left\|\frac{1}{\sqrt{k}}\sum_{t=0}^{k-1}\left(\bar{x}_{t}-x^*\right)-\frac{1}{\sqrt{k}}\sum_{t=0}^{k-1}\left(x_{i,t}-x^*\right)\right\|\right]\\
	&\le \frac{1}{\sqrt{k}}\sum_{t=0}^{k-1}\sqrt{\mathbb{E}\left[\|x_{t}-\mathbf{1}\otimes\bar{x}_{t}\|^2\right]}\le\frac{\sqrt{U_2}}{\sqrt{k}}\sum_{t=0}^{k-1}\alpha_t\rightarrow 0.
	\end{aligned}
	\end{equation*}
	Then  Slutsky's theorem implies (\ref{limit distribution}) if
	\begin{equation}\label{res-asym-norm}
	\frac{1}{\sqrt{k}}\sum_{t=0}^{k-1}\left(\bar{x}_{t}-x^*\right)\stackrel{d}{\rightarrow} N\left(\mathbf{0},\mathbf{H}^{-1}\mathbf{S}\mathbf{H}^{-1}\right)
	\end{equation}
	holds. In what follows, we show (\ref{res-asym-norm}) by Lemma \ref{lem:asym-norm} in Appendix \ref{apd:thm-asym-norm}.

	Firstly, we rewrite the recursion $\bar{x}_k-x^*$ in the form of (\ref{norm form}) in Lemma \ref{lem:asym-norm}. By the equality (\ref{consensus-new-form}),
	\begin{equation}\label{consensus-new-form-1}
	\begin{aligned}
	&\bar{x}_{k+1}-x^*\\
	&=\bar{x}_{k}-x^*-\tilde{\alpha}_k\frac{1}{n}\nabla f(\bar{x}_k)-\tilde{\alpha}_k\left(\bar{y}^{'}_k-\frac{1}{n}\nabla f(\bar{x}_k)\right)\\
	&\quad-\alpha_k\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(y_k^{'}-v\otimes\bar{y}^{'}_k\right)-\alpha_{k}\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\xi_k,
	\end{aligned}
	\end{equation}
	where $\tilde{\alpha}_k\define\frac{u^\intercal v}{n}\alpha_k$.	Denote
	\begin{equation*}\label{sme def-1}
	\Delta_{k}=\bar{x}_k-x^*,\quad h(x)=-\frac{1}{n}\nabla f(x+x^*), \quad V(x)=\frac{1}{2}\|x\|^2
	\end{equation*}
	and
	\begin{align}
	\label{some def-2}&\eta_k=-\left(\bar{y}^{'}_k-\frac{1}{n}\nabla f(\bar{x}_k)\right)-\frac{n}{u^\intercal v}\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(y_k^{'}-v\otimes\bar{y}^{'}_k\right),\\
	&\mu_k=-\frac{n}{u^\intercal v}\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\xi_k.\notag
	\end{align}
	 Then the linear recursion (\ref{consensus-new-form-1}) can be rewritten as
	\begin{equation}\label{consensus-linear}
	\Delta_{k+1}=\Delta_{k}+\tilde{\alpha}_kh(\Delta_{k})+\tilde{\alpha}_k\left(\eta_k+\mu_k\right),
	\end{equation}
	which is in the form of (\ref{norm form}) in Lemma \ref{lem:asym-norm}.

	Next, we verify the conditions $(C0)-(C3)$ of Lemma \ref{lem:asym-norm}. By the definition of $\alpha_{k}$, (C0) holds. By the strong convexity of $f(x)$,
	\begin{align*}
	h(x)^\intercal \nabla V(x)&=\left\langle -\frac{1}{n}\nabla f(x+x^*), (x+x^*)-x^* \right\rangle\\
	&< \frac{1}{n}(f(x^*)-f(x+x^*))<0,\quad\forall x\neq 0,
	\end{align*}
	which implies (C1) of Lemma \ref{lem:asym-norm}.

	Set  $\mathbf{G}=-\frac{1}{n}\nabla^2f(x^*)$. Note that $\nabla^2f(x^*)$ is positive definite as  $f(x)$ is strongly convex and then $\mathbf{G}$ is stable. Subsequently, combining with Assumption \ref{ass:objective} (ii), (C2) of Lemma \ref{lem:asym-norm} holds.

	We are left to verify (C3) of Lemma \ref{lem:asym-norm}. We first verify (\ref{apd:cond-1}) of (C3). By the definition (\ref{auxi-seq}) of $\xi_{k}$,
	\begin{equation*}
	\xi_{k}=\tilde{\mathbf{B}}\xi_{k-1}+\epsilon_{k}-\epsilon_{k-1}=\sum_{t=0}^{k-1}\tilde{\mathbf{B}}^{k-1-t}\left(\tilde{\mathbf{B}}-\mathbf{I}_{nd}\right)\epsilon_{t}+\epsilon_{k},
	\end{equation*}
	and then
	\begin{equation*}
	\sum_{t=0}^{k-1}\mu_{t}=-\sum_{t=0}^{k-1}\frac{n}{u^\intercal v}\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\xi_t=-\sum_{t=0}^{k-1}\mathbf{D}(k-1,t)\epsilon_t,
	\end{equation*}
	where $\mathbf{D}(k-1,t)=\frac{n}{u^\intercal v}\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\tilde{\mathbf{B}}^{k-1-t},~\mathbf{D}(k-1,k-1)=\mathbf{I}_{nd}$.
	Note that $\{\epsilon_k\}$ is a martingale difference sequence and by Assumption \ref{ass:objective} (iii),
	\begin{equation*}
	\sup_k \mathbb{E}\left[\|\epsilon_k\|^2|\mathcal{F}_{k}\right]\le nU_1\quad \text{a.s.}.
	\end{equation*}
	Then by \cite[Lemma B.6.1]{chen2006stochastic}, $\sum_{t=0}^{k-1}\left(\frac{\mathbf{1}^\intercal}{n}\otimes \mathbf{I}_d\right)\epsilon_t=\mathcal{O}(\sqrt{k})$, which implies
	\begin{equation}\label{as-conv-0}
	\tilde{\alpha}_{k-1}\sum_{t=0}^{k-1}\left(\frac{\mathbf{1}^\intercal}{n}\otimes \mathbf{I}_d\right)\epsilon_t=\mathcal{O}(\sqrt{k}\tilde{\al}_{k-1})=\mathcal{O}(k^{1/2-\al})\quad \text{a.s.},
	\end{equation}
	where $1/2-\al<0$.
	Note also that
	\begin{equation}\label{noi-consensus}
	\begin{aligned}
	&\mathbb{E}\left[\left\|\tilde{\alpha}_{k-1}\sum_{t=0}^{k-1}\mathbf{D}_{k-1,t}\epsilon_t-\tilde{\alpha}_{k-1}\sum_{t=0}^{k-1}\left(\frac{\mathbf{1}^\intercal}{n}\otimes \mathbf{I}_d\right)\epsilon_t\right\|^2\right]\\
	&=\tilde{\alpha}_{k-1}^2\sum_{t=0}^{k-1}\mathbb{E}\left[\left\|\left[\mathbf{D}_{k-1,t}-\left(\frac{\mathbf{1}^\intercal}{n}\otimes \mathbf{I}_d\right)\right]\epsilon_t\right\|^2\right]\\
	&\le \tilde{\alpha}_{k-1}^2\sum_{t=0}^{k-1}\left\|\mathbf{D}_{k-1,t}-\left(\frac{\mathbf{1}^\intercal}{n}\otimes \mathbf{I}_d\right)\right\|^2\mathbb{E}\left[\left\|\epsilon_t\right\|^2\right]\\
	&= \tilde{\alpha}_{k-1}^2\sum_{t=0}^{k-1}\left\|\frac{n}{u^\intercal v}\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\left(\tilde{\mathbf{B}}^{k-1-t}-\frac{v\mathbf{1}^\intercal}{n}\otimes \mathbf{I}_d\right)\right\|^2\mathbb{E}\left[\left\|\epsilon_t\right\|^2\right]\\
	&\le  \tilde{\alpha}_{k-1}^2\sum_{t=0}^{k-1}\left(\frac{\|u\|\X}{u^\intercal v}\right)^2 \tau_\Z^{2(k-1-t)}\mathbb{E}\left[\left\|\epsilon_t\right\|^2\right]\\
	&\le \left(\frac{\|u\|\X}{u^\intercal v}\right)^2\frac{n}{1-\tau_\Z^2}U_1\tilde{\alpha}_{k-1}^2,
	\end{aligned}
	\end{equation}
	where $\tau_\Z<1$ is defined in (\ref{consensus-para-1}), the first equality follows from that $\mathbb{E}\left[\epsilon_{t_1}^\intercal\epsilon_{t_2}\right]=\mathbb{E}\left[\mathbb{E}\left[\epsilon_{t_1}^\intercal\epsilon_{t_2}|\mathcal{F}_{t_1+1}\right]\right]=0$ for any $t_1<t_2$, $	\mathcal{F}_0 =\sigma\{x_{i,0},i\in \mathcal{V}\},$
	\begin{equation*}
	\mathcal{F}_k=\sigma\{x_{i,0},\epsilon_{i,t}:i\in \mathcal{V}, 0\leq t\leq k-1\}~(k>0).
	\end{equation*}
	Then
	\begin{equation}\label{MDS -sum}
	\begin{aligned}
	&\sum_{k=1}^\infty\mathbb{E}\left[\left\|\tilde{\alpha}_{k-1}\sum_{t=0}^{k-1}\mathbf{D}_{k-1,t}\epsilon_t-\tilde{\alpha}_{k-1}\sum_{t=0}^{k-1}\left(\frac{\mathbf{1}^\intercal}{n}\otimes \mathbf{I}_d\right)\epsilon_t\right\|^2\right]\\
	&\le \sum_{k=0}^\infty \left(\frac{\|u\|\X}{u^\intercal v}\right)^2\frac{1}{1-\tau_\Z^2}U_1\tilde{\alpha}_{k-1}^2<\infty.
	\end{aligned}
	\end{equation}
	By (\ref{MDS -sum}) and the monotone convergence theorem,
	\begin{equation}\label{as-conv-1}
	\tilde{\alpha}_{k-1}\sum_{t=0}^{k-1}\mathbf{D}_{k-1,t}\epsilon_t-\tilde{\alpha}_{k-1}\sum_{t=0}^{k-1}\left(\frac{\mathbf{1}^\intercal}{n}\otimes \mathbf{I}_d\right)\epsilon_t\rightarrow 0\quad \text{a.s.}.
	\end{equation}
	Combine (\ref{as-conv-0}) and (\ref{as-conv-1}),
	\begin{equation}\label{veri-C2-1-1}
	\begin{aligned}
	&\tilde{\alpha}_{k-1}\sum_{t=0}^{k-1}\mu_{t}\\
	&=\left(\tilde{\alpha}_{k-1}\sum_{t=0}^{k-1}\mu_{t}+\tilde{\alpha}_{k-1}\sum_{t=0}^{k-1}\left(\frac{\mathbf{1}^\intercal}{n}\otimes \mathbf{I}_d\right)\epsilon_t\right)\\
	&\quad-\tilde{\alpha}_{k-1}\sum_{t=0}^{k-1}\left(\frac{\mathbf{1}^\intercal}{n}\otimes \mathbf{I}_d\right)\epsilon_t\\
	&=-\left(\tilde{\alpha}_{k-1}\sum_{t=0}^{k-1}\mathbf{D}_{k-1,t}\epsilon_t-\tilde{\alpha}_{k-1}\sum_{t=0}^{k-1}\left(\frac{\mathbf{1}^\intercal}{n}\otimes \mathbf{I}_d\right)\epsilon_t\right)\\
	&\quad-\tilde{\alpha}_{k-1}\sum_{t=0}^{k-1}\left(\frac{\mathbf{1}^\intercal}{n}\otimes \mathbf{I}_d\right)\epsilon_t
	\rightarrow 0 \quad \text{a.s.}.
	\end{aligned}
	\end{equation}
	By Lemma \ref{lem:noi-asym-norm} in Appendix \ref{apd:thm-asym-norm}, we have
	\begin{equation}\label{veri-C2-1-2}
	\frac{1}{\sqrt{k}}\sum_{t=0}^{k-1} \mu_t\stackrel{d}{\rightarrow} N\left(\mathbf{0},\frac{1}{n^2}\mathbf{S}\right).
	\end{equation}
	Subsequently, (\ref{veri-C2-1-1}) and (\ref{veri-C2-1-2}) verify (\ref{apd:cond-1}) of (C3).

	Note that
	\begin{equation}\label{mu-2-monm}
	\begin{aligned}
	\mathbb{E}\left[\|\mu_k\|^2\right]&=\mathbb{E}\left[\left\|\frac{n}{u^\intercal v}\left(\frac{u^\intercal}{n}\otimes\mathbf{I}_{d}\right)\xi_k\right\|^2\right]\\
	&\le \left(\frac{\|u\|}{u^\intercal v}\right)^2\mathbb{E}\left[\left\|\xi_k\right\|^2\right]\le \left(\frac{\|u\|}{u^\intercal v}\right)^2 \frac{c_b^2}{(1-\tau_\Z)^2}nU_1,
	\end{aligned}
	\end{equation}
	where $c_b=\max\left\{\X^2,\frac{\left\|\mathbf{B}-\mathbf{I}_{n}\right\|}{\tau_\Z}\X^2\right\}$, $\X$ is defined in (\ref{norm-bound-1}), the second inequality follows from the boundedness of $\mathbb{E}\left[\left\|\xi_k\right\|^2\right]$ by Lemma \ref{lem:noi-bound} in Appendix \ref{apd:proof of Lem-rate}. Then $\sum_{s\ge -t}^{\infty}\left\|\mathbb{E}\left[ \mu_t\mu^\intercal_{s+t}\right]\right\|\le c$ (by Lemma \ref{lem:noi-asym-norm} in  Appendix \ref{apd:thm-asym-norm}) and (\ref{mu-2-monm}) imply (\ref{apd:cond-2}) of condition (C3).

	By the definition of $\eta_k$ in (\ref{some def-2}),
	\begin{equation*}
	\begin{aligned}
	\mathbb{E}\left[\|\eta_k\|^2\right]
	&\le \frac{2L^2}{n}\mathbb{E}\left[\|x_{k}-\mathbf{1}\otimes\bar{x}_{k}\|^2\right]\\
	&\quad+2\left(\frac{\|u\|}{u^\intercal v}\right)^2 \mathbb{E}\left[\left\|y_k^{'}-v\otimes\bar{y}_k^{'}
	\right\|^2\right]\\
	&\le\left(\frac{2L^2}{n}c+2\left(\frac{\|u\|}{u^\intercal v}\right)^2c\right)\alpha_{k}^2,
	\end{aligned}
	\end{equation*}
	where the first inequality follows from the Lipschitz continuity of $\nabla f_j(x)$, the last inequality follows from Lemma \ref{lem:rate}. Then (\ref{apd:cond-3}) of (C3) holds.

	Summarizing above results, by Lemma \ref{lem:asym-norm} in Appendix \ref{apd:thm-asym-norm},
	\begin{equation*}
	\frac{1}{\sqrt{k}}\sum_{t=0}^{k-1}\left(\bar{x}_{k}-x^*\right)\stackrel{d}{\longrightarrow} N\left(\mathbf{0},\mathbf{H}^{-1}\mathbf{S}\mathbf{H}^{-1}\right).
	\end{equation*}
	The proof is complete.
\end{proof}

Theorem \ref{thm:asym norm}
shows the asymptotic normality  of
Polyak-Ruppert averaged  $\mathcal{S}$-$\mathcal{AB}$.
One of the applications of asymptotic normality is to do statical inference, such as, constructing the confidence regions of the true solution. For doing so, we need    to estimate the covariance matrix $\mathbf{H}^{-1}\mathbf{S}\mathbf{H}^{-1}$  first.
Motivated by the online plug-in method in the seminal work \cite{chen2020statistical}, we propose the following distributed plug-in method.

\begin{algorithm}[H]
	\caption{Distributed plug-in method.}\label{alg:dis plug-in-1}
	\textbf{Initialization:} For $i\in\mathcal{V}$, set vector $u_{i,0}=[0,\cdots,\underbrace{1}_{i-th},\cdots,0]^\intercal$, matrices $\mathbf{H}_{i,0}=\mathbf{S}_{i,0}=\mathbf{0}\in\mathbb{R}^{d\times d}$.\\
	\textbf{General step:} For any $k=1,2,\cdots$,
	\begin{itemize}
		\item[1:]Update $u_{i,k}$ by
		$u_{i,k}=\sum_{j=1}^na_{ij}u_{j,k-1}$.
		\item[2:]Update $\mathbf{S}_{i,k},~\mathbf{H}_{i,k}$ by
		{\small\begin{align}
		&\mathbf{S}_{i,k}=\dfrac{k}{k+1}\sum_{j=1}^n a_{ij} \mathbf{S}_{j,k-1}^{(1)}\notag\\
		\label{S}&\quad+\frac{\nabla g_{i,k}\left(\nabla g_{i,k}-\nabla g_{i,k-1}\right)^\intercal+\left(\nabla g_{i,k}-\nabla g_{i,k-1}\right)\nabla g_{i,k}^\intercal}{2(k+1)u_{i,k}(i)},\\
		\label{est-H}&\mathbf{H}_{i,k}=\dfrac{k}{k+1}\sum_{j=1}^n a_{ij} \mathbf{H}_{j,k-1}+\frac{\nabla^2 g_{i,k}}{(k+1)u_{i,k}(i)}
		\end{align}}
		respectively, where $u_{i,k}(i)$ is the i-th component of $u_{i,k}$, $\nabla g_{i,k}=\nabla g_i(x_{i,k};\zeta_{i,k})$, $\nabla^2 g_{i,k}=\nabla^2 g_i(x_{i,k};\zeta_{i,k})$.
		\item[]\textbf{End for}.
		\item[]\textbf{Output}: $\mathbf{H}_{i,k},~\mathbf{S}_{i,k}$.
	\end{itemize}
\end{algorithm}
$\mathbf{S}_{i,k}$ and $\mathbf{H}_{i,k}$   in (\ref{S})-(\ref{est-H}) are the estimators of $\mathbf{S}$ and $\mathbf{H}$ respectively and then $\mathbf{H}_{i,k}^{-1}\mathbf{S}_{i,k}\mathbf{H}_{i,k}^{-1}$ is the estimator of $\mathbf{H}^{-1}\mathbf{S}\mathbf{H}^{-1}$.
In  Step 1, we estimate the nonnegative left eigenvector $u/n$ of weight matrix $\mathbf{A}$ by vector $[u_{1,k}(1),u_{2,k}(2),$ $\cdots,u_{n,k}(n)]^\intercal$, which will be used to  eliminate the error induced by weight matrix $\mathbf{A}$.
In Step 2, the rescaling
technology  \cite[Algorithm 1]{MAI201994} is used to eliminate the error, that is, dividing $u_{i,k}(i)$ in the second term of right hand side of equalities (\ref{S})-(\ref{est-H}).
Moreover,  the first terms on the right hand of (\ref{S}) and (\ref{est-H}) are the aggregate steps to collect global information, while the second terms aim at estimating the covariance matrix and Hessian matrix with local information.
\begin{thm}\label{thm:PI-rate}
	Under the conditions of Theorem \ref{thm:asym norm} and that for any $i\in\mathcal{V}$ (a) graph $\mathcal{G}_A$ is strong connected, $\mathbf{A}$ is row stochastic and $a_{ii}>0$, (b) $\Cov\left(\sum_{j=1}^n\nabla g_j(x^*;\zeta_{j})\right)$ has the separable structure, that is,
	\begin{equation*}
	\Cov\left(\sum_{j=1}^n\nabla g_j(x^*;\zeta_{j})\right)=\sum_{j=1}^n\Cov\left(\nabla g_j(x^*;\zeta_{j})\right),
	\end{equation*} (c) there exist positive scalars $L_1$ and $L_2$ such that
	\begin{equation*}
	\begin{aligned}
	&\mathbb{E}\left[\left\|\nabla^2 g_i(x;\zeta_{i})-\nabla^2 g_i(x^*;\zeta_{i})\right\|\right] \le L_1\|x-x^*\|,\\
	&\mathbb{E}[\left\|\nabla^2 g_i(x^*;\zeta_{i})\right\|^2]\le L_2.
	\end{aligned}
	\end{equation*}
	Then for any $i\in\mathcal{V}$, $\mathbf{H}_{i,k}^{-1}\mathbf{S}_{i,k}\mathbf{H}_{i,k}^{-1}$ converges to covariance matrix $\mathbf{H}^{-1}\mathbf{S}\mathbf{H}^{-1}$ in probability.
\end{thm}
\begin{proof}
The proof is similar to \cite[Theorem 4]{zhao2021confidence}.
\end{proof}
\section{Experimental Results}

In this section, we perform a simulation study to illustrate our theoretic findings about convergence rate and asymptotic normality of $\mathcal{S}$-$\mathcal{AB}$.
Consider the ridge regression problem \cite[Part V]{pu2020distributed}:
\begin{equation}\label{sim-pro}
\min_{x\in\mathbb{R}^d} ~f(x)=\sum_{j=1}^n\left(\mathbb{E}\left[\left(w_j^\intercal x-v_j\right)^2\right]+\gamma\|x\|^2\right),\\
\end{equation}
where $f_i(x)\define\mathbb{E}\left[\left(w_i^\intercal x-v_i\right)^2\right]+\gamma\|x\|^2$ is the objective function of agent $i$.
In problem (\ref{sim-pro}),
each agent $ i\in\mathcal{V}$ has access to sample $(w_i,v_i)$ given by the following linear model
$$v_i=w_i^\intercal \tilde{x}_i+\nu_i,$$
where $ w_i $ is the regression vector,  $\nu_i $ is the observation noise and $\tilde{x}_i $ is the unknown parameter. Assume that random variables $w_i $ and $\nu_i$ are independent and then the  unique solution to problem (\ref{sim-pro}) is $$x^*=\left(\sum_{j=1}^n\mathbb{E}[w_jw_j^\intercal]+n\gamma\mathbf{I}\right)^{-1}\sum_{j=1}^n\mathbb{E}[w_jw_j^\intercal]\tilde{x}_j.$$

In this experiment, the setting of parameters follows from \cite[Part V]{pu2020distributed}, the setting of network topology and weighted matrices follow from \cite[Part V]{pu2020robust}. We suppose $\gamma=1$, random variables $w_i \in [1, 2]^3$ is uniformly distributed, $\nu_i$ is drawn from the Gaussian distribution $N(0,1)$ and parameter $\tilde{x}_i$ is evenly located in $[1,~10]^3$ for $\forall i\in\mathcal{V}=\{1,2,...,20\}$.  The directed graph $\mathcal{G}$ is generated by adding random links to a ring network, where a directed link exists between any two nonadjacent nodes with a probability $p=0.3$.
For $\forall i\in \mathcal{V}$,
$\mathcal{G}_\A=\mathcal{G}_\Z=\mathcal{G}$ and
\begin{equation*}
\begin{aligned}
&\mathbf{A}_{ij}=\left\{
\begin{aligned}
&\frac{1}{|\mathcal{N}_{\mathbf{A},i}^{\text{in}}|+1},\quad j\in \mathcal{N}_{\mathbf{A},i}^{\text{in}},\\
&1-\sum_{j\in\mathcal{N}_{\mathbf{A},i}^{\text{in}}}\mathbf{A}_{ij},\quad j=i,
\end{aligned}\right.
\\
&\mathbf{B}_{ji}=\left\{
\begin{aligned}
&\frac{1}{|\mathcal{N}_{\mathbf{B},i}^{\text{out}}|+1},\quad j\in\mathcal{N}_{\mathbf{B},i}^{\text{out}},\\
&1-\sum_{j\in\mathcal{N}_{\mathbf{B},i}^{\text{out}}}\mathbf{B}_{ji},\quad j=i,
\end{aligned}\right.
\end{aligned}
\end{equation*}
where $\mathcal{N}_{\mathbf{A},i}^{\text{in}}\define\{j|(j,i)\in\mathcal{E}_\A,j\ne i\}$ and $\mathcal{N}_{\mathbf{B},i}^{\text{out}}\define\{j|(i,j)\in\mathcal{E}_\Z,j\ne i\}$ are sets of incoming and outgoing neighbors  of agent $i$ respectively, $|\mathcal{N}_{\mathbf{A},i}^{\text{in}}|$ and $|\mathcal{N}_{\mathbf{B},i}^{\text{out}}|$ are the  cardinality of $\mathcal{N}_{\mathbf{A},i}^{\text{in}}$ and $\mathcal{N}_{\mathbf{B},i}^{\text{out}}$.

\begin{figure}[http]
	\centering
	\includegraphics[height=2.1in,width=3.2in]{convergence1-eps-converted-to}
	\caption{ Trajectories of $\frac{1}{n}\sum_{j=1}^n\|x_{j,k}-x^*\|^2$. }
	\label{fig:convergence}
\end{figure}

In Figure \ref{fig:convergence}, we report the  performances of $\mathcal{S}$-$\mathcal{AB}$, distributed stochastic gradient tracking method (DSGT) and  distributed stochastic gradient descent method (DSGD) under  the setting that step-size $\al_k=0.05/(k+1)^{0.6}$ and $x_{i,0}=\mathbf{0}$ for $\forall i\in\mathcal{V}$.  \footnote{DSGT and  DSGD employ a same doubly stochastic weight matrix, which is defined by the Metropolis rule based on the underlying graph of $\mathcal{G}$.} We run the simulations 50 times for $\mathcal{S}$-$\mathcal{AB}$, DSGT and  DSGD and average the results to approximate the expected errors after 5000 iterations,  where the solid curve, dash-dot curve and dashed curve display the average $\frac{1}{n}\sum_{i=1}^n\|x_{i,k}-x^*\|^2$ for $\mathcal{S}$-$\mathcal{AB}$, DSGT and DSGD respectively.
From Figure \ref{fig:convergence},  we may conclude  that the   gradient tracking based method, $\mathcal{S}$-$\mathcal{AB}$ and DSGT, outperform DSGD. On the other hand,   $\mathcal{S}$-$\mathcal{AB}$ is lightly slower  than  DSGT.  However  DSGT and DSGD   require  the weight matrices are doubly stochastic and communication networks are undirected.
We also   test  $\mathcal{S}$-$\mathcal{AB}$, DSGT and  DSGD on the  hand-written classifying problem \cite[Part V]{RX2019} (MNIST dataset \cite{lecun1998mnist}) and record the  performances of the three methods in Figure \ref{fig:convergence-1}. Obviously, Figure \ref{fig:convergence-1} supports the conclusion of Figure \ref{fig:convergence}.\footnote{The step-size $\al_k=0.0001/(k+1)^{0.505}$, initial point $x_{i,0}=\mathbf{0}$ for $\forall i\in\mathcal{V}$ and     the number  of iterations $k=5000$.}

\begin{figure}[http]
	\centering
	\includegraphics[height=2.1in,width=3.2in]{convergence2-eps-converted-to}
	\caption{ Trajectories of $\frac{1}{n}\sum_{j=1}^nf(x_{j,k})$. }
	\label{fig:convergence-1}
\end{figure}

\begin{table}[http]
	\centering

	\caption{Coverage rate($\%$).}\label{tab:cove rate}
	\begin{tabular}{l|l|l|l|l}
		\hline

		\diagbox{\centering {\small Methods}}{{\small Iterations}}& 2000& 5000& 15000&30000\\
		\hline
		PI&81&86.8&92&94.6\cr
		PIave&78.4&86.8&91.4&94.6\cr
		\hline
	\end{tabular}
\end{table}
Next, we employ the  asymptotic normality of $\mathcal{S}$-$\mathcal{AB}$ (Theorem \ref{thm:asym norm}) to construct the   confidence regions of the true solution to problem (\ref{sim-pro}). According to  Theorem \ref{thm:asym norm} and Theorem \ref{thm:PI-rate},
\begin{equation}\label{confidence region}
\left\{y:\left(y-\hat{x}\right)^\intercal\hat{\Sigma}^{-1}\left(y-\hat{x}\right)\le \frac{1}{k}\chi_\beta^2(d)\right\}
\end{equation}
defines an $1-\beta$  asymptotic confidence region for  the optimal solution to problem (\ref{sim-pro}), where  $\hat{\Sigma}$ is the covariance matric returned by Algorithm \ref{alg:dis plug-in-1}, $\hat{x}$ is Polyak-Ruppert average of iterates generated by $\mathcal{S}$-$\mathcal{AB}$, $\chi_{\beta}^{2}(d)$ is defined to be the number that satisfies $P\left(U>\chi_{\beta}^{2}(d)\right)=\beta$ for a $\chi^{2}$ random variable $U$ with $d$ degrees of freedom.   We do 500 Monte-Carlo simulations and record the percentage of the $95\%$ confidence regions containing the true solution in Table \ref{tab:cove rate}, where the rows of PI, PIave record the results for agent 1 and the average of all agents respectively. From the Table \ref{tab:cove rate}, we can see that the coverage rates of PI and PIave at iterations $k=15000$ and $k=30000$ are both nearly $95\%$. Moreover, we can also observe the agreement of $\mathbf{H}_{i,k}^{-1}\mathbf{S}_{i,k}\mathbf{H}_{i,k}^{-1}$ for $i\in\mathcal{V}$ by comparing the performance of PI with PIave.

\section*{Acknowledgment}

The research is supported by the NSFC \#11971090.

\bibliographystyle{IEEEtran}

redpajama_set_name: RedPajamaArXiv


Are you planning a visit to Russia for business, work or as a tourist? Have you heard that getting a visa is difficult, expensive and confusing? Does the whole process worry you? Maybe you have heard about the letters of support or letters of invitation that you need and are unsure how to go about getting them?
It's true, you need a letter of introduction but that need not be an expensive, complex or difficult process. Indeed, our letters of invitation start at $19.99 for a standard tourist visa, and we are also expert in the provision of letters of support for Work and Business visas too.
We are based in St. Petersburg where we are also able to help you with the best deals in accommodation and hotels, but we can help you with visa letters wherever your destination is in Russia.
So if you are looking for cheap accommodation in St. Petersburg, or letters of support for your visa application, we are dedicated to becoming your partner in helping you visit our great country.


text:
PFASs Chemicals – Protecting Our Drinking Water And Our Health
What are PFASs?
PFASs are a class of human-made chemicals which includes Per- and Polyfluoroalkyl substances These chemicals are very long-lived, which means that they remain in the environment and in humans and wildlife for a very long time.
Two of these chemicals, PFOA and PFOS are relatively well known, and manufacturers have voluntarily stopped producing them. However, PFOA and PFAS continue to be detected at contaminated sites, in water, and in our bodies.
PFOA is found in the blood of most Americans at low levels.
PFAS pollution has been documented at 94 sites in 22 states (including industrial plants, military bases, airports and fire training sites) and in the tap water of up to 16 million people in 33 states.
There may be as many as 4,000 other PFASs chemicals, and we know less about where they are made, what they are used for, water contamination or other presence in the environment, , their health effects, and in fact in most cases how to detect them.
How do we use PFASs?
Use in fire fighting foam is common but that's NOT the only use. They have been widely used since 1950s in non-stick cookware, water-repellent clothing, stain resistant fabrics and carpets, some cosmetics, products that resist grease, water, and oil, and food packaging (like pizza boxes and microwave popcorn bags).
What are their health effects?
They are concerning because they are very persistent, don't break down in the environment or the human body and can accumulate over time. Children are particularly at risk.
Earlier studies have shown that PFOA causes cancer, PFOS disrupts thyroid hormones, both can affect the immune system and reduce infant birth weights.
A recent study released by the US Agency for Toxic Substances & Disease Registry (ATSDR) found that PFAS were more harmful than previously thought and that "safe levels" were 10 times less than previously thought. Potential health effects identified in this study include liver damage, increases in cholesterol, increased risk of thyroid disease, asthma, and infertility, decreased response to vaccines, riskier pregnancies, and decreases in birth weight.
How does contamination occur and how are people exposed?
Firefighting foam used at factories or airports can contaminate soil and groundwater.
Waste from PFAS manufacturing or from industrial facilities using PFASs or making products containing them can also contaminate soil and water.
Food packaging containing PFAS can potentially transmit PFAS to the food in the packaging, which we eat.
Use of other products like carpeting, clothing, packaging materials containing PFAS, personal care products, and others can expose our skin to the chemicals.
Working at a facility that produces PFAS or that makes products using these chemicals can also be a source of exposure.
We need better tools to find more of these chemicals in water and to know what drinking water sources, private drinking water wells, and other water is contaminated.
We need to know where these chemicals are made and used and in what products. We also need to identify how they are getting into the environment and to stop that contamination.
We need better information on effects on our health, on wildlife, and on overall environmental quality.
Costs of clean up should not be paid by taxpayers, but by those responsible for the contamination.
Producers of these chemicals need to be accountable for contamination, and for continuing to make and use these chemicals while withholding evidence of health risks.
State and federal government can act. The federal government response should include:
Identify and disclose where PFAS are manufactured, where releases to the environment are occurring, and their use in food packaging and other consumer items
Increase research into human health risks
Increase investment in developing laboratory methods to test for PFAS in drinking water, wastewater, and at toxic sites
Increase investment in research on treatment to remove PFASs in drinking water and at toxic sites
Reform fire fighting foam requirements, including eliminating requirements to use PFASs in training excercises
Use all available authority to control releases into the environment
Increased drinking water monitoring
Some states are taking the lead in addressing PFASs. Examples include:
Washington state has banned PFAS in firefighting foam and will ban it in food packaging starting in 2022 if the state finds there are alternatives available. Washington is also conducting an assessment of alternative chemicals.
California is considering a bill to list ingredients in food packaging so consumers know if chemicals like PFAS are used
New Jersey is setting drinking water limits for several PFASs chemicals.
What is Clean Water Action doing:
Educating policymakers and the public about the risks of PFASs chemicals, how they are used, solutions, and the need to hold polluters accountable.
Focusing on drinking water impacts and identifying research needs, impacts on public water systems, and Safe Drinking Water Act responses to protect public health
Identifying ways to stop releases including stopping the use of these chemicals, through pollution control programs under Clean Water Act, and using other programs.
Working with impacted communities - those with contaminated soil, wells or water sources - to support their efforts to hold polluters accountable, to get water treatment or new water sources, health monitoring, and clean up of contaminated sites.
Working on state regulation and legislation to tell consumers when PFAS is in food packaging and consumer products, to curb their use, stop contamination of water, and hold polluters accountable.
Supporting Safe Drinking Water Act programs to monitor PFASs in drinking water, develop accessible testing and treatment technologies identify health protective drinking water standards, and educate consumers about what they can do to protect themselves and their families from PFASs in drinking water.
Download and share: Fact Sheet - PFAS Chemicals - Protecting our Drinking Water - Clean Water Action November 21 2018.pdf
Civic Engagement and Elections, Water, Toxic Chemicals
Campaign:
Putting Drinking Water First, Safer Chemicals for California, Safer Chemicals for Safer Families and Communities
Region/State:
California, Connecticut, Minnesota, National, Pennsylvania, Rhode Island

redpajama_set_name: RedPajamaCommonCrawl


P.F. Chang's-style chicken lettuce wraps are a low carb, easy and delicious meal that can be made ahead of time for weekly meal prep.
I love lettuce wraps, especially in the summer when I'm craving something lighter. So far we've done fajita lettuce wraps, chicken caesar lettuce wraps, and chicken lettuce wraps. We've especially been eating a lot of chicken lettuce wraps after I came up with my copycat version of P.F. Chang's lettuce wraps.
When you store these, you should store the lettuce and filling separately. That way you can remove your lettuce leaves and reheat just the filling. I photographed the lettuce wraps already assembled because otherwise they don't really look quite as nice when it's just a pile of filling and a separate pile of leaves.
It also gave me an excuse to eat quite a few when I was done with my photos. This is a lunch I definitely look forward to!
P. F Chang's inspired lettuce wraps are a low carb and delicious weekly meal prep.
In a small bowl, add all sauce ingredients and whisk together.
In a large skillet, add 1 tbsp sesame oil and bring to medium high heat. Add in ground chicken and cook and stir until crumbled and browned. Add in water chestnuts and mushrooms.
Reduce heat to medium. Pour in sauce and stir into ingredients until ingredients are evenly coated in sauce. Cover skillet with lid to cook a few minutes and allow ingredients to absorb sauce. Remove lid, taste and make any adjustments as needed. If you prefer a little sweeter, you can add a little bit of brown sugar. If you prefer saltier, you can add a little more soy sauce. When finished, remove from heat.
Stack a few lettuce leaves on one side of your meal prep container. Place some scallions on the top lettuce leaf. Add chicken filling to the other side of your container. Seal containers and store in fridge. When ready to eat, remove stack of lettuce leaves and scallions. Heat the filling. Then assemble your lettuce wraps.
You can omit the mushrooms if you don't like the flavor of shiitake mushrooms but the PF Chang's version does use them.
You can also substitute the shallot with 1/4 of a small onion.
Oh man, how I miss PFChang while reading through your recipe ;). Look delicious.
Can you store in freezer?


The ever loved Oreo dipped in our scrumptious chocolate topped off with a whimsical Easter egg, rabbit, or duck icing. A delightful Easter treat. As pleasing to look at as it is to eat!
Two per package, available in milk or dark chocolate.


Today i felt great and got stuck in to blog posts .. I think iv got my spark back I was M.I.A for a bit but you can read about that here.
Today I will be sharing my 5 favourite body moisturizers… gotta keep that skin hydrated! Lets get to it.
Vaseline 'spray & go' – I love this and probably use this the most out of the 5. It's so easy and drys very quickly. I spray it all over my body , rub it in and its dry in no time. It has a nice coconut kind of smell and its very light on the skin. It also comes in two other scents.
Garnier body 'oil beauty'– I bought this about three weeks ago and i love it. But wouldn't use it daily because it is a heavy moisturizer and takes a bit to dry but it leaves your skin feeling so smooth and smelling great! If i was having a pamper night or just got out of the shower/bath and wanted a good nourish to the skin id definitely us it.
Coco chanel 'moisture mist' – This is a splurge but the smell is amazing ! I love the coco chanel mademoiselle perfume and this is exactly what is smells like. This is a very light mist. Id use it on the go if I wanted to freshing up the skin.
Ghost 'deep night' – The perfume ghost is one of my favourite so i knew id love this. I'm not sure if you can buy it separately as it came with the perfume when I bought it. It smells exactly like the perfume with I love, its kind of a musky scent but girly at the same time …just go smell it haha , This again is a light moisturizer that gives the skin an amazing smell.
Dove silky nourishing body cream– This is a very thick creamy moisturizer like the garnier one. It will give your skin a good hydration after a nice bath.
Here is a swatch of the moisturizers.
Hope you enjoyed this post , Have you tried any of these? Is there any you would recommend? Let me know in the comments don't be shy !!
Aw thanks so much ..i sure will :)!


De Lacy Executive is recruiting for an Experienced Farm Manager to work on a progressive Poultry unit based in the West Midlands. Our client is looking for a dedicated individual to manage their site which comprises of nine sheds. This is the ideal opportunity for someone who feels that they are capable of running a farm on their own. Whilst the candidate would report to the owner, the successful candidate will be in charge of the day to day management of the farm.
This is a family run farm so it would be the perfect opportunity for someone who is looking for a role that would enable them to be a person rather than a figure.
Ensuring all aspects of running the farm is met.
Previous experience in the UK poultry industry.
Tractor driving experience - a telehandler ticket is desirable but not essential.
Remuneration: A competitive remuneration package plus accommodation is offered.
Salary: A competitive remuneration package plus accommodation is offered.


The world's most taught leadership model. To be a leader others want to follow, managers need to set clear and attainable goals, provide the matching leadership style, and give appropriate feedback. The goal of a Situational Leader is to help others develop competence, motivation, and confidence. Situational Leadership® II (SLII®) boosts the effectiveness, impact, and dedication of leaders and teaches them to behave in alignment with their good intentions and with what their people need. Recognized as both a business language and a framework for employee development that transcends cultural, linguistic, and geographical boundaries, the foundation of SLII lies in teaching leaders to diagnose the needs of an individual and then use the appropriate leadership style.
Also includes Master Challenges, THe SLII Playbook and access to course materials, worksheets and videos to support learning for an entire year. $363.00 FREE Shipping.


text:
(function($){
	PNotify.prototype.options.confirm = {
		// Make a confirmation box.
		confirm: false,
		// Where to align the buttons. (right, center, left, justify)
		align: "right",
		// The buttons to display, and their callbacks.
		buttons: [
			{
				text: "Ok",
				addClass: "",
				click: function(notice){
					notice.get().trigger("pnotify.confirm");
					notice.remove();
				}
			},
			{
				text: "Cancel",
				addClass: "",
				click: function(notice){
					notice.get().trigger("pnotify.cancel");
					notice.remove();
				}
			}
		]
	};
	PNotify.prototype.modules.confirm = {
		// The div that contains the buttons.
		buttonContainer: null,

		init: function(notice, options){
			this.buttonContainer = $('
').appendTo(notice.container);

			if (options.confirm)
				this.makeButtons(notice, options);
			else
				this.buttonContainer.hide();
		},

		update: function(notice, options){
			if (options.confirm) {
				this.makeButtons(notice, options);
				this.buttonContainer.show();
			} else {
				this.buttonContainer.hide().empty();
			}
		},

		makeButtons: function(notice, options) {
			var already = false, btn, elem;
			this.buttonContainer.empty();
			for (var i in options.buttons) {
				btn = options.buttons[i];
				if (already)
					this.buttonContainer.append(' ');
				else
					already = true;
				elem = $('<button type="button" class="'+notice.styles.btn+' '+btn.addClass+'">'+btn.text+'</button>')
				.appendTo(this.buttonContainer)
				.on("click", (function(btn){ return function(){
					if (typeof btn.click == "function") {
						btn.click(notice);
					}
				}})(btn));
				if (notice.styles.text) {
					elem.wrapInner('');
				}
				if (notice.styles.btnhover) {
					elem.hover((function(elem){ return function(){
						elem.addClass(notice.styles.btnhover);
					}})(elem), (function(elem){ return function(){
						elem.removeClass(notice.styles.btnhover);
					}})(elem));
				}
				if (notice.styles.btnactive) {
					elem.on("mousedown", (function(elem){ return function(){
						elem.addClass(notice.styles.btnactive);
					}})(elem)).on("mouseup", (function(elem){ return function(){
						elem.removeClass(notice.styles.btnactive);
					}})(elem));
				}
				if (notice.styles.btnfocus) {
					elem.on("focus", (function(elem){ return function(){
						elem.addClass(notice.styles.btnfocus);
					}})(elem)).on("blur", (function(elem){ return function(){
						elem.removeClass(notice.styles.btnfocus);
					}})(elem));
				}
			}
		}
	};
	$.extend(PNotify.styling.jqueryui, {
		btn: "ui-button ui-widget ui-state-default ui-corner-all ui-button-text-only",
		btnhover: "ui-state-hover",
		btnactive: "ui-state-active",
		btnfocus: "ui-state-focus",
		text: "ui-button-text"
	});
	$.extend(PNotify.styling.bootstrap2, {
		btn: "btn"
	});
	$.extend(PNotify.styling.bootstrap3, {
		btn: "btn btn-default"
	});
	$.extend(PNotify.styling.fontawesome, {
		btn: "btn btn-default"
	});
})(jQuery);

redpajama_set_name: RedPajamaGithub


Do wait to be seated.
Do great everyone at your table.
Do use your utensils from: outside - in.
Do place your napkin in your chair when excused.
Do signal your server with eye contact or subtle gestures.
Use these tips to have a great meal! Bon Appétit!


text:
In 2002, we began to investigate overconfidence errors in psychosis. In two pilot studies conducted in Germany (Moritz et al., 2003) and Canada (Moritz & Woodward, 2002), we asked participants to generate words in response to stimuli presented by the experimenter. Previously presented and self-generated words were then read to the participants along with distractor stimuli. Apart from simple recognition (old/new), participants were asked to indicate who they thought had produced the word (source attribution: experimenter or participant) and to estimate their response confidence. In both studies, patients with schizophrenia displayed overconfidence regarding false responses (feedback regarding the correctness of the decision was given later). As expected, healthy controls made fewer errors and were less confident regarding false responses than patients.
We assume that healthy participants tag mental events that are not supported by sufficient evidence as "not trustworthy". Patients with schizophrenia, on the other hand, may have a deficit adjusting their level of confidence for memories that are correct or incorrect (replications by Moritz et al., 2006a; Moritz et al., 2004; Moritz et al., 2005; Moritz et al., 2006b; Moritz et al., 2008; Moritz & Woodward, 2006a, 2006b). Importantly, the word source attribution paradigm discussed above did not employ any material relevant to delusions; all words produced by the experimenter were neutral, precluding tautological inferences (for a review, see Moritz & Woodward, 2006a). Overconfidence appears not to be memory-specific and has also been demonstrated for other cognitive domains, such as social cognition and emotion recognition (Köther et al., 2012, Moritz et al., 2014; Moritz et al., 2012).
Prof. Dr. Steffen Moritz
PD Dr. Christina Andreou
Dipl.-Psych. Birgit Hottenrott
Cooperating Partners
Prof. Dr. Todd S. Woodward (University of British Columbia and Riverview Hospital, Vancouver, Canada)
Dr. Maarten Peters (Maastricht University, Maastricht,The Netherlands)
Dr. Lukasz Gaweda (Medical University of Warsaw, Warsaw, Poland
Köther, U., Veckenstedt, R., Vitzthum, F., Roesch-Ely, D., Pfueller, U., Scheu, F. & Moritz, S. (2012). "Don't give me that look" – Overconfidence in false mental state perception in schizophrenia. Psychiatry Research, 196, 1-8.
Moritz, S., Göritz, A. S., Van Quaquebeke, N., Andreou, C., Jungclaussen, D. & Peters, M. J. (2014). Knowledge corruption for visual perception in individuals high on paranoia. Psychiatry Research, 30, 700-705.
Moritz, S. & Woodward, T. S. (2002). Memory confidence and false memories in schizophrenia. Journal of Nervous & Mental Disease, 190, 641-643.
Moritz, S. & Woodward, T. S. (2006a). The contribution of metamemory deficits to schizophrenia. Journal of Abnormal Psychology, 115, 15-25.
Moritz, S. & Woodward, T. S. (2006b). Metacognitive control over false memories: A key determinant of delusional thinking. Current Psychiatry Reports, 8, 184-190.
Moritz, S., Woodward, T. S. & Chen, E. (2006a). Investigation of metamemory dysfunctions in first-episode schizophrenia. Schizophrenia Research, 81, 247-252.
Moritz, S., Woodward, T. S., Cuttler, C., Whitman, J. C. & Watson, J. M. (2004). False memories in schizophrenia. Neuropsychology, 18, 276-283.
Moritz, S., Woodward, T. S., Jelinek, L. & Klinge, R. (2008). Memory and metamemory in schizophrenia: A liberal acceptance account of psychosis. Psychological Medicine, 38, 825-832.
Moritz, S., Woodward, T. S. & Rodriguez-Raecke, R. (2006b). Patients with schizophrenia do not produce more false memories than controls but are more confident in them. Psychological Medicine, 36, 659-667.
Moritz, S., Woodward, T. S. & Ruff, C. C. (2003). Source monitoring and memory confidence in schizophrenia. Psychological Medicine, 33, 131-139.
Moritz, S., Woodward, T. S., Whitman, J. C. & Cuttler, C. (2005). Confidence in errors as a possible basis for delusions in schizophrenia. Journal of Nervous and Mental Disease, 193, 9-16.
Moritz, S., Woznica, A., Andreou, C. & Köther, U. (2012). Response confidence for emotion perception in schizophrenia using a Continuous Facial Sequence Task. Psychiatry Research, 30, 202-207.