text:
\section{Introduction}
\label{intro}
\vspace{-1mm}

For a deep neural network, architecture \cite{he2016deep,howard2017mobilenets,huang2017densely} and parameter initialization \cite{he2015delving,glorot2010understanding} are two initial elements that largely account for the final model performance. Lots of human efforts have been devoted to finding better answers with respect to the two aspects. To automatically produce better architectures, neural architecture search \cite{zoph2016neural,real2019regularized,chen2018searching} has been a research focus. However, on the other hand, using automated techniques for parameter initialization has rarely been studied.

Previous initialization methods are mostly handcrafted. They focus on finding proper variance patterns of randomly initialized weights \cite{he2015delving,glorot2010understanding,saxe2013exact,mishkin2015all} or rely on empirical evidence derived from certain architectures \cite{zhang2019fixup, huang2020improving, gilmer2021loss,das2021data}. Recently, \cite{zhu2021gradinit,dauphin2019metainit} propose learning-based initialization that learns to tune the norms of the initial weights so as to minimize a quantity that is intimately related to favorable training dynamics. Despite being architecture-agnostic and gradient-based,
these methods do not consider sample-wise landscape.
Their derived quantities lack theoretical supports for being related to model performance. It is unclear whether optimizing these quantities can indeed lead to better training or generalization performance.

In order to find a better initialization, a theoretically sound quantity that intimately evaluates both the training and test performance should be designed. Finding such a quantity is a non-trivial problem and shows many challenges. First, the test performance is mostly decided by the converged parameters after training, while the training dynamic is more related to the parameters at initialization or during training.
Second, to efficiently find a better starting point, the quantity is expected to be differentiable to enable the optimization in the continuous parameter space.
To this end, we leverage the recent advances in optimization landscape analysis \cite{allen2019convergence} to propose a novel differentiable quantity and develop a corresponding algorithm for its optimization at initialization.

Specifically, our quantity is inspired by analyzing the optimization landscapes of individual training samples \cite{zhang2021gradsign}. Through generalizing prior theoretical results on batch-wise optimization \cite{allen2019convergence} to sample-wise optimization, we prove that both the network's training and generalization error are upper bounded by a theoretical quantity that correlates with the cosine similarity of the sample-wise local optima. Moreover, this quantity also relates to the training dynamic since it reflects the optimization path consistency \cite{meuleau2002ant,oymak2019overparameterized} from the starting point. Unfortunately, the sample-wise local optima are intractable. With the hypothesis that the sample-wise local optima can be reached by the first-order approximation from the initial parameters, we can approximate the quantity via the sample-wise gradients at initialization.
Our final result shows that, under a limited gradient norm, both the training and test performance of a network can be improved by maximizing the cosine similarity of sample-wise gradients, named GradCosine, which is differentiable and easy to implement.

We then propose the Neural Initialization Optimization (NIO) algorithm based on GradCosine to find a better initialization agnostic of architecture. We generalize the algorithm from the sample-wise analysis into the batch-wise setting by dividing a batch into sub-batches for friendly implementation. We follow \cite{dauphin2019metainit,zhu2021gradinit} using gradient descent to learn a set of scalar coefficients of the initialized parameters. These coefficients are optimized to maximize GradCosine for better training dynamic and expected performance while constraining the gradient norm from explosion.

Experiments show that for a variety of deep architectures including ResNet \cite{he2016deep}, DenseNet \cite{huang2017densely}, and WideResNet \cite{zagoruyko2016wide}, our method achieves better classification results on CIFAR-10/100 \cite{krizhevsky2009learning} than prior heuristic \cite{he2015delving} and learning-based \cite{dauphin2019metainit,zhu2021gradinit} initialization methods. We can also initialize ResNet-50 \cite{he2016deep} on ImageNet \cite{deng2009imagenet} for better performance.
Moreover, our method is able to help the recently proposed Swin-Transformer \cite{liu2021swin} achieve stable training and competitive results on ImageNet even without warmup \cite{goyal2017accurate}, which is crucial for the successful training of Transformer architectures \cite{liu2019variance,xiong2020layer}.

\section{Related Work}
\subsection{Network Initialization}
Existing initialization methods are designed to control the norms of network parameters via Gaussian initialization \cite{glorot2010understanding,he2015delving} or orthonormal matrix initialization \cite{saxe2013exact,mishkin2015all} with different variance patterns. These analyses are most effective for simple feed-forward networks without skip connections or normalization layers. Recently, initialization techniques specified for some complex architectures are proposed. For example, \cite{zhang2019fixup} studied how to initialize networks with skip connections and \cite{huang2020improving} generalized the results into Transformer architecture \cite{vaswani2017attention}. However, these heuristic methods are restricted to specific architectures. Automated machine learning has achieved success in looking for hyperparameters \cite{bergstra2011algorithms,feurer2019hyperparameter} and architectures \cite{zoph2016neural,real2019regularized,chen2018searching,yang2020ista,yang2021towards,huang2020explicitly}, while similar techniques for neural network initialization deserve more exploration. Current learning based initialization methods \cite{dauphin2019metainit,zhu2021gradinit} optimize the curvature \cite{dauphin2019metainit} or the loss reduction of the first stochastic step \cite{zhu2021gradinit} using gradient descent to tune the norms of the initial parameters.
However, these methods lack theoretical foundations to be related to the model performance. Different from these methods, our proposed GradCosine is derived from a theoretical quantity that is the upper bound of both training and generalization error.

\subsection{Evaluating Model Performance at Initialization}
Evaluating the performance of a network at initialization has been an important challenge and widely applied in zero-shot neural architecture search \cite{abdelfattah2020zero,mellor2021neural,chen2021neural,simon2021neural,zhang2021gradsign,shu2021nasi,li2018optimization} and pruning \cite{wang2020picking,lee2018snip, tanaka2020pruning}. The evaluation quantities in these studies are mainly based on the initial gradient norm \cite{tanaka2020pruning,shu2021nasi}, the eigenvalues of neural tangent kernel \cite{chen2021neural,shu2021nasi}, and the Fisher information matrix \cite{turner2019blockswap,turner2021neural,theis2018faster}. However, these quantities cannot reflect optimization landscape, which is crucial for training dynamic and generalization \cite{brutzkus2017globally,du2018gradient,ge2017learning,soltanolkotabi2017learning,li2017convergence,allen2019convergence}. \cite{allen2019convergence} provided theoretical evidence that for a sufficiently large neighborhood of a random initialization, the optimization landscape is nearly convex and semi-smooth. Based on the result, \cite{zhang2021gradsign} proposed to use the density of sample-wise local optima to evaluate and rank neural architectures. Our study also performs sample-wise landscape analysis, but differs from \cite{zhang2021gradsign} in that our proposed quantity is differentiable and reflects optimization path consistency, while the quantity in \cite{zhang2021gradsign} is non-differentiable so cannot serve for initialization optimization. We make a comparison in more details in Appendix \ref{explanation}.

\begin{figure}[t]
	\vskip 0.2in
	\begin{center}

		\centerline{\includegraphics[width=\columnwidth]{fig1_nips.pdf}}

		\caption{(a) Optimization landscape with sparser sample-wise local optima corresponds to a worse $\theta^*$ (larger loss $l$). (b) Optimization landscape with denser sample-wise local optima corresponds to a better $\theta^* $ (smaller loss $l$). However, the density of sample-wise local optima cannot reflect training path. We further leverage the cosine similarity of sample-wise local optima.  Under the same local optima density, (d) corresponds to a better training dynamic than (c), since (d) enjoys a better optimization path consistency (smaller cosine distance between $\theta^*_1-\theta_0$ and $\theta^*_2-\theta_0$). We give more detailed explanations and discussions in Appendix \ref{explanation}.
		\label{fig1}
	\end{center}
	\vspace{-3mm
\end{figure}

\section{Theoretical Foundations}
\label{theory}
\subsection{Sample-wise Optimization Landscape Analysis}
Conventional optimization landscape analyses \cite{li2017convergence,allen2019convergence} mainly focus on the objective across a mini-batch of samples and miss potential evidence hidden in the optimization landscapes of individual samples. As recently pointed out in \cite{zhang2021gradsign}, by decomposing a mini-batch objective into the summation of sample-wise objectives across individual samples in the mini-batch, they find that the network with denser sample-wise local optima tends to reach a better local optima for the mini-batch objective, as shown in the Figure (\ref{fig1}) (a)-(b). Based on this insight, they propose to use sample-wise local optima density to judge model performance at initialization.

\textbf{Density of sample-wise local optima.}
For a batch of training samples $S = \{(x_i,y_i)\}_{i\in[n]}$, a loss function $l(\hat y_i, y_i)$, and a network $f_\theta(\cdot)$ parameterized by $\theta\in\mathbb{R}^m$, the sample-wise local optima density $\Psi_{S,l}(f_{\theta_0}(\cdot))$ is measured by the averaged Manhattan distance between the pair-wise local optima $\{\theta^*_i\}_{i\in[n]}$ of all $n$ samples near the random initialization $\theta_0$ \cite{zhang2021gradsign}, \emph{i.e.,}
\begin{equation}\label{gradsign}
    \Psi_{S,l}(f_{\theta_0}(\cdot)) = \frac{\sqrt{\mathcal{H}}}{n^2}\sum_{i,j}||\theta^*_i-\theta^*_j||_1,\ \ i,j\in[1,n],
\end{equation}
where $\mathcal{H}$ is the smoothness upper bound: $\forall k\in [m], i\in [n], [\nabla^2l(f_{\theta}(x_i), y_i)]_{k,k}\leq\mathcal{H}$ and $\mathcal{H}$ always exists with the smoothness assumption that for a neighborhood $\Gamma_{\theta_0}$ of a random initialization $\theta_0$, the sample-wise optimization landscapes are nearly convex and semi-smooth \cite{allen2019convergence}. Based on this assumption, the training error of the network can be upper bounded by $\frac{n^3}{2}\Psi^2_{S,l}(f_{\theta_0}(\cdot))$. Moreover, with probability $1-\delta$, the generalization error measured by population loss $\mathbb{E}_{(x_u,y_u)\sim\mathcal{D}}[l(f^*_\theta(x_u), y_u)]$ is upper bounded by $\frac{n^3}{2}\Psi^2_{S,l}(f_{\theta_0}(\cdot))+\frac{\sigma}{\sqrt{n\delta}}$, where $\mathcal D=\{(x_u,y_u)\}_{u\in[U]}$ is the underlying data distribution of test samples and $\sigma^2$ is the upper bound of $Var_{(x_u,y_u)\sim\mathcal D} [||\theta^*-\theta^*_u||^2_1]$ \cite{zhang2021gradsign}.

Although the sample-wise local optima density is theoretically related to both the network training and generalization error, we point out that it may not be a suitable quantity to evaluate network initialization due to the following reasons.
First, it ignores the optimization path consistency from initialization $\theta_0$ to each sample-wise local optimum $\theta^*_i$. As shown in Figure \ref{fig1} (c)-(d), while both (c) and (d) have the same local optima density, in Figure \ref{fig1} (d) the optimization path from the initialization to the local optima of the two samples are more consistent. Training networks on samples with more consistent optimization paths using batch gradient descent naturally corresponds to a better training dynamic that enjoys faster and more stable convergence  \cite{ruder2016overview,oymak2019overparameterized}. Second, the sample-wise local optima in Eq. (\ref{gradsign}) are intractable. It can be approximated by measuring the consistency of sample-wise gradient signs \cite{zhang2021gradsign}. But it is non-differentiable and cannot serve for initialization optimization.

Based on this observation, we aim to derive a new quantity that directly reflects optimization path consistency and is a differentiable function of the initialization $\theta_0$.
Our proposed quantity is based on the cosine similarity of the paths from initialization to sample-wise local optima.

\textbf{Cosine similarity of sample-wise local optima.}
Concretely, our quantity can be formulated as:
\begin{equation}
    \Theta_{S,l}(f_{\theta_0}(\cdot)) = \frac{\mathcal{H}\alpha^2}{n}\sum_{i,j}\left(\frac{\alpha}{\beta} - \cos\angle(\theta^*_i-\theta_0,\theta^*_j-\theta_0)\right),\ \ i,j\in[1,n],
    \label{eq2}
\end{equation}
where $\alpha$ and $\beta$ are the maximal and minimal $\ell_2$-norms of the sample-wise optimization paths, \emph{i.e.,} $\alpha=\max(||\theta^*_i-\theta_0||_2), \beta=\min(||\theta^*_i-\theta_0||_2), \forall i \in [1,n]$,
and $\cos\angle(\theta^*_i-\theta_0,\theta^*_j-\theta_0)$ refers to the cosine similarity of the paths from initialization $\theta_0$ to sample-wise local optima, $\theta^*_i$ and $\theta^*_j$. The cosine term in Eq. (\ref{eq2}) reflects the optimization path consistency. Together with the distance term $\frac{\alpha}{\beta}$, it is also able to measure the density of sample-wise local optima.

In the ideal case, when all the local optima are located at the same point, $\Theta_{S,l}(f_{\theta_0}(\cdot))=\Psi_{S,l}(f_{\theta_0}(\cdot))=0$. Hence, compared with $\Psi$, our $\Theta$ is more suitable for evaluating the initialization quality.

\subsection{Main Results}
\label{s32}
In this subsection, we theoretically illustrate how minimizing the quantity $\Theta$ in Eq. (\ref{eq2}) corresponds to better training and generalization performance. Similar to \cite{zhang2021gradsign}, our derivations are also based on the evidence that there exists a neighborhood for a random initialization such that the sample-wise optimization landscapes are nearly convex and semi-smooth \cite{allen2019convergence}.

\textbf{Lemma 1.} \textit{There exists no saddle point in a sample-wise optimization landscape and every local optimum is a global optimum \cite{zhang2021gradsign}.}

Based on Lemma 1, we can draw a relation between the training error and $\Theta_{S,l}(f_{\theta_0}(\cdot))$. Moreover, we show that the proposed quantity is also related to  generalization performance as the upper bound of population error. We present the following two theoretical results.

\textbf{Theorem 2.} \textit{The training loss $\mathcal{L}=\frac{1}{n}\sum_i l(f_{\theta^*}(x_i), y_i)$ of a trained network $f_{\theta^*}$ on a dataset $S = \{(x_i,y_i)\}_{i\in[n]}$ is upper bounded by $\Theta_{S,l}(f_{\theta_0}(\cdot))$, and the bound is tight when $\Theta_{S,l}(f_{\theta_0}(\cdot))=0$.}

\textbf{Theorem 3.} \textit{Suppose that $\sigma^2$ is the upper bound of $Var_{(x_u,y_u)\sim\mathcal D} [||\theta^*-\theta^*_u||^2_2]$, where $\theta^*_u$ is the local optimum in the convex neighborhood of $\theta_0$ for test sample $(x_u, y_u)$. With probability $1-\delta$, the population loss $\mathbb{E}_{(x_u,y_u)\sim\mathcal{D}}[l(f_{\theta^*}(x_u), y_u)]$ is upper bounded by $\Theta_{S,l}(f_{\theta_0}(\cdot))+\frac{\sigma}{\sqrt{n\delta}}$}.

We provide proofs of these two theorems in Appendix \ref{ap1}. Combining both theorems, we can conclude that $\Theta_{S,l}(f_{\theta_0}(\cdot))$ is the upper bound of both training and generalization errors of the network $f_{\theta^*}$. Therefore, minimizing $\Theta$ theoretically helps to improve the model performance.

Albeit theoretically sound, $\Theta_{S,l}(f_{\theta_0}(\cdot))$ requires sample-wise optima $\theta^*_i$ which are intractable at initialization. To this end, we will show how to develop a differentiable and tractable objective based on Eq. (\ref{eq2}) in Section \ref{sec_GC}, and introduce the initialization optimization algorithm in Section \ref{sec_NIO}.

\section{GradCosine}
\label{sec_GC}

\iffalse
\subsection{Evaluation under Random Initialization}
\label{s41}
By supposing the initialization parameters of networks are drawn from the Gaussian distribution $\mathcal{N}(0,\sigma^2_2)$, we derive the following theorem.

\textbf{Theorem 4} \textit{Given that $\mathbb{E}(\theta_0)=0$ and $Var(\theta_0)=\sigma^2_2$, with probability $1-\delta$ that:}
\begin{equation}
    \Theta_{S,l}(f_{\theta_0}(\cdot)) \leq  \frac{\mathcal{H}\alpha^2}{n}\sum_{i,j}(\frac{\alpha}{\beta} - \cos(\theta^*_i-\theta_0,\theta^*_j-\theta_0)+\frac{C_1}{2\sqrt{\delta}}),
    \label{eq3}
\end{equation}
where $C_1$ is the constant upper bound such that $C_1\geq Var_{\theta_0\sim\mathcal{N}(0,\sigma^2_2)}\cos(\theta_i^*-\theta_0,\theta_j^*-\theta_0)$.
We convert the cosine similarity of sample-wise local optima $\cos(\theta^*_i,\theta^*_j)$ into the similarity of the optimization paths from the randomly initialized points $\cos(\theta^*_i-\theta_0,\theta^*_j-\theta_0)$ which will be shown later that can be approximated by sample-wise gradients. The proof of Theorem 4 is provided in Appendix (\ref{ap1}).
\fi

\subsection{First-Order Approximation of Sample-Wise Optimization}
\label{s42}
Since we are dealing with sample-wise optimization, it is reasonable to calculate each sample-wise optimum by the first-order approximation. We hypothesize that each sample-wise optimum can be reached via only one-step gradient descent from the initialized parameters. Its rationale lies in that it is very easy for a deep neural network to learn the optimum for only one training sample with gradient descent. Based on this hypothesis, we can approximate each local optimum as:
\begin{equation}
    \theta^*_i \approx \theta_0 - \eta g_i,\ \ i\in[1,n],
    \label{eq4}
\end{equation}
where $g_i=\nabla_\theta l(f_\theta(x_i), y_i)|_{\theta_0}$ is the sample-wise gradient at initialization, and $\eta$ is the learning rate. With Eq. (\ref{eq4}), the upper bound quantity in Eq. (\ref{eq2}) can be simplified as:
\iffalse
\begin{equation}
\begin{aligned}
    \Theta &\leq
    \frac{\mathcal{H}\alpha^2}{n}\sum_{i,j}(\frac{\alpha}{\beta} - \cos(\theta^*_i-\theta_0,\theta^*_j-\theta_0)+\frac{C_1}{2\sqrt{\delta}}) \\
    &\leq \frac{\mathcal{H}(\eta g_{max}+C_2)^2}{n}\sum_{i,j}(\frac{\alpha}{\beta} - \cos(g_i,g_j)+\frac{C_1}{2\sqrt{\delta}}),
    \label{eq5}
\end{aligned}
\end{equation}
\fi
\begin{equation}
	\label{eq5}
		\Theta \approx
		\frac{\mathcal{H}g_{max}^2}{n}\sum_{i,j}\left(\frac{g_{max}}{g_{min}} - \cos\angle(g_i,g_j)\right),\ \ i,j\in[1,n],
\end{equation}

where $g_{max}$ and $g_{min}$ are the maximal and minimal sample-wise gradient norms at initialization, \emph{i.e.,} $g_{max} = \max(||g_i||_2), g_{min} = \min(||g_i||_2), \forall i\in[1,n]$.

\subsection{Guidance for Initialization}

Suppose that the sample-wise gradient at initialization is upper bounded by a constant $\gamma$, \emph{i.e.,} $g_{max}\le \gamma$, and then we have:
\begin{equation}
	\label{relax_theta}
	\Theta \le
	\frac{\mathcal{H}\gamma^2}{n}\sum_{i,j}\left(\frac{\gamma}{g_{min}} - \cos\angle(g_i,g_j)\right),\ \ i,j\in[1,n].
\end{equation}
From Eq. (\ref{eq2}) to Eq. (\ref{relax_theta}), we have almost converted the upper bound into a quantity that can be easily calculated by the sample-wise gradient at initialization, except for the term $g_{min}$ and the constraint $g_{max}\le \gamma$. We will explicitly add corresponding optimization objective and constraint in our neural initialization optimization algorithm in Section \ref{sec_NIO} to ensure a small $\gamma/g_{min}$.

From Eq. (\ref{relax_theta}), we can draw some useful guidance for a better initialization:
\begin{itemize}
    \item[(1)] The sample-wise gradient norms of the initialized parameters should be large and close to the maximal value bounded by $\gamma$ to induce a small $\gamma/g_{min}$;
    \item[(2)] The cosine similarity of the sample-wise gradients should be as large as possible.
\end{itemize}

\begin{algorithm}[t]
		\small
\caption{
GradCosine (GC) and gradient norm (GN)
}
\label{alg1}
\begin{algorithmic}[1]
\REQUIRE Initial network parameters $\theta_0$, and a batch of samples $S=\{(x_i,y_i)\}_{i\in[B]}, B = |S|$.
\FOR{i=1 to $B$}
\STATE Compute the sample-wise gradient:\\ $g_i \leftarrow \nabla_\theta l(f_\theta(x_i), y_i)|_{\theta_0}; g_i \in \mathbb{R}^m$
\ENDFOR
\STATE Compute the average of gradient norm:\\ $\textbf{GN}(S,\theta_0)\leftarrow \frac{1}{B}\sum^B_{i=1}||g_i||_2$
\STATE Compute the cosine similarity of gradients:\\ $\phi_{i,j} \leftarrow \frac{g_i\cdot g_j}{||g_i||_2\cdot||g_j||_2}, i=1,...,B, j=1,...,B$
\STATE Compute the average of gradient cosine: \\
$\textbf{GC}(S,\theta_0)\leftarrow \frac{1}{B^2}\sum^{B}_{i=1}\sum^{B}_{j=1}\phi_{i,j}$
\STATE Output \textbf{GN} and \textbf{GC}.
\end{algorithmic}
\end{algorithm}

\textbf{Relations to Favorable Training Dynamic.}
There are significant evidences in existing studies that support the two rules for better training dynamic. The first rule is intimately related to the neural tangent kernel (NTK) \cite{jacot2018neural} that has recently been shown to determine the early learning behavior \cite{xiao2020disentangling,shu2021nasi,chen2021neural,simon2021neural}. Specifically, \cite{shu2021nasi} finds that the training dynamic of a neural network can be characterized by the trace norm of NTK at initialization and further approximates it via the initial gradient norm as a gauge to search for neural architectures. \cite{zhu2021gradinit} initializes a network by maximizing the loss reduction of the first gradient descent step, which also corresponds to gradient norm in the first-order approximation of the loss function. The two rules together prefer a model whose sample-wise gradients have close $\ell_2$ and cosine distances. It is in line with prior observations that the initial gradient variance should be small to enable a large learning rate \cite{liu2019variance,zhuang2020adabelief}. However, gradient variance is not a proper optimization objective due to its high sensitivity to gradient norm.

While the first rule of improving the initial gradient norm has been suggested for better training dynamic in prior studies \cite{zhu2021gradinit,shu2021nasi},
the second rule is completely new. We refer this novel quantity, \emph{i.e.,} $\cos\angle(g_i, g_j)$, for evaluating the network initialization as GradCosine. Intuitively, in the ideal case where the initial gradients of all samples are identical, we have the smallest upper bound of training and generalization error. Moreover, both gradient norm and GradCosine are differentiable and thus enable optimization to find a better initialization. We provide the calculation of gradient norm (GN) and GradCosine (GC) for an initialized network in Algorithm \ref{alg1}.

\section{Neural Initialization Optimization}
\label{sec_NIO}

Based on the GradCosine, we propose our Neural Initialization Optimization (NIO) algorithm. Our algorithm follows \cite{zhu2021gradinit, dauphin2019metainit} to rectify the variance of the initialized parameters in a network via gradient descent. Concretely, we introduce a set of learnable scalar coefficients denoted as $M = \{\omega_1,\cdots,\omega_m\}$. The initial parameters $\theta_0=\{W_1,\cdots,W_m\}$ rectified by these coefficients can be formulated as $\theta_M=\{\omega_1W_1,\cdots,\omega_mW_m\}$. The NIO algorithm mainly solves the following constrained optimization problem:
\iffalse
\begin{equation}
\begin{aligned}
    &\mathop{\text{maximize}}\limits_M \quad \text{GC}(S, \theta_M),\\
    &\text{subject to}  \quad \max_i(||g_{i}||_2) \leq \gamma; i =1,...,B,
    \label{eq6}

\end{aligned}
\end{equation}
\fi
\begin{equation}
	\begin{aligned}
		&\mathop{\text{maximize}}\limits_M \quad \text{GC}(S, \theta_M) + \text{GN}(S, \theta_M),\\
		&\text{subject to}  \quad \max_i(||g_{i}||_2) \leq \gamma,\ \ i = 1,...,B,
		\label{eq7}

	\end{aligned}
\end{equation}
where GC and GN refer to GradCosine and gradient norm, respectively, as calculated in Algorithm \ref{alg1}, and $B$ is the batchsize of the batch data $S$. With the maximal sample-wise gradient norm bounded by $\gamma$, we maximize both GradCosine and gradient norm.

Although the problem in Eq. (\ref{eq7}) is differentiable and tractable, inferring GradCosine and gradient norm in the sample-wise manner can be very time-consuming especially for networks trained on large datasets such as ImageNet.
Moreover, using gradient-based methods to solve Eq. (\ref{eq7}) requires deriving the second-order gradient with respect to the network parameters, which leads to unbearable memory consumption in the sample-wise manner.
To this end, we introduce the batch-wise relaxation.

\iffalse
\subsection{Gradient Norm Constraints}
\label{s51}
To address the intractable term $\alpha/\beta$, we add another constraint for preventing $\beta$ from being too small.
Specifically, instead of maximizing the intractable term $\beta=\min\limits_i(||\theta^*_i||_2)$, we maximize the norm of every sample-wise optimum so that they all have similar norm values. According to the first-order approximation, this can be achieved by maximizing the sample-wise gradient norms.
Since the maximization will be performed under the original norm constraint in Eq. (\ref{eq6}), it ends up with $\alpha$ and $\beta$ having similar values so results in the minimal value of $\alpha/\beta \rightarrow 1$. The new optimization objective can be denoted as follows:
\iffalse
\begin{equation}
\begin{aligned}
    &\mathop{\text{maximize}}\limits_M \quad \text{GC}(S, \theta_M) + \text{GN}(S, \theta_M),\\
    &\text{subject to}  \quad \max_i(||g_{i}||_2) \leq \gamma; i = 1,...,B,
    \label{eq7}

\end{aligned}
\end{equation}
\fi

where $\text{GN}(S, \theta_M)= \frac{1}{B}\sum_i ||g_i||_2 $ is the average of all the sample-wise gradient norms at the initialization point as shown in Line 4 in Algrithm (\ref{alg1}).
\fi

\begin{algorithm}[t]
	\small
\caption{
Batch GradCosine (B-GC) and batch gradient norm (B-GN)
}
\label{alg2}
\begin{algorithmic}[1]
	\small
\REQUIRE Initialized network parameters $\theta_0$, a batch of samples $S = \{(x_i,y_i)\}_{i\in[B]}$, the number of sub-batches $D$, and overlap ratio $r$.
\FOR{$d=1$ to $D$}
\STATE Compute the batch-wise gradient: \\ $g_d \leftarrow \frac{1}{N}\sum_{j\in S_d} \nabla_\theta l(f_\theta(x_j), y_j)|_{\theta_0}$
\ENDFOR
\STATE Compute $\textbf{B-GN}(S,\theta_0)$ and $\textbf{B-GC}(S,\theta_0)$ with batch-wise gradients following Lines (4-6) in Algorithm \ref{alg1};
\STATE Output \textbf{B-GN} and \textbf{B-GC}.

\end{algorithmic}
\end{algorithm}

\subsection{Generalizing to Batch Settings}
To reduce the time and space complexity of gradient calculation, we further generalize GC and GN from the sample-wise manner into a more efficient batch setting. Specifically, for a mini-batch of training samples $S = \{(x_i,y_i)\}_{i\in[B]}$, we split it into $D$ sub-batches $S_1,...,S_d,...,S_D$ with an overlap ratio of $r$ that is used to relieve the violation of the sample-wise setting. The number of samples in each sub-batch is $N=\lceil \frac{B}{D-r} \rceil$, and each sub-batch can be denoted as:
\begin{equation}

    S_d = \{(x_i,y_i)\}_{i=N(d-1)(1-r)+1,...,N(d-1)(1-r)+N},
    \label{eq8}
\end{equation}
 where $d=1,...,D$. The sample-wise input can be regarded as the special case of the batch split when setting $D=B, r=0$. We further illustrate some split examples in Figure \ref{fig2}. We empirically find that making the sub-batches overlapped with each other stabilize the optimization and results in better initialization. As shown in Figure \ref{fig2}, the test accuracies of the models trained using our initialization with different batch settings further confirm this practice.

\begin{figure}[t]
\vspace{2mm}
\begin{center}
\centerline{\includegraphics[width=\columnwidth]{fig2_nips.pdf}}
\vspace{-1mm}
\caption{An illustration of sample-wise inputs and batch-wise inputs with different batch settings. The example experiment is conducted on CIFAR-10 with ResNet-110.}
\label{fig2}
\end{center}
\vskip -0.1in
\end{figure}

The GradCosine and gradient norm under the batch setting are illustrated in Algorithm \ref{alg2}. The main difference is that, in the batch setting, the gradient is computed as the average of gradients in a sub-batch. Our optimization problem in the batch setting can be formulated as:
\begin{equation}
\begin{aligned}
    &\mathop{\text{maximize}}\limits_M \quad \text{B-GC}(S, \theta_M) + \text{B-GN}(S, \theta_M),\\
    &\text{subject to}  \quad \max_i(||g_{i}||_2) \leq \gamma,\ \ i = 1,...,D,
    \label{eq9}

\end{aligned}
\end{equation}
where B-GC and B-GN refer to the batch-wise GradCosine and gradient norm as calculated in Algorithm \ref{alg2}.

\subsection{Main Algorithm}

\begin{algorithm}[t]
\caption{
Neural Initialization Optimization
}
\label{alg3}
\begin{algorithmic}[1]
	\small
\REQUIRE Initialized parameters $\theta_0\in\mathbb{R}^m$, learning rate $\tau$ for scale coefficients $M$, upper bound of gradient norm $\gamma$, lower bound of the scale coefficients $\underline\alpha=0.01$, total iterations $T$, batch size $B$, the number of sub-batches $D$, and overlap ratio $r$.
\STATE $M^{(1)}\leftarrow \textbf{1} $, where $\textbf{1}$ denotes the all-ones vector in $\mathbb{R}^m$;
\FOR{$t=1$ to $T$}
\STATE Sample $S_t$ from the training set;
\STATE Compute $g_i$, $\text{B-GC}(S_t, \theta_{M^{(t)}})$, and $\text{B-GN}(S_t, \theta_{M^{(t)}})$ by Algorithm \ref{alg2};
\STATE Compute the $g_{max}=\max_i(||g_{i}||_2)$, $i = 1,...,D$;
\IF{$g_{max}>\gamma$}
\STATE $M^{(t+1)} \leftarrow M^{(t)}-\tau \nabla_{M^{(t)}}\text{B-GN}
\ELSE{}
\STATE $M^{(t+1)} \leftarrow M^{(t)}+\tau \nabla_{M^{(t)}}(\text{B-GC}+\text{B-GN})$
\ENDIF
\STATE Clamp $M^{(t+1)}$ using $\underline\alpha$;
\ENDFOR
\STATE  Output the rectified initialization parameters $\theta_M{^{(T)}}$.
\end{algorithmic}
\end{algorithm}

The final neural initialization optimization is illustrated in Algorithm \ref{alg3}. It rectifies the initialized parameters by gradient descent to solve the constrained optimization problem in Eq. (\ref{eq9}).

Concretely, we iterate for $T$ iterations. At each iteration, a random batch data $S_t$ is sampled from the training set, and divided into sub-batches according to $D$ and $r$. We calculate B-GN and B-GC by Algorithm \ref{alg2}. If the maximal gradient norm is larger than the predefined upper bound $\gamma$, we minimize the averaged batch gradient norm to avoid gradient explosion at initialization. When the constraint is satisfied, we maximize the GradCosine (B-GC) and gradient norm (B-GN) objectives simultaneously, in order to minimize the upper bound of the quantity $\Theta$ in Eq. (\ref{relax_theta}), which intimately corresponds to not only a lower training and generalization error, but also a better training dynamic.

\section{Experiments}

\subsection{Initialization for CNN}
\label{s61}
\textbf{Dataset and Architectures.}
We validate our method on three widely used datasets including CIFAR10/100 \cite{krizhevsky2009learning} and ImageNet \cite{deng2009imagenet}. We select three kinds of convolution neural networks including ResNet \cite{he2016deep}, DenseNet \cite{huang2017densely}, and WideResNet \cite{zagoruyko2016wide}.
On CIFAR10/100, we adopt ResNet110, DenseNet100, and the 28-layer Wide ResNet with Widen Factor 10 (WRN-28-10) as three main architectures for evaluation. To further show that our initialization method helps training dynamic for better training stability, we also conduct experiments on the same architectures but remove the batch normalization (BN) \cite{ioffe2015batch} layers. Moreover, to illustrate that our method can be extensible to large-scale benchmarks, we test our proposed NIO on ImageNet using ResNet-50.
The detailed settings for different architectures and datasets are described in Appendix \ref{ap2}.

\begin{table}
	\small
	\vspace{-1mm}
	\begin{minipage}[th!]{\textwidth}

		\begin{minipage}[t]{0.48\textwidth}
			\renewcommand\arraystretch{0.9}
			\setlength{\tabcolsep}{2.8pt}
			\centering

			\caption{Test accuracies of three architectures on CIFAR-10. Best
				results are marked in bold.}
\begin{tabular}{l|ccc}
	\toprule
	Model & ResNet-110 & DenseNet-100 & WRN-28-10 \\
	\midrule
	& \multicolumn{3}{c}{w/ BN} \\
	\midrule
	Kaiming    & 95.53$\pm$ 0.19& 95.75$\pm$ 0.13& 97.27$\pm$ 0.27 \\
	Warmup     & 95.56$\pm$ 0.12& 95.73$\pm$ 0.27& 97.30$\pm$ 0.18\\
	Metainit   & 95.45$\pm$ 0.33& 95.75$\pm$ 0.11& 97.26$\pm$ 0.17 \\
	Gradinit   & 95.81$\pm$ 0.29& 95.77$\pm$ 0.22& 97.34$\pm$ 0.15 \\
	NIO  & \textbf{96.13}$\pm$ 0.16& \textbf{96.07}$\pm$ 0.18& \textbf{97.50}$\pm$ 0.21\\
	\midrule
	& \multicolumn{3}{c}{w/o BN} \\
	\midrule
	Kaiming    & 94.83$\pm$ 0.24& 94.78$\pm$ 0.16&  97.15$\pm$ 0.33 \\
	Warmup     & 94.75$\pm$ 0.21& 94.86$\pm$ 0.15&  97.20$\pm$ 0.18\\
	Metainit   & 94.79$\pm$ 0.15& 95.15$\pm$ 0.19&  97.27$\pm$ 0.25 \\
	Gradinit   & 95.12$\pm$ 0.16& 95.31$\pm$ 0.37& 97.12$\pm$ 0.13        \\
	NIO       & \textbf{95.27}$\pm$ 0.19& \textbf{95.62}$\pm$ 0.18& \textbf{97.35}$\pm$ 0.19\\
	\bottomrule
\end{tabular}

			\vspace{2mm}
			\label{tab1}
		\end{minipage}
		\quad \
		\begin{minipage}[t]{0.48\textwidth}
			\renewcommand\arraystretch{0.9}
			\setlength{\tabcolsep}{2.8pt}

			\centering

			\caption{Test accuracies of three architectures on CIFAR-100. Best
				results are marked in bold.}
\begin{tabular}{l|ccc}
	\toprule
	Model & ResNet-110 & DenseNet-100 & WRN-28-10 \\
	\midrule
	& \multicolumn{3}{c}{w/ BN} \\
	\midrule
	Kaiming    & 74.54$\pm$ 0.21& 76.58$\pm$ 0.23& 81.40$\pm$ 0.28 \\
	Warmup     & 74.63$\pm$ 0.33& 76.60$\pm$ 0.32& 81.45$\pm$ 0.25\\
	Metainit   & 74.32$\pm$ 0.26& 76.23$\pm$ 0.28& 81.46$\pm$ 0.20 \\
	Gradinit   & 75.40$\pm$ 0.17& 76.14$\pm$ 0.21& 81.35$\pm$ 0.31         \\
	NIO       & \textbf{75.72}$\pm$ 0.15& \textbf{76.86}$\pm$ 0.26& \textbf{81.83}$\pm$ 0.20\\
	\midrule
	& \multicolumn{3}{c}{w/o BN} \\
	\midrule
	Kaiming    & 73.03$\pm$ 0.23& 71.27$\pm$ 0.25& 79.28$\pm$ 0.26 \\
	Warmup     & 73.10$\pm$ 0.14&  71.50$\pm$ 0.24& 79.58$\pm$ 0.13\\
	Metainit   & 72.60$\pm$ 0.17&  71.68$\pm$ 0.37& 79.15$\pm$ 0.24 \\
	Gradinit   & 72.06$\pm$ 0.31& 71.33$\pm$ 0.21& 79.64$\pm$ 0.23         \\
	NIO       & \textbf{73.17}$\pm$ 0.15& \textbf{72.79}$\pm$ 0.22& \textbf{79.76}$\pm$ 0.26\\
	\bottomrule
\end{tabular}

			\vspace{2.5mm}
			\label{tab2}
		\end{minipage}
	\end{minipage}
	\vspace{-3mm}
\end{table}

\iffalse
\begin{table}[t]
\caption{Best test accuracy over all epochs for three models on CIFAR-10. Best
results in each group are in bold.}
\label{tab1}
\vskip 0.15in
\begin{center}
\begin{small}
\begin{sc}
\setlength{\tabcolsep}{1.4mm}{
\begin{tabular}{l|ccc}
\toprule
Model & ResNet-110 & DenseNet-100 & WRN-28-10 \\
\midrule
& \multicolumn{3}{c}{w/ BN} \\
\midrule
Kaiming    & 95.45$\pm$ 0.28& 95.71$\pm$ 0.24& 97.27$\pm$ 0.27 \\
Warmup     & 95.56$\pm$ 0.12& 95.73$\pm$ 0.27& 97.30$\pm$ 0.18\\
Metainit   & 95.74$\pm$ 0.43& 95.75$\pm$ 0.11& 97.30$\pm$ 0.27 \\
Gradinit   & 95.81$\pm$ 0.29& 95.75$\pm$ 0.21& 97.33$\pm$ 0.35 \\
Ours       & \textbf{96.13}$\pm$ 0.16& \textbf{96.07}$\pm$ 0.18& \textbf{97.51}$\pm$ 0.31\\
\bottomrule
& \multicolumn{3}{c}{w/o BN} \\
\midrule
Kaiming    & 94.83$\pm$ 0.24& 94.98$\pm$ 0.29&  97.15$\pm$ 0.33 \\
Warmup     & 94.85$\pm$ 0.21& 95.05$\pm$ 0.14&  97.20$\pm$ 0.18\\
Metainit   & 94.90$\pm$ 0.13& 95.15$\pm$ 0.19&  97.27$\pm$ 0.25 \\
Gradinit   & 95.13$\pm$ 0.11& 95.31$\pm$ 0.37& 97.32$\pm$ 0.42        \\
Ours       & \textbf{95.32}$\pm$ 0.16& \textbf{95.62}$\pm$ 0.18& \textbf{97.32}$\pm$ 0.24\\
\bottomrule
\end{tabular}}
\end{sc}
\end{small}
\end{center}
\vskip -0.1in
\end{table}

\begin{table}[t]
\caption{Best test accuracy over all epochs for three models on CIFAR-100. Best
results in each group are in bold.}
\label{tab2}
\vskip 0.15in
\begin{center}
\begin{small}
\begin{sc}
\setlength{\tabcolsep}{1.4mm}{
\begin{tabular}{l|ccc}
\toprule
Model & ResNet-110 & DenseNet-100 & WRN-28-10 \\
\midrule
& \multicolumn{3}{c}{w/ BN} \\
\midrule
Kaiming    & 74.54$\pm$ 0.21& 76.58$\pm$ 0.23& 81.40$\pm$ 0.28 \\
Warmup     & 74.63$\pm$ 0.33& 76.60$\pm$ 0.32& 81.45$\pm$ 0.25\\
Metainit   & 75.22$\pm$ 0.46& 76.63$\pm$ 0.23& 81.76$\pm$ 0.21 \\
Gradinit   & 75.40$\pm$ 0.17& 76.63$\pm$ 0.28& 81.81$\pm$ 0.25         \\
Ours       & \textbf{75.72}$\pm$ 0.15& \textbf{77.29}$\pm$ 0.13& \textbf{81.83}$\pm$ 0.20\\
\bottomrule
& \multicolumn{3}{c}{w/o BN} \\
\midrule
Kaiming    & 73.03$\pm$ 0.23& 71.27$\pm$ 0.25& 79.28$\pm$ 0.26 \\
Warmup     & 73.10$\pm$ 0.14&  71.50$\pm$ 0.24& 79.58$\pm$ 0.13\\
Metainit   & 72.90$\pm$ 0.41&  71.68$\pm$ 0.37& 80.68$\pm$ 0.25 \\
Gradinit   & 72.73$\pm$ 0.12& 71.68$\pm$ 0.23& 80.78$\pm$ 0.24         \\
Ours       & \textbf{73.21}$\pm$ 0.11& \textbf{72.80}$\pm$ 0.12& \textbf{80.78}$\pm$ 0.21\\
\bottomrule
\end{tabular}}
\end{sc}
\end{small}
\end{center}
\vskip -0.2in
\end{table}
\fi

\textbf{Experiment Results on CIFAR-10/100.}
We select four different initialization methods for comparison: (1) Kaiming Initialization \cite{he2015delving}; (2) First train the network for one epoch with a linear warmup learning rate, denoted as Warmup; (3) MetaInit \cite{dauphin2019metainit}; and (4) GradInit \cite{zhu2021gradinit}. (3) and (4) are learning-based initialization as ours. We re-implement their methods using the code provided in \cite{zhu2021gradinit}. For MetaInit, GradInit, and our proposed NIO, the initialization is rectified based on the Kaiming initialized parameters. After initialization, we train these models for 500 epochs with the same training setting. Each model is trained four times with different seeds. We report the average and standard deviation numbers of the accuracies on the test set.

As shown in Table \ref{tab1} and \ref{tab2}, compared with the Kaiming initialization, we observe that MetaInit and GradInit do not achieve improvement for some cases. As a comparison, our proposed NIO consistently improves upon the Kaiming initialization on both CIFAR-10 and CIFAR-100 with and without BN. Especially for ResNet-110-BN and DenseNet-100-noBN, the performance improvements upon the Kaiming initialization are more than 0.5\% on CIFAR-10 and 1.0\% on CIFAR-100. Compared with GradInit, we also achieve better results by more than 1.0\% for ResNet-110-noBN and DenseNet-110-noBN on CIFAR-100.

We also make some empirical comparisons between the Kaiming initialization and our NIO with ResNet-110 on CIFAR-10. As shown in Figure \ref{fig3}, compared with the Kaiming initialization, NIO enjoys significantly smaller ratios of the maximal gradient norm to the minimal one, and larger cosine similarities of sample-wise gradients. Therefore, after NIO, the gradient norm ratio is reduced towards 1.0, while the cosine similarity is improved towards 1.0, which indicates that the quantity in Eq. (\ref{eq2}) is minimized, and the sample-wise gradients are close with respect to length and direction.

\begin{figure}[t]
\begin{center}
\centerline{\includegraphics[width=\columnwidth]{fig3_nips_new.pdf}}
\vspace{-1mm}
\caption{Empirical comparisons between the Kaiming initialization (red) and our NIO (blue) using ResNet-110 on CIFAR-10. (a) and (b) show the ratio of the maximal sample-wise gradient norm to the minimal one. (c) and (d) depict the averaged pair-wise cosine similarities between sample-wise gradients. (a) and (c) are calculated for each layer with a random batch. (b) and (d) are calculated for all parameters using 100 randomly sampled batches. Zoom in for a better view.}

\label{fig3}
\end{center}
\vskip -0.2in

\end{figure}

\textbf{Ablation Studies on Batch Setting.} We further test the sensitivity of our method to the batch setting hyper-parameters in Eq. (\ref{eq8}), the number of sub-batches $D$ and overlap ratio $r$. We show the model performances and the time and space complexities of NIO with ResNet-110 on CIFAR-100 using different $D$ and $r$ in Tabel \ref{ablation}. The performances do not vary significantly with the change of $D$ and $r$. The worst accuracy (when $D=2$ and $r=0$ ) 75.56\% in Table \ref{ablation} is still higher than the Kaiming initialization result 74.54\% by 1\%. As $D$ increases, the time and memory consumption of NIO also grows, because more batch gradients need to be  calculated by back propagation. When $D$ is small, a larger $r$ should be adopted to relieve the violation of the sample-wise hypothesis.

\textbf{Experiment Results on ImageNet.} To verify that our method can be extensible to large-scale task, we further evaluate our approach on the ImageNet dataset. Using different initialization methods, we train ResNet-50 for 100 epochs with a batchsize of 256. Alll models are trained under the same training setting. As shown in Table \ref{tab3}, we improve upon the Kaiming initialization by 0.3\%. We also re-implement GradInit using the default configurations in their paper \cite{zhu2021gradinit}, and have a performance close to the Kaiming initialization baseline.  Moreover, NIO takes the shortest initialization time compared with the other two learning-based initialization methods, MetaInit and GradInit. It is because only 100 iterations are needed for NIO in our implementation, while MetaInit and GradInit require 1000 and 2000 iterations, respectively, as their default settings \cite{dauphin2019metainit,zhu2021gradinit}. Compared with the training time, the initialization time of NIO is negligible.

\subsection{Initialization for Transformer}
\label{s62}
Vision Transformer architectures have been popular and proven to be effective for many vision tasks \cite{dosovitskiy2020image,touvron2021training,liu2021swin,xu2021vitae}. But Transformer architectures suffer from unstable training due to the large gradient variance \cite{liu2019variance,zhuang2020adabelief} in the Adam optimizer \cite{kingma2015adam}, so have to resort to the warmup trick, which adopts a small learning rate at initialization to stabilize the training process. In order to test the effectiveness of NIO to improve the training stability, we perform NIO for Swin-Transformer \cite{liu2021swin} with and without warmup on ImageNet. Detailed training and initialization settings are described in Appendix \ref{ap2}. As shown in Table \ref{tab4}, when warmup is not enabled for training, both Kaiming and Truncated normal initialized models cannot converge. As a comparison, the model with NIO achieves a top-1 accuracy of 80.9\%, which is very close to the standard training result with warmup. These results indicate that NIO is able to produce a better initialization agnostic of architecture and dataset.

\iffalse
\textbf{Comparison Results.}
We adopt four experimental settings: (1-2) randomly initialized transformer with or without warm up training; (3-4) transformer initialized by our method and trained with or without warm up. All other training settings remain unchanged. We follow \cite{liu2021swin} to set the warm up epoch as 20 and train the transformer for 300 epochs with 128 batch size. The results are shown in Table (\ref{tab4}). Note that, our implementation is relatively lower than its best performance. Yet some observations can still be made: Firstly, similar to other architectures, our initialization improves the transformer under both settings compared with its original initialization method. Secondly, we find that for the randomly initialized network, training without warm up results in a sharp performance drop. In contrast, our initialized network can stabilize the training and lead to results that are only slightly worse than the baseline. Finally, we visualize the training dynamic of four experimental settings in Figure (\ref{fig4}). As can be seen, our initialization indeed improves the convergence speed at starting epoch.
\fi

\begin{table}[t]
	\small

	\begin{minipage}[th!]{\textwidth}

		\begin{minipage}[t]{0.51\textwidth}
			\renewcommand\arraystretch{1.1}
			\setlength{\tabcolsep}{3pt}
			\centering

			\caption{Model performance and complexity of ResNet-110 on CIFAR-100 with different batch settings. NIO is performed with a batchsize of 128. A smaller number of sub-batches $D$ leads to faster speed and less memory consumption, and almost does not harm the final performance. ``Time'' refers to the time consumed for each iteration in NIO.}
			\begin{tabular}{c|cccc|c|c}
				\hline
				\multirow{2}{*}{$D$}& \multicolumn{4}{c|}{Accuracy (\%)}& \multirow{2}{*}{Time} & \multirow{2}{*}{GPU Mem.} \\\cline{2-5}
				&$r$=0 & $r$=0.2 & $r$=0.4 & $r$=0.6  &  &  \\\hline
				2&  75.56 & 75.71 & \textbf{75.96} &75.72&\textbf{3.75}s &\textbf{5007}M  \\
				3& 75.60 & \textbf{76.00}  & 75.97 & 75.76 &4.07s&6001M      \\
				4& 75.58 & \textbf{76.03} & 75.71 & 75.88&4.68s&7067M \\\hline
			\end{tabular}

			\label{ablation}
		\end{minipage}
		\quad \
		\begin{minipage}[t]{0.51\textwidth}
			\renewcommand\arraystretch{0.87}
			\setlength{\tabcolsep}{5.0pt}

			\centering

			\caption{Top-1 accuracy of ResNet-50 on ImageNet. The results of FixUp and MetaInit are taken from their papers \cite{zhang2019fixup,dauphin2019metainit}, and the other methods are re-implemented by us under the same training setting. FixUp removes the BN layers as their default and the other models are trained with BN layers. }

		\label{tab3}

		\begin{tabular}{l|c|c}
			\toprule
			Method & Accuracy (\%)  & Init. / Train time \tablefootnote{``Train time'' is tested on an NVIDIA A100 server with a batchsize of 256 among 8 GPUs. ``Init. time'' of MetaInit and GradInit is tested for 1000 and 2000 iterations, respectively, according to the default configurations in their papers \cite{dauphin2019metainit,zhu2021gradinit}. NIO is performed for 100 iterations so enjoys a fast initialization process.}\\
			\midrule
			Kaiming \cite{he2015delving}  & 76.43$\pm$0.05 &  - / 8.5h \\
			FixUp \cite{zhang2019fixup} & 76.0 &  - / 8.5h \\
			MetaInit \cite{dauphin2019metainit} & 76.0  & 0.81h  / 8.5h \\
			GradInit  \cite{zhu2021gradinit} & 76.50$\pm$0.05 & 0.21h / 8.5h\\
			NIO (ours) & \textbf{76.71}$\pm$0.07 & 0.03h / 8.5h \\
			\bottomrule
		\end{tabular}

	\end{minipage}
\end{minipage}
\end{table}

\begin{table}[t]
\vspace{1mm}
	\caption{Top-1 accuracy of Swin-Transformer (tiny) on ImageNet with and without warmup. ``fail'' represents that the model cannot converge. Warmup takes up the first 20 epochs when enabled.}
\vspace{-2mm}
	\label{tab4}
	\begin{center}
		\begin{small}

			\begin{tabular}{lcccc}
				\toprule
				& Kaiming & TruncNormal & GradInit\tablefootnote{GradInit did not perform on SwinTransformer in their paper. We adopt their initialization settings for ResNet-50 to perform GradInit on SwinTransformer and then train the model.} & NIO (ours) \\
				\midrule
				w/ warmup  &79.4 & 81.3 & 80.4 & \textbf{81.3} \\
				w/o warmup & fail & fail & 79.9 & \textbf{80.9} \\
				\bottomrule
			\end{tabular}

		\end{small}
	\end{center}
	\vspace{-3mm}
\end{table}

\iffalse
\begin{figure}[t]
\vskip 0.2in
\begin{center}
\centerline{\includegraphics[width=0.5\columnwidth]{swinmac.pdf}}
\caption{Comparing the convergence of Kaiming initialization and our initialization on ImageNet for Swin-Transformer. Our method shows faster and more stable convergence on the starting epochs with and without warm up than Kaiming initialization.}
\label{fig4}
\end{center}
\vskip -0.2in
\end{figure}
\fi

\subsection{Discussions}
\label{dicuss}

\textbf{Why not Gradient Variance?} As indicated by Eq. (\ref{eq2}) and Figure \ref{fig3}, the aim of NIO is to reduce the variability of sample-wise gradients.
But why do we not directly minimize the gradient variance or just the pair-wise Euclidean distances? We point out that gradient variance is highly sensitive to gradient norm. Directly minimizing gradient variance would lead to a trivial gradient norm near zero, which disables the training. So the ability of reducing gradient variance while keeping gradient norm at a proper range is one crucial property of our proposed NIO.

\textbf{Limitations and Societal Impacts.} Similar to \cite{zhu2021gradinit}, the choice of $\gamma$ has not been determined from a theoretical point yet. Calculating GradCosine accurately requires much memory consumption for large models, so it needs the batch setting relaxation. Besides, performing NIO without a dataset will be also a promising breakthrough that deserves future exploration. Our study is general and not concerned with malicious practices with respect to privacy, public health, fairness, \emph{etc}. The proposed algorithm is lightweight and does not bring much environment burden.

\section{Conclusion}

In this paper, we propose a differentiable quantity with theoretical insight to evaluate the initial state of a neural network. Based on our sample-wise landscape analysis, we show that maximizing the quantity with the maximal gradient norm upper bounded is able to improve both training dynamic and model performance. Accordingly, we develop an initialization optimization algorithm to rectify the initial parameters. Experimental results show that our method is able to automatically produce a better initialization for variant architectures to improve the performance on multiple datasets with negligible cost. It also helps a Transformer architecture train stably even without warmup.

\begin{ack}
	Z. Lin was supported by the NSF China (No.s 62276004 and 61731018), NSFC Tianyuan Fund for Mathematics (No. 12026606), and Project 2020BD006 supported by PKU-Baidu Fund.
\end{ack}

\small

redpajama_set_name: RedPajamaArXiv


text:
Film, TV, Gaming, Music, Comics.
10 Actors You Didn't Know Were Coming Back For Movie Sequels
There are few things more exciting than discovering that a favourite actor is set to return for another franchise instalment.
Whether it's a resilient final girl appearing for one more scare in a classic slasher series, a loveable love interest showing their face again for another superhero sequel, or a couple of favourites who haven't been seen together onscreen in decades coming back, nothing confirms the legitimate excitement of a new instalment like a returning fan favourite.
That said, sometimes it seems as if there's no way certain characters will return, regardless of the actor's wishes. Some of these franchise favourites were seemingly offed at the end of their last instalments, whilst others appeared to have retired from the world of acting altogether, never mind retiring individual characters.
With these myriad circumstances in mind, this list is here to bring you ten actors who you weren't even aware were set to return for forthcoming sequels.
Film, Scream, Neve Campbell
kelly kelly royal rumble 2018 bad casting

redpajama_set_name: RedPajamaCommonCrawl


text:
How to Register for the CCMEP™ Exam
The National Commission for Certification of CME Professionals (NC-CME) has developed the first credentialing program exclusively designed for individuals employed in the broad field of continuing medical education (CME). Since its inception in June 2008, nearly 450 persons have earned the designation of Certified CME Professional (CCMEP™). When certifications earned by the 2008 Classes expired, 94% of eligible candidates earned their Extension of Certification, a clear indication that the CCMEP credential is retaining its value over time.
Schroeder Measurement Technologies, an independent test development company, assisted NC-CME volunteers in conducting an initial job analysis and in facilitating the item writing and validation of the ground-breaking certification exam. Exam Forms 02 and 03 were beta tested, validated, and released in 2009 and 2011. The certification exam comprises more than 150 test items that have been validated psychometrically to determine minimum competency in five domains of knowledge, skills, and abilities that define the CME profession. A Practice Exam, representative of topics described in the content outline, was also created with the SMT test developers in 2010. The 45-item practice exam can be accessed online from any computer and is available 24/7.
The NC-CME certification program desvenlafaxine.medsstore.xyz is designed to benefit the public by offering a way to verify the knowledge and skills of CME staff. The program allows employers to monitor and document their ability to be compliant with national guidelines developed by groups such as the Accreditation Council for CME (ACCME), Centers for Medicare and Medicaid Services (CMS), Food and Drug Administration (FDA), and Health and Human Services Office of the Inspector General (HHS OIG).
Successful test takers must achieve a minimum scaled passing score of 600, where the overall range of scores is 200 to 1000. In certification exams of this type, the primary goal is a pass/fail discrimination. The CCMEP credential is awarded to candidates who document an acceptable level of education and experience in the CME field and achieve the minimum score on the examination. Test items capture a generalist view of the profession and assess what CME professionals need to know in order to reflect the highest standards in their work.
Newly designated CCMEPs receive framed certificates and cloisonné lapel pins at ceremonies held in October and January each year in conjunction with national meetings of CME professionals. Volunteers who have participated in NC-CME activities and donors who have contributed to the CCMEP program are also acknowledged at these celebrations. The NC-CME exhibit booth is staffed by CCMEPs who distribute literature and answer questions about the CCMEP program and its network of volunteers.
The certification exam is a computerized secure exam available at more than 200 testing centers throughout the US and Canada during quarterly testing periods: March 13-31, June 13-30, September 13-30, and December 1-31. Registration cut-off dates are the last day of the month preceding each testing period. Candidates may register online 24/7 for future testing periods. Names of individuals who successfully pass the CCMEP exam and also those who earn an Extension of Certification (EOC) are posted in the searchable "National Registry of Certified CME Professionals" on the NC-CME website.
Requirements and benefits to candidates and their employers are outlined under the tab "CERTIFICATION PROGRAM" at NCCME.org
NC-CME is a private, non-profit, tax-exempt organization created in 2006 by an independent group of volunteers committed to serving the public by creating a standard of competence for persons who create, deliver, and/or support CME programs. NC-CME is incorporated in New Mexico and operates under Section 501(c) (3) of the Internal Revenue Code of the United States. For further information go to NCCME.org or email info@nccme.org [rev Jan 1, 2013]


Stage requirement plan – can be sent in advance for engineers.
Standard Road Gig set up – Mackie 1.8KW p. a.
For small acoustic gigs – Trace Elliot TA100's – 2 by 100Watts .
4 – Mackie SRM 450 watt speakers. 2/3 upfront & 1/2 fold back.
2- LR Baggs para acoustic DI pre amp boxes, 1 – Boss Di pre amp – various pedals.
1 – vocal mic pre amp.
1 – Boss 20XL loop pedal.
2 – Samsung CO2 matching acoustic instrument mics.
2 – Senheiser vocal mics.
Stomp Box ; in progress, made from one of the tool box drawers plus parts from the old family motorbike shed. Our shed was so overloaded with old classic British motor bike parts. I built up my first motorbike after diving into a swamp at the age of 12 and dragging out a 125cc BSA Bantam with a rope and grapple . If the stage ain't wooden I need to stomp on real wood to get the sound of an old beaten wooden porch. The number plate and badge is off a 650 BSA motorbike which features in the song Fireball Highway.


The Alligator Snapping Turtle is the world's largest and most dangerous species of freshwater turtle, with one specimen found in 1948 being said to be as large as a kitchen table. Found throughout the Mississippi Basin in the southern USA, the Alligator Snapping Turtle is one of the most ferocious predators in it's natural environment.
They spend the daylight hours buried in the mud on the floors of rivers and lakes where they wait ready with their mouths open to ambush their prey. The Alligator Snapping Turtle has a thread of skin on the end of it's tongue which is used to lure unsuspecting fish right into it's mouth, before it rapidly snaps it's strong, sharp jaws shut.
As the Alligator Snapping Turtle spends much of it's time sitting around waiting for prey to pass, they use up very little oxygen and are therefore able to remain submerged in the thick mud for up to an hour. In fact, although females leave the water to lay their eggs in the sand, male Alligator Snapping Turtles may never venture out at all.
This formidable and prehistoric predator is easily able to defend and protect itself in the face of danger from other animals and therefore has no natural predators within it's native environment. However, the Alligator Snapping Turtle is preyed upon by people who hunt them for both their meat and their shells.
The Alligator Snapping Turtle can live for up to 100 years and is a slow-growing and maturing creature. Increased levels of hunting along with habitat loss or degradation throughout much of their natural range, has led to drastic declines in population numbers. The Alligator Snapping Turtle is now listed as a Threatened species and is protected in most areas.


text:
<?php

/**
 * Ruckusing
 *
 * @category  Ruckusing
 * @package   Ruckusing_Task
 * @author    Cody Caughlan <codycaughlan % gmail . com>
 * @link      https://github.com/ruckus/ruckusing-migrations
 */

/**
 * Ruckusing_Task_Base
 *
 * @category Ruckusing
 * @package  Ruckusing_Task
 * @author   Cody Caughlan <codycaughlan % gmail . com>
 * @link      https://github.com/ruckus/ruckusing-migrations
 */
class Ruckusing_Task_Base
{
    /**
     * the framework
     *
     * @var Ruckusing_FrameworkRunner
     */
    private $_framework;

    /**
     * the adapter
     *
     * @var Ruckusing_Adapter_Base
     */
    private $_adapter;

    /**
     * the migration directory
     *
     * @var string
     */
    protected $_migrationDir;

    /**
     * Creates an instance of Ruckusing_Task_Base
     *
     * @param Ruckusing_Adapter_Base $adapter The current adapter being used
     *
     * @return Ruckusing_Task_Base
     */
    public function __construct($adapter)
    {
        $this->setAdapter($adapter);
    }

    /**
     * Get the current framework
     *
     * @return object
     */
    public function get_framework()
    {
        return $this->_framework;
    }

    /**
     * Set the current framework
     *
     * @param Ruckusing_FrameworkRunner $fw the framework being set
     */
    public function set_framework($fw)
    {
        if (!($fw instanceof Ruckusing_FrameworkRunner)) {
            throw new Ruckusing_Exception(
                    'Framework must be instance of Ruckusing_FrameworkRunner!',
                    Ruckusing_Exception::INVALID_FRAMEWORK
            );
        }
        $this->_framework = $fw;
    }

    /**
     * set adapter
     *
     * @param Ruckusing_Adapter_Base $adapter the current adapter
     *
     * @return Ruckusing_Task_Base
     */
    public function setAdapter($adapter)
    {
        if (!($adapter instanceof Ruckusing_Adapter_Base)) {
            throw new Ruckusing_Exception(
                    'Adapter must be implement Ruckusing_Adapter_Base!',
                    Ruckusing_Exception::INVALID_ADAPTER
            );
        }
        $this->_adapter = $adapter;

        return $this;
    }

    /**
     * Get the current adapter
     *
     * @return object
     */
    public function get_adapter()
    {
        return $this->_adapter;
    }

    /**
     * set migration directories
     *
     * @param string $migrationDir Directory of migrations
     *
     * @return Ruckusing_Task_Base
     */
    public function setMigrationsDirectory($migrationDir)
    {
        $this->_migrationDir = $migrationDir;

        return $this;
    }

}

redpajama_set_name: RedPajamaGithub


This is an important call from Regions Bank. Please return the call at 1-888-231-5461. Our hours of operations are Monday through Thursday from 7:00 AM until 8:00 PM Friday from 7:00 AM until 6:00 PM and Saturday from 8:00 AM to 12:00 PM Central Standard Time. This is an attempt to collect a debt and any information obtained maybe used for that purpose. Thanks bank(?) with region(?).
This is an important call from Regions Bank. Our business hours are Monday through Thursday 7:00 AM to 8:00 PM Friday 7:00 AM to 4:30 PM and Saturday 8:00 AM to 12:00 PM Central Time. Please return our call at 1-800-289-6720 during our regular business hours. Once again this is an important call from Regions Bank. Please return our call at 1-800-289-6720. This communication may be deemed an attempt to collect a debt and any information obtained could be used for that purpose.
This is an important call from Regions Bank. Our business hours are Monday through Thursday 7:00 AM to 8:00 PM Friday 7:00 AM to 6:00 PM and Saturday 8:00 AM to 12:00 PM Central Time. Please return our call at 1-800-333-4104 during our regular business hours. This communication may be deemed an attempt to collect a debt and any information obtained could be used for that purpose. I thought I heard from halo(?) halo mama how you doing.
This is an important call from Regions Bank our business hours are Monday through Thursday 7 a.m.. To 8 p.m.. Friday 7 a.m.. To 6 p.m.. And Saturday 8 a.m.. To 12 p.m.. Central Time. Please return our call at one 800-289-6730 during our regular business hours Regions Bank. Maybe a debt collector under applicable law this communication may be deemed an attempt to collect a debt and any information obtained may be used for that purpose.
This is an important call from Regions Bank. Our business hours are Monday through Thursday 7:00 AM to 8:00 PM Friday 7:00 AM to 6:00 PM and Saturday 8:00 AM to 12:00 PM Central Time. Please return our call at 1-800-333-4104 during our regular business hours regions Bank maybe a debt collector under applicable Law. This communication may be deemed an attempt to collect a debt and any information obtained maybe used for that purpose.


text:
Home Technology I'm not waiting to buy an iPhone 13 on Black Friday –...
I'm not waiting to buy an iPhone 13 on Black Friday – here's why you shouldn't either
It used to be heresy to suggest buying that shiny new iPhone in October. It is, after all, almost a full month before the traditional Black Friday iPhone deals land and offer up what's supposedly the best chance all year to score a bargain.
Delivery delays, back-order backlogs, and general stock issues, however, have plagued the new flagships since launch. It's left me questioning – is it actually worth waiting until Black Friday to get an iPhone 13?
The iPhone 13 family has had no shortage of fantastic deals already, and those backorder queues just seem to be getting longer with no signs of letting up. Maybe you should settle instead of braving a multiple month-long wait until you finally get your hands on that new upgrade.
Especially when several recent reports have suggested that Apple's starting to feel the squeeze in the global chip shortage. No one can hide from the fact – that's bad news if you don't have that iPhone 13 yet.
What normally happens
Cell phone deals in general have traditionally revolved around launch events (like this week's Google Pixel 6 deals) and big sales events like Black Friday or Cyber Monday – with the exception of the odd sale or two from the likes of Amazon or Best Buy.
It used to be the case, especially with iPhones, that lining up your upgrade with a launch window or Black Friday would be your best bet for securing a decent deal. In all truth, I expect this to be the case again when things return to normal. However, certain market factors and reports have led me to believe this year's Black Friday iPhone deals may be disrupted, or at least in shorter supply.
The issue: chip shortages
We've already seen the iPhone 13 on backorder at Verizon and many local Apple stores nationwide
The ongoing global chip and supply chain crisis isn't exactly breaking news – it's been widely reported on and discussed openly for several months now. In fact, it's positively old news for any who have braved the queues over the past year to attempt to secure a coveted PS5 or one of the latest graphics cards.
Phones, so far, have been one category of tech that's largely escaped the indignity of bare shelves and backorders. That could be set to change in the following months if recent reports are anything to go by, however.
Last week, Bloomberg broke the news that Apple is set to cut iPhone 13 production numbers by upwards of 11 percent. The company was set to manufacture 90 million handsets by the end of 2021, but according to 'people with knowledge of the matter' that figure has now been slashed by as many as 10 million.
Apple didn't comment on the news. However, it did state earlier in the year that it would 'would face supply constraints of the iPhone and iPad during the quarter that ended in September'. It's worth noting that's the period directly before Black Friday. If demand goes up over the next month (as we'd expect), then we could be seeing some very lengthy wait times for delivery indeed.
And, we've already seen the iPhone 13 on backorder at Verizon and many local Apple stores nationwide. The iPhone 13 Pro Max in particular is looking at an estimated delivery date of December 3rd via Verizon right now and November 26th via the official Apple store. Unless a large number of devices are somehow delivered by surprise it seems prudent to expect that delivery date to stretch out further from here onwards.
But, will I miss out on deals if I buy early?
Tough question. The short answer is yes. The long answer is, yes, but this year's Black Friday iPhone deals might not be much to shout about anyway.
Back in 2020, when the iPhone 12 was fresh off the presses, in good stock, and in high demand, carrier Black Friday deals weren't actually that great. Verizon and AT&T's maximum saving of up to $700 with a trade-in was essentially the same deal that was offered with a pre-order when these devices were first launched.
This week's iPhone 13 deals are actually better than what was served up on the iPhone 12 last year if you're considering the overall amount of savings on offer.
Both AT&T and Verizon, for example, are offering up to $800 off with a trade-in on the standard iPhone 13 (up to $1,000 off on the Pro), and Apple's maximum trade-in rebate of up to $790 is actually the highest we've ever seen from the retailer.
The kicker here is these deals aren't quite as good as those offered at the initial launch back in September. Existing Verizon customers, for example, could keep their line at launch and simply upgrade for free with an eligible trade-in. Right now you can only get $440 off as an upgrade.
It's hard to say whether 'big upgrade' deals will be back on the iPhone 13 over Black Friday. We hope so, but again it's doesn't seem wise to pin hopes down when the devices themselves may be in short supply.
If you want to be safe and get your device this side of 2022, then it's probably wise to consider beating out the crowds to secure your device by Black Friday. We would advocate at least investigating the excellent range of deals available right now, especially on the more popular 13 Pro and 13 Pro Max models. It's generally a safe bet that trade-in rebates won't get too much higher than they are right now – especially if you're picking up a device on a new line.
If you're after an iPhone 13 mini, however, then you might be able to get away with holding out until Black Friday. This device is the least popular of the four new models and seems to be the least likely to sell out over November. This one's definitely in the 'wait and see' territory and could potentially be worth holding out for. Either way, it's wise to keep an eye on those estimated delivery dates at leading retailers right now.
Previous articleElon Musk tried to troll Apple for a cleaning cloth
Next articleAnalyst Suggests Bitcoin's Bottom Could Be $50K Assuming BTC Surpasses $200K This Cycle – Market Updates Bitcoin News
Cetinje shooting: 11 killed in Montenegro gun attack
According to Montenegrin Police Chief Zoran Brđanin, the suspected shooter first attacked a family that was living in his house as tenants. "He killed...
China is urging families to stock up on food as supply challenges multiply
The country's Ministry of Commerce late Monday issued a notice directing local governments to encourage people to stockpile "daily necessities," including vegetables, oils and...
Bank of America Stock Gains As Lending Drives Q3 Earnings Beat
Bank of America (BAC) - Get Bank of America Corp Report posted stronger-than-expected third quarter earnings Thursday as net interest income and improving loan growth...
Live news: Zelenskyy urges Ukraine's allies to speed up weapons deliveries
Live news: Zelenskyy urges Ukraine's allies to speed up weapons deliveries Source
Inflation relief checks live online: Social Security retirement and disability payments, Bitcoin, FTX
Headlines: Friday 18 November 2022- Estimated $183 bn wiped off crypto markets after FTX collapse- Class action lawsuit filed against FTX founder and numerous celebrity promoters,...


I love it… it so beautifull made.
Oh it is just darling!!!!
Just love it – so pretty!!
Oh. It's so pretty. Scratch that. It's fantastic. Well done you!!!
Hi, Angie! Your colorful blog – this is for me an island of joy and inspiration.
And your new blanket – just no words! Very, very, very nice!
How many different shades of yarn you used for this blanket?
Angie, sorry … I have found already … 24 colors … Thank you!
Wow, I'm in love with this blanket! I would like to make one for me but the only blanket I made is still waiting for the borders!
Sucrette: Just stunning…Your an inspiration!
Absolutely my favorite–I truly love the white background best. It really showcases the happy colors you have chosen. Great job–once again!
absoluutly awesome, and beautiful made !!!!!!!!!!
the finishing edge is wonderfull.
Elle est superbe ! Bravo….
aaaaaah!! What a beautiful fairy tale blanket. Yes!! I'm definitely in love.
What a beautiful blanket. I love your patterns.
I just started a blog with my mom. So far I just have a trial post about a blanket I am working on. So much to figure out about blogging.
Beautiful blanket – love all your work!
this is just so beautiful, it is giving me something to aspire to!!! Mind you I think I might be certified insane after making that many squares!!
Quel travail et quel résultat !
Wow, you are a blanket machine! Super nice this new one, congrats!
B E A U T I F U L!!! What a lovely blanket. Your finished blanket has inspired me to get back to mine. As soon as I finish my flower blanket I am going to get back to work on my white "granny garden" (the one similar to this one). Very nice. I can't wait to see how your other two WIPs grow.
Yes, I love your blanket! All these colours remember me the spring. Beautiful!!!!
Oh wow! That is breath taking! Unbelieveable! I LOVE it!
I love your blanket, it's great!!
Magnifique, splendide, une vraie oeuvre d'art. Je comprends que tu l'aimes.
Encore une petite question : tu les as assemblés au fur et à mesure ?
Beautiful!!!!!!You aren´t crazy because I love making grannys. I begin one blanket similar yours.I´ll show you when I finishe it. Kisses!!!!!!!
It is lovely! I am working on a granny blanket too (actually I am working on 2), but I think it is going to take me years to finish. Are all your granny in different colors? They are all different aren'n they? It's a great blanket!
So so beautiful – I love the edge as well. I'm finishing a little baby stripe blanket for a friend and thinking of using a similar edge that you used. Well done.
Oh Wow! This is one stunning blanket! Je suis dans l'amour… Sigh 🙂 What a work of love it must have been.
the blanket it stunning !!!
That is one of the most beautiful afghans I have ever seen! Just lovely!
Your blog and your blanket is amazing. I can't wait to see your next project.
But I have to say that the blanket on the sofa is also wonderful (with the big squares). I like it so much that I would like to make one just like yours. So I have been looking on the internet all evening for a pattern, but I could't find anything quite like it. Now (in despair) I hope you won't mind me asking if you could help me with a pattern.
I have recently found your blog and I love it! I love the pictures of your colourful home and your beautiful crochet. I'm amazed how quickly you make each project!
I hope you don't mind me asking but how many balls of wool did you use for the white? I am making a simlar blanket and need to buy the white yarn and I am wondering how much to buy?


text:
Also we made ordinances for us, to charge ourselves yearly with the third part of a shekel for the service of the house of our God;
And we made rules for ourselves, taxing ourselves a third of a shekel every year for the upkeep of the house of our God;
Also we made ordinances for ourselves, to charge ourselves yearly with the third part of a shekel for the service of the house of our God;
Bible > Nehemiah > Nehemiah 10 > Nehemiah 10:32
Clarke's Nehemiah 10:32 Bible Commentary
Charge ourselves yearly with the third part of a shekel - According to the law, every one above twenty years of age was to give half a shekel to the sanctuary, which was called a ransom for their souls. See Exodus 30:11-16. But why is one third of a shekel now promised instead of the half shekel, which the law required? To this question no better answer can be given than this: the general poverty of the people, occasioned by their wars, overthrows, heavy tributes, etc., in the land of their captivity: and now on their return, having little property, it was impossible for them to give more; and we know, from the terms of the law in this case, that the poor and the rich were obliged to give alike, because it was a ransom for their souls; and the souls of the poor and the rich were of like value, and stood equally in need of redemption; for all were equally fallen, and all had come equally short of the glory of God.
Though only a third part of a shekel was given at this time, and probably for the reason above assigned, yet when the people got into a state of greater prosperity, the half shekel was resumed: for it is clear that this sum was paid in the time of our Lord, though not to the temple, but to the Roman government. Hence when those who collected this as a tribute came to our Lord, it was for the διδραχμα, didrachma, which was half a shekel; and the coin with which our Lord paid for himself and Peter was a stater, which contained exactly two half shekels. See Matthew 17:24-27.
Barnes's Nehemiah 10:32 Bible Commentary
The third part of a sheckel - This appears to have been the first occasion on which an annual payment toward the maintenance of the temple service and fabric was established. The half-shekel of the Law Exodus 30:13 was paid only at the time of a census (which rarely took place), and was thus not a recurring tax. In later times, the annual payment was raised from the third of a shekel to half a shekel Matthew 17:24.


text:
Movies|'Crazy Rich Asians': Why Did It Take So Long to See a Cast Like This?
'Crazy Rich Asians': Why Did It Take So Long to See a Cast Like This?
From left, the "Crazy Rich Asians" director Jon M. Chu and his cast: Ken Jeong, Awkwafina, Constance Wu, Gemma Chan, Michelle Yeoh and Henry Golding.CreditCreditRozette Rago for The New York Times
By Robert Ito
阅读简体中文版閱讀繁體中文版
WEST HOLLYWOOD, Calif. — In January 2017, the director Jon M. Chu announced an open casting call for Asian and Asian-American actors for his movie adaptation of "Crazy Rich Asians." Recorded in the kitchen of his West Hollywood home (you can see his fridge in the background), the online plea instructed anyone interested in joining his all-Asian cast, from aspiring actors to "cool personalities with hidden talents," to post a two-minute video of themselves on social media. "We are looking for you," he beamed.
The call was an enticing one. The romantic comedy was going to be a major feature film, with a reported budget of $30 million; its inspiration, the best-selling novel by Kevin Kwan, had already sold millions of copies. And then there was the sheer singularity of it all. How often did a Hollywood filmmaker go looking for a whole bunch of Asians for anything? The last time a major Hollywood film set in the present day showcased a majority Asian cast was a whopping 25 years ago, with "The Joy Luck Club" in 1993. Many of the folks Mr. Chu was seeking now weren't even alive then.
For Asian and Asian-American viewers, the film, which opens on Aug. 15, is important not just as something of a cinematic Halley's comet — before "Joy Luck Club," there was "Flower Drum Song" in 1961, and then, what? There's also an eager hope that if this movie succeeds, it just might stave off another quarter-century drought. Producers use something called "comps" — recent films similar to the ones they're pitching — to help sell studios on their ideas and budgets. For the producers of "Crazy Rich Asians," there weren't any. For scores of Asian-themed films to come, the hope goes, "Crazy Rich Asians" could be that comp.
When Mr. Chu made his online pitch, the film seemed a godsend for Asian actors, aspiring or otherwise. Forget a few token parts here and there; in this one film, Asian actors would play everything: the romantic leads and sympathetic sidekicks, the comic foils and cads, the faces in the crowd. With more than 4 billion Asians on the planet, how hard could it be to cast this thing?
[The actors talked about the casting process and working on the film.]
Pretty hard, as it turned out. The talent was there. But as roles were considered and cast, the filmmakers contended with questions about identity amid the Asian diaspora. Could an Asian play an Asian-American? Could a Malaysian play a Singaporean? And what about an Asian guy with a white mom or dad — could he even be in the running?
In Hollywood today, some of the most contentious debates continue to center on just who can play whom. In 2016, the lack of Asian male leads in movies was highlighted by the project #StarringJohnCho, which reimagined "The Martian," "Avengers: Age of Ultron" and other hits as if they starred the gifted Korean-American actor. Last month, Scarlett Johansson dropped out of the coming biopic "Rub & Tug" after her casting as a transgender man sparked a backlash from trans actors and activists. And Hollywood has gotten increasing flak for whitewashing, the practice of casting white actors in roles envisioned as Asian. Think of Tilda Swinton as the Ancient One in "Doctor Strange" or Ms. Johansson (again) as Motoko Kusanagi in "Ghost in the Shell."
Many of these debates wouldn't be nearly as fraught "if there were enough roles to go around," said Nancy Wang Yuen, the author of "Reel Inequality: Hollywood Actors and Racism." She added: "We shouldn't be fighting over scraps. We should be fighting for the system to expand, so that all of us can get access to roles."
The last Hollywood production with a mostly Asian cast was "The Joy Luck Club," in 1993.CreditHollywood Pictures
Indeed, according to a recent study by Ms. Yuen and others, 64 percent of television series in the 2015-16 season didn't have a single Asian-American regular; another report revealed that of the top 100 films of 2017, nearly two-thirds didn't include a single Asian or Asian-American female character.
On a recent morning, Mr. Chu was here in his West Hollywood home, recounting the joys and struggles of casting his all-Asian feature. Sporting a white baseball cap and gray T-shirt, he talked about hiring legends (Michelle Yeoh) and relative unknowns (Henry Golding), Asian stars (Singapore's Fiona Xie) and Asian-Americans on the rise (Awkwafina).
Mr. Chu also explained why he had cast such a wide net in the first place, supplementing the usual Los Angeles and New York casting calls with a five-continent search and an online open call. "We just didn't want to miss anybody," he said. "I wanted this to be the 'Avengers' of Asian actors."
The search began in 2013, when the producers Nina Jacobson ("The Hunger Games"), Brad Simpson ("Pose") and John Penotti ("Awake") secured the rights to Mr. Kwan's book. In the story, Rachel, a Chinese-American professor at New York University, travels with her London-bred professor boyfriend, Nick, to Singapore, where he was born. Once there Rachel discovers two things: Nick's steely mother, Eleanor, thinks Rachel is beneath her, and Nick — and the rest of his family — is obscenely, stupefyingly wealthy.
As it turns out, there were questions about casting even before the book hit stores. Mr. Kwan said a producer who wanted to option the book had suggested that he make Rachel white. Mr. Kwan refused.
"It didn't surprise me," said Constance Wu, the Chinese-American actress who ultimately secured the role and who has been a vocal critic of Hollywood whitewashing. "I'm just glad that Kevin stuck to his guns. It takes a lot of courage to say no to something, especially if you're scared that everything might slip away if you don't say yes."
Early on, Ms. Jacobson and Mr. Simpson had certain actors in mind, like Ms. Yeoh, who was on everybody's wish list for Eleanor. Unlike most of her castmates, Ms. Yeoh had appeared in dozens of films with all-Asian casts, like the Jackie Chan classic "Police Story 3: Super Cop" and the Oscar-winning "Crouching Tiger, Hidden Dragon," so, in some ways, this was nothing new. But those weren't Hollywood productions. "It was very important for me to see an all-Asian cast here," she said. "Twenty-five years? I don't even understand why it's taken so long."
For other roles, the producers sought the help of veteran casting directors with connections to actors throughout Asia, like the Canada-based PoPing AuYeung ("The Forbidden Kingdom"). They also looked at stage actors in New York and London, and checked in with some of the world's top drama schools, particularly in their search for the man who would embody Nick. "Ordinarily when you're casting and maybe looking to break somebody, you go to the big theater programs," Ms. Jacobson said, adding, "But they were like, we have not had a male Asian graduate in years."
When Mr. Chu announced his open call, thousands of videos poured in. Working actors from the United States, Singapore and elsewhere submitted auditions alongside those outside the industry who figured they had nothing to lose.
Among the hopefuls was Cheryl Koh, better known as the YouTube singer Cheryl K, whose videos have garnered more than a million views. Although she was trying out for a role, any role, it was her 15-second, a cappella snippet of Jessie J's "Mamma Knows Best" that convinced the filmmakers to give her a shot at singing the song under the opening credits, a Mandarin and English version of the Beatles hit "Money."
"When I got the news that Jon wanted me to do it, I couldn't stop screaming," she said.
The cast soon became increasingly international. There was Gemma Chan and Jing Lusi from Britain; Chris Pang and Remy Hii from Australia; Pierre Png, Ms. Xie and a host of others from Singapore; Ronny Chieng and Ms. Yeoh from Malaysia; Lisa Lu and Jasmine Chen from China. Nico Santos had spent his childhood in the Philippines before moving to the United States and co-starring on "Superstore"; Mr. Chieng was born in Malaysia and had gone to school in Australia before getting a regular gig on "The Daily Show." Among the American-born crew, there were actors who hailed from Detroit (Ken Jeong); Queens (Awkwafina); and Richmond, Va. (Constance Wu).
"Ken Jeong had to fly out to Kuala Lumpur just to shoot for a couple of days, and it wasn't a ton of money," Ms. Jacobson said. "But he was like, I want to play any part that you have for me in this."
With the entire world to pull from, the filmmakers faced several concerns about who they could cast to play what. There were the accents, of course. Whoever played Nick had to nail a British accent, which left out many of the Chinese box-office draws under consideration.
And then there was the Singaporean sounds of Nick's friends and family. "This is the only accent I can do," said Mr. Chieng, who grew up in Singapore. "I don't know if you've heard many people with this accent, but it's pretty specific, let's put it that way."
With much of the action taking place there, there was a push to get as many Singaporeans as possible. If that wasn't possible, the producers said, they wanted to understand when a non-Singaporean could fill a Singaporean role.
"We definitely always had 'Memoirs of a Geisha' in our heads," said Ms. Jacobson, referring to the 2005 film that starred three prominent Chinese actresses as Japanese women. (Unlike "Crazy Rich Asians," that film, directed by Rob Marshall and based on a book by Arthur Golden, was a period piece, set in Japan before and after World War II.)
In the end, only a handful of YouTube hopefuls got parts in "Crazy Rich Asians" — a woman in a text messaging scene here, a groomsman there. But now, said Mr. Chu, he has "the most amazing database of Asian talent that can speak English."
One of the last actors to be hired was Henry Golding, who auditioned for Nick. On paper, he was perfect. He was handsome and charismatic. Born in Malaysia to a mother from the Iban tribe there and a white British father, he had grown up in London, where he had acquired the requisite British accent, then spent much of the rest of his life in Asia, where he had worked as a travel host for BBC and Discovery Channel.
The fact that he had never acted before seemed to only add to the Cinderella charm of his story, which went something like this: With just a few months to go before filming started and no Nick, an accountant with the production in Malaysia told a line producer about this sexy travel host "all the girls were gaga over." From there, Mr. Golding's name went up the chain.
But when his casting was announced, the actress Jamie Chung, among others, criticized the decision. Ms. Chung, who is Korean-American, had been told by Mr. Chu that he was looking only for Chinese-Americans for the role of Rachel, so casting of a half-white actor as Nick seemed like a betrayal.
Asian-American journalists and bloggers wrote about the controversy, noting Hollywood's historical preference for biracial Asian actors (such as Nancy Kwan and Keanu Reeves) and hinting (sometimes more than hinting) that biracial Asians weren't "real Asians." The controversy became so heated that Mr. Golding and other stars were compelled to address it in interviews and online posts.
Ms. Chung walked back her comments soon after — "I am sincerely sorry about the ignorant comment I made earlier in the year," she tweeted — and apologized to Mr. Golding.
"I think it's fair to question motives of why they're choosing me instead of someone else," Mr. Golding told me. "It ensures that the studio knows, that everyone creating the story knows, that people are watching."
Even so, he said, "I never felt I wasn't suitable for the role because I was half-white. I've always seen myself as Asian, so I never had any qualms about that. I was much more concerned if I could act."
For a while, Mr. Chu admitted, the questions about casting disturbed him, particularly the questions about Mr. Golding. "And then I realized that I was only getting angry at the people who felt that they had been burned. They were people like me who had watched Hollywood whitewash things, and watched roles go away because someone said an Asian man can't be the lead of this or that."
Mr. Golding as Nick in "Crazy Rich Asians."CreditSanja Bucko/Warner Bros.
Much of the current whitewashing, Ms. Yuen said, stems from manga and Marvel films that "draw from these Orientalist tropes, whether it's Asians in the past or Asia in the future. But Hollywood doesn't know how to incorporate Asians or Asian-Americans into that. It's like they want our look and our stories and our martial arts, but they don't want Asians."
With more and more people talking about diversity in casting and calling out examples of whitewashing, Mr. Chu sees positive developments, particularly for Asian-Americans. This week Netflix will begin streaming "To All the Boys I've Loved Before," a coming-of-age tale starring the Vietnamese-American Lana Condor. And on Aug. 24, two years after #StarringJohnCho began hyping the actor as "the star we deserve," there's "Searching," a thriller starring, yes, John Cho.
"If we make a decent showing on that first weekend, there are like six Asian-American lead movies set up at different studios," Mr. Chu said. "They're not greenlit. Everyone's waiting to see how this one does. But if this one does well, we'll immediately have more chances. And if it doesn't, we'll just have to do it again."
Correction: Aug. 8, 2018
An earlier version of this article omitted one of the producers who secured rights to the novel "Crazy Rich Asians." He is John Penotti.
A version of this article appears in print on , Section AR, Page 1 of the New York edition with the headline: 25-Year Wait To See a Cast Like This. Order Reprints | Today's Paper | Subscribe
What Being in 'Crazy Rich Asians' Means to the Movie's Stars
Comics of Asian Descent, Tired of Being Invisible, Put Themselves Onstage
Asian-American Actors Are Fighting for Visibility. They Will Not Be Ignored.
A Bloom in TV's Asian-American Desert


05871506 is referenced by 145 patents and cites 47 patents.
Augmentation of electrical conduction and contractility by biphasic cardiac pacing. A first stimulation phase is administered to the muscle tissue. This first stimulation phase has a predefined polarity, amplitude and duration. A second stimulation phase is then administered to the muscle tissue. This second phase also has a predefined polarity, amplitude and duration. The two phases are applied sequentially. Contrary to current thought, anodal stimulation is first applied and followed by cathodal stimulation. In this fashion, pulse conduction through the cardiac muscle is improved together with an increase in contractility. The technique can also be applied to large muscle tissue stimulation other than cardiac muscle.


Posted November 29th, 2016 by Ms. Jen & filed under news + events.
Posted November 15th, 2016 by Ms. Jen & filed under art + photography, fun stuff.
11.15.16 – Another photo from my October / November 2016 Lightroom folders – posted now to not allow good photos fall into the black hole that is my hard drive. This portrait of my sister, Allison, taken on 10.26.16 with my Nikon D800 and a Nikon 85mm f/1.8D lens.
Enjoy. Double click for larger version(s).
Posted November 14th, 2016 by Ms. Jen & filed under art + photography, nature + environment.
Mon 10.14.16 – In going through my photos taken in October and November 2016 with my Nikon D800 and a Nikon 50mm f/1.4G lens, I have found a few gems to share with you all. This photo was taken on 10.06.16 just south of the Huntington Beach pier from the boardwalk on an evening where the colors of the sunset were fantastic, so I had to pull over while driving on PCH to go out and take photos.


The Chelsea blackout roller blind is one of our most popular designs. A good quality plain blackout fabric in a wide range of colours, that is also moisture resistant and suitable for Kitchens and Bathrooms. Excellent value. If you need a little more information roller blinds, we have a selection of buying guides to help. These cover FAQS on how to make a roller blind order, hot to measure a window for a roller blind, plus maintenance and cleaning advice. View the Buying Guides tab for more information.


text:
class CreateCities < ActiveRecord::Migration
  def change
    create_table :cities do |t|
      t.string :name, null: false
      t.string :description
      t.timestamps null: false
    end
    add_attachment :cities, :image
  end
end