text:
\subsection{Contributions}

\subsection{Terminology}

\subsection{Models}
For NMT, we use Transformer models with 121M parameters and the LAMB optimizer \citep{lamb}\footnote{For complete experiment setup details, see Appendix \ref{appendix:setup}.
}. We train on NMT datasets from the WMT19 machine translation task \citep{barrault-etal-2019-findings}. For evaluation, we decode using beam search and report BLEU \citep{papineni-etal-2002-bleu} scores. For T5, we use the \textit{small} configuration (60M parameters) and the Adafactor optimizer \citep{shazeer2018adafactor}. We use the C4 dataset \citep{raffel2019exploring}. We report loss on the C4 development set and the same evaluation criteria as the original T5 paper for downstream tasks. For all hyper-parameter learning experiments, we use the Adam optimizer \citep{adam} with default settings as meta-optimizer, tuning only the meta-learning-rate ${\hat{\eta}}$. For the guidance set, we use a single held-out training batch\footnote{In preliminary experiments, we found no benefit to a larger guidance set.}. As the hyper-parameters must vary within constrained ranges, $\alpha$ is kept positive by an exponential activation function, and $\beta_1$ is constrained between 0 and 1 by a sigmoid.

\subsection{Neural Machine Translation}
In Figure \ref{fig:lr_sweep}, we compare the evolution of learning the learning rate scalar ($\alpha$) over training for runs with differing meta-learning rates ($\hat{\eta}$) for the German-English language pair. In Figure \ref{fig:beta1_sweep} we do the same for learning $\beta_1$, varying the initialization values in addition to $\hat{\eta}${}. For learning $\alpha$, the guidance optimization drives all runs to as low a learning rate as is allowed by the meta-learning rate, demonstrating the `short horizon bias'. Note that some guided $\alpha$ runs do outperform the baseline, but require tuning of $\hat{\eta}${} to prevent convergence on the guidance objective. In contrast, the learned $\beta_1$ (Figure \ref{fig:beta1_sweep}) runs converge to a similar schedule given a sufficiently high $\hat{\eta}${}, decaying from high to low momentum over the course of training, regardless of the initialization value. All runs with guided $\beta_1$ outperform the baseline.
To evaluate how well these gains generalize, we guide $\alpha$ and $\beta_1$ alone and together for 6 language pairs, setting $\hat{\eta}${} to $3\text{e-}5$ for all runs (Table \ref{tab:combo_bleu}).

\begin{wraptable}{r}{7cm}
  \caption{BLEU scores of baseline vs guided runs across language pairs. $\hat{\eta}${} is set to $3\text{e-}5$.}
  \footnotesize
  \centering
  \begin{tabular}{c|cccccc}
    \toprule

    & \multicolumn{6}{l}{de-en \hspace{0.17cm} en-de \hspace{0.17cm}  fi-en \hspace{0.17cm}  en-fi \hspace{0.17cm}  lt-en \hspace{0.17cm}  en-lt} \\
    \midrule
    base & 38.6 & 37.4 & 27.2 & 18.4 & 27.3 & 11.3 \\
    \midrule
    $\alpha$ & 39.6 & \textbf{39.4} & 27.6 & \textbf{19.7} & 27.7 & 11.7 \\
    $\beta_1$ & 39.8 & \textbf{39.4} & \textbf{28.4} & 19.4 & \textbf{28.0} & 12.1 \\
    $\alpha$\scalebox{1}{+}$\beta_1$  & \textbf{39.9} & 38.8 & 27.5 & 19.6 & 27.8 & \textbf{12.2} \\

    \bottomrule
  \end{tabular}
  \vspace{2mm}
  \label{tab:combo_bleu}
  \caption{BLEU scores of $100$ BO-tuned runs vs un-tuned for baseline and guided runs, on \textit{de-en}. Time is summed runtime in hours.
  \footnotesize
  \centering
  \begin{tabular}{c|c|c|ccc}
    \toprule
    & time & \# runs & $\alpha$ & $\beta_1$ & $\alpha$\scalebox{1}{+}$\beta_1$\\
    \midrule
    base & 7.1 & 1 & 38.6 & 38.6 & 38.6 \\
    + BO & 708 &100 & 39.4 & 39.4 & 39.5 \\
    \midrule
    guided & 43.2 & 4 & 39.6 & 39.8 & \textbf{39.9} \\
    + BO & 1.1k &100 & 39.7 & 39.8 & \textbf{39.9} \\
    \bottomrule
  \end{tabular}
  \label{tab:tuned}
\end{wraptable}

\begin{figure*}
\centering
\begin{minipage}{.5\textwidth}
  \centering
  \includegraphics[width=1\linewidth]{draft_fig_lr_sweep_automl.pdf}
  \captionof{figure}{Learning $\alpha$, varying $\hat{\eta}${} values. \\ German-English NMT}
  \label{fig:lr_sweep}
\end{minipage}%
\begin{minipage}{.5\textwidth}
  \centering
  \includegraphics[width=1\linewidth]{draft_fig_b1_sweep_automl.pdf}
  \captionof{figure}{Learning $\beta_1$, varying $\hat{\eta}${} and init. values. \\ German-English NMT}
  \label{fig:beta1_sweep}
\end{minipage}
\end{figure*}

In order to evaluate the practical applicability of guiding these hyperparameters, we compare the guided runs to a typical hyperparameter optimization scheme, against which we can evaluate both performance and efficiency. We tune both the baseline runs (via the hyper-parameters directly) and the guided runs (via $\hat{\eta}${}) with Bayesian optimization (BO)\footnote{The specific algorithm we use is Gaussian Process Bandits \citep{frazier2018tutorial,Vizier}.} for 100 trials. In Table \ref{tab:tuned}, we find that across guided-parameter settings, the non-BO-optimized guided run outperforms the best BO-tuned baseline model, with some slight gains for the guided $\alpha$ run with further BO-tuning\footnote{We count the 4 different values of $\hat{\eta}${} we tried in Figure \ref{fig:lr_sweep} as tuning runs for the non-BO-tuned guided setup.}. Note the guided $\beta_1$ runs do not require $\hat{\eta}${} tuning to reach best performance.

For all setups, the learned hyper-parameters achieve better performance than Bayesian optimization in fewer training runs and less time. Though the `short-horizon bias' requires tuning $\hat{\eta}${} while learning $\alpha$, doing so still yields performance and efficiency gains over BO-tuning. For $\beta_1$ alone, there seems to be no equivalent bias, as any sufficiently high $\hat{\eta}${} converges to roughly the same useful schedule. The BO-tuned optimal static $\beta_1$ value (0.73) approximates the average $\beta_1$ of the converged runs in Figure \ref{fig:beta1_sweep}, suggesting that the remaining 0.4 BLEU points are only attainable with a $\beta_1$ value that changes over the course of training. Learning both hyper-parameters together does not change their evolution but yields a small additional boost.

\subsection{T5 pretraining}
\begin{wrapfigure}{r}{0.5\textwidth}
  \begin{center}
    \includegraphics[width=0.4\textwidth]{draft_fig_t5_automl.pdf}
    \end{center}
    \caption{Learning $\alpha$\scalebox{1}{+}$\beta_1$ for T5, varying $\hat{\eta}${}.
    }
	\label{fig:t5}
\end{wrapfigure}
We run similar experiments for T5 models, learning $\alpha$, $\beta_1$, and both. For $\alpha$, we see the learning rate scalar decrease prematurely similarly to the NMT setting, demonstrating again the `short-horizon bias' (Appendix, Figure
2), but no guided run outperforms the baseline, even with low $\hat{\eta}${} values\footnote{See Appendix \ref{appendix:t5}
for full details on isolated $\alpha$ and $\beta_1$ experiments.}. For $\beta_1$ alone, we replicate a similar converged schedule as in the NMT setting, but see only minor changes in development set loss across all models, including those varying $\beta_1$ without learning during training (Appendix, Figure
2). This suggests that tuning $\beta_1$ in general is less useful in this setting. Interestingly, when we tune both hyper-parameters together, the evolution of the $\alpha$ parameter changes character (Figure \ref{fig:t5}), and we find a $3$X improvement in speed of convergence relative to baseline and increases in final performance for multiple different settings of $\hat{\eta}${}.

We finetune the baseline and $\hat{\eta}${}=$1\text{e-}5$ models on each of the downstream NLU tasks drawn from the GLUE \citep{wang2018glue} and superGLUE \citep{wang2019superglue} benchmarks, as well as SQuAD \citep{rajpurkar2016squad}, using the same finetuning settings as the original T5 paper\footnote{For details on the setup of finetuning,
see Appendix \ref{appendix:t5_down}. For
full results, including on SQuAD, see  Appendix \ref{appendix:t5_down_results}.}
(Table \ref{tab:t5_down}). We find improvements across 15 of 18 downstream NLU tasks, with average improvements of 0.4 points on GLUE and 1.4 points on superGLUE.

\begin{table*}[b]
    \begin{tabularx}{1\textwidth}{p{0.9cm}|*{9}{X}|c}

    \toprule
    \multirow{2}{*}{GLUE} & CoLA & SST & MRPC & STS & QQP & MNLI & QNLI & RTE & WNLI & \multirow{2}{*}{avg} \\
    & $\phi{}$\textit{corr} & \textit{acc} & $\myovline{\textit{met}}{0.2pt}${} & $\myovline{\textit{met}}{0.2pt}${} & $\myovline{\textit{met}}{0.2pt}${} & $\myovline{\textit{met}}{0.2pt}${} & \textit{acc} & \textit{acc} & \textit{acc} & \\
    \cmidrule(lr){2-11}
    base & \textbf{47.9} & 92.3 & 88.5 & 84.1 & 87.4 & 84.1 & 90.2 & 67.9 & 57.7  & 77.8 \\
    $\alpha$\scalebox{1}{+}$\beta_1$ & 44.5 & \textbf{92.6} & \textbf{90.0} & \textbf{85.1} & \textbf{87.7} & \textbf{84.9} & \textbf{90.5} & \textbf{69.0} & \textbf{59.2}  & \textbf{78.2} \\
  \end{tabularx}

    \begin{tabularx}{1\textwidth}{p{0.9cm}|*{8}{X}|c}
    \midrule
    \multirow{2}{*}{sGLUE} & BoolQ & CB & COPA & MultiRC & ReCoRD & RTE & WiC & WSC & \multirow{2}{*}{avg}  \\

    & \textit{acc} & $\myovline{\textit{met}}{0.2pt}${} & \textit{acc} & $\myovline{\textit{met}}{0.2pt}${} & $\myovline{\textit{met}}{0.2pt}${} & \textit{acc} & \textit{acc} & \textit{acc} & \\
    \cmidrule(lr){2-10}
    base & 73.6 & \textbf{98.4} & 58.0 & 43.7 & 63.8 & 64.6 & 67.2 & 67.3 & 67.1 \\
    $\alpha$\scalebox{1}{+}$\beta_1$ & 73.6 & 95.3 & \textbf{61.0} & \textbf{46.0} & \textbf{64.0} & \textbf{66.4} & \textbf{67.6} & \textbf{74.0}& \textbf{68.5} \\
    \bottomrule
  \end{tabularx}
  \caption{Fine-tuning baseline and $\alpha$\scalebox{1}{+}$\beta_1$ models on the GLUE and superGLUE (sGLUE) NLU tasks. \protect$\myovline{\textit{met}}{0.2pt}${} denotes the mean of the two metrics typically reported for that task, and \protect\textit{avg} takes the average across tasks. Max value of 5 runs shown, see Appendix \protect\ref{appendix:t5_down_results}
  for average results.}

  \label{tab:t5_down}
\end{table*}

\subsection{Guiding $\beta_1$}

In order to understand the function of guided learning we begin with a case study of a single hyperparameter, which yields a guided parameterization with only a single-dimensional degree of freedom.
We start with $\beta_1$ as it is often useful to tune and is not complicated by the application of any schedule. In Figure \ref{fig:beta1_sweep}, we compare the evolution of $\beta_1$ over training for guided runs of differing meta-learning rates and initializations, training on the German-English language pair. The unguided baseline uses the default initialization of $0.9$.
\begin{wrapfigure}{r}{0.5\textwidth}
  \begin{center}
    \includegraphics[width=0.48\textwidth]{draft_fig_b1_sweep.pdf}
  \end{center}
  \caption{Comparison of guiding $\beta_1$ over the course of training, with initialization values $[0.1,0.5,0.9]$ and meta-learning-rates [$3\text{e-}5$, $3\text{e-}4$, $3\text{e-}3$].

  }
  \label{fig:beta1_sweep}
 \end{wrapfigure}

\subsubsection{Analysis}

Figure \ref{fig:beta1_sweep} illuminates the simplicity of the $\beta_1$ guidance-loss landscape; all runs with sufficiently high meta-learning rates converge\footnote{\textit{Convergence} in this sense can be observed in the gradients of the guided parameter, which settle towards $0$ for the runs which have reached the loss-defined optimum.} to remarkably similar schedules, regardless of model or $\beta_1$ initialization. That the different guided runs end up tracing the same schedule and that schedule outperforms not only the baseline, but even a thoroughly BO-tuned constant value, seems to realize a best-case scenario for guided learning: for optimizing $\beta_1$, expensive black-box optimization can be replaced with learning an optimal schedule with little or no tuning of the meta-learning rate.

Further, the converged $\beta_1$ schedule, which gently decreases $\beta_1$ over training, suggests a reasonable explanation. At the start of training, it is useful to have a high momentum as the peculiarities of the individual batches are less important than the general trends across them.
As the model becomes better and training progress levels out, it is better to be sensitive to the peculiarities of the individual training batches rather than treating them all the same. A similar reasoning motivated early explorations of learning rate decay schedules \citep{lecun1991} and has been validated recently by \citet{you2019}. Thus analyzing the results from guided learning allows us to rediscover the intuitions that originally motivated exploration of learning rate decay schedules.

Additionally, the 100-run Bayesian-optimized value of $\beta_1$ is roughly a constant approximation of the learned converged schedule, which is hardly changed by further BO-optimization\footnote{See Appendix \ref{sec:appendix} for plot.}. This suggests that the \textit{true optimal value} of the hyperparameter is following a schedule, and no amount of tuning a single constant value can beat a method that even roughly approximates that schedule. For hyperparameters whose optimal values change over training, methods that produce schedules will have an edge over global tuning methods (gradient-based or otherwise).

\subsection{Guiding $\alpha$}

We now explore the effects of tuning the learning rate ($\alpha$) via guided learning. Unlike $\beta_1$, the learning rate typically follows a predetermined schedule. Rather than throw away the benefits of the hard-coded scheme by training the learning rate value directly, we learn a scalar on the pre-defined schedule, i.e. $\textit{effective learning rate}_t = \textit{scalar} \times \textit{schedule}_t$.

\subsubsection{Analysis}

It is crucial to tune the learning rate and its schedule to achieve good results \cite{Bengio2012}. Unfortunately tuning a schedule is a much more complicated task than tuning a single constant value, so in practice (including in this work), learning rate schedules are frequently borrowed from previous work.
\cite{wu2018understanding} have shown that learning the learning rate greedily is subject to a \textit{short-horizon bias}, whereby the gradient of the guidance loss drives the learning rate too low and thus fails to actually learn, yielding a worse final model. This position is supported by the results of our sweep over meta-learning rates for guiding $\alpha$
, in which the run with the highest meta-learning rate ($3\text{e-}3$) spikes initially and then decays quickly to convergence at a nearly static value of $1\text{e-}5$ for the rest of training. This short-horizon bias means that unlike $\beta_1$, the meta-learning rate cannot simply be set to a high-enough value for convergence, but must be tuned if it is to help performance.

However, even with a modest sweep over meta-learning rates, we are able to substantially outperform the baseline schedule. Indeed we are even able to surpass the best result from a much more expensive Bayesian optimization of the non-guided learning rate. This suggests that tuning the meta-learning rate may still be a more pragmatic approach to optimization rather than tuning the original learning rate.

Thus far we have analyzed guiding individual hyperparameters in isolation, holding the other hyperparameters constant. This simplifies the optimization problem and is helpful for analysis, but misses out on potential gains of jointly guiding parameters.
The max BLEU scores achieved per run are shown in Table \ref{tab:combo_bleu}.

\begin{wraptable}{r}{8cm}
  \caption{BLEU scores of guided runs vs baseline across language pairs. Meta-learning-rate is set to $3\text{e-}5$.}
  \footnotesize
  \centering
  \begin{tabular}{c|cccccc}
    \toprule
    & de-en & en-de & fi-en & en-fi & lt-en & en-lt \\
    \midrule
    baseline & 38.6 & 37.4 & 27.2 & 18.4 & 27.3 & 11.3 \\
    \midrule
    $\beta_1$ & \textbf{39.9} & \textbf{39.4} & \textbf{28.4} & 19.4 & \textbf{28.0} & 12.1 \\
    $\alpha$ & 39.6 & \textbf{39.4} & 27.6 & \textbf{19.7} & 27.7 & 11.7 \\
    $\beta_1$ \& $\alpha$ & 39.8 & 38.8 & 27.5 & 19.6 & 27.8 & \textbf{12.2} \\

    \bottomrule
  \end{tabular}
  \label{tab:combo_bleu}
\end{wraptable}
\subsubsection{Analysis}

In Figure \ref{fig:language_pairs}, we observe that guided learning of $\beta_1$, $\alpha$, or the combination of the two outperforms the baseline across language pairs, which suggests that it may be possible to find ``good recipes'' which generalize across training runs, further simplifying the useful application of guiding hyperparameters. While guiding $\beta_1$ and $\alpha$ together doesn't drastically change the dynamics of their evolution compared to learning each alone, we find that in some cases the combination of the two yields better results than guiding them separately, and when tuned, they outperform the individual setups.

\subsection{Comparing guidance to tuning}

In order to fairly evaluate the practical applicability of guiding these hyperparameters, it is necessary to compare the guided runs not just against the untuned baseline but against the best model produced by typical hyperparameter optimization schemes. This offers the opportunity to evaluate the relative utility of guidance vs tuning in terms of both absolute performance and efficiency. In Figure \ref{fig:tuning}, we compare guiding $\beta_1$, $\alpha$, and their combination against the best model (determined by the minimum of loss on the development set) produced by 100-model run of black-box optimization (BO) over the hyperparameter in question. We find that across guided-parameter settings, the non-BO-optimized guided run slightly outperforms the best BO-tuned model (Table \ref{tab:tuned}). For the setups which involve guiding $\alpha$,
small additional gains are found in tuning the meta-learning rate with the same BO procedure.

\begin{wraptable}{r}{8cm}
  \caption{Comparing BLEU scores of guided runs to baseline and the best of $100$ Bayesian-optimized runs across language pairs. Meta-learning-rate is set to $3\text{e-}5$ for untuned guided runs, and is tuned via BO in the final row.
  \footnotesize
  \centering
  \begin{tabular}{c|c|c|ccc}
    \toprule
    & BO-tuned & \# of runs & $\beta_1$ & $\alpha$ & $\beta_1$ \& $\alpha$\\
    \midrule
    \multirow{2}{*}{baseline} & no & 1 & 38.9 & 38.9 & 38.6 \\
     & yes &100 & 39.4 & 39.4 & 39.5 \\
    \midrule
    \multirow{2}{*}{guided} & no & 4 & 39.6 & 39.7 & \textbf{39.9} \\
    & yes &100 & 39.7 & 39.7 & \textbf{39.9} \\
    \bottomrule
  \end{tabular}
  \label{tab:tuned}
\end{wraptable}

\subsubsection{Analysis}

While the idea-scenario for guiding hyperparameters would involve no tuning of either the initialization or meta-learning-rate values, the \textit{short-horizon} problem faced by the learning rate necessitates tuning of the meta-learning-rate. However, for guided learning to be useful it simply needs to be a more efficient means of discovering useful hyperparameters than common HPO methods. In Figure \ref{fig:tuning} (and Table \ref{tab:tuned} we observe that guiding $\beta_1$, $\alpha$, or their combination actually outperforms Bayesian optimization (BO) in both absolute performance and in efficiency. Relative to BO with 100 training runs, guided learning achieves better results in each setting with a sweep over just 4 meta-learning-rate values. One potential reason is that the BO setup can only tune constant values, while the guided learning setup is capable of evolving hyperparameters over the course of training

\subsection{Guiding T5}

In order to assess the extent to which our observations of guiding learning might generalize, we run similar experiments for T5 models. We run a guided experiments for $\beta_1$, $\alpha$, and the combination of the two. For $\beta_1$ alone, we replicate a similar converged schedule as in the NMR setting, but see only minor changes in accuracy across all models. For $\alpha$, we see the learning rate scalar plummet similarly to the NMT setting, but no guided run outperforms the baseline, even at mild meta-learning rates. For the combination of the two, we run a sweep over meta-learning rates and show results in Figure \ref{fig:t5}.

\begin{wraptable}{r}{7cm}
  \footnotesize
  \centering
  \begin{tabular}{c|c|ccccc}
    \toprule
    & base & \multicolumn{5}{c}{$\beta_1$ \& $\alpha$} \\
    $\hat{\eta}$ & - & $3\text{e-}6$ & $6\text{e-}6$ & $1\text{e-}5$ & $2\text{e-}5$ & $3\text{e-}5$ \\%& $4\text{e-}5$ \\
    \midrule
    \textit{acc} & 63.1 & 63.1 & \textbf{64.1} & \textbf{64.1} & 63.6 & 63.1 \\
    \textit{steps} & 1M & 1M & 600k & 440k & \textbf{290k} & 600k  \\%& - \\

    \bottomrule
  \end{tabular}
  \caption{Comparing final (1M steps) eval accuracy per token (\textit{acc)} and number of steps to reach baseline final accuracy (\textit{steps}) for a sweep of meta-learning rate values guiding $\alpha$ and $\beta_1$ for T5 models.
  \label{tab:t5}
\end{wraptable}

\subsubsection{Analysis}

Our T5 guided experiments on individual parameters replicate the parameter evolution seen in the NMT setting, but make only infinitesimal improvements over the baseline\footnote{See Appendix \ref{sec:appendix_t5} for figures.}. Interestingly, guiding both hyper-parameters simultaneously (Figure \ref{fig:t5}) appears to change the dynamics of their evolution to great effect, and we find a $3$X improvement in speed of convergence and significant increase in final performance for multiple different settings of meta-learning rate (Table \ref{tab:t5}). The plot of $\beta_1$ itself is familiar from the previous NMT experiments but the evolution of $\alpha$ takes on an complicated trajectory that appears related to the schedule of $\beta_1$.

The rough generalization of the guided ``recipe'' from NMT to T5 is encouraging for the possibility of finding generalized guided learning settings which perform well across domains. The demonstrated capacity to improve both speed and performance of T5 pretraining is a significant endorsement of the pragmatic value of the technique to improve over even well-tuned baselines. The superior performance of guiding the hyper-parameters jointly over each individually is a promising and curious result that hints at both the potential and the complexity of learning multiple kinds of hyper-parameters simultaneously; an obvious candidate for further exploration.

\subsection{Guiding $\beta_1$}

In order to understand the function of guided learning we begin with a case study of a single hyperparameter, which yields a guided parameterization with only a single-dimensional degree of freedom. We start with $\beta_1$ as it is often useful to tune and is not complicated by the application of any schedule. In Figure \ref{fig:beta1_sweep}, we compare the evolution of $\beta_1$ over training for guided runs of differing meta-learning rates and initializations, training on the German-English language pair. The unguided baseline uses the default initialization of $0.9$.
\begin{figure}
	\centering
    \includegraphics[width=\linewidth]{draft_fig_b1_sweep.pdf}
        \caption{Comparison of guiding $\beta_1$ over the course of training, with initialization values $[0.1,0.5,0.9]$ and meta-learning-rates [$3\text{e-}5$, $3\text{e-}4$, $3\text{e-}3$]. Baseline (no guidance) uses initialization of $0.9$ and meta-learning-rate of $0$. Loss on the development set and the learned $\beta_1$ values are shown in the top and bottom panels, respectively.}
	\label{fig:beta1_sweep}
\end{figure}

\subsection{Guiding $\alpha$}

We now explore the effects of tuning the learning rate ($\alpha$) via guided learning. Unlike $\beta_1$, the learning rate typically follows a predetermined schedule. Rather than throw away the benefits of the hard-coded scheme by training the learning rate value directly, we learn a scalar on the pre-defined schedule, i.e. $\textit{effective learning rate}_t = \textit{scalar} \times \textit{schedule}_t$. In Figure \ref{fig:lr_scalar_sweep} we show models trained on the German-English language pair, across guided runs coarsely tuning the learning rate scalar with varying meta learning rates.

\begin{figure
	\centering%
    \includegraphics[width=\linewidth]{draft_fig_lr_sweep.pdf}
        \caption{Learning rate scalars, color-coded by meta-learning rate. All guided runs are initialized at 1, and applied as a scalar to the same fixed learning rate schedule.}
	\label{fig:lr_scalar_sweep}
\end{figure}

\subsection{Combining guided parameters}
\begin{figure*
	\centering%
    \includegraphics[width=\linewidth]{draft_fig_combo.pdf}
        \caption{Comparing guiding runs to baseline across language-pairs (columns). Guided runs are the colored lines, the unguided baselines are black lines. The blue curves depict the guiding of the learning rate, the red that of $\beta_1$, and the purple, the combination of the two. The top row shows loss and the bottom shows accuracy on the development sets. All guided runs outperform all baseline runs in accuracy across language pairs.}
	\label{fig:language_pairs}
\end{figure*}
\begin{figure*
	\centering%
    \includegraphics[width=\linewidth]{draft_fig_vizier.pdf}
        \caption{Comparing guided runs for $\beta_1$ and learning rate in isolation and jointly, to unguided models, untuned and tuned, on the German-English language pair. The black solid line is the untuned baseline. The colored lines show a single guided run, learning the value of the variable(s) depicted at the top of the column. The dotted black lines show the best single run out of 100 produced in the course of an application of Bayesian optimization for tuning the variable(s) in question.}
	\label{fig:tuning}
\end{figure*}

Thus far we have analyzed guiding individual hyperparameters in isolation, holding the other hyperparameters constant. This simplifies the optimization problem and is helpful for analysis, but misses out on potential gains of jointly guiding parameters. In Figure \ref{fig:language_pairs}, we learn both $\beta_1$ and learning rate in a single run, setting the meta learning rate to $3\text{e-}5$ for both, and comparing against an un-guided baseline with default values across $6$ different language pairs. The max BLEU scores achieved per run are shown in Table \ref{tab:combo_bleu}.

\begin{table}[h]
  \caption{Comparing BLEU scores of guided runs to baseline across language pairs. Meta-learning-rate is set to $3\text{e-}5$ for all runs.}
  \footnotesize
  \centering
  \begin{tabular}{c|cccccc}
    \toprule
    & de-en & en-de & fi-en & en-fi & lt-en & en-lt \\
    \midrule
    baseline & 37.6 & 36.4 & 26.5 & 18.0 & 26.9 & 11.1 \\
    \midrule
    $\beta_1$ & \textbf{39.0} & \textbf{38.4} & \textbf{27.7} & 18.9 & \textbf{27.2} & 11.8 \\
    $\alpha$ & 38.8 & 38.3 & 27.0 & \textbf{19.3} & 27.1 & 11.4 \\
    $\beta_1$ \& $\alpha$ & 38.9 & \textbf{38.4} & 26.9 & 19.2 & 27.0 & \textbf{12.0} \\

    \bottomrule
  \end{tabular}
  \label{tab:combo_bleu}
\end{table}

\subsection{Comparing guidance to tuning}

In order to fairly evaluate the practical applicability of guiding these hyperparameters, it is necessary to compare the guided runs not just against the untuned baseline but against the best model produced by typical hyperparameter optimization schemes. This offers the opportunity to evaluate the relative utility of guidance vs tuning in terms of both absolute performance and efficiency. In Figure \ref{fig:tuning}, we compare guiding $\beta_1$, $\alpha$, and their combination against the best model (determined by the minimum of loss on the development set) produced by 100-model run of black-box optimization (BO) over the hyperparameter in question. We find that across guided-parameter settings, the non-BO-optimized guided run slightly outperforms the best BO-tuned model (Table \ref{tab:tuned}). For the setups which involve guiding $\alpha$,
small additional gains are found in tuning the meta-learning rate with the same BO procedure.

\begin{table}[h]
  \caption{Comparing BLEU scores of guided runs to baseline and the best of $100$ Bayesian-optimized runs across language pairs. Meta-learning-rate is set to $3\text{e-}5$ for untuned guided runs, and is tuned via BO in the final row. For the guided setting, the $4$ runs correspond to the coarse meta-learning rate tuning shown in Figures \ref{fig:beta1_sweep} and \ref{fig:lr_scalar_sweep}.}
  \footnotesize
  \centering
  \begin{tabular}{c|c|c|ccc}
    \toprule
    & BO-tuned & \# of runs & $\beta_1$ & $\alpha$ & $\beta_1$ \& $\alpha$\\
    \midrule
    \multirow{2}{*}{baseline} & no & 1 & 38.0 & 38.0 & 38.0 \\
     & yes &100 & 38.6 & 38.7 & 38.8 \\
    \midrule
    \multirow{2}{*}{guided} & no & 4 & 39.0 & 38.8 & 38.9 \\
    & yes &100 & 39.0 & 38.9 & \textbf{39.1} \\
    \bottomrule
  \end{tabular}
  \label{tab:tuned}
\end{table}

\subsection{Guiding T5}

In order to assess the extent to which our observations of guiding learning might generalize, we run similar experiments for T5 models. We run a guided experiments for $\beta_1$, $\alpha$, and the combination of the two. For $\beta_1$ alone, we replicate a similar converged schedule as in the NMR setting, but see only minor changes in accuracy across all models. For $\alpha$, we see the learning rate scalar plummet similarly to the NMT setting, but no guided run outperforms the baseline, even at mild meta-learning rates. For the combination of the two, we run a sweep over meta-learning rates and show results in Figure \ref{fig:t5}.

\begin{table}[h]
  \footnotesize
  \centering
  \begin{tabular}{c|c|cccccc}
    \toprule
    & base & $3\text{e-}6$ & $6\text{e-}6$ & $1\text{e-}5$ & $2\text{e-}5$ & $3\text{e-}5$ & $4\text{e-}5$ \\
    \midrule
    \textit{acc} & 63.1 & 63.1 & \textbf{64.1} & \textbf{64.1} & 63.6 & 63.1 & 62.8 \\
    \textit{steps} & 1M & 1M & 600k & 440k & \textbf{290k} & 600k  & - \\

    \bottomrule
  \end{tabular}
  \caption{Comparing final accuracy per token (\textit{acc)} and number of steps to reach baseline final accuracy (\textit{steps}) for a sweep of meta-learning rate values guiding $\alpha$ and $\beta_1$ for T5 models. The best-performing guided models improve by a full accuracy point, and the fastest-converging reaches the same performance as baseline in less than a third the number of steps.}
  \label{tab:t5}
\end{table}

\begin{figure
	\centering%
    \includegraphics[width=\linewidth]{draft_fig_t5.pdf}
        \caption{Guiding learning rate scalar $\alpha$ and Adafactor's $\beta_1$ jointly for a T5 model, with runs color-coded by meta-learning rate.
        }
	\label{fig:t5}
\end{figure}

\subsection{Guiding $\beta_1$}

Figure \ref{fig:beta1_sweep} illuminates the simplicity of the $\beta_1$ guidance-loss landscape; all runs with sufficiently high meta-learning rates converge\footnote{\textit{Convergence} in this sense can be observed in the gradients of the guided parameter, which settle towards $0$ for the runs which have reached the loss-defined optimum.} to remarkably similar schedules, regardless of model or $\beta_1$ initialization. That the different guided runs end up tracing the same schedule and that schedule outperforms not only the baseline, but even a thoroughly BO-tuned constant value, seems to realize a best-case scenario for guided learning: for optimizing $\beta_1$, expensive black-box optimization can be replaced with learning an optimal schedule with little or no tuning of the meta-learning rate.

Further, the converged $\beta_1$ schedule, which gently decreases $\beta_1$ over training, suggests a reasonable explanation. At the start of training, it is useful to have a high momentum as the peculiarities of the individual batches are less important than the general trends across them.
As the model becomes better and training progress levels out, it is better to be sensitive to the peculiarities of the individual training batches rather than treating them all the same. A similar reasoning motivated early explorations of learning rate decay schedules \citep{lecun1991} and has been validated recently by \citet{you2019}. Thus analyzing the results from guided learning allows us to rediscover the intuitions that originally motivated exploration of learning rate decay schedules.

Additionally, the 100-run Bayesian-optimized value of $\beta_1$ is roughly a constant approximation of the learned converged schedule, which is hardly changed by further BO-optimization\footnote{See Appendix \ref{sec:appendix} for plot.}. This suggests that the \textit{true optimal value} of the hyperparameter is following a schedule, and no amount of tuning a single constant value can beat a method that even roughly approximates that schedule. For hyperparameters whose optimal values change over training, methods that produce schedules will have an edge over global tuning methods (gradient-based or otherwise).

\subsection{Guiding learning rate}
It is crucial to tune the learning rate and its schedule to achieve good results \cite{Bengio2012}. Unfortunately tuning a schedule is a much more complicated task than tuning a single constant value, so in practice (including in this work), learning rate schedules are frequently borrowed from previous work.
 \cite{wu2018understanding} have shown that learning the learning rate greedily is subject to a \textit{short-horizon bias}, whereby the gradient of the guidance loss drives the learning rate too low and thus fails to actually learn, yielding a worse final model. This position is supported by the results of our sweep over meta-learning rates for guiding $\alpha$ (Figure \ref{fig:lr_scalar_sweep}), in which the run with the highest meta-learning rate ($3\text{e-}3$) spikes initially and then decays quickly to convergence at a nearly static value of $1\text{e-}5$ for the rest of training. This short-horizon bias means that unlike $\beta_1$, the meta-learning rate cannot simply be set to a high-enough value for convergence, but must be tuned if it is to help performance.

However, even with a modest sweep over meta-learning rates, we are able to substantially outperform the baseline schedule. Indeed we are even able to surpass the best result from a much more expensive Bayesian optimization of the non-guided learning rate. This suggests that tuning the meta-learning rate may still be a more pragmatic approach to optimization rather than tuning the original learning rate.

\subsection{Combination and Generalization}
In Figure \ref{fig:language_pairs}, we observe that guided learning of $\beta_1$, $\alpha$, or the combination of the two outperforms the baseline across language pairs, which suggests that it may be possible to find ``good recipes'' which generalize across training runs, further simplifying the useful application of guiding hyperparameters. While guiding $\beta_1$ and $\alpha$ together doesn't drastically change the dynamics of their evolution compared to learning each alone, we find that in some cases the combination of the two yields better results than guiding them separately, and when tuned, they outperform the individual setups.

\subsection{Guidance vs Tuning}
While the idea-scenario for guiding hyperparameters would involve no tuning of either the initialization or meta-learning-rate values, the \textit{short-horizon} problem faced by the learning rate necessitates tuning of the meta-learning-rate. However, for guided learning to be useful it simply needs to be a more efficient means of discovering useful hyperparameters than common HPO methods. In Figure \ref{fig:tuning} (and Table \ref{tab:tuned} we observe that guiding $\beta_1$, $\alpha$, or their combination actually outperforms Bayesian optimization (BO) in both absolute performance and in efficiency. Relative to BO with 100 training runs, guided learning achieves better results in each setting with a sweep over just 4 meta-learning-rate values. One potential reason is that the BO setup can only tune constant values, while the guided learning setup is capable of evolving hyperparameters over the course of training

\subsection{Guiding T5}
Our T5 guided experiments on individual parameters replicate the parameter evolution seen in the NMT setting, but make only infinitesimal improvements over the baseline\footnote{See Appendix \ref{sec:appendix_t5} for figures.}. Interestingly, guiding both hyper-parameters simultaneously (Figure \ref{fig:t5}) appears to change the dynamics of their evolution to great effect, and we find a $3$X improvement in speed of convergence and significant increase in final performance for multiple different settings of meta-learning rate (Table \ref{tab:t5}). The plot of $\beta_1$ itself is familiar from the previous NMT experiments but the evolution of $\alpha$ takes on an complicated trajectory that appears related to the schedule of $\beta_1$.

The rough generalization of the guided ``recipe'' from NMT to T5 is encouraging for the possibility of finding generalized guided learning settings which perform well across domains. The demonstrated capacity to improve both speed and performance of T5 pretraining is a significant endorsement of the pragmatic value of the technique to improve over even well-tuned baselines. The superior performance of guiding the hyper-parameters jointly over each individually is a promising and curious result that hints at both the potential and the complexity of learning multiple kinds of hyper-parameters simultaneously; an obvious candidate for further exploration.

\section{Broader Impact}\label{sec:impact}
Since \cite{wu2018} described short-horizon bias for greedy methods, work in the gradient-based HPO community has progressed towards more complex methods which seek to address short-horizon bias with longer horizons \citep{micaelli2021} or by other means \citep{donini2019}. Our result showing the absence of bias for learning momentum, and easy performance gains for NMT when doing so, should encourage further evaluation of the behavior of diverse learnable hyper-parameters under greedy meta-optimization. Additionally, we have shown that intuitions about the short-horizon bias do not fully explain the behavior of the learning-rate scalar, which increases at the start of training when learned alongside momentum. These observations, taken together, should encourage further exploration of greedy gradient-based methods.
We do not anticipate this work having potential negative societal impacts beyond those posed by automated methods in machine learning in general.
Rather we hope that it may contribute towards the realization of efficient and general gradient-based HPO, which will help improve the efficiency of training models, reduce energy consumption, and democratize access to machine learning. We hope that our encouraging results and release of the code we used to produce them\footnote{\url{https://www.github.com/google-research/google-research/tree/master/gradient\_based\_tuning}} will facilitate future work within the research community and give practitioners the tools to apply gradient-based HPO in diverse settings.

\section{Related Work}\label{appendix:related_work}
\input{2_related_work}

\section{Algorithm}\label{appendix:algorithm}
\input{algorithm}

\input{9a_setup}
\section{Tuning $\beta_1$ via Bayesian optimization}\label{appendix:tuning_beta1}

\setcounter{figure}{0}
\begin{figure}[!ht]
	\centering%
    \includegraphics[width=0.5\linewidth]{draft_fig_app_b1_vizier.pdf}
        \caption{Learning $\beta_1$ for NMT models, comparison of untuned learned and baseline runs (solid lines) to BO-tuned learned and baseline runs (dotted lines). These runs correspond to those reported in the $\beta_1$ column in Table \ref{tab:tuned}.
        }
	\label{fig:draft}
\end{figure}

\pagebreak

\section{T5 experiments}\label{appendix:t5}

\begin{figure}[!ht]
	\centering%
    \includegraphics[width=0.5\linewidth]{draft_app_lr_t5_automl.pdf}
        \caption{Learning $\alpha$ alone for T5 pretraining, comparison of a sweep over color-coded meta learning rates to baseline (black). The lowest meta-learning rate setting ($1\text{e-}6$, in blue) does outperform the baseline, but only very slightly. The short-horizon bias is evident; note that all learning rate scalars only decrease relative to the baseline learning rate schedule.
        }
	\label{fig:t5alpha}
\end{figure}

\begin{figure}[!hb]
	\centering%
    \includegraphics[width=0.5\linewidth]{draft_app_beta_t5_automl.pdf}
        \caption{Learning $\beta_1$ alone for T5 pretraining, comparison of learned (meta learning rate $3\text{e-}3$) runs vs baseline for initializations [$0.1$, $0.5$, $0.9$], and default baseline $0.0$. Note that learned $\beta_1$ values converge to the same gradually decaying schedule, similar to that of the NMT models in Figure \ref{fig:beta1_sweep}. The runs on that schedule do very slightly out-perform the non-learned hyperparameter runs. However, unlike in the NMT case, none of the changes in $\beta_1$, dynamic or static, have a significant impact upon the accuracy of the model at any point in training. This suggests that this setup is simply insensitive to the value of $\beta_1$.
        }
	\label{fig:t5beta}
\end{figure}

\pagebreak

\subsection{Downstream NLU Task Full Results}\label{appendix:t5_down_results}

\begin{table}[h!]
\centering
  \begin{tabular}{p{0.9cm}|*{8}{c}}

    \toprule
    & GLUE & CoLA & SST & MRPC & MRPC & STS & STS & QQP \\
    & avg & $\phi{}$\textit{corr} & \textit{acc} & \textit{F1} & \textit{acc} & \textit{Pearson} & \textit{Spearman} & \textit{F1} \\% & \textit{acc} & \textit{acc} & \textit{acc} & $\myovline{\textit{met}}{0.2pt}${} & \\
    \cmidrule(lr){2-9}
    base & 77.8 & 47.9 & 92.3 & 90.3 & 86.7 & 84.0 & 84.3 & 85.6 \\
    $\alpha$\scalebox{1}{+}$\beta_1$ & 78.2 & 44.5 & 92.5 & 91.7 & 88.2 & 84.9 & 85.3 & 86.0 \\
    \midrule
     & QQP & MNLI-m & MNLI-mm & QNLI & RTE & WNLI & SQuAD & SQuAD \\
    & \textit{acc} & \textit{acc} & \textit{acc} & \textit{acc} & \textit{acc} & \textit{acc} & \textit{EM} & \textit{F1}  \\
    \cmidrule(lr){2-7}
    \cmidrule(lr){8-9}
    base & 89.2 & 83.8 & 84.5 & 90.2 & 67.9 & 57.7 & 88.2 & 80.1  \\
    $\alpha$\scalebox{1}{+}$\beta_1$ & 89.5 & 84.6 & 85.2 & 90.5 & 69.0 & 59.2 & 88.3 & 80.6 \\
    \toprule
    & sGLUE & BoolQ & CB & CB & COPA & MultiRC & MultiRC & ReCoRD \\
    & avg & \textit{acc} & \textit{F1} & \textit{acc} & \textit{acc} & \textit{F1a} & \textit{EM} & \textit{F1} \\
    \cmidrule(lr){2-9}
    base & 67.1 & 73.6 & 98.1 & 98.7 & 58.0 & 18.9 & 68.5 & 63.3 \\
    $\alpha$\scalebox{1}{+}$\beta_1$ & 68.5 & 73.6 & 94.6 & 96.0 & 61.0 & 22.6 & 69.7 & 63.6 \\

    \midrule
    & ReCoRD & RTE & WiC & WSC & & & & \\
    & \textit{acc} & \textit{acc}  & \textit{acc}  & \textit{acc}  & & & & \\
    \cmidrule(lr){2-9}
    base & 64.3 & 64.6 & 67.1 & 67.3 \\
    $\alpha$\scalebox{1}{+}$\beta_1$ & 64.5 & 66.4 & 67.6 & 74.0 \\
    \toprule
  \end{tabular}
  \caption{Fine-tuning baseline and learned $\alpha$\scalebox{1}{+}$\beta_1$ models on SQuAD, plus the 9 GLUE and 8 superGLUE (sGLUE) downstream NLU tasks. All values are the \textit{max} of 5 separate runs.}
  \label{tab:t5_down_results_a}
\end{table}

\begin{table}[h!]
\centering
  \begin{tabular}{p{0.9cm}|*{8}{c}}

    \toprule
    & GLUE & CoLA & SST & MRPC & MRPC & STS & STS & QQP \\
    & avg & $\phi{}$\textit{corr} & \textit{acc} & \textit{F1} & \textit{acc} & \textit{Pearson} & \textit{Spearman} & \textit{F1} \\% & \textit{acc} & \textit{acc} & \textit{acc} & $\myovline{\textit{met}}{0.2pt}${} & \\
    \cmidrule(lr){2-9}
    base & 79.8 & 47.3 & 92.2 & 89.9 & 86.0 & 83.6 & 84.0 & 87.2 \\
    $\alpha$\scalebox{1}{+}$\beta_1$ & 80.1 & 43.6 & 92.5 & 90.8 & 87.0 & 84.6 & 85.0 & 87.2 \\
    \midrule
     & QQP & MNLI-m & MNLI-mm & QNLI & RTE & WNLI & SQuAD & SQuAD \\
    & \textit{acc} & \textit{acc} & \textit{acc} & \textit{acc} & \textit{acc} & \textit{acc} & \textit{EM} & \textit{F1}  \\
    \cmidrule(lr){2-7}
    \cmidrule(lr){8-9}
    base & 90.5 & 83.8 & 84.4 & 89.1 & 64.8 & 57.5 & 88.1 & 80.0  \\
    $\alpha$\scalebox{1}{+}$\beta_1$ & 90.5 & 84.5 & 85.1 & 90.3 & 66.0 & 57.2 & 88.2 & 80.5 \\
    \toprule
    & sGLUE & BoolQ & CB & CB & COPA & MultiRC & MultiRC & ReCoRD \\
    & avg & \textit{acc} & \textit{F1} & \textit{acc} & \textit{acc} & \textit{F1a} & \textit{EM} & \textit{F1} \\
    \cmidrule(lr){2-9}
    base & 64.8  & 73.1 & 97.9 & 98.4 & 56.4 & 6.4 & 62.1 & 63.1 \\
    $\alpha$\scalebox{1}{+}$\beta_1$ & 67.3 & 72.7 & 93.2 & 93.1 & 57.2 & 21.0 & 69.4 & 63.3 \\

    \midrule
    & ReCoRD & RTE & WiC & WSC & & & & \\
    & \textit{acc} & \textit{acc}  & \textit{acc}  & \textit{acc}  & & & & \\
    \cmidrule(lr){2-9}
    base & 64.1 & 63.8 & 66.1 & 63.5 \\
    $\alpha$\scalebox{1}{+}$\beta_1$ & 64.3 & 65.9 & 66.7 & 73.5 \\
    \toprule

  \end{tabular}
  \caption{Fine-tuning baseline and learned $\alpha$\scalebox{1}{+}$\beta_1$ models on SQuAD, plus the 9 GLUE and 8 superGLUE (sGLUE) downstream NLU tasks. All values are the \textit{average} of 5 separate runs.}
  \label{tab:t5_down_results_b}
\end{table}

\section{Licensing of Data}\label{appendix:license}
WMT (Workshop on Machine Translation) 2019:
\textit{http://www.statmt.org/wmt19/translation-task.html} and downloaded from \textit{https://www.tensorflow.org/datasets/catalog/wmt19\_translate}

\begin{quote}
LICENSING OF DATA (from the statmt.org website) states that it can be used for research purposes:

\textit{The data released for the WMT19 news translation task can be freely used for research purposes, we just ask that you cite the WMT19 shared task overview paper, and respect any additional citation requirements on the individual data sets. For other uses of the data, you should consult with original owners of the data sets.}
\end{quote}

The C4 dataset is released by Google, available at
\textit{https://www.tensorflow.org/datasets/catalog/c4}, licensed by
\textit{Creative Commons Attribution 4.0 License}
\section{Setup}\label{appendix:setup}

\subsection{NMT Experiments}

\subsubsection{Model}
For all experiments, we use the JAX framework \citep{jax2018github}, building off of models from the flax library \citep{flax2020github}. We use Transformer models \citep{vaswani2017attention} and the LAMB optimizer \citep{lamb}, with a 32k sentence-piece vocabulary \citep{kudo-richardson-2018-sentencepiece} for each language pair. Our Transformers have 8 heads and 6 layers with a total of 121M parameters, and for the LAMB optimizer we use the default values of $\beta_1$, $\beta_2$, and $\epsilon$ as $0.9$, $0.999$, and $1\text{e-}6$ respectively.

\subsubsection{Data}
We train on $6$ different language pairs, with training, development, and test sets drawn from the WMT19 machine translation task \citep{barrault-etal-2019-findings}. We tokenize the language pairs into joint 32K subword vocabularies with SentencePiece models \citep{kudo-richardson-2018-sentencepiece}. After filtering the datasets slightly by language ID and with length-based heuristics, we remove a single batch of the remaining data to set aside as a guidance set for each language pair. This is based on our preliminary experiments where we found no change in performance between holding out 1\% of the training data for guidance (iterated through repeatedly over training) or holding out a single batch (applied at every step), so throughout this work we hold out only a single batch for the guidance set\footnote{We likely see no difference because at most we learn two hyperparameters. With higher-dimensional learned hyperparameterizations, overfitting on the guidance set may become a concern that can be addressed by iterating through a larger guidance dataset.}.
The resulting dataset sizes are shown in Table \ref{tab:data}.

\begin{table}[h]
  \footnotesize
  \caption{Comparing dataset sentence count across language pairs. The acronyms de, en, fi and lt refer to German, English, Finnish and Lithuanian, respectively.}
  \centering
  \begin{tabular}{c|cccccc}
    \toprule
    & de-en & en-de & fi-en & en-fi & lt-en & en-lt \\

    \midrule
    train & 32M & 32M & 5.5M & 5.5M & 1.9M & 1.9M \\
    guide & 2165 & 2227 & 2363 & 2337 & 2339 & 2305 \\

    \bottomrule
  \end{tabular}
  \label{tab:data}
\end{table}

\subsection{Training}
We train with dropout and attention dropout both set to $0.1$, and without label smoothing or weight decay regularization. The default learning rate is set to $0.4$, which follows a square-root decay schedule following a a linear warmup of $4000$ steps. We use a training batch size of  ${\sim}2,300$ examples on average. In the experiments where we compare learned hyperparameters to Bayesian HPO \citep{Snoek12}, the objective for the BO is to minimize the loss on the development set, and we select the best of 100 trials for each BO run.

\subsubsection{Evaluation}
We decode with beam search decoding with a beam size of 4, and report BLEU \citep{papineni-etal-2002-bleu} scores calculated using the sacreBLEU tool \citep{post-2018-sacrebleu}.

\subsection{T5 Experiments}

\subsubsection{Model}
For the T5 experiments, we pretrain \textit{T5} models \citep{raffel2019exploring} using the \textit{Adafactor} optimizer \citep{shazeer2018adafactor}. We train a T5 model in the \textit{small} configuration, with 8 layers and 6 attention heads per layer and a total of 60M parameters. For Adafactor we use a default learning rate of $1\text{e-}3$ and a \textit{decay\_rate} of 0.8.

\subsubsection{Data and Training}
For pretraining, we train for 1M steps on the C4 dataset \citep{raffel2019exploring}, using a 32k sentence-piece vocabulary, the same as in the original T5 paper. We use a batch size of 256 packed examples and a maximum input length of 512 sentence-pieces, with dropout set to 0.0. We use a learning rate of 0.01 with 10000 steps of constant value followed by reciprocal square root decay. The unsupervised objective is the same masked language modeling objective that was proposed in the original T5 paper. 15\% of tokens are masked in the input sequence, replacing each masked span with a sentinel token. The model is then trained to predict the missing text for each sentinel token.

\subsubsection{Evaluation}
In pretraining, we report the loss on the C4 development set. For finetuning, we evaluate the appropriate metrics for each of the GLUE and superGLUE tasks. To arrive at the final average for each set of tasks, we follow the T5 paper in averaging the metrics within each tasks (to get the $\myovline{\textit{met}}{0.2pt}${} values shown in Table \ref{tab:t5_down}) and then simply averaging those scores across the tasks of the super-task.

\subsection{Finetuning on Downstream NLU Tasks}\label{appendix:t5_down}

We finetune on downstream NLU tasks from the GLUE and superGLUE meta-tasks. We initialize from the 1M step pretraining checkpoints and train for an additional 250,000 steps with a batch size of 8, mirroring the T5 paper finetuning scheme \citep{raffel2019exploring}.

\subsection{Meta-Optimization}
In our hyperparameter learning experiments, we meta-optimize with Adam and its default hyperparameters. $\beta_1$, $\beta_2$, and $\epsilon$ are set to $0.9$, $0.999$, and $1\text{e-}8$ respectively. For both NMT and T5 experiments, we use a guidance batch size mirroring the size of the training batch in each setting.

While model parameters may be allowed to take positive or negative values, the hyperparameters we study must be bound to a range of appropriate values; the learning rate must be positive and momentum must be between 0 and 1. To achieve this, we pass the learned hyperparameters through an activation function; exponential for learning rate and sigmoid for momentum (Table \ref{tab:activation_functions}). Unlike other hyperparameters, the learning rate is frequently set on a pre-determined schedule. In order to not override the pre-existing schedule, we learn a scalar $\alpha$ on the schedule which is initialized at 1.

\begin{table}[h]
  \footnotesize
  \caption{Learned hyperparameters and their activation functions.}
  \centering
  \begin{tabular}{c|c|c|c}
    \toprule
    hparam & activation fn & domain & init \\
    \midrule
    $\alpha$ & $e^x$ & (0, $\infty$) & 1 \\
    $\beta_1$ & $(1+e^{-x})^{-1}$ & (0, 1) & 0.9 \\
    \bottomrule
  \end{tabular}
  \label{tab:activation_functions}
\end{table}

\section{Hardware}\label{appendix:hardware}
For all experiments, we use TPUv3 with 16 cores. NMT training runs took $\sim$7 hours to train. T5 training took $\sim$48 hours for pretraining and $\sim$3-6 hours for finetuning depending on the task. In both setups, training runs that guided hyper-parameters took approximately 1.5X as long in terms of wall-clock time than baseline runs. The memory requirements of the guided and unguided runs were similar.

\pagebreak

\section{Introduction}\label{sec:introduction}
\input{1_intro}

\section{Method}\label{sec:method}
\input{3_method}

\section{Experiments}\label{sec:experiments}
\input{4_setup_brief}

\input{5_exp_new}

\section{Discussion and Limitations}\label{sec:conclusion}
\input{8a_discussion}

\section{Reproducibility Checklist}

\begin{enumerate}
\item For all authors\dots
  %
  \begin{enumerate}
  \item Do the main claims made in the abstract and introduction accurately
    reflect the paper's contributions and scope?
    %
    \answerYes{See Sections \ref{sec:experiments} and \ref{sec:conclusion}}.,
    %
  \item Did you describe the limitations of your work?
    %
    \answerYes{See latter portion of Section \ref{sec:conclusion}.}
    %
  \item Did you discuss any potential negative societal impacts of your work?
    %
    \answerNA{We anticipate no specific potential negative impacts beyond those of improving automated machine learning methods in general. We state this in Section \ref{sec:impact}.}
    %
  \item Have you read the ethics author's and review guidelines and ensured that your paper
    conforms to them? \url{https://automl.cc/ethics-accessibility/}
    %
    \answerYes{We do not violate the guidelines.}
    %
  \end{enumerate}
  %
\item If you are including theoretical results\dots
  %
  \begin{enumerate}
  \item Did you state the full set of assumptions of all theoretical results?
    %
    \answerNA{We present no theoretical results.}
    %
  \item Did you include complete proofs of all theoretical results?
    %
    \answerNA{We present no theoretical results.}
    %
  \end{enumerate}
  %
\item If you ran experiments\dots
  %
  \begin{enumerate}
  \item Did you include the code, data, and instructions needed to reproduce the
    main experimental results, including all requirements (e.g.,
    \texttt{requirements.txt} with explicit version), an instructive
    \texttt{README} with installation, and execution commands (either in the
    supplemental material or as a \textsc{url})?
    %
    \answerNA{We will release the code prior to the publication of the work. While this is clearly not the same as releasing it now (at submission time), we intend to do so as open-sourcing the code is a main aspect of the intended impact of the work.}
    %
  \item Did you include the raw results of running the given instructions on the
    given code and data?
    %
    \answerNA{See above.}
    %
  \item Did you include scripts and commands that can be used to generate the
    figures and tables in your paper based on the raw results of the code, data,
    and instructions given?
    %
    \answerNA{Close analogues of the figures in this paper will be automatically generated by the training code.}
    %
  \item Did you ensure sufficient code quality such that your code can be safely
    executed and the code is properly documented?
    %
    \answerYes{The code, which will be released prior to publication, will be well documented.}
    %
  \item Did you specify all the training details (e.g., data splits,
    pre-processing, search spaces, fixed hyperparameter settings, and how they
    were chosen)?
    %

    \answerYes{See Appendix \ref{appendix:setup}.}
    %
  \item Did you ensure that you compared different methods (including your own)
    exactly on the same benchmarks, including the same datasets, search space,
    code for training and hyperparameters for that code?
    %
    \answerYes{We took care to ensure our experiments comparing methods were fair, including in these mentioned categories.}
    %
  \item Did you run ablation studies to assess the impact of different
    components of your approach?
    %
    \answerYes{We vary $\hat{\eta}${}, combination of hyper-parameters, and in comparing NMT to T5 pretraining, we vary optimizer, model, and task.}
    %
  \item Did you use the same evaluation protocol for the methods being compared?
    %

    \answerYes{See section \ref{sec:experiments} and Appendix \ref{appendix:setup}.}
    %
  \item Did you compare performance over time?
    %
    \answerYes{See Figures in Section \ref{sec:experiments}.}
    %
  \item Did you perform multiple runs of your experiments and report random seeds?
    %
    \answerNo{We did perform multiple runs of the experiments but do not report random seeds.}
    %
  \item Did you report error bars (e.g., with respect to the random seed after
    running experiments multiple times)?
    %
    \answerNo{We do not report error bars, we report the max and average metric values over repeated runs.}
    %
  \item Did you use tabular or surrogate benchmarks for in-depth evaluations?
    %
    \answerNA{We do not employ NAS approaches.}
    %
  \item Did you include the total amount of compute and the type of resources
    used (e.g., type of \textsc{gpu}s, internal cluster, or cloud provider)?
    %

    \answerYes{See Appendix \ref{appendix:hardware}.}
    %
  \item Did you report how you tuned hyperparameters, and what time and
    resources this required (if they were not automatically tuned by your AutoML
    method, e.g. in a \textsc{nas} approach; and also hyperparameters of your
    own method)?
    %
    \answerYes{See Section \ref{sec:experiments}.}
    %
  \end{enumerate}
  %
\item If you are using existing assets (e.g., code, data, models) or
  curating/releasing new assets\dots
  %
  \begin{enumerate}
  \item If your work uses existing assets, did you cite the creators?
    %
    \answerYes{See Section \ref{sec:experiments}.}
    %
  \item Did you mention the license of the assets?
    %

    \answerYes{See Appendix \ref{appendix:license}.}
    %
  \item Did you include any new assets either in the supplemental material or as
    a \textsc{url}?
    %
    \answerNo{We will include a link to the code at publication time.}
    %
  \item Did you discuss whether and how consent was obtained from people whose
    data you're using/curating?
    %
    \answerNA{Our experiments were performed on publicly available datasets.}
    %
  \item Did you discuss whether the data you are using/curating contains
    personally identifiable information or offensive content?
    %
    \answerNo{None of our datasets contains personally identifiable information or offensive content.}
    %
  \end{enumerate}
  %
\item If you used crowdsourcing or conducted research with human subjects\dots
  %
  \begin{enumerate}
  \item Did you include the full text of instructions given to participants and
    screenshots, if applicable?
    %
    \answerNA{}
    %
  \item Did you describe any potential participant risks, with links to
    Institutional Review Board (\textsc{irb}) approvals, if applicable?
    %
    \answerNA{}
    %
  \item Did you include the estimated hourly wage paid to participants and the
    total amount spent on participant compensation?
    %
    \answerNA{}
    %
  \end{enumerate}
\end{enumerate}

\begin{acknowledgements}
The authors would like to thank Andrew Chou, Felix Stahlberg, Ji Ma,
and the anonymous reviewers, for their helpful comments.
\end{acknowledgements}

redpajama_set_name: RedPajamaArXiv


Our 1,100sqm building at Durham operates 30 fumehoods, including dedicated walk-in facilities, for milligram to kilo scale synthesis. Out staff are trained in safely handling pressure vessels, separation and purification techniques, corrosive materials (including HF) and potentially toxic reactions. We can work to full cGMP or traceable manufacture, if required.
We operate at scales of up to 30L in jacketed glassware at temperatures -40 – 180ºC . Our high pressure reactors are 7.5L and 10 bar maximum. Separation and purification techniques include semi-preparative and preparative chromatography (Isoleras/Biotage) and tangential crossflow filtration techniques.
2019 saw the opening of our dedicated Research facility at the North-East Technology Park (NETPark) in Sedgefield. This purpose-built facility for PhD and Graduate chemists is aimed at developing new compounds and collaborations with our academic and corporate customers.


View of the dam and water works at Fairmount, Philadelphia 1 print : engraving ; 24 x 42.4 cm (sheet) | Print shows a bird's-eye view below the dam at the Fairmount water works, in Philadelphia, with small boats and a steam ferry on the waterway.
Masonic Hall Chesnut St. Phila'a. erected A.D. 1813 destroyed by fire A.D. 1819 1 print : chromolithograph, color ; 65.8 x 55.2 cm (sheet) | Print shows the Masonic Hall from the street, with horse-drawn coach and pedestrians in the foreground. Includes "inscription on corner stone" at center, below image.
Life in Philadelphia. Plate 11 1 print : etching, hand-colored ; sheet 19.2 x 15.8 cm. | Print shows an African American woman, wearing a very large hat, in a shop trying to purchase a pair of stockings.
Life in Philadelphia. Plate 11 1 print : etching, hand-colored ; sheet 18.9 x 16 cm, on mount 25.6 x 21.2 cm. | Print shows an African American woman, wearing a very large hat, in a shop trying to purchase a pair of stockings.
Life in Philadelphia. Plate 13 1 print : etching, hand-colored ; 19.8 x 19 cm (sheet) | Print shows an African American couple walking on the street; the woman asks the man whether he likes the waltz.
Life in Philadelphia. Plate 14 1 print : etching, hand-colored ; 17.6 x 15.3 cm (sheet) | Print shows an African American woman in a millinery shop trying on a large bonnet; she asks the advice of a man who tells her he doesn't like it because it is too large and hides her face.


This entry was posted on Friday, February 20th, 2015 at 4:08 pm and is filed under Buildings & Grounds, Closings. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.


text:
Q: using memory barriers correctly I am trying to understand how do memory barriers work and how useful they are. I am aware of the fact that they are not worth complicating my code in most cases, but it's still an interesting topic. Trying to build a mental model around it but I don't quite succeed yet.
I have probably succeeded in understanding release and acquire enough to be able to correctly, or at least more correctly, use or understand volatile fields. My problem is usage of full memory barriers. For example, here is a link to some description:
https://afana.me/archive/2015/07/10/memory-barriers-in-dot-net.aspx/
In the section called "cache coherency" there is an example where cpu2 may not see the store to y even when the memory barrier is executed after memory barrier of cpu1. I get what happens here, most likely. But the fixed example contains two memory barriers, an additional one is after both stores in cpu1 and before both loads in cpu2. My specific question here is: isn't it enough to add a barrier before both loads and after both stores, but removing this one in between? why/why not? I will probably have more related question as this is not easy to get right even theoretically...

redpajama_set_name: RedPajamaStackExchange


text:
[Letterhead of Uber Technologies, Inc.]
Via EDGAR and Courier
Office of Trade & Services
Division of Corporation Finance
U.S. Securities & Exchange Commission
100 F Street, NE
Re: Uber Technologies, Inc.
Registration Statement on Form S-4
File No. 333-242307
Request for Effectiveness
Dear Mr. Morris and Ms. Jaskot:
Reference is made to the Registration Statement on Form S-4 (File No. 333-242307) filed by Uber Technologies, Inc. (the "Company") with the U.S. Securities and Exchange Commission on August 7, 2020, as amended on September 25, 2020, October 26, 2020 and November 12, 2020 (the "Registration Statement").
The Company hereby requests that the effective date for the Registration Statement be accelerated to 4:00 p.m., Eastern Time, on November 18, 2020, or as soon as possible thereafter, pursuant to Rule 461 promulgated under the U.S. Securities Act of 1933, as amended.
Please contact Andrew J. Nussbaum of Wachtell, Lipton, Rosen & Katz at (212) 403-1269 or Mark A. Stagliano at (212) 403-1060 with any questions you may have concerning this letter or if you require any additional information. We request that we be notified of the effectiveness of the Registration Statement by a telephone call to Mr. Nussbaum or Mr. Stagliano and that such effectiveness also be confirmed in writing.
/s/ Keir Gumbs
Keir Gumbs
cc: Tony West, Uber Technologies, Inc.
Bastian Lehmann, Postmates Inc.
Tad J. Freese, Latham & Watkins LLP
Chad G. Rolston, Latham & Watkins LLP
Andrew J. Nussbaum, Wachtell, Lipton, Rosen & Katz
Mark A. Stagliano, Wachtell, Lipton, Rosen & Katz

redpajama_set_name: RedPajamaCommonCrawl


Dr. David E. Wilhite is Professor of Christian Theology at Baylor's Truett Seminary. In this episode, he joins us to talk about Tertullian the African and his upcoming book on Ancient African Christianity. Read more about Dr. Wilhite here.


text:
The magick of National Socialism
The years following the Immolation of 56yf have witnessed much speculation concerning "occult" connections with National-Socialism. Most of the theories have been propounded in order to discredit National-Socialism by linking it with the "Black Arts", whilst others are well-meaning musings on such organisations as the Thule Society.The simple truth regarding National-Socialism and Occultism is that no such connections ever existed. They did not exist because National-Socialism did not then and does not now need some form of external occult aid or conspiracy in order for it to achieve victory. This is because National-Socialism is a complete world-view, an organic whole incorporating all aspects of the Aryan psyche. It is not an outer or exoteric shell which needs something esoteric or hidden to give it existence: it is the very essence itself - it is the cosmos made manifest in this world. It is both numinous and practical.
National-Socialism regards conventional forms of occultism as harmful - not because National-Socialism is fearful or ignorant of such things, but simply because it reasons that such things draw Aryans away from the spiritual purpose of their existence.This spiritual purpose, as will be explained, lies in an individual Aryan achieving a union with Nature, and thus an awareness of the truth of Racial identity and Destiny.National-Socialist Germany did away with 'occult' pursuits because, quite simply, such things were not needed.
With the triumph of a future Reich, such things again will not be needed: they will gradually fade from a National-Socialist society, because National-Socialism itself is the quintessential religious and "magickal" Western Way.The Meaning of Magick It is true to say that National-Socialism possesses "magick" - that is, it is true to say that it possesses the capacity, inherent in its various aspects, to cause significant Change. This form of Change is what occultists term as "magick" because it describes what is first desired and then willed into being. As to whether this magick is ultimately creative or destructive depends on whether an individual seeks to bring Change for selfish, personal ends, or whether that individual desires to aid the course of Nature which exists beyond all personal things. In choosing the latter, the individual "magickian" is positively aiding evolution.
National-Socialism therefore expresses the highest form of magick because this Change is willed not just for individuals but on a mass scale, in harmony with Nature via the quest to aid Racial survival and extend the Aryan legacy.This simple description of magick - the Will to Change - has been forgotten by those seeking gothic and elaborate explanations for the numinosity of National-Socialism. What was "magickal" in the most profound sense about National-Socialist Germany resided in its symbols, the speeches of its most inspiring activists, the heroism of the SS, the codes of behaviour founded primarily on Honour, the places where mass gatherings were held, and most significantly of all, in the very being of Adolf Hitler himself.
Is it any wonder that Berchtesgaden was completely razed to the ground, and every piece of rubble, even the surrounding trees, were removed by the "allies"? Is it any wonder the swastika is banned in most European countries? Is it any wonder the life of Adolf Hitler has become utterly defamed? All these things, and more, had and still have the power to cause Change - far greater than any "forbidden" rituals or ancient "words of power".Aryan Magickal Rites - The Simple TruthThe magick of National-Socialism is cosmological: that is, its magick is the life force of the cosmos, and to become a National-Socialist - a real Aryan - is to wield real power. But this power is only of use within the personal life if that personal life is in every respect dedicated to the Aryan cause. There is no personal gain, in terms of riches and selfish pleasures, to be had from this magick; no power over others, and no possibility of establishing a kingdom on Earth to the glory of oneself. Likewise, there are no demons to petition, no angels to converse with, and no wraiths to send out into the world to do one's bidding.
All those things belong to jewish magick - to the 'qabalistic' . The methods of Aryan magick are instead very simple, and would be a matter of course for the folk of an healthy Aryan civilization. Rather than comprising of complex and arcane ceremony, the genuine Aryan 'rites' of magick essentially involve the individual in striving to live the following nine National-Socialist principles:
1.) In everything that you do or undertake, strive for excellence.
2.) Do your duty by placing the welfare and well-being of your race before your own self-interest, and seek to preserve and extend your race by marrying among your own kind, and by producing/nurturing healthy children.
3.) Uphold the noble ideal of honour in your own personal life, and strive to live, and die, in an honourable way.
4.) Strive to uphold the noble, human, ideals of fairness and courtesy by being fair and courteous toward others, and strive to treat animals in a humane way.
5.) Be loyal to those you have sworn loyalty to, if necessary unto death. Your word, once given, should not be broken since to break your word is a dishonourable act.
6.) Be intolerant of what is harmful and unhealthy to, and what endangers, your race, and what is detrimental to the other creations of the divine.
7.) Reverence Nature and be respectful toward what reveals or expresses the divine.
8.) Always be ready, willing and physically fit enough to defend yourself and your family - and thus your own personal honour - and always carry a defensive weapon to enable your honour to be saved.
9.) Seek always to make the world a better, a more noble, place by striving to make others aware of the noble ideals of honour, loyalty and duty.
The tenets of the nine rites1 are the keys to enable Aryans to unlock the power of Aryan magick, and thus re-awaken the most awesome force this planet has ever witnessed. To re-iterate: there are no secrets behind the glorious but brief reign of the first National-Socialist Reich, and no secrets regarding the future emergence of a second National-Socialist Reich - only and simply what is described above by the nine principles, and the will to implement those principles.Thus it can be seen that National-Socialism acknowledges only the living supra-personal Being of the cosmos, revealed in Racial well-being and expansion. National-Socialism recognises the organic whole of Nature, and that we Aryans are an expression of this organic whole. When we live the above nine tenets - when we practice Aryan magick - we are this organic whole, conscious and evolving. Thus, to set up cosmic divisions in terms of "entities" - both "good" and "evil" - that exist independent of us and that require appeasing or controlling via "spells" or whatever, is absolutely against the Aryan character and against the Aryan wholistic understanding of Nature.Likewise it is against Aryan nature to waste life pouring over tomes of complex and elitist esoteric analyses of the cosmos (as in the many qabalistic writings).
National-Socialism instead seeks union with the cosmos according to the high standards of Aryan character and conduct, and the cultivation of an understanding of Nature as Nature is, without the sentimental and unnatural projections so prevalent in this modern age. Aryan "worship", as explained below, is expressed via practical, simple and spontaneous acts of reverence - acts that do not require individuals to suffer decades of ritualised ordeals and study in order to practice. All Aryans, from the moment they swear an oath of allegiance to National-Socialism, possess the full power of Aryan magick. Aryan Reverence"Worship" in the Aryan sense differs in all respects from the "worship" that characterises the judaeo-nazarene disease.
The practice of Aryanism, the National-Socialist religion, does not mean bowing and trembling before effigies of Adolf Hitler and others, as it does not mean that devotees of the religion march around wearing replica SS uniforms. For National-Socialists , worship invloves a natural expression of reverence for the example Adolf Hitler and others set us in the fight for Aryan well-being and evolution. It implies a re-awakening of the Aryan soul of which Adolf Hitler and others were the most recent and most profound expression. Aryanism reveres Adolf Hitler in the sense that he was a gifted mortal - as human as we are, but possessed of a greater awareness of the cosmic Being than others.
For National-Socialists and the devotees of Aryanism, he is a great Aryan prophet: he made conscious the revelation of Race, and revealed what was necessary in order to create a divine Aryan society. He is also a Hero2 in the greatest Aryan sense because he strove to create that society by the power of his will and by using his innate gifts and talents. Adolf Hitler therefore is revered as the greatest example we Aryans have of Aryan character. When we Aryans revere an individual we do so because they have, by the means of their life, set us an heroic example: thus we seek to emulate and measure our own conduct by that example (hence the practical purpose of the gods of our past). Usually, that individual is someone who still seems alive to us, who feels relevant - someone with whom we can imagine holding a conversation. Thus, as in Nature, we Aryans need renewal if the spirit of evolution is to guide our Race towards the genuine "philosopher' s stone": the goal of Wisdom. Consequently, it is the sacred duty of each new Aryan era to strive to spawn a new hero for our reverence.For when we Aryans worship, we draw forth from ourselves a new, living expression of the Aryan soul - as Adolf Hitler and others in their time brought forth a new expression via their own unique characters and insights. Fundamentally, when we Aryans worship - when we are living spiritually and physically as Aryans - there no longer exists for us a separation of the physical from what is sacred or divine: no division between the "gods" and Man. And thus, Heroes are born.Because of National-Socialism, "magick" has been taken out of the bedsit "temple" with its collection of seedy and superstitious artifacts and practices, and returned to the unified Aryan world of racial consciousness, creative supremacy, and the promise of the stars.
Because of the revelation of Adolf Hitler, a new, modern Aryan religion is born to guide us towards the attainment of living Divinity.A Race of Gods deserves nothing less.
Posted in: Anti Zog Media,Editorial,Eugenics,Indo-European,WW2


Is it a financial hurdle which is troubling you in the mid of the month? Are you looking for cash loans? If yes then don't look any more and apply with Bad Credit Loans Quick Decision! With us you will able to find these funds at better terms that too in least possible time!
Right after meeting these terms you will be approved without any delay! Upon approval, the amount will be electronically wired to your account!
Cash loans are short term funds which are needed to be repaid within 14 to 30 days. With these funds you will be able to raise amount up to AU$1000. We at Bad Credit Loans Quick Decision will definitely make sure to find you a productive deal! Apply with us and avail a tailor-made deal for your condition!
Those having imperfect credit will find it very simple to apply with us for cash loans. With Bad Credit Loans Quick Decision, you will be approved easily with just any kind of credit! Apply today and leave behind your bad credit records!
You need not fax even a single paper for getting cash loans with us! At Bad Credit Loans Quick Decision you will have fax less application process! Also, with us there are no application charges too!
We at Bad Credit Loans Quick Decision are available all day to serve you! To apply with us for cash loans, you just need to complete our free online application and nothing more. Our experts will contact you with the perfect deal right away!
Copyright © 2010. www.badcreditloansinstantdecision.com.au All rights reserved.
You Borrow - $400 You Pay - $496 Representative APR 1223.6% (Variable).


text:
Often asked: Are Anthony Rapp And Adam Rapp Related?
1 Is Adam Rapp married?
2 How old is Anthony Rapp?
3 How tall is Anthony Rapp?
4 Is the sound inside a musical?
5 How much is Kevin Spacey?
6 Who plays the female Klingon in discovery?
7 How old is Kevin Spacey?
8 Who plays Paul Stamets?
Is Adam Rapp married?
Mr. Rapp grew up in Joliet, Ill., a place he describes as not unlike Midlothia. Like Tobin, he is by now a longtime New Yorker, but he has not been married (he lives in the East Village with the actress Katherine Waterston) and his career is thriving.
How old is Anthony Rapp?
Anthony Rapp, Lt. Paul Stamets on Star Trek: Discovery, was born in Chicago and raised in Joliet, Illinois. An acclaimed actor and singer, he is most recognized for playing Mark Cohen in the off-Broadway and Broadway productions of the musical Rent, as well as in the film adaptation.
How tall is Anthony Rapp?
Actor Anthony Rapp (of Star Trek: Discovery) fronts a rock band called Albinokid, but is not albinistic, being a normally pigmented, blonde Caucasian. Former Mayor of London Boris Johnson, known for his pale hair, has also been mistakenly called albino.
Is the sound inside a musical?
Adam Rapp's play transfers to Broadway in a rivetingly dark and detailed production by David Cromer. 90 min. Or at least it does in Cromer's flawless production of "The Sound Inside," a play by Adam Rapp that opened at Studio 54 on Thursday.
How much is Kevin Spacey?
In spite of being cut out of movie and TV parts, his current net worth is $100 million, according to Celebrity Net Worth.
You might be interested: Readers ask: Are Antonio Cipriano And Renee Rapp Daing?
Who plays the female Klingon in discovery?
SFX Make up artists transform actress Mary Chieffo into Head of the Klingon High Council, L'Rell on Star Trek: Discovery.
How old is Kevin Spacey?
Paul Stamets is a character in the fictional Star Trek franchise. He appears in the television series Star Trek: Discovery. Stamets is portrayed by actor Anthony Rapp. The character is one half of the first openly gay regular character couple in a Star Trek television series.
Who plays Paul Stamets?
These days, Spacey is reportedly residing with his manager Evan Lowenstein, a former member of the boy band Evan and Jaron, per a THR source. As for where he's living, that's up for debate. One source told the outlet that he'd been holed up in London near the Old Vic theater he used to run.
Often asked: A.E Rapper Girlfriend?
Readers ask: Are Chance The Rapper Concerts 18 Year A?


Botanical Watercolour Painting workshop at my home studio, with a focus on summer flowers from the garden.The day includes full tuition, demonstration of techniques, support materials and refreshments, including my home made cakes.
Charges: £55 for the day. Paper can be purchased on the day and a palette of the colours I use can be 'rented'.


Home U.S. West - Photographs, Manuscripts, and Imprints Carmichael's Cut, Granite Canon.
Title Carmichael's Cut, Granite Canon.
Notes Recto: [imprinted] Carmichael's Cut, Granite Canon. Cut sixty feet from the surface to the level of the Railroad track, through hard red granite rock, and is nearly eight hundred feet in length.
Add tags for Carmichael's Cut, Granite Canon.


text:
NewsLafayette Parish
Two injured, one arrested in shooting on Wilkie St. in Lafayette
KATC
Lafayette Police are continuing to investigate a shooting Wednesday afternoon on Wilkie Street in which two people were injured.
Officers say they were dispatched to a burglary in progress at a residence in the 200 block of Wilkie Street around 12:03 pm on Wednesday. The victim was inside his residence when two people forced entry into his home, police say.
The victim armed himself with a handgun and fired shots, striking the two individuals, police say. They then fled the home and sought medical attention for their injuries at a local hospital. They remain in the hospital as of Wednesday night, a spokesperson says.
A third person, Antonio McClelland, 21, has been arrested and booked into the Lafayette Parish Correctional Center on one count of principal to aggravated battery and one count of conspiracy to commit armed robbery.
Police originally said three people were in custody. They say the investigation is ongoing and further arrests are expected.


text:
De Boeing 747-200 is de verbeterde versie van het vliegtuigtype Boeing 747-100. De eerste bestelling werd op 19 december 1968 door de Koninklijke Luchtvaart Maatschappij (KLM) geplaatst. De roll-out van dit type was op 10 september 1970. De eerste vlucht op 11 oktober 1970. Het eerste exemplaar voor de KLM, de PH-BUA met de naam Mississippi, werd afgeleverd op 15 januari 1971. Dit toestel werd op 25 november 1973 door drie Palestijnen gekaapt en in 1996 gesloopt. Een andere Boeing 747-200 van de KLM, de PH-BUK, is tegenwoordig te zien in het Nederlandse Luchtvaartmuseum Aviodrome.

De Boeing 747-200 werd ook gebruikt voor het vervoer van een vliegtuigmotor. Deze hing dan aan een van de vleugels, dicht bij de romp, maar had geen functie. Op de plaats van bestemming kon deze vijfde motor er weer afgehaald worden. Deze motoren werden soms bij reguliere passagiersvluchten meegenomen, bijvoorbeeld als een motor defect was en voor reparatie naar de plaats moest waar dat vliegtuig naartoe ging. Het was feitelijk een deel van de vracht die het vliegtuig vervoerde.

In de jaren 80 werd de Boeing 747-200 opgevolgd door de Boeing 747-300, die een stuk minder populair was, er werden van de 300-variant slechts 81 gebouwd, terwijl van de 200-versie uiteindelijk bijna 400 stuks zijn gebouwd. Wel zijn enkele Boeing 747-200's tot een -300 omgebouwd, waaronder tien 747-200's uit de KLM-vloot. In 1989 was de productie van de Boeing 747-400 al begonnen, deze opvolger werd wel een succes.

De laatste 747-200 (de VC-25A) werd in december 1990 afgeleverd aan de United States Air Force. De laatste 747-200F werd november 1991 afgeleverd aan Nippon Cargo Airlines.

Typen
Er zijn zes Boeing 747-200-typen geproduceerd.
747-200, de normale passagiersversie
747-200B, Better, de verbeterde passagiersversie.
747-200SUD, StretchedUpperDeck, een passagiersversie waarbij het bovendek is verlengd.
747-200F, Freighter, de normale vrachtversie.

747-200C, Convertible, een vliegtuig dat als vracht- en passagiersvliegtuig kan worden gebruikt.
747-200M, Mixed, een versie waarbij het toestel als vracht- en passagiersvliegtuig tegelijk dient.

Verkeersvliegtuig
0747-200

redpajama_set_name: RedPajamaWikipedia


Causeview can help you to represent the connections and associations between individuals and organizations accurately using accounts and affiliations. An account is used to specify an organization which employs an individual, or which an individual represents when interacting with your organization. When an account is in place, an individual is seen as a part of the organization. An affiliation, on the other hand, is used to indicate connections an organization has with other organizations and individuals outside of the group or company.
While both accounts and affiliations allow you to show a connection between an individual and an organization, only one account may be specified for an individual, while many affiliations can be formed. This is largely because an individual is expected to only represent or be employed by one organization at any time. However, an individual may be in professional or personal contact with many organizations at a time, and it can be useful to keep track of how your individual and organization constituents are connected.
Accounts and affiliations act differently within the Causeview application, and in order to understand which connection to use it is helpful to know how. An individual affiliated with an organization will appear on the organization's related list of affiliated contacts, and the organization will appear on the individual's related list of organization affiliations. This is the only impact of an affiliation upon either constituent. When an individual falls under an account, the name of the organization appears in the "Account" field near the top of the individual's detail page, the individual is shown in the related list of individuals on the organization, an automatic affiliation may be created between the organization and individual, and the organization will appear as the account for any opportunities created from the individual. Perhaps most importantly, the organization will receive transactional credit for any opportunities the individual is involved in if the individual has an account. The functional differences between affiliations and accounts reflect their inherent conceptual differences.