text:
\section{System Design and Execution}
\label{sec:system}

The baseline system comprises a server multi-core CPU connected to multiple GPU devices via the PCIe-link.
The inter-GPU communication happens through NVLink~\cite{nvlink}.
Figure~\ref{fig:system} shows the \hotline system; it places the accelerator on the \emph{low profile PCIe slot} not being used by a GPU.
The accelerator can access the DMA engine via the PCIe switch and can directly communicate with the main memory whilst entirely avoiding the high-overhead CPU pipeline.
\hotline does not require any changes to CPU and GPU devices.
The accelerator employs two phases during training: (1) The `access learning' phase identifies the \highaccess embeddings.(2) The `runtime accelerated' phase pipelines popular and non-popular mini-batches to the GPU.

\if 0
\begin{figure}
	\centering
	\includegraphics[width=0.45\textwidth]{body/Graphs/Sampling/samplinghotline.pdf}
	\vspace{-1ex}
	\caption{Determining the mini-batch sampling rate.}
	\label{fig:sampling}
	\vspace{-2ex}
\end{figure}
\fi

\subsection{Hotline Training Execution Phases}
\subsubsection{Access Learning Phase}
\label{sec:access_learning}
During training runtime, in the first phase, called the access learning phase, the \hotline accelerator samples mini-batches to determine which embeddings are \highaccess.
This is performed \emph{only} in the first epoch \emph{only} for a \% of mini-batches.
Our experiments show that, the accelerator only needs to sample 5\% of the mini-batches for datasets like Criteo Kaggle~\cite{criteokaggle} and Criteo Terabyte~\cite{criteoterabyte}.
It only needs to sample 15\% and 20\% of the mini-batches for Avazu~\cite{avazu} and Taobao Alibaba~\cite{alibaba} datasets, respectively.
Such low-overhead sampling can identify $>$90\% of \highaccess embeddings.

During this phase, all the embedding entries are obtained from the CPU main memory and gradually some of these entries are classified as \highaccess.
Once the first phase completes, the contents of the \highaccess embeddings are replicated across all the GPUs.
The \hotline accelerator only stores the \emph{indices} of the \highaccess embeddings and does not need to store the data contents of \highaccess embeddings.
The \highaccess embedding indices are used by the accelerator to classify popular and non-popular inputs into mini-batches.
The detailed execution of the accelerator during this phase is described in Section~\ref{sec:architecture}.

\subsubsection{Runtime Accelerated Phase}
Once the \highaccess embeddings are replicated on each GPU, the \emph{runtime accelerated phase} commences.
The \hotline accelerator reorders popular and non-popular inputs across a collection of mini-batches.
We refer to these mini-batches as the working set of the accelerator.
The \hotline accelerator creates \emph{new} mini-batches from this working set.

Now a mini-batch contains inputs that only accesses the \highaccess embeddings (popular mini-batch) or contain inputs that access both \highaccess and not-\highaccess embeddings.
As the GPU memories contain the \highaccess embeddings, the popular mini-batches can be directly scheduled onto the GPUs.
The accelerator obtains the embeddings from both the CPU and GPU memories while GPUs are executing popular minibatches and hides the embedding gather latency.
CPU and GPU memories have the most updated not-\highaccess and \highaccess embeddings respectively.
\hotline does not encounter any synchronization or cache coherence overheads because the not-\highaccess embeddings and \highaccess embeddings are always accessed from the CPU main memory and GPUs respectively.

\subsection{Pipelined Execution at Runtime}
Figure~\ref{fig:hotlinepipe} shows an example of the pipelined execution across the CPU, the \hotline accelerator, and the GPU.
All the training inputs are obtained from the CPU main memory.
If we have a working set of four mini-batches, on average, three mini-batches worth of inputs would be popular.
This is because, on average, $>$75\% of the inputs are popular and they can be directly scheduled onto the GPUs.
While the popular mini-batches are executing on the GPU, the accelerator obtains the embeddings for the non-popular minibatch from CPU and GPU memories.
Once the GPU is done executing popular mini-batches, the accelerator schedules the non-popular mini-batch to the GPUs along with its working parameters.

The forward and backward pass for the entirety of the recommender model executes on GPUs for both the popular and non-popular mini-batches
This is because, now the required parameters are always in the local GPU memory.
As the non-popular mini-batch is executing on the GPU, the accelerator begins to read the next set of mini-batches from the CPU main memory.
The details of how the accelerator segregates mini-batches and gathers embeddings are provided in Section~\ref{sec:hotlineexample}.

\section{Background and Challenges}
\label{sec:background}

\subsection{A Primer on Recommendation Systems}
\subsubsection{Machine Learning Model:}

Recommender models use categorical features and continuous features. Typically, large embedding tables store the categorical features for each user and item.
On the other hand, the continuous features are processed by reasonably-sized neural network layers which range from Multi-layer Perceptrons (MLP) to DNNs.
Recommendation models are currently among the largest ML models that are being executed in data centers.
A recently released recommendation model by Meta contains 12 Trillion parameters~\cite{softwarehardware} in comparison to GPT-3 with 175 Billion parameters~\cite{gpt3}.

\subsubsection{Training Setup:}
Large recommender models can be trained in two distributed modes.
First, the embeddings are stored and gathered on CPU and the neural networks are executed on GPUs in a data-parallel fashion.
Now, the training throughput is throttled by large data transfer and low bandwidth CPU main memory.
Figure \ref{fig:motivation} shows the training time being spent on these operations in 4 real-world models and datasets.
We observe that the communication and embedding-bag operation can be as high as 75\% for large datasets such as Criteo Terabyte.
This presents an opportunity for acceleration.

\begin{figure}[h!]
	\centering
	\includegraphics[width=1\columnwidth]{body/Graphs/motivation/motivation.pdf}
	\caption{The breakdown of the training time for an Intel-optimized DLRM with 1 GPU in a hybrid CPU-GPU training setup. The dotted parts of the bar are executed on CPU and present an opportunity for GPU-based acceleration.}
	\label{fig:motivation}
\end{figure}

\if 0
\begin{figure}[h!]

	\centering
	\includegraphics[width=0.45\textwidth]{body/Graphs/communication/cpu_gpu_comm.pdf}
	\vspace{-0.05in}
	\caption{Percentage of training time spent on CPU-GPU communication. For large datasets like Criteo Terabyte, the embedding transfer time can be as high as 40\%.}
	\label{fig:comm}
\end{figure}
\fi

Second, we can use as many GPUs as required to store the embeddings fully while enabling data-parallel neural-network execution.
However, this mode has low compute utilization, because the number of GPUs scale based on embedding sizes and not with the requirements of the neural network.
On the contrary, our paper exploits the semantics of user behaviour, i.e. training input, in recommendation models to efficiently use the memory capacity and improve compute throughput.

\subsubsection{Popularity in Training Inputs:}

\hotline exploits a key property of recommendation system inputs -- that certain users and items observe very high popularity.
This implies that some embedding entries are accessed much more frequently.
Figure~\ref{fig:ip_emb_acc} shows that, across real-world datasets, a small number of \highaccess embeddings have $>$100$\times$ more accesses than others.
Our experiments show that simply tracking up to 512~MB of embeddings can help cater to $>$75\% of training inputs.
Tracking beyond 512~MB of \highaccess embeddings only provides diminishing returns.
However, exploiting input popularity at a fine granularity is non-trivial.
\begin{figure}[h!]
	\centering
	\includegraphics[width=0.5\textwidth]{body/figures/fig3_updated.pdf}
	\caption{The frequency of accesses per embedding entry for one training epoch. The \highaccess embeddings tend to have more than 100$\times$ more accesses compared to others. Inputs with rate of 1-in-every-100000 embedding accesses are labeled as \highaccess.}
	\label{fig:ip_emb_acc}
\end{figure}

\subsection{Challenges in Exploiting Input Popularity}

\niparagraph{Limitation -- Segregation and Scheduling with CPU:}
One could simply employ CPU-based multi-processing to segregate \highaccess and not-\highaccess embeddings and schedule the corresponding mini-batches. Figure~\ref{fig:hotline_cpu_algo} shows that an Intel-Xeon CPU, even after using \emph{all its cores} for multi-processing, has up to 2.5$\times$ higher latency as compared to NVIDIA-V100 GPU(s) executing 4 training iterations. Additionally, as we execute larger mini-batch sizes, the CPU processing overhead and its latency increases proportionately. Therefore, typical CPUs \emph{cannot} complete segregating popular and not-popular mini-batches and schedule them before the GPU(s) finish execution.

\begin{figure}[h]

	\centering
	\includegraphics[width=1\columnwidth]{body/Graphs/hotline_cpu_algo/hotline_cpu_algo.pdf}
    \vspace{-1ex}
	\caption{Comparison of multi-process CPU-based segregation and scheduling time using an Intel Xeon CPU for 4 training iterations with NVIDIA-V100 GPU(s). We use mini-batches of 1K, 2K, and 4K for 1, 2, and 4-GPU execution respectively. Even after using multiprocessing, the CPU has a higher latency and cannot finish segregating and scheduling popular and not-popular mini-batches before the GPU(s) finish execution.}
	\label{fig:hotline_cpu_algo}

\end{figure}

\niparagraph{Challenge 1 -- Tackle Changing Access Frequency:}
Prior works use an \emph{offline} profiler to identify the \highaccess embeddings. They use two suppositions~\cite{fae, recshard}.
First, that training data is fully available before training.
Second, that \highaccess embeddings do not change over time.

We observed that training data is typically updated on a daily basis.
Thus, one cannot access the full training dataset ahead in time.
Furthermore, as the dataset evolves daily and a recent work, called BigPipe~\cite{bagpipe}, has shown the set of \highaccess embeddings also rapidly change over time.

\niparagraph{Challenge 2 -- Dynamically Capture the Access Skew:}
We need to dynamically determine the \highaccess embedding entries at very low latency.
This helps avoid extra overheads in the case of online training where training samples could arrive sporadically.
We must also adapt to the changing popularity of inputs and update the \highaccess embedding entries continuously with very low overheads.

\begin{tcolorbox}
\textbf{Our Approach}: \hotline dynamically determines the \highaccess embeddings by processing only a small portion of the training inputs.
\hotline uses a novel accelerator to enable fine-grained parallel embedding index tracking across mini-batches.
The \hotline accelerator can re-calibrate the \highaccess embeddings as the training data changes over time.
\end{tcolorbox}

\niparagraph{Challenge 3 -- Training with Mixed Mini-Batches:}
Each mini-batch can require a mix of \highaccess and other embeddings.
Thus, the working parameters can be spread across the CPU main memory (for not-\highaccess embeddings) and GPU's (for \highaccess embeddings) local memory.
Consequently, embeddings from the main memory would need to be collected and sent to the GPU before computation.
Gathering working parameters using GPUs is not ideal as they are not only inefficient at collecting irregularly scattered embeddings, but would also waste their precious compute cycles on intricate fine-grained operations.
\begin{tcolorbox}
\textbf{Our Approach}: The \hotline accelerator uses immensely parallel compute engines that gather several inputs. The accelerator then creates mini-batches of inputs that only access \highaccess embeddings and dispatches them to GPUs. While the \highaccess inputs are being executed, the accelerator gathers the embeddings from CPU main memory for the remaining not-\highaccess embeddings inputs thereby hiding their communication overheads.
\end{tcolorbox}

\if 0
\begin{figure}[ht!]
	\vspace{-1ex}
	\centering
	\includegraphics[width=0.45\textwidth]{body/figures/recomodel.pdf}
		\vspace{-1ex}
	\caption{Typical recommender systems~\cite{dlrm,neuralcf,tbsm} model. Models comprise of an interconnection of compute-bound Neural Networks like DNNs and MLPs in tandem with the memory-bound (size and bandwidth) Embedding Tables.}
	\label{fig:recomodel}
	\vspace{-2.5ex}
\end{figure}

Most commonly, data parallel~\cite{dataparallel} training is deployed, where a mini-batch of training inputs is processed in parallel across devices and model parameter gradients are computed.
Subsequently, these gradients are synchronized across the entire mini-batch and fed into the optimizer to obtain the final feature update.
As the size of DNNs is increasing, one single GPU or accelerator cannot handle the entire model, thus in addition to data parallel, model~\cite{modelparallel} and pipeline~\cite{pipedream, gpipe} parallel modes are also being employed.
Intermediate activations and results are transferred over the network to the next device in the pipeline.
Such techniques work efficiently in the case where these splits are load-balanced~\cite{deviceplacement:neurips} across devices to maximize throughput, i.e, each split often exhibits a high computational and memory intensity, thus utilizing resources effectively.
This is because, the device-to-device, especially GPU-GPU communication costs for some of these models can be as high as 60\% on average~\cite{pipedream, nvlink, blink}, especially if compute and communication are not overlapped.
\fi
\section{Conclusions}
\label{sec:conclusion}

Production-scale recommendation models deploy DNNs along with capacity and bandwidth-constrained embedding tables.
In this paper, we present \hotline, a heterogeneous acceleration pipeline based on the insight that small portion of embedding tables are frequently accessed.
\hotline aims to dispatch popular inputs that only access these \highaccess embeddings directly to GPU, while obtaining the not-\highaccess embedding data for non-popular inputs.
To perform the dynamic segregation and dispatching, we offer a novel accelerator that can perform the fine grained tasks of determining which inputs require what embedding entries, and if needed gather them from main memory and dispatch the inputs.
As such, \hotline hides the data transfer latency of obtaining non-\highaccess embedding data behind GPU execution of popular inputs.
Results on real-world datasets and recommender models show that \hotline reduces the average training time by \avgperfimprfourgpudlrm and \avgperfimprfourgpuhugectr in comparison to Intel optimized CPU-GPU DLRM and HugeCTR optimized GPU-only baseline, respectively.

\section*{Acknowledgements}

This project is part of the \textit{STAR Lab} at the University of British Columbia (UBC).
We express our thanks to the entire team of the Advanced Research Computing center at UBC~\cite{sockeye}.
This work was partially supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) [funding reference number RGPIN-2019-05059].
The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of NSERC, the Canadian Government, Microsoft, or UBC.

\balance

\section{Evaluation Methodology}
\label{sec:eval}

\subsection{Recommender Models}
Table~\ref{table:datasets} illustrates the specifications of four well-established recommender models used to evaluate \hotline.
The number of sparse parameters varies from 5.1M (\bench{RM1}) to 266M (\bench{RM3}).
The number of dense parameters is around 5 orders of magnitude less than the number of sparse parameters.
The bottom MLP computes on dense inputs, whereas the top MLP generates the final recommendation for the user.
\bench{RM1} is a neural network dominated model with top and bottom MLP layers and a deep learning attention layer.
\bench{RM2} and \bench{RM3} models, instead, are embedding dominated as they consist of more sparse features and bigger embedding tables.
\bench{RM4} has an average size of dense neural network and sparse embedding tables.
We use open source implementation of Deep Learning (DLRM)~\cite{dlrm} and Time-based Sequence Models (TBSM)~\cite{tbsm} benchmarks to train the above mentioned models.
TBSM trains the \bench{RM1} model and DLRM is used to train \bench{RM2}, \bench{RM3} and \bench{RM4} models.
TBSM consists of an embedding layer implemented using DLRM and a time-series layer resembles an attention mechanism and contains its own neural networks.

\subsection{Datasets}

We use four real-world datasets to train the above mentioned models.
Table~\ref{table:datasets} shows the details of each dataset.
Taobao Alibaba~\cite{alibaba} is a user behaviors dataset from Taobao, for recommendation problem with implicit feedback offered by Alibaba.
Criteo Kaggle~\cite{criteokaggle} dataset is obtained from Display Advertising Challenge and contains advertising data from Criteo to capture user's preference by predicting click-through-rate.
Criteo Terabyte \cite{criteoterabyte} is the largest publicly available dataset for users click logs.
Avazu~\cite{avazu} dataset is taken from click-through-rate prediction competition by Kaggle.

\subsection{Software libraries and setup}
DLRM and TBSM are configured using Pytorch-1.9 using Python-3.
We use the \emph{torch.distributed} backend to support scalable distributed training and performance optimization~\cite{pytorch}.
For all the backend GPU-to-GPU communication such as gather, scatter, and all-reduce collective calls, we use the fast NVIDIA Collective Communication Library (NCCL)~\cite{nccl} on NVLink-2.0.
We compare against DLRM and TBSM implementation on XDL~\cite{xdl}, Intel-optimized DLRM~\cite{dlrm-intel} and FAE~\cite{fae}.
We use Tensorflow-1.2~\cite{tensorflow} as computation backend for XDL-based implementation.

\begin{figure*}[t]
	\centering
	\includegraphics[width=1\textwidth]{body/Graphs/norm_perf/norm_perf.pdf}

	\caption{The performance comparisons \hotline with XDL, Intel optimized DLRM, and FAE implementations (normalized to a 1-GPU XDL). On average, even a 1-GPU \hotline provides \avgperfimpronegpudlrm higher performance as compared to the baseline.}
	\label{fig:performance}
\end{figure*}

\subsection{Server Architecture}

Table~\ref{tab:systemspecs} shows the details of the server.
They comprise 24-core Intel Xeon Silver 4116 (2.1 GHz) processor with Skylake architecture.
Each server is connected to 4 NVIDIA Tesla-V100 GPUs.
Every GPU and the \hotline accelerator communicates with the rest of the system via a 16x PCIe Gen3 bus.
We perform experiments on a single server.

\input{body/tables/system_config}

\subsection{Runtime measurements.}
For the CPU and GPU executions we measure the wall clock time to train the entire model based on its convergence criterion.
For \hotline accelerator, we synthesize the architecture implemented in Verilog RTL using Synopsys DC with 45$nm$ technology and validated 350~MHz as the frequency.
We obtain the access time of the memory components in the accelerator using Cacti~\cite{Cacti}.
We used a RTL-simulation for compute cycles and access cycles from Synopsys DC and Cacti to calculate the accelerator runtime.
Real system DMA latency and HBM latency are being used for estimating working parameters gathering time.
Also we empirically observed that fetching \highaccess embeddings alternatively from each GPU device would rule out the GPU HBM contention and results in balanced memory load on each GPU device.
The details of the accelerator are provided in Table~\ref{tab:accelspecs}.
We observe that a size of 2.5~MB for the input eDRAM is sufficient to store 16~K input non-popular mini-batch. The 0.5~KB embedding vector buffer can store an embedding vector with a dimension of 128. We chose the eDRAM~\cite{eDRAM} technology because of its high storage density.
We use a 4~MB embedding access logger (EAL).

\input{body/tables/accel_config}

\subsection{Area/Energy measurements of \hotline}

We use Cacti~\cite{Cacti} to estimate the area/energy of memory components (40~$nm$ technology).
We implemented the Verilog RTL and synthesized with Synopsys Design Compiler (45~$nm$ FreePDK process) for other components.

\if 0
\input{body/tables/arch}
\fi
\begin{figure*}[t]
	\centering
	\includegraphics[width=1\textwidth]{body/Graphs/perf_breakfown/perf_breadown_new.pdf}
	\caption{Latency breakdown of 1, 2, and 4 GPU implementations of software frameworks and \hotline. The overhead of CPU-GPU communication time increases as number of GPUs scale because of inter-GPU communication. Note that, unlike \hotline, FAE incurs a reasonably high overhead from embedding synchronization.}
	\label{fig:latencybreakdown}
\end{figure*}

\section{Results and Analysis}
\subsection{Training Accuracy}
\hotline segregates the mini-batches into popular and non-popular ones within the working set.
Hence, we evaluate the accuracy implications of \hotline.
Figure~\ref{fig:accuracy} illustrates the Area Under Curve (AUC) accuracy metric for Criteo Kaggle, Criteo Terabyte, and Avazu datasets as established by MLPerf~\cite{mlperf, mlperf_rec}.
For the Taobao dataset, we use the test accuracy -- as AUC metric is not offered for the TBSM model.
We use full-precision DLRM and TBSM model implementations using Intel's optimized DLRM~\cite{dlrm-intel} for accuracy comparisons.
We observe that \hotline completely follows the baseline test and train accuracy.
Table~\ref{table:accuracy} compares the testing accuracy, AUC, and cross-entropy loss for all workloads.

\input{body/tables/accuracy.tex}

\subsection{End-to-End Performance}
\subsubsection{Performance benefits over hybrid CPU-GPU Baselines}
Figure ~\ref{fig:performance} demonstrates the end-to-end training benefits obtained due to \hotline in comparison to three state of the art software implementations with varying number of GPUs.
First, XDL~\cite{xdl} is an Industrial Deep Learning Framework that uses a parameter server approach. Here the GPU improves the efficiency of Advanced Model Server by using a faster embedding dictionary lookup on GPU while CPU is used as backend workers.
Second, an Intel-Optimized DLRM~\cite{dlrm-intel} implementation, where the embeddings are executed on CPU using lock free updates and neural networks on GPUs in data parallel fashion.
Third, a software framework called FAE~\cite{fae}, that uses input popularity but combines \textit{offline pre-processing} with \textit{CPU based scheduling without pipelining}.
For the data-parallel approach, we deploy the weak scaling technique where we scale the mini-batch size with number of GPUs, i.e., 1K for 1-GPU setup, 2K for 2-GPU setup, and 4K for 4-GPU.
All the numbers are normalized to XDL 1-GPU setup.

As shown in Figure~\ref{fig:performance}, \hotline across all the models, datasets, and settings, reduces the overall training time as it performs the entire recommender model compute on the GPU(s).
On average \hotline obtains a speedup of \avgperfimpronegpuxdl and \avgperfimpronegpudlrm for 1-GPU, \avgperfimprtwogpuxdl and \avgperfimprtwogpudlrm for 2-GPU, \avgperfimprfourgpuxdl and \avgperfimprfourgpudlrm for 4-GPU over XDL and optimized DLRM implementations. As compared to a recent contemporary work called FAE~\cite{fae}, \hotline obtains a speedup of \avgperfimpronegpufae, \avgperfimprtwogpufae, and \avgperfimprfourgpufae for the 1-GPU, 2-GPU, and 4-GPU implementations respectively.

\subsection{Latency Breakdown}
Figure~\ref{fig:latencybreakdown} shows the latency breakdown of the two hybrid baselines and \hotline.
\hotline incurs a negligible overhead due to its efficient pipelining.
For the Criteo Kaggle and Terabyte datasets, which are more embedding and memory intensive, \hotline significantly reduces the CPU-GPU communication time.
For the Taobao dataset, which is neural network centric, the deep learning execution overshadows the communication time.
Broadly, \hotline always reduces the total compute time.

\subsection{Comparison Studies}
We compare \hotline against a GPU-only Model-Parallel and a CPU-based \hotline implementation.
\subsubsection{GPU-only Model-Parallel Baseline:}
We use HugeCTR~\cite{hugectr} by Nvidia as the GPU-only baseline.
HugeCTR scales the number of GPUs to fit the entire model within GPUs' local memories and performs model parallel training for embeddings while using data parallel training for the neural network portion.
As shown in Figure~\ref{fig:hugectr_perf}, we compare the performance of \hotline with HugeCTR for two models, Criteo Kaggle and Criteo Terabyte -- the only ones offered by the current HugeCTR evaluation.
We observe that for a small model like Criteo Kaggle, HugeCTR is able to fit and train the model on a single GPU.
The Criteo Kaggle results are thus normalized to 1-GPU HugeCTR.
For a large model like Criteo Terabyte, HugeCTR is unable to fit the model within 1 or 2 GPUs; throws an Out of Memory (OOM) error, and requires a minimum of 4 GPUs.
Hence, Criteo Terabyte results are normalized to 4-GPUs.

\begin{figure}[h!]
	\centering
	\includegraphics[width=1\columnwidth]{body/Graphs/hugectr_perf/hugectr_perf.pdf}
	\caption{The speedup of \hotline normalized to HugeCTR. \hotline provides a speedup of up to \avgperfimprfourgpuhugectr.}
	\label{fig:hugectr_perf}
\end{figure}

As shown in Figure~\ref{fig:hugectr_perf}, \hotline removes all-to-all communication associated with model parallel embedding layer of HugeCTR and outperforms in all the cases.
Moreover, for popular minibatches \hotline completely execute embeddings and neural layers in data parallel fashion, which mitigates any CPU-GPU communication overheads.
For non-popular mini-batches it incurs an overhead of obtaining embeddings from CPU memory.
However, this latency is hidden behind the execution of popular mini-batches.

\subsubsection{Comparison over CPU-based \hotline:}
CPU can be used as an alternative to the \hotline hardware accelerator for reordering mini-batches, gathering working set parameters, and as a pipeline scheduler.
To enable this, we map the \hotline computation (forward, backward and optimizer) to high throughput GPUs and try to provide working parameters to these GPUs at a rate faster than GPU execution -- to avoid stalls.
We observe that mini-batch re-ordering and working parameter gathering are memory intensive operations.
This results in data stalls within the CPU it acts as bottleneck.
\begin{figure}[h!]
	\centering
	\includegraphics[width=1\columnwidth]{body/Graphs/hotline_cpu/hotline_cpu.pdf}
	\caption{\hotline speedup normalized to CPU-based \hotline implementation. The \hotline hardware accelerator provides up to 3.5$\times$ speedup as compared to a CPU-based implementation.}
	\label{fig:hotline_cpu_perf}
\end{figure}

Figure~\ref{fig:hotline_cpu_perf} shows the performance benefits of \hotline accelerator over a CPU that implements the features of \hotline using multi-processing.
As the mini-batch and number of GPUs scale, performance of CPU-based \hotline degrades because of increased data stalls.

\if 0
\begin{table}
    \begin{minipage}{0.4\linewidth}

        \centering

        \label{table:syn_models}
        \begin{tabular}{l|c}
            \hline
            \multicolumn{2}{c}{\textbf{\revision{Model: SYN-M1}}} \\
            \hline
            \revision{Dense Features} & \revision{54} \\
            \revision{Sparse Features} & \revision{102} \\
            \revision{Size (GB)} & \revision{196} \\
            \hline
            \multicolumn{2}{c}{\textbf{\revision{Model: SYN-M2}}} \\
            \hline
            \revision{Dense Features} & \revision{102} \\
            \revision{Sparse Features} & \revision{204} \\
            \revision{Size (GB)} & \revision{390} \\
            \hline
        \end{tabular}
        \caption{\revision{Synthetic Model's Configuration}}
    \end{minipage}
    \hfill
    \begin{minipage}{0.55\linewidth}
        \centering

        \includegraphics[width=0.9\linewidth, right]{Hotline-ISCA-2022-Revision/Revision/body/Graphs/syn_perf/syn.pdf}
        \captionof{figure}{\revision{Performance comparison of \hotline with Intel Optimized DLRM across synthetic models using 1-GPU.}}
        \label{fig:syn_perf}
	 \end{minipage}
\end{table}

\fi

\subsection{Throughput Improvements}
\hotline improves the end-to-end training throughput by processing a pipeline of mini-batches.
As Figure~\ref{fig:throughput} shows, for a 4-GPU system \hotline can process, on average, \avgthroughputimp more epochs/hour as compared to optimized DLRM baseline.
\hotline throughput improves at a higher rate with increasing mini-batch than the baseline, because with a larger mini-batch, \hotline can create bigger popular mini-batches.
\begin{figure}[h!]
	\centering
	\includegraphics[width=1\columnwidth]{body/Graphs/Throughput/throughput.pdf}
	\caption{Varying throughput with 4-GPU executions.}
	\label{fig:throughput}

\end{figure}

\if 0

\begin{figure}[h!]
\vspace{-0.1in}
  \centering

	\begin{minipage}[t]{0.235\linewidth}
	   \centering
	   \includegraphics[width=\textwidth]{body/Graphs/Throughput/kaggle.png}
	\end{minipage}

	\begin{minipage}[t]{0.22\linewidth}
	   \centering
	   \includegraphics[width=\textwidth]{body/Graphs/Throughput/alibaba.png}
	\end{minipage}

	\begin{minipage}[t]{0.22\linewidth}
	   \centering
	   \includegraphics[width=\textwidth]{body/Graphs/Throughput/terabyte.png}
	\end{minipage}

	\begin{minipage}[t]{0.23\linewidth}
	   \centering
	   \includegraphics[width=\textwidth]{body/Graphs/Throughput/avazu.png}
	\end{minipage}
\caption{Varying throughput with 4-GPU executions.}
\label{fig:throughput}
\end{figure}

\fi

\subsection{CPU to GPU embedding transfer latency}
Table~\ref{table:cpu_gpu_time} shows the absolute time spent by baseline and \hotline to perform CPU to GPU embedding transfers.
\hotline reduces the transfer time, on average (4-GPU), by \cpugpucommtime, as popular inputs do not need embeddings from CPU memory.

\input{body/tables/CPU_GPU_time}

\subsection{Performance improvement for synthetic models}
To understand the efficacy of \hotline as the size recommendation models increase, we generated synthetic models and datasets.
Figure~\ref{fig:syn_perf} shows the performance of \hotline across two synthetic models.
\hotline continues to provide up to 2.5$\times$ speedup even as model sizes scale.

\begin{figure}[h!]
	\centering
	\includegraphics[width=1\columnwidth]{body/Graphs/syn_perf/syn.pdf}
	\caption{Performance of \hotline versus Intel-Optimized DLRM across synthetic models for a 1-GPU system.}
	\label{fig:syn_perf}
\end{figure}

\subsection{Design Space Exploration}
\niparagraph{Varying the number of inputs in a mini-batch:}
Figure~\ref{fig:minibatch_perf} shows the performance benefits of \hotline are accentuated with larger mini-batch sizes.
This is due to two reasons,
scheduler has to issue input dispatch commands fewer times.
Secondly, larger mini-batches offer more  parallelism opportunities, and \hotline enables more compute on GPU(s).

\begin{figure}[h!]
	\centering
	\includegraphics[width=1\columnwidth]{body/Graphs/mbs_perf/mbs_perf.pdf}
	\caption{\hotline speedup with varying mini-batch sizes.}
	\label{fig:minibatch_perf}
\end{figure}

\if 0

\subsubsection{Comparison with Static Profiling}

\hotline, at runtime, segregates popular embeddings and popular inputs.
This segregation can also be done statically as a part of data pre-processing.
Prior work FAE~\cite{fae} performs static scheduling, however, incur the embedding synchronization between GPU and CPU that degrades the training speedup with frequent jumps from popular to non-popular mini-batches and vice versa.
Figure~\ref{fig:fae_perf} shows \hotline speedup over FAE normalized to 1-GPU FAE for all datasets.

\begin{figure}[h!]
	\centering
	\includegraphics[width=0.45\textwidth]{body/Graphs/fae_perf/fae_perf.pdf}
	\vspace{-1ex}
	\caption{\hotline speedup normalized to FAE.}
	\label{fig:fae_perf}
	\vspace{-2ex}
\end{figure}

\fi

\niparagraph{Mini-batch working set:}
Figure~\ref{fig:mbs_working_set} compares the impact of the mini-batch working set on the execution time.
For a small working set, e.g. 1, even if there is a single input which accesses not-\highaccess embeddings, the accelerator will have to gather the working parameters for the whole mini-batch.
This data transfer latency cannot be hidden behind the popular input execution and causes a bottleneck.
We observe that a 4 mini-batch working set is sufficient to completely fill the pipeline and extract maximum throughput.

\begin{figure}[h!]
	\centering
	\includegraphics[width=1\columnwidth]{body/Graphs/minibatch_working_set/mbs_working_set.pdf}
	\caption{Effect of varying working set on unit iteration time.}
	\label{fig:mbs_working_set}

\end{figure}

\niparagraph{Embedding Access logger size:}
Figure~\ref{fig:emb_logger_size} shows the percentage of inputs accessing \highaccess embeddings captured by varying the logger size.
For Criteo and Avazu datasets with more skewness, even a small 2MB logger can capture most of the \highaccess indices.
However for Taobao dataset, that has a lower skew, once the logger size is increased beyond 4~MB we observe diminishing returns.

\begin{figure}[h!]

	\centering
	\includegraphics[width=1\columnwidth]{body/Graphs/emb_logger_size/emb_logger_size.pdf}
	\caption{Percentage of inputs accessing \highaccess embeddings captured with varying logger size.}
	\label{fig:emb_logger_size}

\end{figure}

\if 0

\begin{figure}[h]
  \centering

	\begin{minipage}[t]{0.48\linewidth}
	   \centering
	   \includegraphics[width=0.95\textwidth]{body/Graphs/minibatch_working_set/mbs_working_set.pdf}
	\end{minipage}

	\begin{minipage}[t]{0.49\linewidth}
	   \centering
	   \includegraphics[width=0.95\textwidth]{body/Graphs/emb_logger_size/emb_logger_size.pdf}
	\end{minipage}
\caption{Accelerator design space exploration for EAL SRAM size and Input eDRAM size.}
\label{fig:mbs_working_set}
\vspace{-0.1in}
\end{figure}

\fi

\input{body/tables/acc_breakdown}

\niparagraph{Hotline Accelerator Area and Power Breakdown.}
Table~\ref{table:acc_breakdown_table} shows the area and power division across different components of \hotline accelerator.
Embedding access logger consumes the most area and power as it is a 4-way set associative structure with replacement policy logic.
The next significant area consuming component is Input eDRAM as this memory structure is used to store non-popular inputs.
Finally, the lookup engine consumes the least percentage of area and power as it only performs lookup logic

\section{Introduction}

\paragraph{} Recommendation models are increasingly becoming one of the deployed Machine Learning (ML) workload.
Meta reports that 50\% of their training cycles are devoted to training deep learning based recommendation models~\cite{acun2020understanding}.
Recommender models are a disaggregated conflation of compute-intensive neural networks and memory-intensive embedding tables~\cite{facebookemb}.
The embedding tables store the user and item features.
The size of the embedding tables is expected to increase as more users and items interact.
For instance, a recent work from Baidu~\cite{zhao2020distributed} shows that production-scale recommendation models are reaching 10~TBs in size.

Deep Learning (DLRM)~\cite{dlrm} and Time-Based Sequence (TBSM)~\cite{tbsm} Models are often trained in a hybrid setup of CPUs and GPUs.
Figure~\ref{fig:baselineflow} shows the training flow of a recommender model in the hybrid mode with the CPU handling the embedding data by offering its high memory capacity.
The GPUs offer high compute throughput and perform data parallel execution of the DNN phase~\cite{largescalegputraining}.
A training iteration includes reading the inputs, fetching corresponding embedding data, neural network and embedding computations, and any data transfer required to update the model.

The hybrid execution has low efficiency as (1) GPUs rely on the CPU to provide all the embeddings to execute the DNN; (2) the low-bandwidth main memory acts as a bottleneck during embedding-bag operation and embedding-weights' optimization; and (3) it does not fully utilize the GPU for the entirety of the training process -- as the embeddings are executed on the CPU.
Alternatively, one can also use multiple GPUs to store a single-copy of the embeddings and train in a model-parallel fashion.
For instance, HugeCTR~\cite{hugectr} uses GPU memories to store the embeddings.
Such techniques require the number of GPUs to scale with the embedding size while the GPUs remain under-utilized by the neural portion.

\begin{figure}[t]
	\centering
	\includegraphics[width=1\columnwidth]{body/figures/baseline_flow.pdf}
	\caption{The execution flow of a typical recommendation model in the hybrid CPU-GPU mode. Due to their large sizes, the  embedding tables are stored and processed on CPUs. The GPUs process the neural layers.}
	\label{fig:baselineflow}
\end{figure}

Instead in this work we offer a novel \emph{throughput-optimized} heterogeneous acceleration pipeline called \hotline.
\hotline aims to to efficiently use the system resources - memory and compute - to pipeline the end-to-end training of large-scale recommender models.
\emph{\hotline is the first work to enable the GPU-only execution of the entire training of the recommender model, i.e, forward pass, backward pass, loss function, and optimizer for both embeddings and DNN portion, without requiring to store the entirety of embeddings on GPUs.}
As \hotline does not require each GPU to act as a bank for \emph{all} embeddings the number of GPU devices does not scale with the model size.
The pipeline also mitigates the inefficiencies of the CPU-GPU hybrid execution mode by hiding the latency of gathering embeddings from the main memory.

\niparagraph{Access-aware memory layout for embeddings.}
To offer \hotline, we leverage the insight that real-world recommender models train on inputs that exhibit \emph{extremely} high popularity skew~\cite{fae, recshard, wave, popsimilar, bagpipe}.
This causes certain embedding entries to be accessed several orders-of-magnitude more frequently than others.
We refer to these embeddings as \emph{\highaccess entries}.
The \highaccess entries have a small memory footprint while being computationally significant.
Keeping this access property in mind, \hotline foremost offers an optimized \emph{access-aware memory layout for embeddings} that is utilized by a novel \emph{runtime scheduler to perform mini-batch scheduling onto GPUs to ensure the parameter gathering of embeddings is pipelined with compute.}
Our framework stores the \highaccess embeddings on GPUs to mitigate their transfer overheads.

\niparagraph{Layout-aware runtime scheduling.}
To utilize this memory placement, \hotline requires a runtime scheduler that can tackle the challenge that a minibatch of inputs (2K-32K) can access a combination of \highaccess and not-\highaccess embeddings.
While \highaccess embeddings are on GPU, accessing the not-\highaccess embeddings from main memory can stall the pipeline.
The runtime scheduler mitigates this by reforming a small set of mini-batches into two classes.
The first class only accesses \highaccess embeddings and are directly scheduled onto GPUs.
This is because the local GPU memory contains the required working parameters.
The second class contains the remaining inputs that access not-\highaccess embeddings.
To hide the data transfer latency, the scheduler gathers the required working parameters from the CPU memory while the \highaccess inputs are being executed on GPUs.
The second task needs to be performed in a high-throughput manner to ensure that GPU is not starving during the non-popular inputs.
The scheduler overcomes this by offering a novel runtime \emph{accelerator} that sits between the GPUs and the main memory.
Thus the accelerator schedules the input mini-batches in a data and model-aware manner and by efficiently gathering embedding parameters from both the GPU (\highaccess) and CPU-based main (not-\highaccess) memory.

\vspace{0.05in}
\noindent \textbf{Contributions:} We make the following contributions:

\begin{enumerate}[noitemsep]
\item Identify the \highaccess embeddings \emph{dynamically} at runtime with negligible overhead.
\item Offer a dynamic data and model-aware scheduler to efficiently pipeline the mini-batch dispatch onto GPUs while concurrently obtaining not-\highaccess embeddings from the CPU-based main memory.
\item Offer a runtime framework that increases the training throughput by stitching the recommender model execution onto GPUs -- \emph{without} impacting fidelity.
\end{enumerate}

We evaluate \hotline with publicly available deep learning (DLRM) and time-sequence (TBSM) based recommendation models.
We compare against two baselines.
Namely, a hybrid CPU-GPU baseline where CPUs store and compute on embeddings and a GPU-only embedding processing baseline.
In both these cases, multiple GPUs parallelize the neural network computations,

For the CPU-GPU hybrid baseline, we compare \hotline against state-of-the-art deep learning framework XDL~\cite{xdl} and Intel~\cite{dlrm-intel} optimized DLRM.
On average, the \hotline acceleration pipeline provides \avgperfimprfourgpuxdl speedup over the 4-GPU XDL and \avgperfimprfourgpudlrm speedup over Intel optimized DLRM while maintaining the baseline accuracy.
\hotline is able to perform training of larger models such as Terabyte even while using a single GPU, whereas GPU-only baseline requires at least 4 GPUs to store its embeddings.
On average, \hotline obtains \avgperfimprfourgpuhugectr performance benefits over the GPU-only baseline, as it eliminates data movement between the GPU devices.

\if 0

\fi
\section{Related Work}
\label{sec:related}

Industry-scale recommender datasets show that accesses depict a Power or Zipfian distribution~\cite{yin2021ttrec, fae}.

\niparagraph{Recommendation models and designs:}
In this work we focus on deep learning based recommender models as they are commonly used in production and consume large number of compute cycles in data centres~\cite{fbtrainingreco, facebookreco:hpca}.
Even with their popularity, much of the research has focused on optimizing the inference phase of these models~\cite{centaur, DeepRecSys, nvidiareco, tensordimm, facebookreco:hpca}.
Unlike inference, training is throughput optimized and tends to be more compute and memory intensive.
Certain solutions that offer training optimizations~\cite{mixedde, compemd} or acceleration~\cite{tabla:hpca, recnmp, cosmic:micro} do not maximize throughput by facilitating better memory and bandwidth utilization of a distributed GPU system.

\niparagraph{Embedding parameter placement:}
Work in~\cite{zhao2020distributed, acun2020understanding} offers different embedding table placements either through hierarchical parameter server or on different devices, however, treat all embedding entries equally, without the concept of \highaccess embeddings and do not deal with perceptive input pre-processing to reduce the overhead of communication between devices.
Work in ~\cite{bandana} propose techniques to store embedding tables in non-volatile memories and allocate a certain portion of DRAM for caching.
Recent work in ~\cite{mixedde, recnmp, compemd, softwarehardware} has also proposed solutions to accelerate near-memory processing for embedding tables.
All of these works do not facilitate distributed training of entire recommender models using GPUs.
\hotline, instead offers an entirely runtime solution to efficiently pipeline embedding gather and store operations from external memory.

\niparagraph{Optimizations to mitigate memory intensive training:}
Prior work~\cite{nvopt}, optimizes training by modifying the model either through mixed-precision~\cite{mixedprecision} training or eliminating rare categorical variables to reduce the embedding table size.
Such approaches that change the data representation and/or embedding tables, require accuracy re-validation across a variety of models and datasets.
More work has been done on compression~\cite{compressreco, gist, compressreco2} and sparsity~\cite{sparsemat} to reduce the memory footprint of machine learning models in general.
\hotline, on the other hand, performs full-precision training without employing overheads such as compression/decompression and sparse operations by leveraging the skewed access patterns for embeddings.
\hotline is orthogonal to these techniques and can be used with them to improve the memory efficiency.

\niparagraph{Machine learning accelerators:}
There has been a plethora of work on designing accelerators to execute compute portion of deep learning models~\cite{tpu, tabla:hpca, cosmic:micro, maeri, minerva, eyeriss, brainwave, pudiannao, dnnweaver:micro}.
Certain works have also offered accelerators for recommender models, especially collaborative filtering based models~\cite{tabla:hpca, cosmic:micro}.
\hotline does not aim to design a specialized architecture to optimize the compute of deep learning based recommender models.
\hotline facilitates training by efficiently dispatching and gathering working parameters and input mini-batches to an already established GPU based system.
Nevertheless, \hotline accelerator can be used to enhance the overall benefits of deep learning accelerators by pipelining their execution efficiently.

\if 0
\niparagraph{Distributed machine learning training.}
Distributed training has become the norm to train large scale models.
Data parallel training~\cite{dataparallel} forms the most common form of distributed training as it only requires synchronization after the gradients generated in backward pass of training.
However, as models become bigger and bigger~\cite{tnlg, megatron}, model parallelism~\cite{modelparallel, gpipe} and pipeline parallelism~\cite{pipedream} are becoming common as they split a single model onto multiple devices.
Nonetheless, the techniques employed to automatically split the models~\cite{flexflow, deviceplacement:neurips}, offer model parallelism solutions to enable training of large model with size constrained by the accelerator memory capacity.
Unlike DNNs, for recommender models, even though embedding tables are becoming larger, splitting them across devices just for memory capacity (especially GPUs) is sub-optimal and does not effectively utilize GPU resources.
In this work, in contrast to past model parallelism work of splitting model, we use the fact that embedding table does not have to be fully replicated to perform data parallel training, instead only hot embedding entries can be sent to all the devices.
Nonetheless, our optimization is orthogonal to these past techniques, as it can be employed in tandem with pipeline and model parallel distributed training to reduce the memory footprint further of embedding table per device.
\fi
\section{Design: Hotline Accelerator}
\label{sec:architecture}
Figure~\ref{fig:acceleratorflow} shows the block diagram of the \hotline accelerator.
We describe the micro-architectural details of each component and then provide an example execution flow.
\begin{figure}[h!]
	\centering
	\includegraphics[width=1\columnwidth]{body/figures/Accelerator-Overview.pdf}
	\caption{Block diagram of the \hotline hardware accelerator. The hardware accelerator connects to the low-profile PCIe slot via a Streaming Interface.}

	\label{fig:acceleratorflow}

\end{figure}

\subsection{\hotline Accelerator: High-Level Flow}
We describe the high-level flow during the access learning and the runtime accelerated phases:

\niparagraph{Access learning phase.} The data dispatcher block reads input mini-batches from CPU main memory via the streaming interface, sends them to the lookup engine block, and updates the embedding access logger (EAL) block.
The role of the EAL block is to keep track of the amount of times an embedding index has been accessed across mini-batches.
Based on the access pattern of the embedding indices across first few mini-batches, the \hotline accelerator can determine which entries are \highaccess.
Once this phase completes, all entries in the EAL block becomes read-only.

\niparagraph{Runtime accelerated phase.} The data dispatcher and the scheduler play a vital role.
The data dispatcher directs each input in the mini-batch to the lookup engine.
The output of the lookup engine then flows into the input classifier.
The input classifier marks each incoming input as either popular or non-popular.
The non-popular inputs are collected in the \emph{input eDRAM} while the popular inputs are dispatched to the GPU(s).
Once a mini-batch worth of non-popular inputs are collected and their working parameters are fetched from GPU and CPU memories, the scheduler dispatches this mini-batch to the GPU devices.

\subsection{\hotline Accelerator: Components}

\subsubsection{Data Dispatcher}
Figure~\ref{fig:datadispatcher} shows the block diagram of the the Data Dispatcher block.
The \emph{Address Registers} store the base address of each embedding table in the main memory and GPU memory.
The \emph{Memory Controller} generates the address using the Address Registers to access the embeddings within the CPU and GPU memories.
The \emph{Input Classifier} segregates incoming inputs as popular and non-popular based on availability of embeddings within EAL.
While the popular mini-batches are being executed on GPU(s), the dispatcher issues the non-popular mini-batch from the input eDRAM to the Lookup Engine.
This enables the \hotline accelerator determine the working parameters of the non-popular mini-batch at runtime.
\begin{figure}[h!]
	\centering
	\includegraphics[width=1\columnwidth]{body/figures/Data-Dispatcher.pdf}
	\caption{The Address Registers, Memory Controller, and Input Classifier constitute the Data Dispatcher block.}
	\label{fig:datadispatcher}
\end{figure}

A non-popular minibatch consists of inputs that require a mixture of not-\highaccess and \highaccess embeddings.
For accessing the not-\highaccess embeddings (present in main memory), the Memory Controller uses the Address Registers to calculate the embedding address based on the embedding row index and its sparse dimension size.
The memory controller then issues a direct memory access (DMA) request to the DMA-engine for not-\highaccess embeddings.
While for \highaccess embeddings, a GPU read memory request is issued.
Thereafter, the working parameters obtained from CPU and GPU memory are processed by the Reducer block.
This block reduces multiple embedding rows into a single embedding vector and stores it in the \emph{Embedding Vector Buffer}.

\subsubsection{Embedding Access Logger (EAL)}

During the access learning phase, the \emph{Embedding Access Logger} (EAL) uses counters to track the access frequencies of embedding entries.
The logger only stores the indices and not the contents of the embedding entries.
As Figure~\ref{fig:EALentry} shows EAL consists of a Multi-Banked SRAM, Controller, and an input/output Queue.

\begin{figure}[h]
	\centering
	\includegraphics[width=1\columnwidth]{body/figures/EALentry_updated.pdf}
	\caption{The block diagram of the components within the Embedding Access Logger (EAL). It consists of an Multi-Banked SRAM, Controller, and Queue sub-blocks.}
	\label{fig:EALentry}
\end{figure}

\niparagraph{Naive Embedding Tracking.} A naive way to track frequency of accesses into each embedding entry is to use a separate counter for each sparse parameter.
As number of sparse parameters can be in the range of billions, this method of tracking would require gigabytes of on-chip storage.
Alternatively, storing the frequencies in the CPU/GPU memory will incur 3 accesses - one to obtain the embeddings, one to read the frequency, and another to update the frequency.

\niparagraph{Efficient embedding frequency tracking:} EAL is designed to dynamically track only the \highaccess embeddings by using their indices.
Our design is motivated by two observations.
First, compared to the overall embedding size (10-100~GBs), the size of \highaccess embeddings is under 512 MB.
Second, the access skews for these entries is extremely high ($>$ 100$\times$ accesses).
Thus, EAL is designed as a cache-like structure that keeps track of \highaccess embedding indices.
For the purpose of this paper we assume a 4-way set associative SRAM cache.
Assuming an embedding dimension of 64, with each entry being 4~Bytes, 512~MB worth of \highaccess embeddings contains 2 million indices.
Thus, the \hotline accelerator uses a 4~MB SRAM cache with 2 million blocks.
As we need to classify embeddings based on their access frequency, we could use the Least Frequently Used (LFU) replacement policy.
However, this would would require significant area overheads, as each of these 2 million cache blocks would require 24-bit counters (Figure~\ref{fig:ip_emb_acc} shows embeddings can have up to 10 million accesses).

To minimize area overheads, EAL uses the static Re-reference Interval Predictor (SRRIP) replacement policy with a 2-bit Re-reference Predictor Value (RRPV) counter~\cite{srrip}.
As \highaccess embeddings have $>$100$\times$ more frequent accesses, a 2-bit RRPV counter with insertions at RRPV-1 value captures $>$ 99\% of the \highaccess embeddings.
Figure~\ref{fig:ip_hotness_roofline} compares SRRIP based optimal logger to an Oracle which has infinite-sized access counters for each embedding entry.

\begin{figure}[h!]
	\centering
	\includegraphics[width=1\columnwidth]{body/Graphs/oracle_vs_replacement/ip_hotness_roofline.pdf}
	\caption{SRRIP-based tracker as compared to the Oracle LFU scheme. On average, the SRRIP-based tracker can track 70\% of the \highaccess embeddings.}
	\label{fig:ip_hotness_roofline}
\end{figure}

The SRRIP based EAL SRAM cache has 70\% tracking capability.
Note that, even if a non-\highaccess embedding is mis-classified as \highaccess or vice versa, model correctness does not change.

\niparagraph{Multi-Banked SRAM for multi-embedding lookup.}
Each input accesses multiple embeddings within each embedding table.
\hotline enables these parallel lookups by dividing EAL into multiple banks.
Figure~\ref{fig:queuesize} shows our empirical design space exploration wherein the average number of requests issued as the number of banks (n) increase while varying input queue size (m).
On average, a 512-sized queue with 64-banks enables 60 parallel requests into unique banks per iteration.
The EAL uses a controller to schedule the incoming requests from lookup engine block to the 512-entry queue.
\begin{figure}[h!]

	\centering
	\includegraphics[width=1\columnwidth]{body/Graphs/banks/banks.pdf}
	\vspace{-1ex}
	\caption{Impact of Queue Size and Banks on \# of parallel requests per iteration. Embedding Access Logger uses a 512 queue with 64-banks to enable 60 parallel requests.}
	\label{fig:queuesize}
\end{figure}

Once the \highaccess embeddings are determined, the EAL is switches to stops updating and provides these embedding indices during the runtime accelerated phase. EAL periodically switches back to the access learning phase and captures the changes in popularity,

\subsubsection{Lookup Engine}

Lookup engine is a parallel 2D compute network that processes on every training input and extracts their embedding entries.
For instance, if an input requires 26 distinct embedding tables, the accelerator can achieve 26x throughput per input.
Furthermore, the 2D compute network enables exploiting parallelism across training inputs within the mini-batch working set.
The Lookup engine, during the access learning phase, provides EAL with the indices accessed by each input for its updates.
During the runtime accelerated phase, the Lookup Engine tags inputs as popular if all embedding indices are in EAL as \highaccess.

\begin{figure}[h!]
	\centering
	\includegraphics[width=1\columnwidth]{body/figures/lookupengine_updated.pdf}
	\caption{Architecture of the Lookup Engine. The Lookup engine array determines the embedding entries that are used by a training input.}

	\label{fig:pe}
\end{figure}

\begin{figure*}
	\centering
	\includegraphics[width=1\textwidth]{body/figures/hotline_dataflow_updated.pdf}

	\caption{Dataflow through the Hotline accelerator. The accelerator is responsible for re-arranging popular/non-popular inputs, gathering their embeddings, and scheduling them on the GPU.}
	\label{fig:hotlineflow}

\end{figure*}

Figure~\ref{fig:pe} shows the components within a single lookup engine.
 %
Each engine consists of registers containing embedding table number, embedding index, hot embedding index register and a randomizer.
The embedding table number and index for the input are merged and sent to the randomizer.
The randomizer creates a hash of the (Embedding Index, Embedding Table) tuple.
This hash scatters embedding index values across EAL to avoid trashing during the access learning phase.
We implement our randomizer block using a low-latency Fiestal Network implementation~\cite{fiestal}.

\if 0

\subsubsection{Input eDRAM and Scheduler}

Input eDRAM stores the non-popular inputs.
Once a mini-batch worth of non-popular inputs are collected in input eDRAM, and their respective working parameters are fetched from GPU and CPU memories, the scheduler sends these inputs to the GPU devices using the memory controller sub-block in the data dispatcher.
\fi

\subsubsection{Reducer}
The Reducer performs sparse length sum element-wise  operation using an array of simple arithmetic units.
It reduces multiple embedding rows a table into a single vector (embedding pooling operation) and stores the result in embedding vector buffer.

\if 0
\begin{figure}[h]
	\centering
	\includegraphics[width=0.45\textwidth]{body/figures/reducer.pdf}
		\vspace{-1ex}
	\caption{Embedding Reduction Array}
	\label{fig:reducer}
	\vspace{-2ex}
\end{figure}
\fi

\subsection{\hotline Accelerator: Instruction Set Architecture}
As the \hotline accelerator interacts with both the main memory and GPU via the PCIe link, it requires a brief set of instructions (Table~\ref{table:isa}) to read/write the required data into these devices.
The \hotline ISA facilitates communication with the DMA engine and GPUs.

\input{body/tables/isa}

\subsection{\hotline Accelerator: Tying it together}
\label{sec:hotlineexample}

Figure~\ref{fig:hotlineflow} shows an example execution of the accelerator during the runtime accelerated phase.
The \hotline accelerator reads mini-batches from the main memory and segregates them into popular and non-popular sets.
In this example we show a working set of 4.
Input eDRAM just stores the non-popular mini-batch while popular mini-batches are sent to the GPU(s).
The \emph{Scheduler} block organizes the timeline to issue non-popular mini-batch after the popular mini-batches complete execution.
Popular mini-batches execute in a data parallel fashion as all the model parameters are present in the local memory of each GPU.
Simultaneously, the scheduler sends the inputs of the non-popular mini-batch from \emph{input eDRAM} to the \emph{Lookup Engine} block.
The \emph{Lookup Engine} block checks whether the indices are located in the EAL, if so, they are classified as \highaccess and obtained from the GPU(s), else are obtained from the main memory.

Each embedding entry obtained from main or GPU memory goes through the \emph{Reducer} block.
For multiple embedding lookups within an embedding table, the \emph{Reducer} block applies embedding pooling operation to convert all the lookups into a single embedding vector.
Thereafter, the embedding vector is temporarily stored in the \emph{embedding vector buffer}.
Once the all the working parameters of non-popular mini-batch have been fetched, the non-popular mini-batch is dispatched along with the embedding vectors.
The entire execution including the forward, backward pass, and the optimizer execute on the GPU for all the mini-batches.
Once the embedding updates are generated by the optimizer, the updated not-popular embeddings are stored are written to the CPU main memory.
Thus, the CPU maintains the most up-to-date non-\highaccess embeddings whereas the GPU stores the \highaccess embeddings for the entirety of the training process.

\begin{table*}
\centering
\caption{Recommender Model Architecture and Parameters}
\input{body/tables/models}
\label{table:datasets}
\end{table*}

\begin{figure*}[t]
  \centering
  \subfloat[Criteo Kaggle]{
	\begin{minipage}[t]{0.48\textwidth}
	   \centering
	   \includegraphics[width=\textwidth]{body/Graphs/Accuracy/kaggle_auc.pdf}
	\end{minipage}}

  \subfloat[Taobao Alibaba]{
	\begin{minipage}[t]{0.48\textwidth}
	   \centering
	   \includegraphics[width=\textwidth]{body/Graphs/Accuracy/alibaba_auc.pdf}
	\end{minipage}}
	\\

  \subfloat[Criteo Terabyte]{
	\begin{minipage}[t]{0.48\textwidth}
	   \centering
	   \includegraphics[width=\textwidth]{body/Graphs/Accuracy/terabyte_auc.pdf}
	\end{minipage}}

  \subfloat[Avazu]{
	\begin{minipage}[t]{0.48\textwidth}
	   \centering
	   \includegraphics[width=\textwidth]{body/Graphs/Accuracy/avazu_auc.pdf}
	\end{minipage}}
\caption{AUC comparison of \hotline with a baseline system while performing full-precision training. \hotline maintains the training fidelity throughout execution.}
\label{fig:accuracy}
\end{figure*}

\if 0
the execution flow within the \hotline accelerator by using control signals.
Additionally, the hotline-controller is also responsible for managing the communication with the CPU and GPU devices.
The read and write queue within the hotline-controller, for our design point, is 512 entries in size.
The primary task of the controller is to exploit the parallelism within training inputs by routing them through the PE array.
The inputs from the CPU are placed into the read queue. Thereafter, these inputs are forwarded into the PE array and the input deck.
For each input sent into the PE array, we obtain information if the input accesses only popular embedding entries.
The input classifier within the hotline-controller updates the popularity information in the input deck.
This helps form minibatches with popular inputs to be being executed on GPUs.
While these hot-minibatches are being executed, the hotline-controller helps gather the embedding entries of the not-popular minibatches within the input deck.
Thus the the hotline controller helps hide the latency bottleneck of not-popular mini-batches.
\fi

redpajama_set_name: RedPajamaArXiv


text:
Q: how to incorporate grunt-vulcanize into the chromeapp yeoman generator's Gruntfile.js? I am trying to make a chromeapp incorporating some polymer elements - polymer-list and one of my own.
I started with the yeoman chromeapp generator, but it didn't take long before I ran into some csp issues and I found that I should be using the vulcanize tool. When I try that using the command line version, it seems to work nicely.
Then I started to want that done automatically for me in the build process, and so I had a look at adding it to the Gruntfile.js which was generated by the chromeapp yeoman generator. I found the grunt task and it provides instructions for adding it to the Gruntfile.js.
However, one thing I learned about the Yeoman generator is that the developer is supposed to debug their app in the app/ directory, and only 'build' when building an optimised version (presumably for distribution). Thinking about it, the vulcanize step seems to be necessary even for debugging (since polymer won't work out-of-the-box in a chromeapp due to the csp), and so it is an intermediate step. As such, it's a bit more effort for a beginner (part of the audience for Yeoman, I presume).
So, now I'm wondering how to do this. It does seem like something someone's already done, but I can't find any examples.
I have worked on some projects that use a 2-step grunt build process (due to having several targets mostly), so that the flow is something like this : app/ -> build/app -> app/crx. IINM, the first step simply brings all the bits together plus substitutes app names and version numbers/etc; and the second step does the optimisation (uglify/minify/etc).
I guess the solution to this question is to do something similar, and debugging/development is done on the build/app in the middle step.
I've done some fiddling with grunt in the webapp projects I reference (and even wrote one), but the plugins used by this generator aren't very familiar to me.
My question, then, is has anyone has done this already, or corrections if I have misunderstood anything/etc. Also, if someone has suggestions on how to integrate the grunt vulcanize task into the Gruntfile.js provided by the chromeapp Yeoman generator, then I'll give is a try.

A: I worked on a solution myself and came up with this.
The solution is somewhat involved but does essentially add an intermediate step, where a grunt 'build' target is a step used to 'compile' (just vulcanize) app/ into build/, and the old build target became 'dist' which optimized build/ into dist/.
The intention is that development is now done in build/ and the optimized code in dist/ is used to build packages (crx) for distribution.
It's still somewhat unfinished and certainly isn't ready to push upstream to the yeoman generator, but I hope it helps someone.

redpajama_set_name: RedPajamaStackExchange


text:
The Bach Project
Blog April 20, 2009
Johann Sebastian Bach's birthday last month, inspired Science Cheerleader contributor Stephen Zachary to examine yet another example of how science and art coexist so beautifully.
Here's Stephen's report:
Filmmaker Michael Lawrence's documentary The Bach Project, is currently in post-production. Though we tend to think that the arts and science occupy separate realms, Lawrence went to a researcher and a mathematician for insights into Bach's ingenuity and the structure of his music. In the clip above, Dr. Charles Limb explains some of his research into the neural basis of musical improvisation and demonstrates how he goes about scanning musicians at work in an fMRI machine. Limb found increased activity in the medial prefrontal cortex while musicians engaged in spontaneous performance. While the prefrontal cortex as a whole is recognized as the site of complex cognitive function, the medial area seems to play a key role in several important facets of human uniqueness, including internally-motivated behavior and a sense of the self.
The vast majority of neuroscientific research focuses on either perception or brain abnormalities, but Limb is designing elegant experiments that shed light on the very personal and mystifying phenomena of creativity. As he explains in The Bach Project, Limb firmly believes that Bach experienced brain states similar to those he observed in his subjects. Because he's tapped into a novel line of research, Limb hopes to conduct similar experiments focusing on diverse musical genres, so it may not be long before he's comparing the creative processes of, say, classical musicians and freestyle rappers.
Lawrence also added some interesting commentary to the film by bringing in Harlan Brothers, an amateur mathematician who is interested in fractals. The defining characteristic of a fractal is that its component parts mirror the structure of the whole. In other words, fractals are "self-similar" or "self-reflective." Brothers did the math, so to speak, on Bach's work and published a paper identifying fractal elements in the Cello Suite #3. The self-reflective nature of Bach's music provides its rich and layered texture, and it is very likely that fractal elements play a significant part in Bach's appeal.
Lawrence interviewed over thirty people for the film, all with valuable perspectives on Bach and many performing renditions of well-known Bach pieces. In a documentary focused on art and expected to last only ninety minutes, why would he spend precious time on the analytical perspectives of a scientist and a mathematician? First, Lawrence's undertaking may be the most in-depth examination of Bach ever captured on film, and he firmly believes in exploring any worthwhile avenue of inquiry. Second, both Limb and Brothers are impressive musicians in their own right, and their emotional attachment to Bach's music comes through in their interviews no less than any other contributor to the film. Taken together, all these perspectives create a picture of Bach that is nearly as layered as the music itself.
For more information on The Bach Project, see the film's website.

redpajama_set_name: RedPajamaCommonCrawl


text:
You are here: Home / Blogs / Singapore: Global City or Regional Hub?
Singapore: Global City or Regional Hub?
In a Straits Times article titled "Global city or regional centre?", Emeritus Senior Minister Goh Chok Tong said those living in Singapore must now help to decide what kind of city it wants to be, as well as the kind of values it wants to hold.
Mr Goh raised the issue of whether Singapore should continue having cosmopolitan values in society as immigrants from all over the world settle down here, or revert to being merely 'Singaporean' with just Chinese, Malay, Indian and Eurasians in the country.
The former prime minister said an increasing diversity of immigrants, including Australians, Italians, Britons and Cambodians, have chosen to settle down in Singapore, unlike the previous wave of immigrants who were mainly from India, China and Malaysia. He also said Singapore's future and the challenges facing it were more complex now than before.
Related Read: Why Singapore is the place to do business?
Rikvin finds that both globalisation and regionalism are both workable models for Singapore, as it is well-placed to take advantage of both.
Singapore is an active participant in the global economy, with an international voice and influence which is disproportionately larger than its size.
It has been ranked among the top countries in the world for doing business, and is also noted for having the right mix of stable governance, macroeconomic buffers and effective regulation for companies setting up here.
Hence, investors have parked large amounts of their capital in Singapore, placing trillions of dollars in securities, mutual funds, retirement plans, life insurance and other financial instruments.
International MNCs have also marked Singapore as their Asia-Pacific headquarters and placed an increasing numbers of top executives here to manage the booming Asian market. These companies are also lured by Singapore's robust financial and logistics hub.
Politically, Singapore has proven adept at engaging global powers like the United States, or connecting with countries at the far side of the globe, like Luxembourg and the Eurozone, and helped them keep Asian interests in mind.
Considering that trade and financial flows are sometimes concentrated in regions, Singapore is an active part of regional pacts like the ASEAN free trade agreements, and active trading partners with North-East Asian countries like Taiwan, Hong Kong, Japan, South Korea and China.
The previous waves of immigrants were also mainly from regional countries, which increases the likelihood of these immigrants facilitating business deals or connecting with their home economies with Singapore's to increase the volume and speed of trade and financial flows.
The smaller number of participating countries in regional pacts also does away with the complicated decision-making and planning process which has stumped international organisations like the World Trade Organisation. This allows for the 'real' solving of issues instead of endless rounds of discussions which sweep issues under the carpet.
Co-existence
Co-existence of globalisation and regionalism is possible, especially if the government and those who live in Singapore are able to work together to harness the two and meld them into a single driving force.
There is space for Singapore to tap into the two forces of globalisation and regionalism, and those who wish to set up their businesses and homes here will be able to take full advantage of Singapore's unique position to do so.
This Place was Designed to Do Business
Singapore Banks Raise Minimum Monthly Balance
Get a quote Request Call Back Have a question?
How can we help you?Company IncorporationOffshore IncorporationCorporate Secretarial ServiceAccountingTaxationImmigrationEmployee PayrollOther service
Select Service:
Employment PassS PassEntrepreneur PassDependant PassLOCLooking for Job
Incorporating a Company in Singapore
Singapore Immigration and Work Visas
Singapore Taxation
Singapore Business Infographics
Singapore Industry Guides
Rikvin Download Guides
Embracing the Pandemic with A Hybrid Workforce In Singapore
A Guide to Filing Corporate Taxes in 2021 – Form C-S Lite
Why Start an eCommerce Business in Singapore?
SAFE vs KISS in Singapore
Overview of Convertible Instruments in Singapore


When water is bubbling from cracks in the street it usually means the supply line beneath the street is leaking. Please call us immediately at (714) 738-6897. A crew will be dispatched as soon as possible to repair the leak before it undermines the street or causes other damage. During repairs, every effort will be made to maintain the water supply to as many customers as possible.


Fun boutonnieres that work and stay on your shirt.
One of my pet peeves at a destination wedding are the boutonnieres. If the guys are wearing shirts and not jackets then the chances of a flower boutonniere staying in place is slim to none. I have no idea why most guys don't simply ditch them at that stage, but the reality is 99% of our joy comes from seeing the joy on her face. So in a way I get it.
Anyhow, I love these fun boutonnieres on Etsy. And if it is for a shirt only, think light. Super light!


Using state of the art equipment and cameras we can supply 4k quality for broadcast. Professional sound and voice over can be utilised. If you require a short video contact us for a free consultation and appraisal of your needs.
Riley RM F from Lee Robertson on Vimeo.


text:
Qatar weather: Meteorologists warn of unstable weather from Friday
GT Digital
The Department of Meteorology on Thursday said in a report that the weather charts indicate a gradual increase in the amount of clouds from Friday. The weather will be partly cloudy to cloudy with a chance of scattered rain of varying intensity at intermittent intervals from Friday evening until Tuesday, January, 18, 2022 that may turn into thundery at times.
The wind will be Southeasterly - Southwesterly, with speeds ranging between 14-24 KT daytime, with gusts exceeding 30 KT over some areas at times causing blowing dust and reducing horizontal visibility to less than 3 KM at times. Winds are expected to shift to Northwesterly at the beginning of Monday, 17/1/2022, and will continue for several days until the end of next week, at moderate to fresh speed reaching strong daytime on Tuesday and Wednesday 18-19 /1/2022.
The marine condition is not suitable for all marine activities, as the wave height is expected to exceed 10 FT during the period of weather fluctuations.
The Northwesterly winds are also accompanied by a noticeable decrease in temperatures with a greater feeling of coldness, especially during the night and early morning hours, where maximum temperatures are expected to range between 16-22° C and minimums between 8-15° C and even cooler in the southern and exterior areas. It's worth noting that this period coincides with the entry of the (Shabbat Season), which is considered the coldest winter season locally, and it starts from mid-January and lasts about 26 days. Qatar Meteorological Department urges all to be careful and avoid all marine activities during this period.
Amir issues decision appointing an assistant to the Head of the Planning and Statistics Authority
Amir issues decision restructuring the Qatar University Board of Trustees
Amir issues decision to renew the term of the Board of Directors of Qatar Development Bank
Department of Meteorology warns of thundery rain, strong wind and high sea
Book highlights implementation of recycled aggregate in construction
US embassy showcases 10,000 books at DIBF
FM meets UN Special Envoy for Syria


text:
Scarborough disability charity launches Crowdfunding page
Scarborough Disability Action Group have launched a fundraising campaign with Aviva Community Fund and Crowdfunder.
By Louise Hollingsworth
Monday, 29th November 2021, 5:00 pm
Scarborough DAG are appealing for help with their fundraising campaign. The campaign will allow the charity to continue providing free activities. (Credit: Scarborough DAG)
The fundraiser is to raise money for the Scarborough Disability Action Group (DAG) TREATS project.
DAG are asking the local community for their support to help raise funds for the project as the deadline nears. The fundraising campaign will help the charity to continue providing a variety of free leisure activities for local disabled and disadvantaged people.
The activities include accessible cycling, arts and craft sessions, keep fit and chair based exercises.
Ian Whitfield, DAG chairman, said: "We're set a target of £10,000 which we are hoping to raise to cover the costs of the TREATS project. We are hoping through the money raised we will be able to maintain our TREATS activities helping support the wellbeing of the TREATS participants who are lonely or feel isolated".
Mark Smith, DAG treats co-ordinator, said: "We regularly get over over 80 people attend treats activities/sessions over the year and we want to ensure we are able to maintain this level of support, even during the lockdown we held online activities such as quizzes and virtual cuppas to make sure we still had activities running."
"We're very excited to launch our Crowdfunding page and hope the local community will support DAG, so we can continue our work across Scarborough. We appreciate that the ongoing pandemic has caused financial uncertainty for everyone and especially for charities, which is why we're so grateful for any and all support we receive".
The Crowdfunding page ends on Monday December 6.
To pledge your support for the campaign, please visit their Crowdfunding page. You can also donate via the DAG office.


Roger Smith has written a report on his attendance at the 15th Annual Technology Initiative Grants Conference of the Legal Services Corporation in the USA. This was held recently in San Antonio, Texas and Roger's report highlights some of the latest worldwide developments and discussion around the use of technology in the provision of legal services. To view the report, click here.


text:
Home > Ancestors Abroad > Scottish Marriage and Family History Research
Scottish Marriage and Family History Research
Researching marriages to trace your family history in Scotland is very similar to tracing those in England or Wales. But it's a game of two halves. After 1855 with compulsory registration, tracing marriages is quite easy, with all records in the General Register Office. Prior to that, you're relying on church records, which can present problems, as not all marriages were in church (or even at the church door, the most common place to marry).
Scottish Marriage Records Post 1855
From the beginning of 1855, all Scottish marriages had to be registered, just like those in England and Wales. The Scot's marriage certificates offer a wealth of information that will help you with your family history research, including not only the complete names of both parties, their birthplaces, ages, occupations and residences, but also the names of parents (including the maiden names of the mothers), father's occupation, and how many prior marriages each spouse had had, as well as the number of children (previous marriage details and birthplace were quickly dropped). If you find the notation "RCE" by an indexed entry, this stands for Register of Corrected Entries, a separate volume showing where an entry had to be changed for some reason. You can search the records online at Scotlands People , although they currently only go up to 1931 for legal reasons. You will, however, have to pay for copies of certificates.
Scottish Marriage Records Before 1855
Before 1855 you're in the hands of church records. These can also be useful for family history information and those of the Church of Scotland, the old parochial registers, can be found online at www.gro-scotland.gov.uk, with other denominations at the National Archives.
The problem is that with the registers you'll get either the date of the marriage itself or the reading of the banns, but you'll have no idea which it is, as it won't be noted. However, they will give the names of the parties and the father of the bride.Additionally, there were many marriages that simply never entered the record books at all, because they were deemed "irregular." Scottish law recognised these; they could be as simple as an exchange of vows anywhere (witnesses didn't even have to be present) or what we call common-law – simply living together. If your ancestors engaged in this, then you'll be out of luck on finding any written documentation.
Handfasting in Scotland: A kind of Trial Marriage
There's a myth surrounding the idea of handfasting, namely that it was a kind of trial marriage, supposedly lasting for a year and a day. If, after that time, the couple had no children and wanted to go their separate ways, the handfasted marriage just became null and void.
However, according to several researchers, this simply wasn't true. Instead, handfasting was probably the equivalent of an engagement, signifying a betrothal, an intent to marry, not an actual marriage itself.
You can find records of handfasting as a betrothal into the 17th century and beyond. By the end of the 18th century the term was no longer used for betrothals. It should also be noted that if a handfasted couple had sex, they then automatically became married.
The fact that Scotland has done such a thorough job of computerising its indexes (and you can also find many of them at the International Genealogical Index means that searching is easy, even from home.
Employing a Foreign Genealogist
Ancestors in Other British Colonies
American and Nordic Roots
The Clearance and Scottish Emigration
The Irish Potato Famine and Emigration


Google Chrome now natively supports the ability to redirect Cortana or Windows 10 desktop searches to your preferred search engine, when Chrome is set as your default browser. Learn how to turn on this feature after the jump.
Yesterday, Google released version 50 of Google Chrome to the stable channel. Among numerous improvements, it includes one seriously useful feature for Windows 10 users: the ability to use your default search engine in Chrome for all your Cortana or desktop queries. For most people, this means just using Google for searching the web from Cortana or the desktop.
This is now a built-in Chrome feature, meaning you don't need to install our previously recommended Chrometana extension anymore.
After you ensure Google Chrome is selected as your default web browser – from Settings > Default Browser – follow the steps below to enable this feature on your Windows 10 PC.
To redirect Cortana searches to default Chrome search engine, launch Google Chrome, and type in chrome://flags into the address bar. This will bring up a long list of experimental features you can try out. Scroll down, and you'll find Windows desktop search redirection. Open the drop-down menu and select Enabled to use your search engine in Chrome for all desktop searches.
Keep in mind that this is an experimental feature, so it may or may not work all the time. In fact, we've seen reports of Google Chrome crashing on enabling Windows desktop search redirection. Do let us know in the comments section how you experience is like.
Google has taken its sweet time implementing this feature. Windows 10 has been out in the market for almost a year now. In that time, a number of Chrome extensions have popped out to allow users to search Google using Cortana. Even Mozilla Firefox had this feature right around the time Windows 10 launched.
Is Google getting lazy with Chrome? Does Google not like Microsoft? Am I making stuff up? We won't know. We're just glad this feature is here, and it is finally native to the Internet's favorite web browser.


text:
5 things that could trip up the spending bill
David Nather
Trump next to empty seats for Schumer and Pelosi at an aborted spending meeting last month. Photo: Susan Walsh / AP
Sure, Congress wants to go home for the holidays. But it also has to fund the government after Dec. 22. And there are a lot of things that could go wrong.
Here's how plugged-in appropriations experts are ranking the dangers over the next two weeks:
Immigration: This is the biggest danger. Democrats will push hard for legal status for the "DREAMers" — people who are in the country illegally who came here as children. And there's no sign that Republicans are willing to put it in the year-end bill. "I don't see how a majority of this Congress leaves without some resolution," said Jim Dyer, a former House Appropriations Committee Republican staff director.
No tax deal before the spending bill: If Republicans can't pass a final version of the tax bill by Dec. 22, President Trump could easily pressure Congress to stay in town by refusing to sign a government funding extension until lawmakers finish the tax bill.
Defense vs. domestic spending: Democrats could make trouble if a funding bill includes a big boost for defense spending, but nothing for domestic programs.
CHIP funding: The Children's Health Insurance Program has expired, and Republicans had said the year-end bill was where its funding would be extended — but they still haven't figured out how to pay for it. Expect pushback from Democrats, and maybe governors, if it doesn't happen.
ACA payments: Republican Sen. Susan Collins wants two fixes for the Affordable Care Act — a bipartisan renewal of payments for insurers and a separate set of "reinsurance" funds — as a condition for her to vote for a final tax bill. That could cause problems if the House doesn't go along, which it probably won't.
One more thing: A larger problem, Dyer said, is "the general level of rhetorical excess" — partly driven by the White House.
His point: Most of the problems are solvable. It just takes cooler heads.


To my delight most of the vegetation was a challenge even for our gardeners. I went around tasting every leaf, flower and fruit that came my way. Chaangeri or Indian Sorrel was thus discovered. I called it Khatti Patti, going by the deliciously sweet-sour taste. I did not know that it was a well documented Ayurvedic herb useful in Vitamin-C deficiency, Rhuematoid Arthritis, indigestion and diarrhea.
The flowers, fruit and leaves are completely edible and can be used to make green chutney by blending with coriander leaves, mint leaves, salt and a dash of raw mango or lemon juice.
Indian Sorrel or Chaangeri springs up in spring and lasts through the rains, almost all over India. It loves sunlight but doesn't mind a shady patch under the shrubs. I know it to be of two kinds, one has larger leaves and pink-mauve flower; the other has smaller leaves and yellow flowers.
Many thanks to the annual flower show at Rajbhawan, Lucknow this year, where I reconnected with my childhood friend -- Chaangeri. Feeling great that I chewed on the leaves of this delectable herb as a kid, trusting nature so completely, without fear of harm. The reward was not in taste alone. It built my relationship with nature and Mother Earth.


Sheer shimmer to intense metal, this water based blendable eye colour goes on wet and dries quickly for a long wearing metallic look.
This unique formula has a gelly texture that feels amazing and wears beautifully. Once dry it is non sticky, smudge and crease resistant.
Plus, it can be used alone, or on top of a powder.


The Licensing area on our Support Portal aims to provide you with all the information you need to understand how your license works, useful information on advanced licensing and what to do if you encounter any problems.
If you can't find the information you're looking for, please contact our Support team by creating a ticket through the Support Portal, and we will be happy to help.
You can also find detailed instructions on installing and using node-locked and floating licenses in the Foundry Licensing Tools User Guide, which is available to download here.
Problems will occur if you choose to use a MacBook Pro as a server, as it will not meet criteria point 2 above, and it will change hostname constantly when plugged into different networks. You may need to discuss this with your Network or Systems teams if you encounter hostname resolution issues.
If you encounter problems with your licensing, and you can't find the solution in either the Known Issues, Licensing Knowledge Base or Subscription and Login based Licensingsections of the Licensing area on our Support Portal then please follow the instructions in the article below to create a license diagnostic file and contact the Support team.


This Vector Logo Car Customizing Club Crossed has 1500 x 1416 pixel resolution with jpeg format. Vector Motors Wx8, Vector Car For Sale, Vector W8 Car, Vector Wx8 Price, Vector Motors Wx8, Vector Car For Sale, Vector W8 Car was related topic with this Vector Logo Car Customizing Club Crossed . You can download the Vector Logo Car Customizing Club Crossed picture by right click your mouse and save from your browser.