text:
\section{Appendix}

\end{document}

\section{Artifacts Appendix}

\subsection{Getting Started}

The artifacts of this paper are provided as a docker image and can be found at the Zenodo archive: \\https://doi.org/10.5281/zenodo.4290558

To run the image, the only prerequisite is to have docker installed in your local machine.
To download and run the image, the following commands can be used:

\begin{lstlisting}
$ wget \
  https://zenodo.org/record/4290558/files/ppopp.tar
$ docker load --input ppopp.tar
$ docker run -h debian --name artifact \
  -it artifacts/ppopp:1.0.0
\end{lstlisting}

After running the earlier commands, you will be presented with an interactive prompt in an image with all the prerequisites to run the benchmarks installed.

The full suite of benchmarks can take several hours to run completely, and requires a system with a large amount of main memory (+ 32 GB), as some problem sizes are big.
To test the functionality of the artifacts, a small suite of benchmarks is provided, which will generate the granularity scaling plots with smaller problem sizes, and can be run in a few minutes.
The small suite also does only one execution of each benchmark, providing no standard deviation information.

To run the reduced set of benchmarks, the following script is provided which can be executed from \texttt{/home/user}:

\begin{lstlisting}
user@debian:~$ ./run-small-suite.sh
\end{lstlisting}

After running the benchmarks, the results corresponding to the comparison between the optimized Nanos6 runtime compared with
GNU OpenMP and LLVM OpenMP will be stored in the \texttt{/home/user/output/} folder.
To retrieve the results back to the local machine, the following command can be used \textbf{outside the container}:

\begin{lstlisting}
$ docker cp artifact:/home/user/output .
\end{lstlisting}

Which will create a folder named \texttt{output} in the current path and retrieve the plots in pdf format.

\subsection{Step by step}

The artifact is prepared to be flexible and all the sources as well as scripts to re-build all of the software are included. In this section we will explain the structure of the image, how to run the full benchmark suite, and how to change and re-build the experiments and software.

In case it is needed to install more software on the docker image, the password for the user of the machine is \texttt{user}.

\subsubsection{Full benchmark suite}
It is possible to run the full benchmark suite with the same parameters that were used on the original paper. However, be warned that the expected runtime is several hours, and that not all machines may be able to handle the input sizes due to lack of memory. To do so, run the following command:

\begin{lstlisting}
user@debian:~$ ./run-full-suite.sh
\end{lstlisting}

\subsubsection{Directory structure}

The image has the following structure on which the relevant files can be found:
\begin{itemize}
    \item \texttt{Sources}: Contains the source code for the Nanos6 runtime as well as its dependencies (Mercurium, Jemalloc and TAMPI), and GCC 9.3.0 for the benchmarks.
    \item \texttt{Benchmarks}: Contains the source code and binaries for the benchmarks, each one in a separate folder with a standalone (and working) Makefile.
    \item \texttt{Install}: Contains the built binaries for Nanos6 and its dependencies.
    \item \texttt{Automate}: Contains the Pyhton scripts and JSON configuration files that are used to execute the benchmarks.
    \item \texttt{output}: Output directory, where the plots of granularity and efficiency for each of the benchmarks are saved after running the benchmark suite.
\end{itemize}

\subsubsection{Nanos6 Sources}

Although the sources used in the article evaluation are included in the docker image, Nanos6 is free software and the most up-to-date version of the sources is publicly available on the following GitHub repository: https://github.com/bsc-pm/nanos6

We suggest using the last version from git if you want use Nanos6 in your research.

\subsubsection{Rebuilding the software}

In the root folder, an \texttt{install-all.sh} script is provided which will re-extract all the sources and re-build Nanos6, its dependencies, and the GCC 9.3.0 toolchain.
This is provided as a way to make the image re-usable, as it is possible to change the sources and re-build the whole stack.

In case you want to install Nanos6 bare-metal in a machine, we suggest to refer to the official Nanos6 documentation which can be found on the GitHub repository or inside the included source tarball, which will guide you through all the configuration and building process.

\subsubsection{The OmpSs-2 Programming Model}

The BSC Programming Models research group routinely supports other researchers that want to use or improve the OmpSs-2 programming model, the Nanos6 runtime or any of our tools. The specification for the OmpSs-2 programming model can be found online at https://pm.bsc.es/ompss-2, and you can reach us by email at \texttt{pm-tools@bsc.es}.

\subsection{Supported claims}
This artifact is designed to support the claims done in Section 6 of the paper, specifically the performance comparison between the Nanos6 runtime, containing all of the novelties presented on the paper, and other state of the art OpenMP runtimes.

The goal of the artifacts is to facilitate the reuse of our research by others, and allow access to a functional and reusable version of the sources and benchmarks referenced in the paper. Reproducing exactly the results obtained in the paper would require access to the exact same machines and software that was used, which is not in the scope of the artifact.

Running the benchmark suite will generate the performance plots based on task granularity, using the same method that was used to generate the original plots.
Raw results are also extractable, and are available on the \texttt{Automate/scaling} folder (or the \texttt{Automate/scaling\_small/} for the small suite).
However, there are some caveats and claims that are not supported by the artifacts:

\begin{itemize}
    \item The artifacts do not include non-free software that was used during the evaluation. Specifically, the following software is not included and thus not evaluated in the comparison:
    \begin{itemize}
        \item The Intel MKL library. Instead, the BLAS and LAPACK kernels used in the benchmarks have been linked against the OpenBLAS library, which may affect the results. If desired, the user can download the Intel MKL libraries from debian's non-free repositories and link the benchmarks against them.
        \item The Intel Compiler and OpenMP runtimes, which require Intel licenses to use.
        \item The AMD Optimizing C/C++ Compiler and OpenMP runtime, which is available on AMD's website.
        \item The ARM Performance Libraries, which were used on the Graviton2 machine.
    \end{itemize}
    \item Performance evaluation was done bare-metal in all the machines, without container overhead and always on exclusive nodes. Caution is advised when drawing conclusions from the results obtained running the artifacts, as similar conditions need to be achieved for the results to be valid.
\end{itemize}

\section{Conclusion and Future Work}
\label{sec:conclusion}
The proliferation of many-core architectures and workloads with irregular parallelism and load imbalance have shifted the focus
from traditional fork-join parallelism to task-based parallelism.
Nevertheless, task management costs are still an important source of overhead, especially when using fine granularities.
Throughout this paper, we enhance two critical components that bound the ability of runtime systems to manage fine-grained tasks:
the dependency system and the scheduler. We combine both with a state of the art memory allocator to achieve very competitive performance.

We have introduced a novel wait-free approach to implementing dependency management inside a parallel runtime.
We have also defined the \textit{Atomic State Machine} concept and its restrictions and formalized its wait-freedom.
We believe the ASM concept is applicable to similar models and runtimes that use a data-flow execution model.

Additionally, we proposed a novel \textit{Delegation Ticket Lock} that delivers very good performance compared to other
state-of-the-art locks, while keeping the simplicity in the development of scheduling internals and policies.

We also identified the critical contention bottleneck caused by memory management and tackled the problem by leveraging the \textit{jemalloc} state-of-the-art scalable memory allocator.

Finally, we implemented a highly-detailed instrumentation to provide information from both application and kernel level, while introducing
minimal overhead. Such a tool is crucial to identify and analyze bottlenecks in modern runtime systems.

Our evaluation assesses the performance of the different components separately and together, showing important performance improvements compared to
(1) the previous version of the runtime system, and (2) state-of-the-art runtime systems such as Intel OpenMP, GNU GOMP and LLVM OpenMP.

As future work, we plan to investigate extensions of the DTLock interface to support flat combining \cite{10.1145/1810479.1810540}.
This interface will require the ability to access and unblock several waiting threads simultaneously to be able to combine their operations.

\section{Data dependency system}
\label{sec:dependencysystem}

The data dependency system is one of the limiting factors in the scalability of task-based runtimes, especially when running programs with very fine-grained tasks.
Such programs register large amounts of small tasks with dependencies that take a short time to execute.
Thus the overhead in the dependency system can significantly impact the overall application performance.

In this paper, we apply our optimizations to the Nanos6 runtime \cite{bsc2019nanos6} for the OmpSs-2 programming model.
Compared to OpenMP, the model for data dependencies in OmpSs-2 is more complex \cite{JMWeaks, reductions-ferran, bsc2020ompss2}.
The main complexity increase is because the dependency domains of tasks on different nesting levels can share dependencies,
 which complicates the locking scheme used to implement the model.
Reductions also are treated as data dependencies on OmpSs-2 tasks, unlike OpenMP where they are defined at a task group level.

\subsection{Dependencies in Nanos6}

\begin{figure}
	\centering
	\includegraphics[width=.8\columnwidth]{figures/FigureDeps.pdf}
	\Description{The graph of task dependency accesses on an OmpSs-2 program}
	\caption{The graph of task dependency accesses on an OmpSs-2 program. The program (left) results in the dependency graph (right) between the accesses to location A.}
	\label{fig:dependencygraph}
\end{figure}

In Nanos6, a task is a \textit{sibling} of another when they are at the same nesting level.
Tasks declared inside another one (nested) are considered \textit{child} tasks.

The dependencies of a task in Nanos6 are represented as accesses, which are composed of a memory address and an access type, e.g., read or write.
Two accesses have a {\it successor} relation when they share the same memory address and their tasks have a {\it sibling} link.
Similarly, two accesses have a {\it child} relation if they share the same memory address and their tasks have a {\it child} link.
Task access relations on OmpSs-2 programs form binary trees between the linked tasks, as shown in Figure \ref{fig:dependencygraph}.
Note that on OpenMP users cannot express dependencies crossing nesting levels, and the \textit{child} relationship is not considered to determine the dependencies between tasks.

During an OmpSs-2 program, several tasks can be created and finished concurrently.
They have to propagate dependency information through the data structures to determine if the added tasks are ready and if the recently finished tasks allow any successor to become ready.
\subsection{Wait-free data dependencies}

The previous implementation of dependencies inside Nanos6 was based on fine-grained locking, but it was very complex to avoid possible deadlocks.
Instead of protecting the data structures that hold the information about the dependencies through mutual exclusion, our alternative is to adapt some wait-free programming concepts to create a data structure capable of supporting the concurrency we need.
Otherwise, if we had decided to build a data structure based on mutual exclusion, we would have to compromise either with the complexity of fine-grained locking or the performance degradation of coarse-grained locking.
The main goal is not to have wait-freedom as a requirement but to provide fast and scalable dependency registration.

In this section, we describe the concept behind the dependency implementation we propose for the Nanos6 runtime.
In the following section, we will formalize the approach and prove its wait-freedom property.

When a program creates a task, all the dependencies are registered inside the Nanos6 runtime using the \textit{DataAccess} structure, shown in Listing \ref{lst:dataAccess}.
The \textit{Task} structure stores all task-related information, including a pointer to the array of its accesses.
Each access has an atomic \textit{flags} field that stores its current state, indicating if the dependency is currently satisfied (not preventing the task execution) and whether the satisfiability information has propagated to its successor and child accesses.
The access also stores a pointer to its \textit{successor}, which is the next access (belonging to a successor task) to the same address in the current nesting level, and a pointer to the \textit{child}, which points to the first access to the same address that belongs to a child task.

The flags field represents a Finite State Machine in which the state diagram has no cycles, so there are starting and final states.
Since we only modify this data structure with atomic operations, we have named it \textit{Atomic State Machine} or ASM.
Note that there is one instance of this state machine for each access.

\begin{lstlisting}[language=C, label=lst:dataAccess, caption=Relevant fields in the Task and DataAccess structures]
	struct Task {
		...
		DataAccess *dataAccesses;
	};

	struct DataAccess {
		void *address;
		std::atomic<access_flags_t> flags;
		DataAccessType type;
		DataAccess *successor, *child;
	};
\end{lstlisting}

The only way for an ASM to transition from one state to another is through receiving a data access message.
The structure of a message, shown in Listing \ref{lst:dataAccessMessage}, contains two flags fields.
One field contains the flags to set in the target access.
The other has flags that have to be set on the message's originator as a delivery notification.
The ASM's transitions have to be done as a single atomic operation, optimally through a \textit{fetch\&or}.
Based on the values before and after the transition, the ASM may generate additional messages to deliver to its child or successor accesses.
All the messages that are still pending to deliver are stored in a simple per-thread queue called MailBox.
We illustrate this process in Figure \ref{fig:mailbox}.

\begin{lstlisting}[language=C, label=lst:dataAccessMessage, caption=DataAccessMessage structure]
	struct DataAccessMessage {
		access_flags_t flagsForNext, flagsAfterPropagation;
		DataAccess *from, *to;
	};
\end{lstlisting}

Figure \ref{fig:mailbox} represents the basic structure of the algorithm.
While the \textit{MailBox} has undelivered messages, we pop one from the container and \textit{deliver} it to the destination access.
Upon receiving the message, the access atomically updates its \textit{flags} field.
The atomic update provides us with the flags' exact values before and after the message was received.
As flags cannot be unset, we know each transition happens only once in the access lifetime.
With this information, we decide if it is needed to generate more messages (to propagate information about satisfied accesses, for example).
Finally, we atomically update the \textit{flags} field of the originator access of the message to notify the delivery of the information.
We use this last atomic update to determine we can safely delete an access.

\begin{figure}
	\centering
	\includegraphics[width=1\columnwidth]{figures/FigureMailbox.pdf}
	\Description{Atomic state machine dependency propagation}
	\caption{Atomic state machine dependency propagation}
	\label{fig:mailbox}
\end{figure}

\subsection{Formalization}

In this section, we introduce several relevant definitions and finally prove wait-freedom.

\theoremstyle{definition}
\begin{definition}{\textit{Access Flags.}}
We can define the set of all possible flags an access can have as set $F$.
Then, the set $F_a$ of flags that an access $a$ has is defined as $F_a \subseteq F$.
When $a$ is created, flags are initialized as $F_a = \emptyset$.
\end{definition}

\theoremstyle{definition}
\begin{definition}{\textit{Delivery.}}
	The only operation that can be done on the flags $F_a$ of an access $a$ is the delivery of a message $M \subseteq F$.
	A delivery operation is defined as:
		\[F_{a, i+1} = M \cup F_{a, i}\]
	Assuming the message M follows the two restrictions:
		\[ M \cap F_{a, i} = \emptyset \]
		\[ M \neq \emptyset \]
\end{definition}

The restrictions on the content of the messages are part of what provides the wait-freedom assurance.
Informally, bits in the flag field can only be set, and the field size is limited.
Additionally, each message that an access receives has to contain at least one flag, and none of the flags can be already set in the access.
By those properties, we can deduce that each access can receive only a limited number of messages, which in the worst case is $|F|$.

\begin{lemma}
	\label{lem:waitfreedelivery}
	The delivery of a message $M$ to an access $a$ is non-blocking and wait-free.
\end{lemma}

\begin{proof}
	To establish wait-freedom, we need to prove that there is a bound on the time a delivery operation can take regardless of any other threads.
	Assuming that the operation is performed using a \textit{CAS} primitive, we can assume constant time for the \textit{CAS}, but it can fail in case of conflict with another thread.
	Hence, to prove wait-freedom, we need to bound the number of failures due to conflict a delivery operation can suffer.

	A \textit{CAS} can fail if and only if another thread modified the memory location during the delivery operation.
	However, in our system, the only way to change the flags $F_a$ of an access $a$ is to deliver another message.
	As we established our restriction that a message cannot be empty, the maximum number of messages an access $a$ can receive if each message only contained one flag would be:
		\[M_{max} = \{ \{x\} \mid x \in F \}\]

	Then, the maximum number of \textit{CAS} operations that have to be done until one succeeds, assuming a worst case scenario, is $|M_{max}|$, which trivially $|M_{max}| = |F|$
	We can establish that the maximum number of tries to deliver a message $M$ is $T_m \leq |F|$.
	It is possible to get a closer bound on the retries if we consider the number of flags in the message to be delivered, but this is sufficient for us to prove Lemma \ref{lem:waitfreedelivery}.
	The time needed to deliver a message is clearly bounded to a constant number of \textit{CAS} operations.
\end{proof}

\theoremstyle{definition}
\begin{definition}{\textit{Unregister.}}
For a \textit{Task} $t$ that has a set of accesses $A_t$, the unregister operation on a Task is defined as delivering a specific message $M$ to each access $a$ so that $a \in A_t$.
\end{definition}

A task is unregistered once it finishes its execution, and the message delivered to the access indicates this condition.
An unregister operation will thus do $|A_t|$ delivery operations.
As we have proved that a delivery is wait-free, the unregister operation will be wait-free because it does a finite and known number of delivery operations.

\section{Evaluation}
In this section, we evaluate the effects of the different optimizations that we present in this paper.
To evaluate their impact, we have prepared different versions of the Nanos6 runtime system, where each one removes one of the three optimizations.
This methodology allows for a better understanding of which optimizations have the most significant impact on runtime performance.
To prove that these optimizations make Nanos6 one of the lowest-overhead task runtimes, we also compare our most optimized version with the most relevant OpenMP implementations.
We conduct our experiments on three HPC machines.

\subsection{Methodology}

To evaluate the task-based runtimes and check the capability of scaling to more finely partitioned work, we will use the following benchmarks, running constant problem sizes and varying the task granularity. These are (1) a \textbf{Dot product} between two arrays that uses a task reduction to aggregate the results from each block, (2) an iterative \textbf{Gauss-Seidel} method solving the heat equation of a 2-D matrix in blocks and task reductions to calculate the residual of each time step, (3) a taskified \textbf{HPCCG} with with several kernels using task reductions and multi-dependencies, (4) a taskified version of \textbf{Lulesh} 2.0 \cite{lulesh}, (5) a taskified \textbf{miniAMR} that mimics the different patterns of Adaptive Mesh Refinement applications \cite{miniAMR, sala2020towards}, (6) a classic parallel blocked \textbf{Matmul}, (7) an \textbf{NBody} benchmark that mimics dynamic particle system simulations, and (8) a blocked \textbf{Cholesky} decomposition that is generally compute bound.

We ran our experiments on various HPC platforms featuring very different architectures:
(1) the \textbf{Intel Xeon} with 2x Intel Xeon Platinum 8160 (Sky-lake) for a total of 48 cores,
(2) the \textbf{ARM Graviton2} with 64 ARM Neoverse N1 cores, and
(3) the \textbf{AMD Rome} with 2x AMD EPYC 7H12 processors for a total of 128 cores and 256 hardware threads.

For all benchmarks, the parallelization is implemented using tasks with OpenMP and OmpSs-2 versions that feature the same amount of parallelism.
However, the kernels used inside each task are sourced from the best available vendor library for each machine, to guarantee competitive performance.
In the AMD and Intel machines this library was Intel MKL, and on the ARM Graviton2 we used the ARM Performance Libraries.
We ran each benchmark a minimum of five times to extract each measurement.

Following this evaluation, we will compare our optimized Nanos6 runtime with the most relevant OpenMP implementations for each machine, including GOMP 9.2.0 \cite{GOMPSource}, LLVM 10 \cite{LLVMSource}, Intel OpenMP and the AMD AOCC depending on availability for each platform.
It is worth noting that both the LLVM, AMD AOCC and Intel OpenMP runtime are based on a work-stealing scheduler, which will allow us to determine if our centralized delegation-based implementation can outperform work-stealing runtimes.

Note that some combinations might not be available depending on the platform because of incompatibilities, non-implemented OpenMP features or compiler bugs. For brevity, we only show the four most relevant benchmarks for each machine.

\subsection{Results}

The best way we found to evaluate objectively how each component of the runtime affects the overall performance is to remove the optimizations we have described selectively.
To present the results, we use a metric \cite{taskbench} we will refer to as \textit{efficiency}.
It is calculated by dividing the performance of a specific run of a benchmark by the peak performance obtained across all executions.
This \textit{efficiency} provides a view of how close to peak performance is a specific run while being agnostic to benchmark specific units.
Combining this metric with varying task granularity \cite{OpenMPGranularities, WorksharingMarcos} gives a good view of each runtime version's scalability.
The granularity is expressed in instructions executed per task, which gives an approximation of the task's size.
We chose this unit instead of using time or cycles because the scheduling policies used by each of the runtimes can affect the execution time of a task, and thus it could result in unfair comparisons.

The runtime version without the wait-free dependencies uses the previous dependency implementation based on fine-grained locking.
The variant without the DTLock has a simple mutual exclusion mechanism (based on the PTLock) protecting the scheduler.
Finally, the version without jemalloc uses the standard system allocator for each of the machines.

\begin{figure}
	\centering
	\subcaptionbox*{}{
		\includegraphics[width=\columnwidth]{results/Legend.pdf}
	}\\[-2.5ex]
	\setlength{\tabcolsep}{0pt}
	\subcaptionbox*{}{
		\begin{tabular}{p{2.5mm} c}
			\multirow{2}{*}{\hspace{-1mm}\rotatebox[origin=lc]{90}{\footnotesize{Efficiency}}} &
			\includegraphics[width=.45\columnwidth]{results/lulesh.pdf}
			\\
			&\includegraphics[width=.45\columnwidth]{results/dotp.pdf}
			\\
			& \multicolumn{1}{r}{\footnotesize{Task}}
		\end{tabular}
	}
	\subcaptionbox*{}{
		\begin{tabular}{c}
			\includegraphics[width=.45\columnwidth]{results/miniamr.pdf}
			\\
			\includegraphics[width=.45\columnwidth]{results/cholesky.pdf}
			\\
			\multicolumn{1}{l}{\footnotesize{granularity}}
		\end{tabular}
	}
	\vspace{-2em}
	\caption{Efficiency vs task granularity of the Nanos6 runtime with and without the described optimizations on Intel Xeon (higher is better)}
	\Description{Efficiency vs task granularity of the Nanos6 runtime with and without the described optimizations on Intel Xeon (higher is better)}
	\label{fig:results-nanos}
\end{figure}

Figure \ref{fig:results-nanos} displays our benchmarks running in the Intel Xeon platform.
The different versions allow us to explore precisely how each optimization affects the scalability for different benchmarks.
The results also confirm that for every benchmark at least one optimization greatly increases the performance for fine-grained tasks.

\begin{figure}
	\centering
	\subcaptionbox*{}{
		\includegraphics[width=\columnwidth]{results/Legend.pdf}
	}\\[-2.5ex]
	\setlength{\tabcolsep}{0pt}
	\subcaptionbox*{}{
		\begin{tabular}{p{2.5mm} c}
			\multirow{2}{*}{\hspace{-1mm}\rotatebox[origin=lc]{90}{\footnotesize{Efficiency}}} &
			\includegraphics[width=.45\columnwidth]{results/amd/nbody.pdf}
			\\
			&\includegraphics[width=.45\columnwidth]{results/amd/hpccg.pdf}
			\\
			& \multicolumn{1}{r}{\footnotesize{Task}}
		\end{tabular}
	}
	\subcaptionbox*{}{
		\begin{tabular}{c}
			\includegraphics[width=.45\columnwidth]{results/amd/miniamr.pdf}
			\\
			\includegraphics[width=.45\columnwidth]{results/amd/matmul.pdf}
			\\
			\multicolumn{1}{l}{\footnotesize{granularity}}
		\end{tabular}
	}
	\vspace{-2em}
	\caption{Efficiency vs task granularity of the Nanos6 runtime with and without the described optimizations on AMD Rome (higher is better)}
	\Description{Efficiency vs task granularity of the Nanos6 runtime with and without the described optimizations on AMD Rome (higher is better)}
	\label{fig:results-nanos-amd}
\end{figure}

Figure \ref{fig:results-nanos-amd} shows a similar picture on the AMD Rome system.
This system has a much larger number of CPUs, which can increase the performance degradation caused by heavily contended locks.
Illustrating this point, we see that the scheduler optimization is much more relevant than in the Intel Xeon.
The clearest example is seen on the \textit{miniAMR} benchmark, which we analyze with detailed traces in subsection \ref{sec:traces}.

\begin{figure}
	\centering
	\subcaptionbox*{}{
		\includegraphics[width=\columnwidth]{results/Legend.pdf}
	}\\[-2.5ex]
	\setlength{\tabcolsep}{0pt}
	\subcaptionbox*{}{
		\begin{tabular}{p{2.5mm} c}
			\multirow{2}{*}{\hspace{-1mm}\rotatebox[origin=lc]{90}{\footnotesize{Efficiency}}} &
			\includegraphics[width=.45\columnwidth]{results/arm/heat.pdf}
			\\
			&\includegraphics[width=.45\columnwidth]{results/arm/hpccg.pdf}
			\\
			& \multicolumn{1}{r}{\footnotesize{Task}}
		\end{tabular}
	}
	\subcaptionbox*{}{
		\begin{tabular}{c}
			\includegraphics[width=.45\columnwidth]{results/arm/miniamr.pdf}
			\\
			\includegraphics[width=.45\columnwidth]{results/arm/matmul.pdf}
			\\
			\multicolumn{1}{l}{\footnotesize{granularity}}
		\end{tabular}
	}
	\vspace{-2em}
	\caption{Efficiency vs task granularity of the Nanos6 runtime with and without the described optimizations on ARM Graviton2 (higher is better)}
	\Description{Efficiency vs task granularity of the Nanos6 runtime with and without the described optimizations on ARM Graviton2 (higher is better)}
	\label{fig:results-nanos-arm}
\end{figure}

Finally, Figure \ref{fig:results-nanos-arm} shows the same benchmarks running on an ARM Graviton2 system.
Results are similar to our Intel Xeon evaluation, although some benchmarks have different behaviors due to the lack of NUMA effects on this platform.

Overall, we have seen that our optimizations achieve significant performance gains, especially on small task granularities.
Depending on the benchmark, the wait-free dependencies or the scheduler are the most important optimizations.
However, the scalable memory allocator also delivers some notable performance improvements, especially on the Intel Xeon and AMD Rome machines.

\subsection{Comparison versus OpenMP}

To give context to the results, we compare the optimized Nanos6 runtime to current state-of-the-art OpenMP runtimes on the same three systems.
Our baseline for every benchmark will be the GOMP runtime, distributed with the GCC Compiler, and the LLVM OpenMP Runtime.
However, on Intel Xeon we use the Intel OpenMP runtime, and on AMD Rome the runtime provided by the AMD AOCC, as long as they implement all the features needed by the benchmark.
To ensure a fair comparison, we expressed the same parallelism in the OmpSs-2 and OpenMP versions of the benchmarks.
We also used available kernels in Intel MKL or the ARM Performace Libraries, to prevent noise introduced by the compilers to alter the results.

\begin{figure}
	\centering
	\subcaptionbox*{}{
		\includegraphics[width=.8\columnwidth]{results/LegendOmp.pdf}
	}\\[-2.5ex]
	\setlength{\tabcolsep}{0pt}
	\subcaptionbox*{}{
		\begin{tabular}{p{2.5mm} c}
			\multirow{2}{*}{\hspace{-1mm}\rotatebox[origin=lc]{90}{\footnotesize{Efficiency}}} &
		\includegraphics[width=.45\columnwidth]{results/heat-omp.pdf}
		\\
		&\includegraphics[width=.45\columnwidth]{results/dotp-omp.pdf}
		\\
		& \multicolumn{1}{r}{\footnotesize{Task}}
		\end{tabular}
	}
	\subcaptionbox*{}{
		\begin{tabular}{c}
		\includegraphics[width=.45\columnwidth]{results/miniamr-omp.pdf}
		\\
		\includegraphics[width=.45\columnwidth]{results/cholesky-omp.pdf}
		\\
		\multicolumn{1}{l}{\footnotesize{granularity}}
		\end{tabular}
	}
	\vspace{-2em}
	\caption{Comparison of performance between the current Nanos6 version and the main OpenMP runtimes on Intel Xeon (higher is better)}
	\Description{Comparison of performance between the current Nanos6 version and the main OpenMP runtimes on Intel Xeon (higher is better)}
	\label{fig:results-omp}
\end{figure}

Figures \ref{fig:results-omp}, \ref{fig:results-omp-amd} and \ref{fig:results-omp-arm} feature the results of comparing the current OpenMP implementation versus the optimized variant of the Nanos6 runtime.
The results are really positive, as in all of the machines, and all of the benchmarks, the best performance in small granularity tasks is provided by the Nanos6 runtime.
In some cases, a higher peak performance is also achieved.
This happens when the ideal block size for a specific benchmark is small enough that performs better in one runtime than another.

As for the other runtimes, the LLVM OpenMP implementation comes second in most benchmarks in terms of scalability, and ties on AMD Rome with the runtime provided with the AOCC compiler.
However, this is expected because the AOCC compiler is based on LLVM 10.

\begin{figure}
	\centering
	\subcaptionbox*{}{
		\includegraphics[width=.8\columnwidth]{results/amd/LegendOmp.pdf}
	}\\[-2.5ex]
	\setlength{\tabcolsep}{0pt}
	\subcaptionbox*{}{
		\begin{tabular}{p{2.5mm} c}
			\multirow{2}{*}{\hspace{-1mm}\rotatebox[origin=lc]{90}{\footnotesize{Efficiency}}} &
		\includegraphics[width=.45\columnwidth]{results/amd/hpccg-omp.pdf}
		\\
		&\includegraphics[width=.45\columnwidth]{results/amd/nbody-omp.pdf}
		\\
		& \multicolumn{1}{r}{\footnotesize{Task}}
		\end{tabular}
	}
	\subcaptionbox*{}{
		\begin{tabular}{c}
		\includegraphics[width=.45\columnwidth]{results/amd/miniamr-omp.pdf}
		\\
		\includegraphics[width=.45\columnwidth]{results/amd/matmul-omp.pdf}
		\\
		\multicolumn{1}{l}{\footnotesize{granularity}}
		\end{tabular}
	}
	\vspace{-2em}
	\caption{Comparison of performance between the current Nanos6 version and the main OpenMP runtimes on AMD Rome (higher is better)}
	\Description{Comparison of performance between the current Nanos6 version and the main OpenMP runtimes on AMD Rome (higher is better)}
	\label{fig:results-omp-amd}
\end{figure}

\begin{figure}
	\centering
	\subcaptionbox*{}{
		\includegraphics[width=.7\columnwidth]{results/arm/LegendOmp.pdf}
	}\\[-2.5ex]
	\setlength{\tabcolsep}{0pt}
	\subcaptionbox*{}{
		\begin{tabular}{p{2.5mm} c}
			\multirow{2}{*}{\hspace{-1mm}\rotatebox[origin=lc]{90}{\footnotesize{Efficiency}}} &
		\includegraphics[width=.45\columnwidth]{results/arm/heat-omp.pdf}
		\\
		&\includegraphics[width=.45\columnwidth]{results/arm/hpccg-omp.pdf}
		\\
		& \multicolumn{1}{r}{\footnotesize{Task}}
		\end{tabular}
	}
	\subcaptionbox*{}{
		\begin{tabular}{c}
		\includegraphics[width=.45\columnwidth]{results/arm/miniamr-omp.pdf}
		\\
		\includegraphics[width=.45\columnwidth]{results/arm/matmul-omp.pdf}
		\\
		\multicolumn{1}{l}{\footnotesize{granularity}}
		\end{tabular}
	}
	\vspace{-2em}
	\caption{Comparison of performance between the current Nanos6 version and the main OpenMP runtimes on ARM Graviton2 (higher is better)}
	\Description{Comparison of performance between the current Nanos6 version and the main OpenMP runtimes on ARM Graviton2 (higher is better)}
	\label{fig:results-omp-arm}
\end{figure}

\subsection{Detailed traces}
\label{sec:traces}

\begin{figure*}
	\centering
	\includegraphics[width=\textwidth]{figures/SchedulerLockTrace.pdf}
	\caption{Scheduler lock comparison}
	\Description{Scheduler lock comparison}
	\label{fig:scheduler-trace}
\end{figure*}

Figure \ref{fig:scheduler-trace} shows two miniAMR traces obtained with the new Nanos6 CTF instrumentation backend comparing the Partitioned Ticket Lock (PTlock) and the combination of wait-free queues with the Delegation Ticket Lock (DTLock).
The view displays running tasks (in red), specific runtime subsystems such as task creation (in cyan), or other generic runtime parts (in deep blue) along time (X axis) for a number of cores (Y axis).
The total length is 500us and the zoomed areas (black rectangles) are 10us long, approximately.
Hint {\it A} points to cores running a "task creation" task.
The wait-free version (above) allows created tasks to be queued independently of other cores requesting ready tasks to run.
Instead, in the PTLock version (below), adding and getting a ready task requires obtaining a shared lock which leads the "task creation" task to undergo heavy pressure as other cores requesting a ready task also attempt to acquire the same lock.
Consequently, the number of available ready tasks cannot match the task completion rate and most cores starve (in khaki green).

Hint {\it B} points to DTLock task serving periods.
Yellow arrows depict single tasks being served by the delegation lock owner to waiting cores.
When no more ready tasks left, the lock owner moves ready tasks from the wait-free queues (in green) and continues serving tasks.

\begin{figure}
	\centering
	\includegraphics[width=\columnwidth]{figures/ServerInterruptKernelTrace.pdf}
	\caption{Operating System noise effect on Scheduler}
	\Description{Operating System noise effect on Scheduler}
	\label{fig:scheduler-interrupt}
\end{figure}

Figure \ref{fig:scheduler-interrupt} exemplifies the effect that the operating system noise can incur on the runtime system.
The upper trace displays runtime threads (in deep blue) and hardware interruptions (purple).
The trace below shows a view similar to Figure \ref{fig:scheduler-trace}, where a considerable delay is introduced in the server which causes all cores to stall but the "task creator" core.
Note the yellow lines pattern difference before (irregular) and after (regular) the interrupt.
While the serving thread was stalled in the interrupt, a provision of ready tasks was accumulated.
The surplus of tasks is enough to feed all cores leading to long periods of red (tasks) without yellow lines.
As the extra reserve of tasks lowers, the chances of at least two idle cores requesting a task simultaneously increase, and extra yellow lines appear.
In conclusion, combining OS events with runtime events allows us to complete the whole picture and to identify the source of problems better.

\section{Instrumentation}
Analyzing runtime performance and finding problems or bottlenecks in the different component requires a mechanism to collect fine-grained instrumentation data.
This instrumentation must have a very low overhead, which is difficult to achieve on a very optimized runtime.
Additionally, runtime systems are sensitive to OS noise (such as thread preemptions), making exploiting kernel internals particularly useful when evaluating latency-critical features, such as those presented in this article.
For this reason, we have developed a new tracing backend aiming at minimum overhead and with both runtime and kernel tracing capabilities.

The backend generates traces in the Common Trace Format (CTF) \cite{ctf}, which strives for fast data writes.
Instrumentation overhead is minimized by storing events on lock-free NUMA-aware per-core circular buffers.
Each buffer is divided into page-aligned sub-buffers that, when full, are periodically flushed to a tmpfs backed file by Nanos6 threads between tasks execution.
Each file contains a time-ordered event subset of the final trace, with either kernel or user events.

Nanos6 threads write events on the lock-free per-core buffer they are pinned to.
User-selected Kernel events are obtained from a per-core memory-mapped circular buffer exported by the Linux Kernel through the {\it perf\_event\_open()} system call.
Between task executions, Nanos6 threads read and format the kernel events according to the CTF specification and move them to an exclusive kernel-events Nanos6 per-core circular buffer.

\section{Introduction}
\label{sec:introduction}
Due to diminishing returns on modern CPUs' single-thread performance, the industry has shifted towards many-core and heterogeneous architectures \cite{DarkSiliconMulticore,TheisMoore,BorkarFuture}.
The recent focus on energy efficiency has increased the interest in systems with numerous processing elements with lower frequencies.
Those parallel systems can achieve huge performance figures, but their limiting factor is the scalability of the software.

One of the most widely used standards for programming shared-memory systems in both industry and academy is OpenMP\cite{openmpindustry}.
OpenMP initially had only a \textit{fork-join} execution model, where programmers explicitly define parallel regions.
The fork-join model is an efficient way to exploit well-structured parallelism, but it is not well suited to exploit irregular, dynamic, or nested parallelism.
In recent years, task-based parallelism has been introduced in OpenMP to overcome these limitations.

The task-based paradigm can exploit more fine-grained, dynamic, and irregular parallelism than the fork-join model.
Additionally, it minimizes the need for global synchronization points, and it naturally copes with load-imbalance.
These features make the paradigm especially promising to exploit modern many-core architectures \cite{taskBased1, taskBased2}.

Moreover, the introduction of task data dependencies was the critical element to truly move forward to a \textit{data-flow} execution model that relies on fine-grained synchronizations between tasks.
This model reduces further the need for global synchronization points and allows the runtime to exploit data-locality between tasks.
However, task management inside the runtime might incur some non-negligible overhead, especially for fine-grained tasks on large many-core systems.

This paper presents optimized designs for the main components of a task-based runtime system.
We also present a lightweight and integrated instrumentation system, which we used to analyze in detail how the scalability of each component affects the global scalability of the runtime.

A task-based runtime system has three main components: the dependency system, the task scheduler, and the memory allocator, which tightly interact with each other.
The first stage of a task's life cycle is its creation, which involves the memory allocator.
The runtime then checks its data dependencies to determine if the task is ready or blocked based on the previous tasks' dependencies.
Once all its dependencies are satisfied, the task becomes ready and is added to the scheduler, which will eventually schedule it on an available core.
Once the task has executed, it releases its dependencies so that its successor tasks may become ready.
Note that the three components require a synchronization mechanism as they have to deal with multiple requests simultaneously.
Thus, the application developer has to strike a balance in task granularity: it has to be small enough to provide sufficient work for all available cores while being coarse enough to evade runtime system overheads \cite{OpenMPGranularities}.
However, as applications scale out to more cores (or nodes) and the problem size remains constant, task granularities naturally decrease.
At some point in the scaling process, tasks can become so small that the application is overhead-bound, and the scalability depends on the ability of the parallel runtime to handle small tasks.

Our contributions are to (1) present a novel wait-free data structure and algorithm to support complex dependency models in a task-based runtime; (2) provide a scalable task scheduler that works well under high contention and uses delegation instead of work-stealing; and (3) analyze and create a detailed performance profile of the task-based runtime with a lightweight instrumentation framework.
\section{Memory management}

When optimizing a runtime to achieve the lowest overhead, every operation that requires synchronization between thre\-ads quickly becomes a bottleneck.
This is the case for memory allocation.
Some general-purpose allocators are not well suited to handle a high volume of memory requests in many-core systems.
Many implementations require the serialization of every allocation in the system.
Additionally, the operating system may introduce even more overhead when allocators request more memory areas through system calls.

In the Nanos6 runtime case, removing contention from the scheduler and dependencies caused an even more significant bottleneck on memory allocation.
However, the current state of the art techniques for scalable memory allocation can be applied to any software \cite{hoard, jemalloc}, solving most of the contention problems.
To solve this bottleneck and achieve the performance presented in this article, we had to substitute the default allocator in Nanos6 for Jemalloc, a widely used scalable memory allocator.

\section{Related work}
\label{sec:relatedwork}

Other dependency system implementations have been described in previous literature, such as the implementation for the OpenUH compiler \cite{PrototypeDependencySupport}.
The GOMP library and LLVM's OpenMP runtime are also available online as Open Source software \cite{GOMPSource}\cite{LLVMSource}.
The topic of overhead in dependency resolution has also been tackled from other angles, such as the TurboBLYSK framework, to create dependency patterns \cite{TurboBLYSK}.

Previous research has also aimed at applying a lock-free approach to dependency resolution, with \cite{DataFlowAnalysis} analyzing several generic dependency resolution schemes and concluding that a ticket-based lockless scheme provided the best performance in their benchmarks.
On the same line, another lock-free dependency system was implemented for the OMPi OpenMP/C Compiler \cite{LocklessListDeps}, which supported OpenMP 4.0 and was based on the same patterns as lock-free lists.
However, our implementation offers a stronger wait-free guarantee.

Regarding task scheduling, there are several studies on alternatives to centralized lock-based scheduling.
Many have studied work-stealing techniques for hierarchical and partitioned schedulers in shared-memory systems \cite{prokopec2013near,guo2010slaw,wang2014adaptive}.
Olivier et al. \cite{olivier2012openmp} proposed a hierarchical scheduler featuring a lock-free ready task queue per socket.
Once the socket's queue is empty, only one of the threads in that queue can try to steal tasks from other sockets, while the others wait.
Similarly, Muddukrishna et al. \cite{muddukrishna2015locality} proposed a lock-based queue per NUMA node, but in this case, workers from the same NUMA can steal at the same time.
In contrast, Vikranth et al. \cite{vikranth2013topology} implemented a task queue per thread, grouped into stealing domains (e.g., one per socket).
Workers always first try to steal from queues in the local domain before trying in the rest.
However, all these approaches can suffer the same bottlenecks as centralized schedulers when running in modern many-core systems.
The ready task queues of consumer threads tend to be empty, so they steal tasks from the creator's queues (usually few), producing contention in those sections.
Also, hierarchical schedulers often have complicated implementations, so developing new scheduling policies becomes an arduous task.

Even though, notice that these hierarchical approaches with work-stealing could use our DTLock (Section \ref{sec:dtlock}) to protect the access to task queues.

The Linux Kernel static tracing infrastructure is used by several backends such as LTTng, ftrace, perf, SystemTap or eBPF.
Yet, only LTTng focuses on efficient user and kernel correlated static tracing \cite{lttng}.
When tracing on the context of runtime systems for HPC, LTTng has two main drawbacks.
On the one hand, tracing the kernel requires installing an out-of-tree Linux Kernel module that usually clashes with operating policies of data centers and supercomputing facilities.
On the other hand, LTTng relies on server daemons to collect and write tracepoints from both user and kernel space.
Such daemons might oversubscribe runtime threads leading to undesired noise.
Therefore, we concluded that our particular case needed an ad-hoc tracing solution.
\section{Task Scheduling System}
The scheduling system orchestrates the execution of ready tasks on worker threads.
Throughout this section, we assume that exactly one worker thread is bound to each CPU for simplicity but without loss of generality.
When a task becomes ready, it is forwarded to the scheduling system. Then, when a core becomes idle, it calls the scheduler to ask for more work.
If there are ready tasks, the scheduling system will determine the best task that can be executed on this specific core.
It is worth noting that multiple ready tasks can be added to the scheduler concurrently and that several worker threads can simultaneously call the scheduler.
Thus, it becomes mandatory to add some kind of synchronization on the scheduler to prevent data-races.

Most task-based runtime systems rely on multiple ready task queues combined with work-stealing to mitigate the above-mentioned problems.
However, on the typical application design pattern in which a single thread creates all tasks, work-stealing behaves similarly to the global lock approach because most threads need to steal work from a single creator queue.
In contrast, the approach described in this section adapts the global lock concept to handle both single creator and multiple creator cases efficiently.

Usi a global lock is the most straightforward approach to synchronize the scheduler.
In this case, the lock is acquired to add ready tasks to the scheduler and to schedule tasks to worker threads.
When task granularity is coarse enough, this approach works well and keeps the scheduling system's design simple and the scheduling policies accurate.

However, when task granularity is fine-grained and the system has many cores, the core that is creating new tasks might not be fast enough to feed all other cores.
In this scenario, many worker threads will busy-wait on the global lock.
This has two adverse effects.
Firstly, it increases contention on the cache subsystem due to the additional cache coherence traffic.
Secondly, it prevents ready tasks from entering the scheduler fast enough because the task creator has to fight with all of the worker threads to get the global lock.

A well-known technique to mitigate lock con\-ten\-tion on the scheduler is to let the worker threads spin for a while, and if they do not get any ready task, block its thread using a mutex until a ready task becomes available.
We avoid using this approach because it adds extra work to the thread that is creating tasks. When ready tasks are added to the scheduler, it has to check if there are blocked threads and wake them up with an expensive system call.

\subsection{Optimizing task insertion}
\label{sec:spsc}
To avoid the stagnation of ready tasks in the scheduler, we have decoupled the actions of adding and scheduling ready tasks. When a task becomes ready, we do not directly add it to the scheduler but a bounded wait-free single-consumer single-producer (SPSC) queue working as a buffer.

The number of SPSC queues can be configured from a single one to one per core.
In the first case, we would need a lock to synchronize all task additions, while in the latter, no locking is needed at all.
We use the lock to synchronize between producers, but the synchronization between producer and consumers remains wait-free.
In our experiments, we use one SPSC queue and lock per NUMA node.

When a worker threads enters the scheduler, it first drains all SPSC queues and inserts the ready tasks into the global ready queue.
With this approach, we ensure that any contention generated by many worker threads calling the scheduler does not affect the performance of cores that are creating tasks.
We can implement this optimization because the actual addition of tasks can be safely delayed until a core becomes idle and calls the scheduler.
Notice that this delegation technique is compatible with any lock implementation.

\subsection{Scalable lock designs}
The scheduler system has to be extensible, and adding new scheduling policies should be easy.
We have discarded a wait-free or lock-free scheduler because of its complexity and difficulty to maintain, as each scheduling policy would require a new ad-hoc design and implementation.

Our scheduling system relies on a global lock to protect its internal data structures, making it easy to develop new scheduling policies.
Ticket Locks \cite{10.1145/359060.359076} are fair and provide strict FIFO ordering, but they have contention problems under high-load conditions, so they are not suitable for our centralized scheduler.
Partitioned Ticket Locks \cite{10.1145/1989493.1989543} (PTLocks) extend Ticket Locks with a padded array used to do busy-waiting by idle threads.
If the size of this array is equal to the number of CPUs, then each core will busy-wait in a different array slot, reducing the cache coherence traffic to the minimum.
We use PTLocks as a building block of our optimized lock design presented in the next section.

\lstinputlisting[language=C++, label=lst:PTL, caption=Implementation of a PartitionedTicketLock, firstline=6, lastline=26]{code/PartitionedTicketLock.hpp}

Listing \ref{lst:PTL} shows the implementation of a Partitioned Ticket Lock. For the sake of clarity, we have omitted the padding of the fields to prevent false sharing, and the memory order constraints of all atomic operations. The {\it \_waitq} (line 6) is an array of unsigned 64-bit integers used to implement a circular buffer representing an infinite virtual waiting queue. The {\it \_head} and {\it \_tail} fields (lines 4 and 5) are used to index the {\it \_waitq} array. The {\it \_head} represents the index of the latest slot in the virtual waiting queue and the {\it \_tail} is the index of the next slot that will be able to acquire the lock. When the lock is free and no thread is waiting to acquire it, {\it \_tail == \_head+1}.

We initialize the lock such that {\it \_waitq[\_head\%Size] == \_head}, guaranteeing that the first thread that arrives will be able to acquire it.
The {\it lock()} operation (line 14) consists of just two calls. The first one is {\it \_getTicket()} (line 8), which performs an atomic fetch and increment of the {\it \_head} field to obtain the last ticket (line 9). The second call is {\it \_waitTurn} (line 11) that receives the ticket as a parameter. The current thread busy-waits on the {\it \_waitq[ticket \% Size]} position until it matches (or exceeds) the ticket value. The {\it unlock} operation (line 17) calculates the next slot index that will be able to acquire the lock.  Then it increments {\it \_tail} and writes {\it \_tail-1} in the computed slot to release the lock.

PTLocks perform as well as more complex designs such as MCS~\cite{10.1145/103727.103729} or Ticket Locks Augmented with a Waiting Array (TWA)~\cite{TWA}, however it requires more memory space.

\subsection{Delegation Ticket Lock (DTLock)}
\label{sec:dtlock}
Another well-known technique to improve the performance of data structures that are not amenable to fine-grained locking is delegation \cite{10.1145/3132747.3132771}.
The main idea behind this method is that protected data structures are accessed only by one privileged thread, which executes all the operations on behalf of the other threads.
Delegation relies on a lightweight and optimized communication protocol between the privileged thread and the rest of the threads to be able to delegate operations and forward results back.
A drawback of this approach is that it requires a dedicated core for each independent set of data structures that has to be protected, making it unpractical in many situations.

There are variants of the delegation technique that use queues to delegate work to the threads currently inside the lock \cite{queuedelegationlock}.
These are better suited to use in centralized schedulers as they do not require dedicated cores for each lock.
In order to build our scalable centralized scheduler, we have developed a novel Delegation Ticket Lock (DTLock) that builds on state of the art delegation techniques and extends our implementation of the PTLock with support for fine-grained and dynamic delegation of operations.
The DTLock supports the standard {\it lock}, {\it unlock} and {\it trylock} operations, as well as {\it lockOrDelegate}, {\it empty}, {\it front}, {\it popFront} and {\it setItem}.
The {\it lockOrDelegate} operation either acquires the lock if it is free or delegates the operation to the current lock's owner.
However, it is the current owner that will decide if it executes or not the delegated operations.
Suppose the current owner releases the lock without performing a pending delegated operation.
In that case, the thread that originally delegated that operation will eventually acquire the lock and execute it by itself.
The {\it empty}, {\it front}, {\it popFront} and {\it setItem} operations can only be called by the thread that owns the lock and are used to manage the threads that are waiting outside the lock.

The main advantages of the DTLock are that it does not require a dedicated core and its additional operations can be freely combined with traditional {\it lock}, {\it unlock} and {\it trylock} operations.
Additionally, the DTLock allows for threads to remain inside the critical section of the lock executing delegated operations.
This can be leveraged to minimize operation latency when there is not enough work to keep all cores busy.

	\lstinputlisting[language=C++, label=lst:DTL, caption=Implementation of a Delegation Ticket Lock, firstline=5, float]{code/DelegationLock.hpp}

Listing \ref{lst:DTL} shows the implementation of a DTLock in C++, which inherits the PTLock's {\it \_head}, {\it \_tail} and {\it \_waitq[]} members, as well as, {\it lock}, {\it unlock} and {\it tryLock} operations.
The DTLock extends the PTLock with two additional arrays. The {\it \_logq} array (line 3) is used to register waiting threads while the {\it \_readyq} array (line 4) is used to store the result of delegated operations, i.e. ready tasks in our case.

The first parameter of the {\it lockOrDelegate} operation is a unique \textit{id} that identifies the thread that is calling this function. This \textit{id} should be in the range {\it 0..Size-1} as it is used to index the {\it \_readyq} array. Thus, we need to know in advance the maximum number of threads that can call the DTLock. If the {\it lockOrDelegate} operation is finally delegated, the second parameter is used to store the result.

The first thing done in {\it lockOrDelegate} is to obtain a ticket (line 7).
Then, the thread is registered on the {\it \_logq} (line 8) array with just one store operation that combines the ticket and calling thread's id.
The values written on the {\it \_logq} array cannot be overrun because it is guaranteed that there will be at most {\it Size} threads calling the {\it lockOrDelegate} operation, so that each thread will have their own position.
Once the thread has been registered, it just busy-waits (line 9) until it acquires the lock (lines 11-12), or the operation has been delegated and the result is stored in the {\it \&item} parameter (line 14).

The {\it empty} operation (line 17) returns {\it true} if there is no thread registered in the {\it \_logq} array and {\it false} otherwise. We check the first position of the {\it \_logq} array, and if the value is smaller than {\it \_tail}, we know there is no thread waiting yet. Otherwise we would see the value written in line 8. Notice that this operation is intrinsically racy but harmless.

If a call to {\it empty} returns {\it false}, then the owner of the lock can call {\it front} (line 20) to obtain the id of the thread that is waiting. To that end, we only need to do the inverse of the operation done on line 8, subtracting the {\it \_tail}'s value to obtain the thread id (line 21).

Then, the {\it setItem} operation assigns a result {\it T} to a registered thread using its id to index the {\it \_readyq} array. First, it sets the {\it item} field (line 27) and then the {\it ticket} to the {\it \_tail} value, marking the entry as valid. We use the {\it ticket} field in line 10 to determine if an operation was delegated or not.
Finally, the {\it popFront} operation wakes up the first thread busy-waiting on the {\it \_waitq} by executing the {\it unlock} operation.

\subsection{Synchronized Scheduler}
This section presents a synchronized scheduler that leverages the SPSC queues and the DTLock described in sections \ref{sec:spsc} and \ref{sec:dtlock}, respectively.

\lstinputlisting[language=C++, label=lst:SynchSched, caption=Implementation of the Synchronized Scheduler using a Delegation Ticket Lock, firstline=11]{code/SyncScheduler.hpp}

Listing \ref{lst:SynchSched} shows the implementation of a synchronized scheduler using a DTLock (line 2) and a SPSC wait-free queue (line 5) to synchronize the {\it getReadyTask} and {\it addReadyTask} operations, respectively. The SyncScheduler is a wrapper of the unsynchronized scheduler (line 3), which implements the actual scheduling policy.

The PTLock (line 4) protects the producer side of the SPSC queue in the {\it addReadyTask} function (line 14), while the DTLock protects the consumer side in the {\it processReadyTasks} function. In the {\it addReadyTask} function, if there is no free space on the SPSC queue, the thread will try to acquire the DTLock with a {\it tryLock} operation (line 17). If it succeeds, it will call the {\it processReadyTasks} function (line 18), which removes all tasks from the SPSC queue and inserts them to the unsynchronized scheduler (line 8).

We use the {\it lockOrDelegate} operation (line 25) of the DTLock to synchronize {\it getReadyTask} operations.
If the operation is delegated because another thread owns the DTLock, the calling thread will busy-wait until it gets a task (lines 25-26).
Otherwise, it will eventually acquire the lock and call the {\it processReadyTasks} (line 27) to add the tasks that are waiting on the SPSC queue into the unsynchronized scheduler.
Then, it will try to schedule a task for each of the threads that are waiting on the DTLock (lines 28-33) until there are no more waiting threads (line 28) or no ready tasks are left (line 31).
At that point, it will try to get a task for him (line 35) and then release the DTLock (line 36).
In our simplified design, the thread inside the scheduler leaves as soon as there are no more tasks to schedule.
However, it is easy to extend our design in a way that {\it processReadyTasks} is called when no tasks are left inside the scheduler.

\begin{figure}
    \centering
    \includegraphics[width=.7\columnwidth]{figures/TimelineSyncScheduler.pdf}
	\caption{Timeline of five threads using a DelegationLock to add and get ready task into the scheduler.}
	\Description{Timeline of five threads using a DelegationLock to add and get ready task into the scheduler.}
    \label{fig:timelinesyncscheduler}
\end{figure}

Figure \ref{fig:timelinesyncscheduler} shows a timeline of five threads creating and scheduling tasks using the SyncScheduler and a simple FIFO scheduling policy.
$Th_0$ has already created and inserted three tasks ($T_0$ - $T_3$) that are inside the SPSC queue before creating and inserting four additional tasks ($T_4$ - $T_7$). $Th_1$ to $Th_4$ call the {\it getReadyTask} function, one after the other, to obtain a ready task.
The call to {\it lockOrDelegate} of $Th_1$ acquires the lock, while the other threads delegate and busy-wait. Once $Th_1$ is inside the lock, it calls {\it processReadyTasks} and inserts tasks $T_0$ to $T_3$ into the actual scheduler. Then, $Th_1$ schedules one ready task for each of the waiting threads, and finally, it gets a ready task for itself. When $Th_3$ finishes the execution of $T_1$, it calls again to {\it getReadyTask}, and just after that, $Th_2$ does the same. $Th_3$ acquires the lock and calls {\it processReadyTasks}, moving the tasks from the SPSC queue ($T_4$ and $T_5$) to the actual scheduler. Finally, $Th_3$ schedules $T_4$ for $Th_2$, and then, it executes $T_5$.

In microbenchmarks, we found a fourfold speedup on task scheduling using a DTLock compared to a PTLock, and a twelvefold speedup compared to serial task insertion thanks to the SPSC queues.

redpajama_set_name: RedPajamaArXiv


text:
Стодолище (станция) - станция на железнодорожной линии Смоленск - Рославль. Расположен в деревне Стодолище Починковского района Смоленской области.

Пригородные поезда
По состоянию на 2015 год для пригородного сообщения не используется.

С 2019 года используется для пригородного сообщения.

Примечания

Железнодорожные_станции_и_платформы_Смоленской_области
Станции_Смоленского_региона_Московской_железной_дороги

redpajama_set_name: RedPajamaWikipedia


text:
Castner Scrapbook v.2, Theatres, page 4
Historical Images of Philadelphia Castner Scrapbook Collection
Item No: pdcc01791
Title: Castner Scrapbook v.2, Theatres, page 4
Historic Street Address: 6th & Chestnut Streets
Historic Street Address: Fifth & Chestnut Streets
Media Type: Scrapbooks
Source: Print and Picture Collection
Item pdcc00071, top [image dimensions 14.5 cm x 19.0 cm]:
http://www.jstor.org/stable/988747
The first Chestnut Street Theatre (originally named the New Theatre) was often referred to as one of the Seven Wonders of America. Begun in 1794, the georgian building which was modeled after the Theatre Royal in Bath, England, wasn't completed until 1805. In 1816 it became the first American theatre to be illuminated by gas fixtures rather than candlelight or oil lamps. Four years later, however, the building along with its library, scenery and costumes was destroyed by a suspicious fire. The image, c.1820, shows the theatre's Chestnut Street facade, along with well-dressed citizens and a canopied horse-drawn carriage.
Building demolished: YES 1820
Item pdcc00072, bottom right [image dimensions 16.0 cm x 12.5 cm]:
Drawn and engraved for the Sunday Dispatch expressly to illustrate Westcott's "History of Philadelphia."
Item pdcc00073, bottom left [image dimensions 6.1 cm x 7.3 cm]:
The second Chestnut Street Theatre rose from the ashes of the first Chestnut Street Theatre in 1822, which had burned down two years earlier. Its marble facade was embellished with statues of Comedy and Tragedy, sculpted by William Rush.
Geocode Latitude: Geocode Longitude:-75.150969
Geocode Latitude:39.949228
Call Number: A917.481 P536 v.2
Creator Name: Castner, Samuel, Jr., 1843-1929 - Compiler
Rush, William, 1756-1833
Chestnut Street Theatre
Horse-drawn vehicles
Proscenium arches
Go to Hip Map
Inside front cover
99 (inside back cover)
Castner, Samuel, Jr., 1843-1929 - Compiler, Rush, William, 1756-1833. Castner Scrapbook v.2, Theatres, page 4. [Scrapbooks]. Retrieved from https://libwww.freelibrary.org/digital/item/55777
Castner, Samuel, Jr., 1843-1929 - Compiler, Rush, William, 1756-1833. Castner Scrapbook v.2, Theatres, page 4. Scrapbooks. Free Library of Philadelphia: Philadelphia, PA. https://libwww.freelibrary.org/digital/item/55777. (accessed Jan 28, 2023)

redpajama_set_name: RedPajamaCommonCrawl


I'm at that point in my young life where I'm thinking about who in my family gets what. Some of you know and can relate to this. Anyway, most kids today don't read books, they gat their info off the web in some form or other. My daughters don't want these books and sine my wife passed away, frankly I don't have the kind of hold on my 'stuff' as I may have once did.
The following books are either first edition or collectors edition as best I can tell. Anyone who wants any of these, let me know. I'm giving them away. Just cover shipping.
I would like to have #2 and 4 if still avalible.
Allen, may I have #5?
1 and 6, if still available. Let me know by PM your address and what I owe you.
Allen, What a kind offer, and it looks like these books are going to good homes. I have struck up a friendship with a young man who occasionally hunts with me. We have no children and, like you, those younger family members -- nieces and nephews -- do not hunt and would see our "possessions" for what cash they might bring them. I'm hoping I can leave some my collectable sporting books to our University library, but I doubt they have the room or inclination to house such a collection.
I hope you continue to find good homes for your very nice books.
Mills, you may have #6. Pm me your address.
All books spoken for now. Thank you gents.
On a similar note, i have a beautiful African Kudu mount from a so africa safari some yrs ago. My daughter had it in her great room until they sold their home and downsized. My home will not accommodate it either. I tried to donate this near mint condition full shoulder mount to Harvards Museum of Natural History for their African but their Dir of Donations won't return my calls and text messages. I shot this animal and So Africas premier taxidermy shop, 'Lifeform' mounted it. I need to find a good home for this Kudu. Its quite large. Anyway, if someone wants it, come and get it.
I didn't see #3 go. It must have been a "curb trade".
Very generous, Allen; all of the books have a secondary market value.


Removing the customer via Plesk UI also deletes local backups that belong to that customer.
This is expected behavior. Such behavior was implemented since Plesk Onyx 17.0.15.
Vote for the related feature request "Add an opportunity to choose will the customer's backups be removed together with the customer or not" at User Voice.
Top-ranked suggestions is likely to be implemented in next product releases.
As a workaround it is possible to set up Plesk to store backups at FTP storage.
Customer's backup files stored on the FTP will not be removed when the customer is removed via Plesk interface.


US Vice-President Joe Biden was in town this week.
as compared to a US Presidential visit to Jerusalem, Israel.
But this visit was not like last time.
and for an extra layer of security, this blue tarp was erected.
Flags, flowers and podiums were arranged.
Hamentashen were among the sweets served to munch.
Key players were in their positions.
and the United States Vice-President.
International press lined the way ready to report.
right after the official hand shake.
while security shades blocked the view.
everyone knew that Joe Biden was here this time.
another one of those Arab girls!
and there was an Arab girl having a driving lesson.


text:
Donaldson Creative
CREATIVE STRATEGY ON A WORLD STAGE
DONALDSON CREATIVE WAS FORMERLY TRADING AS UNSPUN CREATIVE AND THE PROJECTS REFERRED TO REFLECT WORK UNDERTAKEN AS SUCH.
We are creative strategists, agitators and alchemists; captivating audiences on intimate and global stages.
To develop creative strategies equal to the greatest ambitions of our clients.
To thrill and transport our many audiences, through unforgettable encounters.
Concept creation
We dream up the big ideas that connect with people; bringing stories to life in unexpected and arresting ways.
We create high-profile ceremonies, events and launches,designing the complete journey of audience experience: from arrival to departure via extraordinary encounters.
We build relevant, compelling campaigns - from narrative development and winning strategy to 360° support and execution.
SARA DONALDSON OBE
"Create an experience to inspire wonder, and in no small way you will have changed the world."
Sara has overseen the delivery of many of the UK's highest profile events and launches. These include projects such as London's successful bid for the 2012 Olympic and Paralympic Games, the Royal Opening of Heathrow's Terminal 5 and Terminal 2, the launch of the new concourse at King's Cross Station, and Camelot's Olympic Handover in the Mall. Her career in staging major events developed from coordinating sporting championships for the British Universities Sports Federation. In 1997 she became National Programme Director for the New Millennium Experience Company and in 2000, was awarded an OBE for her services to the Millennium Celebrations.
Sara helped Singapore win the 2010 Youth Olympic Games; led the strategic direction of England's bid to host the 2018 Football World Cup; supported Glasgow 2014 in developing their bid narrative and co-produced the Glasgow 2014 Commonwealth Games Flag Handover Ceremony in Delhi. Sara was the Chief Operating Officer of the production company for the London 2012 Olympic and Paralympic Opening and Closing Ceremonies. Sara was on the Board of Directors for the Inaugural Invictus Games, and Exec Producer of the Ceremonies for the 2014 and 2106 events. In September 2015 Sara was the Exec Producer for the England 2015 Rugby World Cup Opening Ceremony. In 2017 Sara oversaw key elements of the 2017 Athletics World Championships and in 2019 and 2020 she was Championships Director of the World Skateboarding Championships.
Experience on a global scale
From governments and world leaders to major charities and blue-chip organisations, our clients confidently entrust their most prestigious events to our unrivalled expertise.
Fostering an intimate connection
At personal level and for smaller organisations, we provide individual consultancy and bespoke events. We are proud to be known as facilitators and enablers - seeking always to help people connect.
Pricing tailored to you
Our made-to-measure pricing policy enables us to take on smaller projects that ignite our passions, alongside our large-scale work.
Anything but ordinary, at Donaldson Creative we dare our clients to dream big:
pushing the boundaries of creative possibility to discover uniquely inspiring approaches.
Combining:
penetrating strategy with boundless creativity;
meticulous delivery with bravery of vision
From the moment the right idea is ignited we are unstoppable, driven to excel with total integrity. We don't rest until a creation is extraordinary in every detail; lending the weight of our unrivalled experience and intuition to guide an evolving vision toward all-out success.
A desire to delight infuses our every choice. From micro moments of surprise and joy to historic spectaculars on an epic scale, our ambition is ever to awaken people to the moment; touching each audience with a brilliance all at once captivating, memorable, and very, very effective.
London 2017 World ParaAthletics Championships
Donaldson Creative devised, produced the Medal Plaza delivering 213 Medal Ceremonies for the World ParaAthletics Championships held in the Queen Elizabeth Olympic Park in July 2017.
Located in the Hero Village (sponsor activations), this was the first time the ceremonies were delivered outside of the Stadium to ensure that they were not a distraction to competition taking place.
With a complex schedule, a daily programme of the Hero Village and all its activity was created to ensure Partner activations did not disrupt the medal ceremonies and there was a consistent programme of activity both on and off stage throughout the day to create a fun and energetic atmosphere.
A dedicated and highly experienced team was recruited to address the need for flexibility and speed due to number of ceremonies.
Sentebale Fundraising Concerts
Two unique concerts at Kensington Palace and Hampton Court Palace to raise awareness of the charity's work - supporting orphans and vulnerable children in Lesotho, many of whom are affected by the HIV/AIDS epidemic.
In 2016, exclusive performances took place by Coldplay, Laura Mvula, George the Poet and the Basotho Children's Choir flown in from Lesotho to perform with Joss Stone. In 2019, Rita Ora headlined along with B Positive choir and Lesotho performer Morena Leraba.
Donaldson Creative provided consultancy support to develop multiple hospitality packages within the Palaces and the creation of a funding model in order to secure partners. In the delivery phase, we developed the overarching running order, created visual content including film and graphics and produced all scripts. We worked with the concert promoters to deliver all technical production and site infrastructure. We also provided stage management during the show for all creative elements and speakers including HRH Prince Harry.
Careful management of stakeholders including the Royal Household, headline talent and sponsors was required as well as guidance on guest experience including catering, venue management, entertainment and spectator experience.
Invictus Games Orlando 2016
Following on from our successful involvement in the inaugural Invictus Games in London, we provided significant consultancy and production support for the 2016 Invictus Games that took place in May 2016 at the ESPN Wide World of Sports Complex in Orlando, Florida. The consultancy began with a feasibility study showcasing Invictus best practice.
We were appointed to produce the Opening and Closing Ceremonies, overseeing the complicated sports presentation for 10 disciplines in 5 venues, as well as a producing a number of Symposium events for the George W. Bush Institute taking place ahead of the games. In addition to this support central project management was provided, including the production of critical path documentation and other project management tools including management action registers and status documentation.
The Opening Ceremony took place on the 8th May in front of an audience of 14,000 and a global television audience of many millions.
Click here to view the Opening Ceremony: http://www.continuumlive.com/invictus-games-orlando-2016-opening-ceremony/
RWC 2015 Opening Ceremony
To excite the world and set the tone for the Tournament, our objective was to create an Opening Ceremony that would engage and inspire a live audience of 82,000 at Twickenham Stadium and a broadcast audience in the hundreds of million. The production played a significant role in realising the vision for the Tournament and in presenting England as a proud, warm and welcoming host nation.
This needed to be more than a Ceremony, this needed to be an experience; a moment in time to break new ground via an extraordinarily ambitious production. We placed the grand, the formal and the received traditions of ceremony to one side and got right to the point, the vision, the SCRUM. The Opening Ceremony set the tone for the Tournament; passionate, succinct, bold and bursting with excitement. Most crucially the field of play was cleared in just 10 minutes, allowing teams plenty of warm up time for the Opening Match. The show had a fantastic response from the audience, the media and the client - a truly unforgettable experience.
England 2018 Football World Cup Bid
England's Bid Team asked us to help develop the narrative, messaging and their effective communication for the bid to host the FIFA World Cup in 2018 or 2022 by telling a compelling story for why England should host the World Cup.
We created an engaging, relevant and compelling launch event with both supporting films and brand animation; we orchestrated multiple brand activations including a Back the Bid scarf around Admiral Nelson's neck and the Fly the Flag tour.
We leveraged the impact of celebrity supporters, including Prince William, Prime Minister David Cameron, David Beckham and Wayne Rooney through a 'Back the Bid' campaign which was then opened up to the public.
We were also fundamental in delivering a flawless final presentation in Zurich developing content, co-ordinating logistics and speech coaching.
Although the bid team were unsuccessful in winning the right to host the 2018 Football World Cup, FIFA president Sepp Blatter praised an "excellent and remarkable" presentation from the bid team in Zurich
Salvation Army 150th Anniversary Congress
To celebrate the 150th Anniversary of the founding of The Salvation Army, we partnered with Executive Producer Christopher Laue and Corporate Magic to deliver a 7-day programme of full-scale productions, concerts and exhibitions across four venues at The O2, London.
The production included 7 arena shows honouring their rich-storied heritage while at the same time celebrating their innovative future in front of an audience of 15,000 each time. We provided project management for all aspects including brand activations, creative production, complex logistics, stakeholder management and onsite delivery at an iconic but challenging venue.
Invictus Games London 2014
Sara Donaldson sat on the Board of Directors of the inaugural Invictus Games, responsible for ceremonies. The Opening Ceremony had a live audience of 10,000 and broadcast audience of millions. Sara led the team to manage every aspect from developing the creative concept and a compelling narrative to managing full production, logistics, scripting, onsite delivery, entertainment and military elements whilst liaising with multiple stakeholders including the Royal Household, within a challenging venue and a stringent budget. Highlights included persuading Chris Martin to compose an Invictus anthem, Idris Elba performing a reading and the Foo Fighters playing an exclusive headline set to close the games.
Donaldson Creative was also awarded the contract to oversee Sports Presentation, Opening, Closing and Medal Ceremonies for the 2016 Invictus Games hosted in Orlando in May 2016.
National Lottery 3rd License Bid
We worked on the bid by National Lottery operators Camelot to retain their licence for a third, ten-year term seeing off competition from Indian conglomerate Sugal and Damani. Among the various tender criteria, the rival bidders needed to convey to the independent National Lottery Commission (NLC) that their plans would maximise revenue for Good Causes, with the funding demands of the London 2012 Olympic and Paralympic Games providing the backdrop to the contest. Camelot asked us to manage and produce the initial media launch and the final bid presentation.
The media launch was a headline-grabbing event in London with key content distilled into three films. The official bid presentation took place in a more formal environment behind closed doors. This presented a rare challenge, with no opportunity for on-site support. As such, all scripts, speaker performances and technical aids were developed to a stage where flawless delivery was assured.
The Commission unanimously decided that although it had received two strong bids, Camelot's clear sales and marketing strategies meant they were better placed to generate returns for good causes and were confirmed as the preferred partner.
Between 2002 and 2014, we continued to work with Camelot on a number of projects including their 10th Anniversary celebrations, Lotto Launch and the National Lottery Showcase on the Olympic Park.
A member of the Camelot team called our contribution to the bid as "vital" specifically heralding the "military precision" with which events were planned, finalised and produced.
Heathrow - Launch of T5 & T2
March 2008 & June 2014
To launch the new landmark buildings with purpose and pride required world-class ceremonies. We provided full service event support co-ordinating every aspect, from writing speeches and producing films, to co-ordinating Her Majesty's schedule and ensuring that up to 1,100 VIP guests passed seamlessly through security checks whilst working within a live Terminal environment.
We included creative, bold and unexpected elements alongside the official proceedings to create unique events and also addressed various challenges such as high security levels, strict royal protocol and the buildings themselves.
BBC Sports Personality of the Year 2012
2012 was a historic year for British sport; hot on the heels of the Olympic Games, the BBC's Sports Personality of the Year 2012 Awards was the most hotly anticipated show of its 59 year history. The BBC needed to deliver its biggest and best celebration yet so we created an event that wowed the 16,500 live arena audience and millions of viewers through live transmission.
Our innovative set design created a dynamic canvas that allowed unlimited possibilities for lighting transformation as well as diverse staging possibilities for an exciting show. In addition to the main arena for the Awards ceremony, we were also charged with the design, implementation and management of four high-end hospitality areas and the after-show party featuring a unique musical performance by Bradley Wiggins.
London 2012 Bid & Ceremonies
In order to bring the Olympic and Paralympic Games to London, the bid committee asked us to help them define a brand that would set London apart from the other candidate cities. It was vital to build nationwide and international recognition & support from sporting organisations, government, public and media and communicate the essential messages with consistency, clarity, credibility and passion to compel the IOC to vote for London.
Working both embedded inside the bid team and as consultants, we developed the creative concepts and core narratives from the launch event, through to the development of the bid book, and the delivery of all events including the Evaluation Commission and the final presentation to the IOC.
We helped to define the bid narrative, designed innovative environments, developed content and produced marketing collateral and films. We wrote scripts and trained speakers. We helped to manage and brought on board stakeholders and the general public.
"Your contribution to London's Olympic bid was unrivalled. It was clear from the beginning that you would go to extraordinary lengths to ensure every event was an unqualified success. Whatever the brief, the team were completely dependable. We couldn't have won without you." Marketing Director, London 2012
As a partner in a joint venture company, Flow Productions Ltd and our team established and managed London 2012 Ceremonies Limited - the dedicated production company that delivered the Opening and Closing Ceremonies of the Olympic and Paralympic Games. Sara Donaldson and Christopher Laue served as joint Chief Operating Officer with the remainder of our leadership team serving as Procurement Director, Finance Director and Producer of the Ceremonies.
The creative concept of the Ceremonies brought to life the key themes and priorities of the London 2012 Games – 'Inspiring a generation' - through sport, youth and urban transformation. It was a Ceremony 'for everyone', celebrating the most innovative, revolutionary and iconic examples of Britain's rich history and unique humour. Led by a visionary creative team that included Danny Boyle (Slumdog Millionaire), Kim Gavin (Take That The Circus Tour) and Stephen Daldry (Billy Elliott), these distinctly British ceremonies were embraced by more than 4 billion viewers worldwide in 204 countries, and received widespread critical acclaim.
Glasgow 2014 Handover Ceremony
To bring to life Glasgow 2014 and the city's unique personality, we delivered an 8-minute segment of the Closing Ceremony of the Delhi 2010 Commonwealth Games, working across geographies and cultures with a limited budget. The objective was to deliver a strong and stylish performance; meaningful to the home audience, entertaining to the live audience, and memorable for the global broadcast audience. This was achieved by identifying established symbols that led the audience on a journey through traditional Scotland to modern Glasgow and by placing the proud people of Glasgow at the heart of the performance. This was highly visible in the form of tartan, kilts, bagpipes, the Loch Ness monster and a 350-strong mass choreographed cast from every corner of Scotland, from all backgrounds and all walks of life. We provided the overall creative concept from inception through to delivery as well as creative direction on all elements including choreography, original music, protocol, design, and co-ordinating a team of world-class experts to lead an inexperienced in-house team. We were also responsible for the technical and operational delivery.
Medal Ceremonies
Fundraising Concerts
Orlando 2016
Football World Cup Bid Campaign
150th Anniversary Congress
Launch of T5 & T2
Handover Ceremony
sdonaldson@donaldsoncreative.co.uk
Sara Donaldson


& World BASS Culture Family.
Two Soundsystem in action on the night.
Playing Deep Roots from early and leading into 2016 Steppas styles.
Irieland's pioneers bringing you on a mystic audio visual journey..
Special guest the beautiful and talented FYahRed.


I won the competition for the helluin nickname character Burnstorm and a week after the announcement of the winners, I finally gave my suit. BUT alas, and there was a problem in the news you wrote that the award suit will be with such characteristics as hp 600 pm 600 in fact you issued the most common suit with 200 hp and 200 mp. I again wrote in support of Facebook and I again promised that in a week my suit will receive the characteristics of 600 have already been 3 updates 3 weeks I'm waiting for the correction but my suit still has the characteristics of 200 hp and 200 mp.
Hi DenisBolo. For what contest is this? Can you PM me the following?
I'll try to ask for updates from Support or the Requiem Team.
Edited by VModStrawberry, 17 November 2017 - 01:07 AM.
Edited by VModStrawberry, 17 November 2017 - 01:34 AM.
Omitted account info for security.
Thank you for the info! I removed your account name and character name from your reply, for security (I asked you to PM me the info, hehe) But don't worry, I have the info you provided, and I'll be asking for updates about this. I'll send you a PM or reply here once I get an update.
The Requiem Team informed me that the costume rewards will be replaced with the ones with the correct stats [ 600 HP 600 MP 60 HP Recovery 60 MP Recovery ] during this week's maintenance.


Bowman sales opportunities tx support recent past n'STILLWATER, Okla. (AP) mike Bowman put to 397 meters but two touchdowns tx technological was concluded fixture wholesale nba jerseys a nine dropping talent oklahoma to talk about having wining 41 17 along the No. 15 cowboys forward tuesday night.The payout is going to be third precisely provided a season starting deterioration on the way to Mississippi regarding inflammed Raiders (3 1, 1 0 tremendous 12), as their work for triumph additional okla nation were only available in 2008.Bowman performed 35 relating to 46 travels in her big 12 very first, powering the continent the best criminal offense to 621 total lawns. Demarcus Felton greater a couple of flowing touchdowns on colorado technical or carried out with 121 showrooms holds 12.probably not uncertain for a little daughter male as well as what normally he doing to, nevada technological bus Kliff Kingsbury described. serious. I held it's place in his sandals in addition coached at wholesale jerseys from China an early age fellas, And he just separates your idea loose fitting. red Raiders diagnosed passes during the obtain, having Ja top class at the forefront catches eight along with gardens 79 plus austin tx Antoine Wesley polished Zach each seven.rights mountain hurried 12 things for 111 metres and maybe a landing regarding that cowboys (3 1, 0 1), usually the 10th fixed on the internet game senior is bound to have obtained. Tylan kids jerseys Wallace held seven records in support of 123 gardens.even after leading ok assert to a enlighten Boise the other day, ok state qb Taylor Cornelius effective creating polishing off 18 related with 38 passing towards 258 lawns by a touchdown and thus interception.simple, ok phase fitness instructor paul Gundy talked about. obtained Stephen Curry jersey youth completly taught and moreover played out, And we were looking at stronger in comparison with what folks. setting up, I add really know all other way to install it. win had to be the first from the boys in support of Kingsbury, Who now is 1 5 versus okla state as well as,while Gundy.had gotten connected with selection years, Kingsbury assumed. for those of you dudes to be removed but play the way they achieved, i just now proud in their energy and efforts additionally eventually we will craft separate from everything.of the takeawayof which wasn a 605 back garden passing abilities for Bowman, the wholesale nfl jerseys same as this morning that the texas technological indication owner get the larger 12 freshman checklist in a conquer Houston. a new cowboys don't ever could very well rather get your piece of synchronize within tx computer, on the other hand a favorable plan as opposed to the next three weeks may replica nba jerseys help them come up with amends consisting of voters.
Richard Wong : Great shoe, this is my third pair! they hold up a very long time and are very comfortable. I do prefer the fit of the Helsinki though.


text:
'Polypill could avoid millions of premature deaths, heart attacks and strokes'
A combination of medications to treat high blood pressure and high cholesterol could prevent millions of premature deaths, heart attacks and strokes each year, experts have said.
They suggest global health has paid a deadly price for not using simple, low-cost blood pressure lowering drugs, statins and aspirin widely in the form of a single tablet known as the polypill.
According to two leading cardiologists, widespread availability of the drugs would substantially reduce the risk of cardiovascular disease (CVD) and be affordable for most people around the world.
Professor Fausto Pinto, president of the World Heart Federation, said: 'Despite substantial scientific evidence of the high effectiveness, safety and affordability of the polypill, few such combined products are available, and in the few countries where they are available, use is low .'
They suggest global health has paid a deadly price for not using simple, low-cost blood pressure lowering drugs, statins and aspirin widely in the form of a single tablet known as the polypill
Professor Salim Yusuf, executive director of the Population Health Research Institute and Professor Emeritus of Medicine at McMaster University, said: 'This systemic failure is a global tragedy, as many premature deaths from CVD could be avoided.'
There is currently no polypill licensed in the UK, the UK's Medicines and Healthcare products Regulatory Agency said.
Some 18 million people die from cardiovascular diseases each year, 80 per cent of whom are in low-income and middle-income countries, and three times as many people have non-fatal cardiovascular diseases, it is estimated.
The majority of heart attacks and strokes afflict people who have no prior CVD, meaning primary prevention of the first heart attack or stroke is essential.
While secondary prevention for people who already have CVD is also important, it does not have the same impact, the experts suggest.
Prof Yusuf said: 'The current strategy for primary and secondary prevention of CVD has only been modestly successful in most countries, including high-income countries.
What even IS a polypill?
The polypill is also known as fixed-dose combination therapy.
It is a combination of blood pressure lowering agents, a statin for lowering low-density lipoprotein cholesterol, and low-dose aspirin.
The idea was proposed in the early 2000s as a means to reduce CVD substantially and at low cost.
'Even in these countries, fewer than half of patients with prior CVD, and fewer than 20 per cent without prior CVD, receive effective preventive treatments.'
The polypill, also known as fixed-dose combination therapy – with the combination of blood pressure lowering agents, a statin for lowering low-density lipoprotein cholesterol, and low-dose aspirin – was proposed in the early 2000s as a means to reduce CVD substantially and at low cost.
'The answer is now clear and resounding, with data from three independent, large, and long-term trials in primary prevention and one in secondary prevention showing its life-saving significance,' Prof Yusuf said.
Various publications from him have shown that heart attacks and strokes were cut by 35 per cent to 50 per cent through the use of a polypill.
'It is time to use the polypill widely to save millions of lives each year,' Prof Pinto said.
The researchers say large pharmaceutical companies should invest in developing and testing polypills.
They also say the medication should be included on the World Health Organisation's Essential Medicines List, and that the pill should be combined with lifestyle advice for CVD prevention delivered by experts.
The comment is published in the Lancet journal.
Paul Leeson, professor of cardiovascular medicine at the University of Oxford, said: 'As highlighted by the article, there is now robust trial evidence that shows a polypill approach – in which low doses of several 'heart protective' drugs are combined in a single tablet – can reduce the chance that someone develops heart disease.
'However, the science, which is widely accepted, is not the real focus of the article.
'Scientific trial results, alone, do not change the health of people or populations.
'The real impact only comes if new treatments start to be used.
'This requires those responsible for healthcare delivery in different countries, and the people themselves, deciding they want to use a new treatment.
'This is a much more complex decision process that involves politics, economics, geography and commerce, as well as patient behaviors and choice.
WHAT SHOULD A BALANCED DIET LOOK LIKE?
Meals should be based on potatoes, bread, rice, pasta or other starchy carbohydrates, ideally wholegrain, according to the NHS
• Eat at least 5 servings of a variety of fruit and vegetables every day. All fresh, frozen, dried and canned fruit and vegetables count
• Base meals on potatoes, bread, rice, pasta or other starchy carbohydrates, ideally wholegrain
• 30 grams of fiber a day: This is the same as eating all of the following: 5 servings of fruit and vegetables, 2 whole-wheat cereal biscuits, 2 thick slices of wholemeal bread and large baked potato with the skin on
• Have some dairy or dairy alternatives (such as soya drinks) choosing lower fat and lower sugar options
• Eat some beans, pulses, fish, eggs, meat and other proteins (including 2 portions of fish every week, one of which should be oily)
• Choose unsaturated oils and spreads and consuming in small amounts
• Drink 6-8 cups/glasses of water a day
• Adults should have less than 6g of salt and 20g of saturated fat for women or 30g for men a day
Source: NHS Eatwell Guide
Categories Health Tags dailymail, health
Zimbabwe Tertiary Learning Institution Developing Central Bank Digital Currency – Bitcoin News
5 Reasons Why Sweatcoin's Cryptocurrency Launch Will Revolutionize Fitness


Antique 1901 book written in Norwegian, The Life and Distinguished Services of Our Martyred President William McKinley, by Murat Halstead. The book is filled with illustrations and photos. Many of the pages have been written on by previous owners. The book measures 9 3/4" x 7 1/4". Enjoy!