text:
\section{Introduction}
\label{sec:introduction}

\IEEEPARstart{C}{onvolutional} neural networks (CNNs)
are being used in several visual analysis tasks such as image recognition \cite{tan2019efficientnet, howard2017mobilenets, Sandler2018},
object detection \cite{liu2016ssd, Xiong_2021_CVPR}, and segmentation \cite{He_2017_ICCV}.
In CNNs, computation grows proportional to the input size, and it eventually leads to higher latency.
Although reducing the number of parameters and precision of variables help in improving the latency,
they often come at the cost of lower accuracy \cite{howard2017mobilenets,tan2019mnasnet}.
Moreover, the complexity of deep learning models and their applications are only to grow over foreseeable future, and hence, overall they require even more resources.

Model distribution across multiple machines is one viable solution that has consequently attracted growing attention especially when considering the fact that there are typically quite a number of local compute nodes around us that can help get the jobs done.

\begin{figure}
  \centering
\subfloat[\footnotesize Model Parallelism\label{fig:mp_architecture}]
      \includegraphics[width=0.25\textwidth]{mp_architecture}}
	~
\subfloat[\footnotesize Class Parallelism\label{fig:cp_architecture}]
      \includegraphics[width=0.25\textwidth]{cp_architecture}}

	\\
\subfloat[\footnotesize Variant Parallelism\label{fig:vp_architecture}]
      \includegraphics[width=0.35\textwidth]{vp_architecture}}
\caption{Comparison of different model distribution methods on atomic input data.}
\label{fig:mp_cp_cp_aio}
\end{figure}

For atomic data that cannot be split into further pieces \cite{MLSYS2021_sensai}, two major solutions are often used to decompose a given model and distribute it among different compute resources; namely, model parallelism (\modelp{}) and class parallelism (\classp{}).
\modelp{}  decomposes a model into multiple sequential or parallel slices, each sent over to a separate compute node for processing, then lots of intermediate data values are returned to the master to combine and complete the model.
Slicing in \modelp{}  can be inter- or intra-layer (Fig.~\ref{fig:mp_architecture}).
\classp{}  decomposes the model into multiple disjoint models, each aiming to predict only one or a few non-overlapping classes (Fig.~\ref{fig:cp_architecture});
thus, contrary to \modelp{}, \classp{}  runs a monolithic model on each computing node.
\classp{}  improves efficiency of each model mostly by applying class-aware pruning policies \cite{MLSYS2021_sensai}.

Both solutions have their own limitations. In \modelp{}, intermediate tensors have to be communicated between different nodes.
In current CNNs, and of course depending on the available bandwidth,
the time needed to transmit intermediate tensors can be orders of magnitude higher than inference time itself.
Hence, \modelp{}  would drastically increase end-to-end latency due to link saturation.
\classp{}  does not suffer from this issue since it only returns a few numbers from each computing node;
these numbers represent the prediction of classes allocated to the model executed on that computing node.
Verbatim \classp{}  has a firm restriction on scalability that is limited to the number of classes in the application \cite{MLSYS2021_sensai}, which can debatably be extended by choosing overlapping subsets of classes instead.
However more importantly, both paradigms intrinsically suffer from multiple instances of single point of failure (SPOF) issue because every compute node has to complete its job and send the result back to the requesting node before the top-level classification task can be fully finished.
Otherwise, the master node cannot generate the final network prediction.
Therefore, failure in a single node compromises availability of the whole system.
The scenario of interest in this paper is where we use the free times of IoT devices to do a distributed inference job.
Failure is a common problem in distributed systems, especially in this scenario of cycle-stealing from nearby IoT devices, where worker nodes only contribute in idle times and at any times may preempt the inference task to process their main jobs \cite{isard2009quincy}.
Also for instance, some nodes may seriously slow down due to various unexpected reasons \cite{dean2013tail}. Additionally, communication issues such as network congestion, lead to missed deadlines in real-time applications.
These concerns motivate the need for a fault tolerant mechanism for distributed inference.
Obviously, in both \modelp{}  and \classp{}, higher availability may be achieved by introducing redundant nodes, but of course this comes at the expense of large impact on communication and computation cost.

In this paper, we propose a different distribution technique and we call it Variant Parallelism (\variantp{}).
Both \modelp{}  and \classp{}  can be viewed as applying a ``top-down" model slicing approach: they try to break a baseline model into pieces, although differently in \modelp{}  vs \classp{}, and distribute each piece across multiple nodes.
In contrast, we in \variantp{} employ a ``bottom-up" approach:  we generate multiple lightweight variants (described below) from a chosen basic architecture
  and then combine their results in an aggregator component (Fig.~\ref{fig:vp_architecture}).
In fact, \classp{}  and \variantp{} are both special types of ensemble learning models, but with different intents and approaches.

In \variantp{}, each variant is an independent model that can be deployed on a different machine or accelerator. Unlike \classp{}, our variants have full-class prediction heads, i.e., each worker node predicts probabilities for all classes, and this is the key in achieving a fault-tolerant architecture.
Consequently, in a system with $n$ worker nodes, \variantp{} can still get a full response even if $n\mhyphen 1$ nodes fail, while having the variants helps to improve accuracy over na\"ive replication when more than one worker node is working.

\variantp{}'s variants can differ in input resolution and input offset.
To achieve faster inference time, we apply three network optimizations: narrowing network width, reducing complexity of the compute intensive layers, and replacing with faster operations when possible.

Since each machine generates full-class prediction, the data transmission amount would be higher than \classp{}, resulting in risking a higher response time.
To avoid this, we put a compression/decompression module. The compression layer attaches to each worker's classification head.
In the compression part we select $k$ classes with the highest confidence score and send them to the master together with their corresponding indices.
In \variantp{}, each machine executes a lightweight classification model, and sends its compressed result to the master. The master machine then applies a decompression step followed by a score scaling to weight each prediction based on its corresponding model capacity.
It eventually combines results via an aggregation module.
There is no trainable parameter in the combination modules. Otherwise, it reduces system's flexibility and can compromise availability.
For evaluation, we consider a smart home scenario and demonstrate \variantp{}'s robustness on six object recognition datasets.

Our major contributions can be summarized as follows:
\begin{itemize}
\item We introduce a bottom-up model distribution called Variant Parallelism which contrary to \modelp{}  and \classp{}  is fault-tolerant, flexible, and therefore, can be freely distributed across several sporadically available compute nodes such as the case of IoT devices or across multiple accelerators within a single machine.
\item Our technique generates multiple variants each having different number of parameters, multiply-accumulate operations (MACs), latency, and model size that can be deployed on different machines based on their computing capacities.
\item We propose an aggregation method that is resilient to failures, while matching or exceeding the accuracy of the baseline model.
\item We propose a fast and simple technique to reduce bandwidth usage. It can compress the output size of each node by up to $10\times$ while losing less than $0.5\%$ accuracy.
\item Our proposed method can have $5.8\mhyphen 7.1\times$ fewer parameters, ${4.3\mhyphen 31\times}$ fewer MACs, and ${2.5\mhyphen 13.2\times}$ lower response time on an atomic input compared to the baseline while achieving comparable or higher accuracy.
\end{itemize}

\section{Related Work}
\label{sec:related_work}

\textbf{Model Parallelism (\modelp{}):} In model parallelism (Fig.~\ref{fig:mp_architecture}) a deep learning model is sliced into multiple sub-models \cite{dean2012large,huang2019gpipe,DBLP:conf/mlsys/YuC20}. Each of them can be deployed and concurrently executed on parallel machines. These sub-models have to transfer a huge amount of data from intermediate tensors. The time to transfer these tensors can be several orders of magnitude higher than the time to compute operations of the neural network itself.

\textbf{Class Parallelism (\classp{}):} As its name suggests, class parallelism (Fig.~\ref{fig:cp_architecture}) decouples a basic deep learning model into multiple independent sub-models, each can predict one or a few non-overlapping classes. Structured pruning is then applied on each sub-model to reduce their latency. The ultimate result can be prepared by combining the prediction of all of these sub-models \cite{MLSYS2021_sensai,yang2020robust}.
\classp{}  does not have the issue mentioned for \modelp{}, and can improve latency compared to a single model. However, it has a hard limitation on the number of sub-models. More importantly, since both \modelp{}  and \classp{}  require results of all machines, they suffer from mutiple SPOFs.

\textbf{Data Parallelism (\emph{DP}):}
There is also another parallelism paradigm called data parallelism, in which all machines run the same model, but the data that they observe is different \cite{MLSYS2019_c74d97b0,DBLP:conf/mlsys/WangVPTDS20}. For example, each may process a different part of a single image. However, for atomic live data which is impossible to be further split, \emph{DP} cannot be leveraged.

Variant parallelism (\variantp{}) aims to work on atomic live data and by design can tolerate fault in up to $n\mhyphen 1$ nodes. \variantp{} gives us the flexibility to generate enough variants based on different objectives including latency constraints, model size, and compute resources. \variantp{} can be easily combined with other parallelism schemes.

\section{Variant Parallelism}
\label{sec:variant_parallelism}

\begin{figure}[!t]
    \centering
    \includegraphics[width=0.8\columnwidth]{workflow}
	\vskip -0.15in

    \caption{Overview of our proposed workflow including offline parts (variant generation and model training) and online actions (dispatcher and aggregator).}
    \label{fig:workflow}

\end{figure}

In \variantp{}, we consider a basic model, and generate multiple lightweight and fast models based on it.
Each of these variants can have different storage sizes, parameters, inference times, and accuracies.
We add inference-time augmentation. It leads to feeding slightly different inputs for each model.
Each variant can be treated as an independent full-class predictor.
Every machine can concurrently execute one or more of these variants in-parallel with other machines.
Depending on the use-case, these machines may be different GPUs on a single node or different connected edge devices, which is the case of interest in this paper.
If there is an idle compute node with limited storage space or computation constraints, \variantp{} can generate a lighter variant which provides the opportunity to utilize the available compute capacity.
\variantp{} aims to reduce response time to enable faster decision making via a distributed deep learning architecture while maintaining availability in presence of failure.
Here, we first concisely overview \variantp{}'s workflow. We then explain our design decisions, and how each module works.

\subsection{Overview}
\label{sec:vp,sebsec:overview}

Our workflow starts with one basic architecture, but is straightforward to extend to more than one model.
As shown in Fig.~\ref{fig:workflow}, we generate different optimized variants from the base model(s).
We then add classification head, compression, decompression, and scaling modules to each variant.
Variants are retrained on the desired dataset.
These variants are independent and can be separately deployed on different machines for distributed in-parallel execution.
Every node has its own model. Once deployed, the master node multicasts live data to all contributing workers, and waits for a predefined period (proportional to the desired deadline).
The workers feed the data into their models and send the compressed result back to the master node.
The master then decompresses, scales and combines them through the aggregation module to prepare the final prediction (Fig.~\ref{fig:system_overview}).

\subsection{Reducing Model Complexity}
\label{sec:vp,subsec:optimization}

\begin{table}[b]
	\vskip -0.2in
	\caption{Comparison of different models' characteristics}
	\vskip -0.2in
	\label{table:models_characteristics}
	\begin{center} \begin{small} \begin{tabular}{lccc}
		\hline
		Model				& Input Size		& \#Params	& \#MACs	\\
		\hline
		MobileNetV2			& $3\times224^2$	& 2.27M		& 300M		\\
		$V_5$				& $3\times224^2$	& 370K		& 54M		\\
		$V_4$				& $3\times192^2$	& 360K		& 40M		\\
		$V_3$				& $3\times160^2$	& 340K		& 27M		\\
		$V_2$				& $3\times128^2$	& 330K		& 17M		\\
		$V_1$				& $3\times96^2$		& 320K		& 10M		\\
		\hline
		\end{tabular} \end{small} \end{center}

\end{table}

For variant generation, we reduce complexity of the basic architecture through the following three steps: (1) reducing width factor, (2) reducing complexity of the last few layers, and (3) replacement with faster operations when possible.
These steps help us to improve per machine model size, computation requirements, and latency.
Firstly, we reduce width factor of depth-wise separable convolution layers (was first introduced in \cite{howard2017mobilenets}) by $2.85\times$.
It drastically improves network runtime.
We recover the lost accuracy by combining multiple parallel variants.
Secondly, we reduce complexity of the last few layers of each variant based on its input size.
The intuition is that in the current efficient CNNs, the last few layers take considerable portion of the computation time.
However, it should be proportional to the input resolution and the achievable accuracy.
As shown in Table~\ref{table:models_architectures}, each model uses convolutional layers with the number of filters ($f$) proportional to its input size ($\rho_i$).
For example, input size of $V_3$ ($i=3$) is $160$ and its last convolution layer has $384$ output filters while $V_1$ ($i=1$) with input size of $96$ has only $256$ output filters.
We empirically select the number of these output filters so that they are divisible by $64$ for better performance.
Lastly, we employ faster operations when possible, e.g., replacing $FullyConnected$, $AveragePooling$, and $Average$ layers by $1\times1 Convolution$, $MaxPooling$, and $Add$ layers, respectively.
We generate five different models based on these optimizations.
Their characteristics are illustrated in Table~\ref{table:models_characteristics}.

\begin{table}[b]
	\vskip -0.2in
	\caption{Variants model architecture.}
	\vskip -.15in
	\label{table:models_architectures}
	\begin{center} \begin{small} \begin{tabular}{p{2.5cm}llccc}

    \hline
		Input									& Operator			& $t$	& $f$				& $n$	\\
		\hline
		$3\times\rho_i^2$							& conv2d			& -		& 32				& 1		\\
		$32\times\rho_i^2/4$				& bottleneck		& 1		& 16				& 1		\\
		$16\times\rho_i^2/4$				& bottleneck		& 6		& 24				& 2		\\
		$24\times\rho_i^2/8$				& bottleneck		& 6		& 32				& 3		\\
		$32\times\rho_i^2/16$				& bottleneck		& 6		& 64				& 4		\\
		$64\times\rho_i^2/32$				& bottleneck		& 6		& 96				& 3		\\
		$96\times\rho_i^2/32$				& bottleneck		& 6		& 160				& 3		\\
		$160\times\rho_i^2/64$			& bottleneck		& 6		& 320				& 1		\\
		$320\times\rho_i^2/64$			& conv2d $1\times1$	& -		& ${64\times(3+i)}$	& 1		\\
		$64\times(3+i)\times\rho_i^2/64$	& globalmaxpool		& -		& -					& 1		\\
		$64\times(3+i)$			& conv2d $1\times1$	& -		& $C$				& -		\\
		$C$									& compression 		& -		& $k$				& -		\\
    \hline

		\end{tabular} \end{small} \end{center}
    {\raggedright \footnotesize Similar to the original MobileNetV2, we apply a convolution layer on input followed by $19$ depth-wise convolutional residual bottlenecks. Each row is sequentially applied $n$ times, and $t$ is the expansion factor of intermediate layers in bottleneck blocks. $C$ is the number of classes. \par}

\end{table}

\begin{figure}[!t]
    \centering
    \includegraphics[width=\columnwidth]{variantP}
	\vskip -0.15in
    \caption{The flow of data and the online operations on them, for the inference task.}
    \label{fig:system_overview}

\end{figure}

\subsection{Compression and Decompression}
\label{sec:vp,subsec:compress_decompress}

In \classp{}, for tasks with only few classes (e.g., CIFAR-10), each machine returns only one floating point, and for tasks with more classes (e.g., CIFAR-100), 10 to 20 floating point values are returned by each.
In contrast, our variants are independent models that can return full-class prediction. This feature increases the system reliability in a faulty environment, but raises a challenge in communication time.
To address the mentioned issue, we leverage the $Top\mhyphen k$ idea in which from a prediction vector $\psi_i$, we select $k$ elements having the highest confidence scores and pack them along with their corresponding indices.
Fig.~\ref{fig:comp_decomp_scaling} depicts a toy example with 6 classes, assuming $k=2$.
The compression step shrinks the vector size to two vectors of size $k$: One stores floating point predictions with the highest score ($0.51$ and $0.4$), and the second keeps their integer indices ($1$ and $3$).
After receiving compressed vectors, the master reconstructs a tensor of shape ($batch\_size, V, C$), where $V$ and $C$ are the number of total contributing machines and classes, respectively.
It then scatters values of compressed vectors based on their indices. This procedure is fast yet robust.
The intuition behind is that in each vector except a few elements with high confidence scores, others are close to zero.
More importantly, our aggregation method (\S~\ref{sec:vp,subsec:aggregation}) amplifies both actual information and noise at the same time.
Hence, by removing values with lower confidence score, we actually mitigate noisy information to some degrees, and consequently achieve higher accuracy.
The $Top\mhyphen k$ technique also gives us the flexibility to set $k$ as a hyperparameter, trading accuracy for transmission time.

\subsection{Scaling and Aggregation}
\label{sec:vp,subsec:aggregation}

Master is responsible for balancing the prediction vectors of different variants.
After decompression step, we apply a $Softmax$ operation to transform the reconstructed prediction vectors into a probability distribution (Fig.~\ref{fig:comp_decomp_scaling}).
We then need to scale each prediction vector based on its network capacity.
For example, if variant $V_i$ has more parameters, and thereby returns more accurate predictions than variant $V_j$, then it deserves to have a higher weight in the aggregated result.
We use a proxy of the network architecture itself to scale values of every prediction vector via eq.~\ref{scaling_eq}.

\begin{equation}
\label{scaling_eq}
\psi_i = (\frac{\rho_i}{\max(P)})(\frac{d_i}{\min(D)})^\alpha.\psi_i
\end{equation}

\begin{equation}
where ~~ \rho_i \in P ~~, and ~~ d_i \in D
\end{equation}

Where $\psi_i$ is the prediction vector of variant $V_i$ with input shape $\rho_i$ and depth multiplier $d_i$, and $\alpha$ is a parameter to equalize the impact of input shapes and depth multiplier.

In aggregation module we combine prediction results that are received from compute nodes including the master itself.
We add the vectors together and apply a $Softmax$ afterwards.
Using FullyConnected ($fc$) layers instead, although might improve accuracy, degrades flexibility.
Also, with the same amount of neurons as the number of classes $C$, and $V$ variants contributing in the aggregation step,
the computation cost of using only a single $fc$ layer would be $O(CV)$ which is $O(V)$ times higher.

\begin{figure}[t!]
	\centering
	\includegraphics[width=0.8\columnwidth]{output_compression_decompression}
	\vskip -0.15in
	\caption{An example of compression, decompression and scaling steps assuming $C=6$ and $k~(in\:Top\mhyphen k)=2$.}
	\label{fig:comp_decomp_scaling}

\end{figure}

\section{Experiments}
\label{sec:experiments}

\subsection{Experimental Setup}
\label{sec:experiments,subsec:setup}

\noindent\textbf{Compute Platform:}
We use an in-house server (Table~\ref{table:server_spec}), to design, train and evaluate our models.
We selected the commonly reported vision datasets that are trainable on our server in a reasonable time.

\begin{table}[!b]
	\vskip -0.2in
	\caption{Server specification}
	\vskip -0.2in
	\label{table:server_spec}
	\begin{center} \begin{small} \begin{tabular}{ll}
		\hline
		CPU		& Intel Xeon E5-2630-v3 (x86\_64)				 	\\
				& Frequency: 2.4GHz (1.2-3.2GHz)				   	\\

		GPU 	& NVidia GeForce GTX 1080 Ti					 	\\
			   	& Total Memory: 11GB							   	\\

		Memory	& 32GB (1600MHz)								   	\\

		\hline
		\end{tabular} \end{small} \end{center}

\end{table}

\noindent\textbf{Dataset:}
We evaluated our proposed method on six different vision datasets (Table~\ref{table:datasets}), namely CIFAR-10 \cite{cifar_paper}, CIFAR-100 \cite{cifar_paper}, Fashion MNIST (F-MNIST) \cite{xiao2017fashion}, MNIST \cite{lecun2010mnist}, SVHN \cite{netzer2011reading}, and Food-101 \cite{bossard2014food}.

\begin{table}[b!]
	\vskip -0.2in
	\caption{Datasets used for evaluation}
	\vskip -0.2in
	\label{table:datasets}
	\begin{center} \begin{small} \begin{tabular}{p{1.5cm}p{1.2cm}p{1.5cm}cr}
		\hline
		Dataset     & $C$ & Input Size 	  & \#Trains & \#Tests \\
		\hline
		Food-101    & 101		& $3\times256^2 $ & 75,750 	 & 25,250  \\
		SVHN		& 10		& $3\times32^2 $  & 73,257	 & 26,032  \\
		CIFAR-10	& 10		& $3\times32^2 $  & 50,000	 & 10,000  \\
		CIFAR-100	& 100		& $3\times32^2 $  & 50,000	 & 10,000  \\
		MNIST		& 10		& $28^2 $  & 60,000   & 10,000  \\
		F-MNIST		& 10		& $28^2 $  & 60,000	 & 10,000  \\
		\hline
		\end{tabular} \end{small} \end{center}

\end{table}

\noindent\textbf{Edge Device:}
For latency evaluation of different models, we use a mid-range tablet (Table~\ref{table:edge_spec}), and leverage Google TFLite native benchmark tool \cite{tflitebenchmark} to profile the performance.

\begin{table}[hb]
	\vskip -0.2in
	\caption{Edge device specification}
	\vskip -0.2in
	\label{table:edge_spec}
	\begin{center} \begin{small} \begin{tabular}{p{1.2cm}p{6cm}}
		\hline
		\multicolumn{2}{c}{\textbf{Samsung Galaxy Tab A 8.0 with S Pen (2019, SM-P205)}} \\
		\hline
		CPU		& Octa-core big.LITTLE (2x1.8 GHz Cortex-A73 and 6x1.6 GHz Cortex-A53) 	\\

		GPU 	& ARM Mali-G71 MP2								   	\\
		Memory	& 3GB LPDDR4X									   	\\

		Network	& 2G/3G/4G(LTE) \& Wi-Fi 802.11						\\
		\hline
		\end{tabular} \end{small} \end{center}

\end{table}
\noindent\textbf{Baseline Model:}
We consider MobileNetV2 \cite{Sandler2018} as our basic architecture (Table~\ref{table:models_characteristics}) and compare our results with it.

\subsection{Training} \label{sec:experiments,subsec:training}

We trained all models for $135$ epochs which is reportedly $32.5\%$ fewer than the number of epochs required in \classp{}  ($200$ epochs \cite{MLSYS2021_sensai}).
We port pre-trained weights from ImageNet-1k \cite{Russakovsky2015} whenever available.
Our training procedure has two steps. For the first 30 epochs, all layers except the last two convolutions were frozen. We used batch size of 128 in this step, and 64 (32 in Food-101) for the rest of 105 epochs. We used Adam optimizer with $\beta_1=0.9$ and $\beta_2=0.999$. Learning rate decays by a factor of $0.98$ once the accuracy stagnates.
Since our tasks are classification, and each model can independently predict all of the classes, we apply a separate categorical cross entropy for each model.
Contrary to \classp{}, our variants can observe the same data during training.
Therefore, we train multiple models along with the combination components all together.
The number of models that can be trained simultaneously is a parameter that can be set based on the computation capacity.
Having more models requires more memory, but improves GPU utilization as most parts of our data pipeline executes once.
We apply random horizontal flip, random rotation, and MixUp augmentations.

\begin{table*}[]
	\vskip -0.2in
	\caption{Accuracy (\%) of single models on different datasets}
	\vskip -0.2in
	\label{table:single_models_acc}
	\begin{center} \begin{small} \begin{tabular}{p{1.7cm}lcccccc}
		\hline
		Dataset		& Metric	& MobileNetV2	& Ours($V_1$)	& Ours($V_2$)	& Ours($V_3$)	& Ours($V_4$)	& Ours($V_5$)	\\
		\hline
		CIFAR-10	& Top-1		& 95.16			& 91.76		& 93.07		& 93.81		& 94.43		& 94.62		\\
					& Top-5		& 98.57			& 97.24		& 97.87		& 98.20		& 98.26		& 98.29		\\
		\hline
		MNIST		& Top-1		& 99.37			& 99.38		& 99.41		& 99.41		& 99.42		& 99.46		\\
					& Top-2		& 99.87			& 99.92		& 99.95		& 99.89		& 99.92		& 99.95		\\
		\hline
		CIFAR-100	& Top-1		& 80.51			& 69.62		& 73.57		& 74.46		& 74.67		& 75.20		\\
					& Top-5		& 94.9			& 91.17		& 93.36		& 93.47		& 93.41		& 93.67		\\
		\hline
		SVHN		& Top-1		& 95.52			& 93.81		& 95.16		& 95.34		& 95.89		& 95.91		\\
					& Top-2		& 98.37			& 97.47		& 98.21		& 98.25		& 98.46		& 98.32		\\
		\hline
		\multirow{2}{1.7cm}{Fashion-MNIST}
					& Top-1		& 94.41			& 93.38		& 94.16		& 94.60		& 94.67		& 94.55		\\
					& Top-2		& 98.82			& 98.30		& 98.51		& 98.76		& 98.84		& 98.82		\\
		\hline
		Food-101	& Top-1		& 79.68			& 58.78		& 65.35		& 70.13		& 71.43		& 74.23		\\
					& Top-5		& 93.5			& 83.50		& 87.70		& 90.19		& 90.78		& 92.43		\\
		\hline
		\end{tabular} \end{small} \end{center}

\end{table*}

\subsection{Accuracy of Single Models}	\label{sec:experiments,subsec:accuracy}

First, we evaluate accuracy of each variant and compare it to the accuracy of our baseline.
To have a fair comparison, we trained the baseline with the same training policy, and made sure it has the same or higher accuracy compared with original reports, if applicable.
For each dataset, we report the accuracy of five different models. We also report $top\mhyphen 2$ and $top\mhyphen 5$ in aggregator for datasets with ${C<100}$ and ${C>=100}$, respectively. Here, by $top\mhyphen m$ we mean the ground-truth label is among top $m$ class predictions with the highest confidence scores.
As depicted in Table~\ref{table:single_models_acc}, as we go from the variant $V_1$ to $V_5$, the accuracy improves. For the datasets with few number of classes, the accuracy of $V_5$ is close to the baseline. Nevertheless, our $V_5$ has ${\sim}6\times$ fewer parameters and MACs.

\subsection{Impact of Compression Method}
\label{sec:experiments,subsec:compression}

In this section, we analyze the effect of our compression algorithm on accuracy and the required bandwidth.
We feed the same input data to all variants, and evaluate the impact of different values for $k$ in our compression module.

\label{sec:experiments,subsec:aggregate_acc}

\noindent\textbf{Bandwidth Saving:}
Output size is a linear function of $k$.

\begin{equation}
	\label{k_outputsize}
	h(k) =
	\begin{cases}
		\min(k\times[\text{size}(fp)+1], \text{size}(fp) \times C) & k > 1 \\
		1 & k = 1
	\end{cases}
\end{equation}

Where $h(k)$ is the output size in byte, $\text{size}(fp)$ is the number of bytes required for storing a floating point value and $C$ is the number of classes.
Among different $k$ values, $k=1$ is a special case. Because the compressor selects only one element, we can further reduce communication cost, sending only an integer representing its index.
It can be found that choosing $k>\frac{C}{2}$ is not logical as transmitting the whole vector actually costs less than compressing it.

\label{sec:experiments,subsec:aggregate_acc}
\begin{figure*}[t!]
    \centering
    \subfloat[\footnotesize CIFAR-10 aggregator Top-1\label{fig:cifar10_top1}]
      \includegraphics[width=0.25\textwidth]{k_experiment_cifar10_top1}}
    ~
    \subfloat[\footnotesize CIFAR-100 aggregator Top-1\label{fig:cifar100_top1}]
      \includegraphics[width=0.25\textwidth]{k_experiment_cifar100_top1}}
    ~
	\subfloat[\footnotesize Food-101 aggregator Top-1\label{fig:food101_top1}]
    \includegraphics[width=0.25\textwidth]{k_experiment_food101_top1}}
	\\
	\subfloat[\footnotesize CIFAR-10 aggregator Top-2\label{fig:cifar10_top2}]
	\includegraphics[width=0.25\textwidth]{k_experiment_cifar10_top2}}
    ~
	\subfloat[\footnotesize CIFAR-100 aggregator Top-5\label{fig:cifar100_top5}]
    \includegraphics[width=0.25\textwidth]{k_experiment_cifar100_top5}}
	~
	\subfloat[\footnotesize Food-101 aggregator Top-5\label{fig:food101_top5}]
    \includegraphics[width=0.25\textwidth]{k_experiment_food101_top5}}

    \caption{Impact of different $k$ values on accuracy of different benchmarks. $k$: the number of top predictions returned by each node.}
    \label{fig:k_experiment}
    \vskip -0.15in
\end{figure*}

\noindent\textbf{Ensemble Accuracy:}
In this experiment, we evaluate how different values of $k$ can impact the final accuracy.
Fig.~\ref{fig:k_experiment} illustrates a significant jump in $top\mhyphen 1$ accuracy from $k=1$ to $k=2$. For $k>2$, the accuracy does not significantly increase.
Another interesting observation is on $top\mhyphen m$ accuracy: sending the prediction with the highest confidence score per variant improves $top\mhyphen m$ accuracy.
The reason is that when aggregating multiple prediction vectors, classes with lower scores behave like noise. By omitting these elements from our prediction vectors, we indeed achieve higher $top\mhyphen m$ accuracy.
It appears to be a trade-off between higher $top\mhyphen 1$ and $top\mhyphen m$ accuracies.
Apparently, $k=2$ is a good choice and we use it for the rest of our experiments.

\subsection{Scaling Accuracy}	\label{sec:experiments,subsec:model_counts}

We generated five variants each having unique characteristics.
Here, we analyze the impact of combining different variants on the ensemble accuracy (Fig.~\ref{fig:scaling_experiment}).
We show results on CIFAR-10, CIFAR-100, and Food-101. The other three datasets behave similarly to CIFAR-10.
Although our variants are independent and can be executed in-parallel on separate machines, for simpler comparison,
we show the final accuracy based on the \textit{combined} MACs.
We also added results from the combination of the variants $V_1$ through $V_5$ while using ${k=5}$ (in the compression module) and without compression as well.
It is worth noting that even the combined number of parameters and MACs of our models are fewer than the baseline.

On CIFAR-10, we almost reach the same accuracy as the baseline by combining $V_1$, $V_2$, and $V_3$. Addition of $V_4$ and $V_5$ improves the accuracy incrementally and enables us to achieve even better results than the baseline.
On CIFAR-100, comparable accuracy can be achieved by aggregating the results of all five variants.
The most MACs is required by $V_5$ which is ${\sim}6\times$ fewer than the original MobileNetV2.
The Food-101 dataset is rather more complex as the size of its original images are two orders of magnitude bigger than the other benchmarks.
To achieve comparable results, we generated $V_6$ and $V_7$ (Table~\ref{table:v6_v7_characteristics}) with $448$ and $512$ filters in their last convolution layers, respectively.
We reach similar accuracy by combining the first \textbf{six} variants. Inclusion of $V_7$ further improves the result, however, at the cost of more computation.
In all benchmarks, increasing $k$ in the compression step or transmitting the predictions without any compression gives better results but the differences are not significant.
Intrestingly, by increasing the number of variants to ${>=}6$ in the Food-101, this gap disappears.
This can be due to the increase in the number of variants, particularly one or two more accurate ones.
\subsection{Effect of Every Variant on Accuracy}	\label{sec:experiments,subsec:model_counts}

\begin{table}[b]
	\vskip -0.2in
	\caption{Characteristics of additional variants for Food-101.}
	\vskip -0.2in
	\label{table:v6_v7_characteristics}
	\begin{center} \begin{small} \begin{tabular}{lccc}
		\hline
		Model	& Input Size		& \#Params	& \#MACs	\\
		\hline
		$V_6$	& $3\times256^2$	& 390k		& 71M		\\
		$V_7$	& $3\times320^2$	& 410K		& 112M		\\
		\hline
		\end{tabular} \end{small} \end{center}

\end{table}

To find out whether each variant actually impacts the final accuracy, we evaluate the predictions returned by the combination of our variants while excluding only one of them.
As shown in Fig.~\ref{fig:scaling_exclude_experiment}, in all datasets, omitting each variant has a negative impact on the final accuracy.
This is especially interesting in less accurate variants. For example, on CIFAR-100, $V_1$ has ${\sim}70\%$ accuracy itself which is more than $10\%$ lower than the baseline, yet when included in the aggregation, it can improve the final accuracy.
Same is true in the other benchmarks.

\begin{figure}[t]
	\centering
	\begin{minipage}{0.49\columnwidth}
	  \subfloat[\footnotesize CIFAR-10\label{fig:scaling-cifar10}]

		\includegraphics[width=\columnwidth]{scaling-cifar10-macs}}
	  \\
	  \subfloat[\footnotesize CIFAR-100\label{fig:scaling-cifar100}]

	  \includegraphics[width=\columnwidth]{scaling-cifar100-macs}}
	  \\
	  \subfloat[\footnotesize Food-101\label{fig:scaling-food101}]

	  \includegraphics[width=\columnwidth]{scaling-food101-macs}}
		\caption{Impact of different variants on the final accuracy. (Please zoom in)}
		\label{fig:scaling_experiment}
	\end{minipage}\hfill
	\begin{minipage}{0.49\columnwidth}
		\centering
		\subfloat[\footnotesize CIFAR-10\label{fig:scaling-cifar10}]

		  \includegraphics[width=\columnwidth]{scaling-excluding-cifar10-macs}}
		\\
		\subfloat[\footnotesize CIFAR-100\label{fig:scaling-cifar100}]

		\includegraphics[width=\columnwidth]{scaling-excluding-cifar100-macs}}
		\\
		\subfloat[\footnotesize Food-101\label{fig:scaling-food101}]

		\includegraphics[width=\columnwidth]{scaling-excluding-food101-macs}}
		\caption{How omitting a variant affects the final accuracy. (Please zoom in)}
		\label{fig:scaling_exclude_experiment}
	\end{minipage}
	\vskip -0.15in
\end{figure}

\subsection{Availability in Presence of Failure}	\label{sec:experiments,subsec:availability}

One serious drawback of utilizing idle nodes is preemption in presence of higher priority jobs (e.g., their main tasks). Missing real-time deadlines due to unexpected communication or computation issues are other reasons that indicate the demand for a fault-tolerant parallelism approach. Nevertheless, \modelp{}  and \classp{}  need to get the results of all contributing machines to provide the final outcome.
By design, our architecture is fault-tolerant even if $n\mhyphen 1$ worker nodes are not able to return their predictions.
Our aggregation module also supports this behavior.
As shown in Fig.~\ref{fig:scaling_experiment}, we can still have results by contribution of a fraction of variants being executed on different machines. As the number of faulty nodes increases (i.e., fewer contributing variants), the final accuracy decreases. The achievable accuracy depends on the variants remained for the aggregation. In the worst case, all nodes fail except the one that executes $V_1$ as it has the lowest accuracy among all, which is still better than \modelp{}/\classp{}  that fail to provide any predicton. Our \variantp{} can be viewed as providing a graceful degradation in presence of increasing levels of faults/preemtions in the environment.

\subsection{Other Performance Metrics: Inference Time, MACs, Model Size}
\label{sec:experiments,subsec:other_perfs}

We report the performance of our variants on the other common metrics. We convert each variant designed for and trained on Food-101 to TFLite \cite{tflite}, and execute them for $100$ iterations. The results are illustrated in Table~\ref{table:perf_metrics}.
Since each variant has different characteristics, the slowest one determines the eventual response time. Therefore, depending on the available combination of nodes, one can get $2.5\mhyphen 13.2\times$ speedup compared to the baseline.
Considering a smart home scenario with a star topology and the round-trip time ${RTT=2ms}$, the estimated speedup would be $2.4\mhyphen 10.7\times$

\subsection{Variant Parallelism vs. Class Parallelism}

We briefly compare our method with the current state-of-the-art deep learning parallelism algorithm, \textit{sensAI} \cite{MLSYS2021_sensai} which is a \classp{} technique.

\noindent\textbf{Fault-Tolerance:}
After sending requests to workers, the master node waits for a time window to receive as many responses as possible. Thus, in the event of preemtion due to processing high priority tasks or an issue in a worker or the network, the master node treats them as a fault.
We demonstrated that in case of failure on even $n\mhyphen 1$ worker nodes, we can still get a response.
In \classp{}, however, since each model predicts different class(es), all contributing machines have to successfully process and send their results to the master.
One can improve it by executing replications on additional machines, but it increases both complexity and cost.
Another idea is to apply overlapping class-aware pruning \cite{yang2020robust_sensai}.
In this type of replication, in presence of failure in one machine, the results can be aggregated from other machines with accuracy drop. Both ideas increase computation and communication costs.

\noindent\textbf{Response Time, MACs, Model Size, Training Time:}
As depicted in Table~{\ref{table:perf_metrics}, by leveraging class parallelism, \textit{sensAI} can reach up to ${\sim}2\times$ speedup using $10$ parallel machines.
We showed that variant parallelism gets ${\sim}2.5\mhyphen 3.2\times$ speedup compared with the baseline while requiring roughly half of in-parallel compute nodes compared to \textit{sensAI}. Further note that, we trained our variants for $135$ epochs which is $32.5\%$ less than class parallelism, yet we achieve comparable or higher accuracy. However, since our infrastructures and scheduling policies are different, direct comparison might be unfair.

\begin{table}[t]
	\vskip -0.2in
	\caption{Comparison of our variants with the baseline and \textit{sensAI}.}
	\vskip -0.2in
	\label{table:perf_metrics}
	\begin{center} \begin{small}\begin{tabular}{p{1cm}p{1.4cm}p{0.8cm}p{1.7cm}p{1.8cm}}
		\hline
		Model			& Time (ms)			 & Speedup		& \#MACs Gain	& \#Params Gain		\\
		\hline
		MBNet2		 	& $226\pm0.5$		 & $1{\times}$	& $1{\times}$	& $1{\times}$	\\
		\hline
		$V_6$		 	& $91\pm0.4$	     & $2.5\times$	& $4.3\times$	& $5.8\times$	\\
		$V_5$		 	& $71\pm0.2$	     & $3.2\times$	& $5.6\times$	& $6.1\times$	\\
		$V_4$		 	& $53\pm0.2$	     & $4.2\times$	& $7.6\times$	& $6.3\times$	\\
		$V_3$		 	& $38\pm0.2$	     & $5.9\times$	& $10.9\times$	& $6.7\times$	\\
		$V_2$		 	& $26.5\pm0.1$	     & $8.6\times$	& $17.5\times$	& $6.9\times$	\\
		$V_1$		 	& $17\pm0.05$	     & $13.2\times$	& $31\times$	& $7.1\times$	\\
		Aggregate	  	& $0.05\pm0$	     & -			& -				& -				\\
		\hline
		\textit{sensAI}	& -					 & $2\times$	& $3.5\times$	& $5\times$		\\
		\hline
		\end{tabular} \end{small} \end{center}

\end{table}

\noindent\textbf{Output Size:}
In the tasks with few classes (e.g. CIFAR-10, SVHN, and MNIST) almost the same number of values must be transmitted. However, for the datasets with ${C>=100}$ (e.g. CIFAR-100 and Food-101), we transmit ten bytes per variant which depending on the number of clusters in \classp{}, it can be up to $2\times$ less than \textit{sensAI}.

\noindent\textbf{Flexibility:}
Since each variant is being executed independently, it gives us more flexibility than \classp{}  to achieve different goals. We can generate additional variants based on more than one basic architecture, or design a highly customized variant for contributing machines. We can have several variants with different characteristics to be deployed on heterogeneous compute machines. \variantp{} can also be combined with other distribution schemes including \classp{}.

\section{Conclusion \& Future Directions}
\label{sec:disscusion}

In this paper, we presented Variant Parallelism, a more flexible and fault-tolerant distribution scheme based on ensemble learning. Our evaluation on six different object recognition datasets demonstrates that our method can improve performance in number of parameters, MACs, and response time compared to the baseline and the current state-of-the-art.

We note the issues to be considered in future works.
Although \variantp{} has enough flexibility, it has yet to be evaluated on more complicated benchmarks.
In addition, we intend to extend it on more complicated tasks such as object detection and semantic segmentation. We believe more performance gain can be achieved when tasks become more complicated.

Both \variantp{} and \classp{}  are based on the well-known ensemble learning techniques. Therefore, they can have similar weaknesses and strengths. For example, in \variantp{} combining variants with identical characteristics, may not help achieving significant accuracy boost unless we retrain each of them with different random seeds \cite{allen2020towards}
Contributing machines in \classp{}  and \variantp{} observe the same data.
In some scenarios, e.g., a smart city with long latency or bandwidth bottlenecks, transmitting input data can impact the end-to-end latency. Combining split computing paradigm
\cite{kang2017neurosurgeon, sbai2021cut}
with \classp{}  or \variantp{} can help mitigate the problem. Thus, in future work we decide to optimize the input communication in \classp{}  and \variantp{}.

redpajama_set_name: RedPajamaArXiv


text: Biaggi letter objects to 2800 Bruckner as a 822 program Transportation Centers Workers Tell Lawmakers at Town Hall of Challenges Accessing Health Insurance State Sen. Alessandra Biaggi, Assembly Member Alicia Hyndman and 32BJ SEIU President Hector Figueroa led the discussion on providing benefits supplement compensation for transportation centers' workers to use to acquire quality, affordable health insurance Jordany Bueno, a 28-year-old La- Guardia Airport wheelchair agent, told a panel that his employer's health insurance is not an option to treat his epilepsy because it is simply too expensive but that the costs and expenses of going without health insurance are piling up. "I had a seizure at work one day and was rushed by ambulance to the emergency room," Bueno said. "The ambulance bill alone came to $1,000. I am still paying that and the hospital bills. I kept getting bills and bills." State Sen. Alessandra Biaggi and Assembly Member Alicia Hyndman have introduced the landmark Healthy Terminals Act bills to provide workers benefits supplement compensation that thousands of workers at New York's transportation hubs can use to acquire this health insurance. If the bills become law, they would apply to workers at the John F. Kennedy International Airport, LaGuardia Airport, New York Stewart International Airport, the Port Authority Bus Terminal, Pennsylvania Station and Grand Central Station. "At the airports, we fought long and hard to improve our wages and we finally won," Bueno continued. "But one of the unintended consequences of our victory is that I am no longer eligible for the city health insurance that I was using to get treatment and pay all the expenses for my epilepsy medications. "I see my doctor every three months. That is $125 each visit. My epilepsy medication, plus a calcium supplement to beat back lupus, which is a possible side effect of the medication, cost me $75 discounted. I also get terrible headaches, another side effect of the epilepsy. It costs $200 to get the medication for the headaches. I take a blood test every three months so my doctor can calibrate my epilepsy medication. There is a cost for that. I also have to have annual MRI and other scans, which I've been putting off, because of the costs." Bueno was one of the workers who spoke at a Queens town hall today convened by lawmakers to address challenges workers at New York transportation hubs face trying to acquire the quality, affordable health insurance they desperately need while working in public service, high interaction jobs. 32BJ SEIU President Hector Figueroa pointed out one study that showed four out of 10 Americans say they couldn't pay for a bill of $400 or more if they had an unexpected medical expense, and the same goes for Terminal workers. "A large share of workers in this country have medical debt which leads to financial hardship, including bad credit scores that inhibit working BRONX TIMES REPORTER, J 18 UNE 21-27, 2019 BTR families and poverty that further cripples our communities," Figueroa said. "The New York Healthy Terminals Act will get workers at these vital transportation centers out of this morass and provide a path to achieving quality, affordable health insurance." Workers at the town hall shared personal stories of crushing debt and struggles to find quality, affordable health insurance in New York City. Juniya Montomery, a 55-year-old wheelchair agent, said he took his job at JFK Airport two years ago specifically because he wanted health insurance. "Before I became eligible for my company's insurance, I had not been to a doctor in six years," he said. "I felt I had to look out for my health. I had never had any of the screening and tests that a man my age was supposed to have had. No screening for colon cancer, diabetes, prostate, heart, whatever." He took his employer's health insurance as soon as he became eligible on January 1, 2018. "My company's health insurance cost me $58.85 each pay period," Montomery said. "That's $235.40 during a four-week month and $294.25 in a fiveweek month. That plan, I believe, has a $5,000 deductible. So, that's more than $10,000 before I can get meaningful healthcare. It is unaffordable but I take it and pay for it because my health is a priority for me right now." Horace Foster, 61, battled depression and homelessness before undergoing chemotherapy treatment for throat cancer last year. He said he has the opposite problem to Montomery. "I don't have health insurance because I cannot afford my employer's plan," he said. "I don't want to be sick again. I don't want to be depressed again. I don't want to be homeless again. But, without health insurance, like the one being proposed in the Healthy Terminals Act, I am afraid that all those things may be in my future." The lawmakers discussed how transportation center workers could use their proposed law to address the health insurance challenges they described. "Why fight for economic justice only to have the lack of quality, affordable health insurance push you back into poverty?" Sen. Biaggi asked. "A stable, trained, experienced workforce that knows the facilities, protocols, and chain of communication is essential to the safe and secure operation of New York's major transportation centers. That is why we introduced in the New York State Senate and the Assembly the New York Healthy Terminals Act." Assembly Member Hyndman said the new New York Healthy Terminals Act will establish higher minimum standards for pay and benefits—including healthcare, holidays and vacations— at travel hubs across the city. Starting Sept 1, 2020, the new law will establish an employer-paid benefits supplement that has already been proven effective with the Service Contract Act, which currently provides $4.48, with small annual increases. Worker could then use this $4.48 for health insurance or employees working under collectively bargained contracts could negotiate to use this supplement to provide affordable, quality health insurance directly. "I stand with the members of 32BJ SEIU in their fight for affordable and adequate healthcare insurance. As one of the largest transit hubs in the world, New York needs a well-equipped, healthy workforce to support our local economy," Assemblywoman Hyndman said. "By establishing minimum wage rates to assist with the high costs of healthcare, the New York Healthy Terminals Act would improve retention, lower turnover and create an environment where experienced workers can envision long-term success." With 175,000 members in 11 states, including 9,000 airport workers in New York and New Jersey, 32BJ SEIU is the largest property service workers union in the country. BY PATRICK ROCCHIO Another elected offi cial has gone on record opposing a proposed therapy program for those suffering from substance abuse issues and eating disorders. Miracle City has plans to establish a NYS Offi ce of Alcoholism and Substance Abuse Services licensed 822 program at 2800 Bruckner Boulevard in Throggs Neck. The latest opposition letter came from Senator Alessandra Biaggi, who in a Friday, June 7 letter to OASAS, stated she wouldn't support the facility because of its proximity to These Our Treasures, an early education program and its remote location on a oneway street. "Expanding Miracle City's Current Program will be detrimental to the atmosphere of the neighborhood and disrupt the fabric of the community where the residents and children need to live independently and thrive," said Biaggi. Her opposition, long awaited, came weeks after a Friday, May 3 letter by Assemblyman Michael Benedetto and Councilman Mark Gjonaj, who where the fi rst to oppose the facility. The assemblyman and councilman opposed the facility citing a lack of police resources to support the facility, its distance from public transportation and concerns about loitering at other nearby similar facilities, in addition to the reasons Biaggi gave. The Throggs Neck Homeowners Association contacted OASAS via email on Friday, May 10, to express its opposition to the planned program citing the past track record of Miracle City. Gerbino said that the membership took a unanimous vote at a recent meeting to oppose the facility, with one person abstaining because she is a judge. The civic organization is concerned about the credibility of the facility owners and managers, which was referenced in the letter and in an email by Robert Barbarelli, TNHOA vice president. Bob Jaen, president of the Throggs Neck Merchants, took a more measured stance in explaining his own support for the facility. "I am against any kind of drugs being dispensed out of any locations other than a pharmacy, but if there is counseling and off premise social work, what would be wrong with it?" he said, adding since Miracle City is suggesting it won't dispense medication. He feels the community should monitor the program before drawing conclusions. He believes that misinformation is being spread about Miracle City's intentions. "As far as the merchants, we treat all the members the same, until we see them doing something wrong," said Jaen, who noted he was in fact the co-leader of the Throggs Neck Strong coalition in 2017 that opposed a rumored homeless shelter at the same location. As previously reported in the Bronx Times, local homeowners and residents are concerned that once an 822 license is issued for counseling purposes, Miracle City could easily add the dispensing of medications like methadone. Community activist Edgido Sementelli, who hosted several rallies at the location, has collected countless signatures on paper (and more online) with the names and address of individuals opposing "a drug rehab facility at 2800 Bruckner Boulevard." An anti-drug clinic rally drew a broad cross-section of people on June 8th. Photo courtesy of Egidio Joseph Sementilli

redpajama_set_name: RedPajamaCommonCrawl


text:
Q: How to send a multi line string to URL - Objective C I have a Text View which will be used to input data on multiple lines. The string will then be posted to an URL to a php file and then to a MYSQL Database.
The problem i am having is that the string will not be send if i use multiple lines. I used the following to get around the fact that (Spaces) could not be sent via a URL:
hostStr = [hostStr stringByReplacingOccurrencesOfString:@" " withString:@"%20"];

Is there anything similar to this I can do to get the string to return with multiple lines?

A: Have you tried [hostStr stringByAddingPercentEscapesUsingEncoding:] ?

redpajama_set_name: RedPajamaStackExchange


Would this include access for the Telstra AFL/NRL Live Pass?
gluewell14 commented on Mandatory to Renew Lease?
Are the tickets GA or reserved seating?
I'm just finishing off the last of the variety pack. Best to worst for me: Mocha, Aroma, Organic, Flair (didn't like this one by a mile).
gluewell14 was awarded a badge.
That's the card I have and am enrolled in the Membership Rewards Gateway program.
Looks like it's back online now, just activated it.
How do you know its 12 months free if no support from CS?
Ughh do I really need this or not..
> The Xiaomi airdots [@MostlyMelbourne](https://www.ozbargain.com.au/comment/6821203/redir) Will you be stocking these?


text:
package org.apache.flink.api.common.typeutils.base;

import java.io.IOException;

import org.apache.flink.annotation.Internal;
import org.apache.flink.core.memory.DataInputView;
import org.apache.flink.core.memory.DataOutputView;

@Internal
public final class CharSerializer extends TypeSerializerSingleton<Character> {

	private static final long serialVersionUID = 1L;

	public static final CharSerializer INSTANCE = new CharSerializer();

	private static final Character ZERO = Character.valueOf((char)0);

	@Override
	public boolean isImmutableType() {
		return true;
	}

	@Override
	public Character createInstance() {
		return ZERO;
	}

	@Override
	public Character copy(Character from) {
		return from;
	}

	@Override
	public Character copy(Character from, Character reuse) {
		return from;
	}

	@Override
	public int getLength() {
		return 2;
	}

	@Override
	public void serialize(Character record, DataOutputView target) throws IOException {
		target.writeChar(record.charValue());
	}

	@Override
	public Character deserialize(DataInputView source) throws IOException {
		return Character.valueOf(source.readChar());
	}

	@Override
	public Character deserialize(Character reuse, DataInputView source) throws IOException {
		return deserialize(source);
	}

	@Override
	public void copy(DataInputView source, DataOutputView target) throws IOException {
		target.writeChar(source.readChar());
	}

	@Override
	public boolean canEqual(Object obj) {
		return obj instanceof CharSerializer;
	}
}

redpajama_set_name: RedPajamaGithub


A luxurious facial for all skin types. Gentle steam, mild exfoliation and a facial massage with essential oils custom blended for your skin type, followed by a hydrating masque. Warm hand conditioning included. This escape will relax, refresh and rejuvenate your total well-being.
Designed to treat premature and aging skin conditions. This treatment includes a non-irritating vitamin ampoule and a vitamin masque. This will noticeably improve skin elasticity, tone and texture.
This treatment provides deep pore cleansing that is great for acne and acne prone skin types with a combination of lactic acid, fruit enzyme and pore vacuuming.
Beginning with gentle steam, this soothing facial is followed by a deep cleanse and hot towel exfoliation. A relaxing facial massage then a calming masque tones and firms the skin while relieving razor burn.
Leaves skin refined while promoting new cell growth and collagen production. Beneficial for those with aging, acneic, sun damaged or scarred skin. This treatment includes deep cleansing with steam, neck and shoulder massage and a hydrating restoring masque.
Swedish massage consists of long relaxing movements to warm and stretch muscles. A full body Swedish massage covers the back, glutes, legs, feet, arms, chest, and head. You may request light or firm pressure, full body, or concentration on a particular area.
This deeper massage is designed for individuals looking for more detailed body work on specific muscle groups using the techniques of Trigger Point Therapy, Soft Tissue Release and Sports Massage to relieve spasms and chronic pain.
There is nothing more relaxing, more stress reducing, more luxurious and more addictive than a Hot Stone Massage. Smooth Basalt stones are heated in hot water and applied to the body in energetic patterns during the session. Stones are also used as tools for deeper, more penetrating massage of thicker muscular areas.
Using overhead bars for support, we provide soothing strokes that are applied to all areas of the back and legs.
the body's own self-healing capabilities.
This 90 minute "Ultimate" session combines the best of the Hot LavaStone Massage and the Ashiatsu Oriental Bar Therapy. A true melt down experience!
Raindrop Technique combines the art of aroma- therapy with the techniques of Vita Flex and massage in the application of essential oils to various areas of the body. This collection provides a revolutionary means of bringing balance and harmony to the body – physically, mentally and emotionally.
One of our most popular body treatment combos! Your skin will be moist, plump and exfoliated. Your muscles and joints will feel like they just came back from a complete overhaul. These two treatments go together like a sweet and salty treat.


text:
Jamfiction The Beautiful Wife Of The Whirlwind Marriage novel – Chapter 1120 – Good Gracious, You're So Intimate with Each Other finicky female share-p3
memoir of an eventful expedition in central america
Chapter 1120 – Good Gracious, You're So Intimate with Each Other plug pin
"Yes, I'll test my best, for sure!" Mu Feiran sighed and shook her head.
Nan Gongyu berated her when she had to be berated. It designed her really feel much better.
the castle inn of the lost coast
Nan Gongyu berated her when she must be berated. It built her experience greater.
"Tch. You are not thinking of my emotions at all."
kris longknife audacious inquiry
But even though he was quoted saying so, he was acting with Mu Feiran when he possessed only just started. There was clearly no way he could overlook her status as being a top actress so promptly. Whether or not a high actress was creating her comeback, she was still a high celebrity. She got numerous followers and was so famous in those days. Her several years of knowledge also spoke for themselves. Any person could be concerned.
Just what the h.e.l.l? He really were built with a method of undertaking things…
Within, Mu Feiran stated, "It's also mainly because I'm not approximately par. The director is absolutely not terrible. He's not too strong."
She considered that probably they are often purchased ahead of time.
However it was no diverse.
Dark colored Hawk?
Lin Che failed to go with the launching wedding service, due to the fact she understood that a great many reporters would certainly be there, however she had not been able to seem to be there presently.
"Hey, major-photo producer. You're not gonna hinder this, very, ideal?"
They checked up…
Anyone on this page belonged for the manufacturing team. A lot of them were personnel of your firm, therefore they were definitely also aware about her health. The others acquired approved non-disclosure agreements also. They did not dare to spread rumors in the industry possibly, lest people discover out they were the people spreading the rumors. At some point, they could have got a poor standing and people will not dare to indulge their products and services once again.
She looked at him to determine him say calmly, "I did not ought to order them upfront."
Nan Gongyu let the two girls go walking about the established first as he journeyed off and away to meet with the male direct about his displays.
She could only say, "Never head, by no means mind. It is not a thing."
Possibly it was actually actually because a single thing done by an individual you preferred can be cute.
He transferred aside to adjust him self. He believed he could not spend everyone's time for that reason. He was required to fine-tune him or her self into the predicament.
Mo Jinyan reported, "I advised my adult men to look there early every morning and convey to him I wanted anything. In the beginning, he declined to promote these to me. I mentioned that he could determine between reselling the crooks to me or having his retail store wrecked and trashed into the container. He marketed those to me finally."
Mu Feiran questioned sheepishly, "Why never we rehea.r.s.e a further time? Let's make an effort to evoke our thoughts once again."
Mu Feiran reported, "Don't express that. I'm no different from a rookie right now. Did not you can see it? The director even yelled at me earlier on. Whilst we're acting, we'll only do stuff as outlined by our assignments. I will only be Sibling Feiran when we're outdoors. You're the strong male steer from the series. I am the female direct who has to count on you for emergency."
Additionally, he experienced done it in such an open approach. There is almost nothing anyone could do concerning this.
joan of naples alexandre dumas
Mu Feiran said, "Next time, we are able to give everyone food items that is a bit quicker to get. They are delighted too."
the life and times of kateri tekakwitha
Perhaps it turned out actually for the reason that something carried out by somebody you loved can be attractive.
Chapter 1120: Very good Gracious, You're So Intimate with one another
From The Darkness Cometh The Light
As Mu Feiran experienced not recorded for a long period, she still necessary to familiarize herself with the course of action. As for the male lead, he was also somewhat inflexible because he was operating with Mu Feiran and believed genuinely worried.
Beautiful Bitch
But though he was quoted saying so, he was acting with Mu Feiran as he obtained only just started. There was no way he could forget about her rank being a top notch actress so quickly. Regardless of whether a top-notch actress was helping to make her return, she was still a top celebrity. She possessed so many followers and was famous back then. Her years of working experience also spoke on their own. Any individual could be tense.
The younger idol quickly claimed, "Yes, of course. It is truly a great honor. I'm just scared that we could make mistakes and hinder Sister Feiran."
She may not have recognized it, why performed she obtain him lovable when he was silly?
In fact, most of the directors whom Mu Feiran achieved can be very respectful towards her and will be particularly subservient to her. That made her feel awkward as an alternative.
Lin Che smiled when taking a look at him. "He's a newbie from our organization. Director Nan Gongyu, how would you discover him? He's pretty good, correct?"
"Yes, I'll check out my greatest, beyond doubt!" Mu Feiran sighed and shook her brain.
Topgallantfiction The Beautiful Wife Of The Whirlwind Marriage read – Chapter 1208 – How Dare She Make Him Jealous lying cloudy recommendation-p2


text:
Updated on 3rd May, 2022.
This fair play policy ("Policy"), which forms an integral part of the terms and conditions that govern the use of Gamezy ("Platform"), is intended to ensure that all games played on the Platform adhere to high standards of fairness, competition, skill, and sportsmanship. Accordingly, the Policy sets out mandatory standards of conduct that is expected from users of the Platform ("Users" or "You") and discloses, without limitation, specific measures implemented to ensure the aforesaid standards on the Platform.
GamesKraft Technologies Pvt. Ltd (hereafter, "GamesKraft", or "We" or "Us" or "Our") reserves the right to take necessary actions, determined at its sole discretion, to enforce this Policy on all games occurring on the Platform, and hence Users are advised to read this Policy carefully before using playing a game on the Platform.
1. Fair Play Violations
1.1 A fair play violation ("Fair Play Violation") means any activity of a User that violates this Policy, and includes:
activity that uses unfair means to manipulate the outcome of a game on the Platform, irrespective of whether this in favour of the user adopting such means;
activity that is fraudulent, illegal, or unfair activity, as determined by Gameskraft in its sole discretion;
activity that is intended solely to transfer funds from one person to another person, or between distinct sources, thereby amounting to money transferring;
activity that involves collusion between two or more players to achieve a result during a game, thereby amounting to syndicate playing; and
activity that falls under the Prohibited Conduct set out in Clause 2 below.
1.2 By registering on and using the Platform, Users agree that they will not indulge in any Fair Play Violation, by any means. Any User found guilty of a Fair Play Violation shall be subject to the consequences set out in Clause 3 below.
2.1 The following conduct is expressly prohibited on the Platform:
2.1.1 use of multiple accounts to play the same Game or at the same table, as may be applicable;
2.1.2 use of multiple accounts or colluding with other Users to unfairly achieve a collective outcome while playing any sports-based fantasy game;
2.1.3 intentionally losing a particular round while playing a Poker game on the Platform in order to deliberately transfer chips to another User;
2.1.4 use or availing of any form of external assistance to play a game on the Platform, and shall participate in games using their solely own skill and abilities;
2.1.5 use of unauthorized components, create or use cheats, exploits, bots, hacks, or any other third-party software designed to modify, intercept, decrypt, or mine the Platform in any manner;
2.1.6 dissemination or uploading of spam, unlawful content, or any unauthorised communication to other players on the Platform, or employees of Gameskraft;
2.1.7 use of the Platform as a means of money transfer between persons or separate accounts, or any conduct that may amount to money laundering or other unlawful activities under Indian law;
2.1.8 uploading any content to the Platform, or registering a user ID, which is racist, casteist, profane, vulgar, offensive, or unlawful in any manner.
2.2 Users must ensure that all KYC documentation required at the time of registration on the platform is accurate and valid, and the submission of any false, forged, or other fraudulent documentation shall invite immediate termination of the User account, along with related legal action.
2.3 All communication must be strictly pertaining to matters related to the Platform, and Users found to be indulging in offensive or unlawful communication with any person on the Platform may be terminated and barred from accessing the Platform. The following types of communication is specifically prohibited:
2.3.1 Vulgar or abusive language, racist remarks, offensive comments about religion or caste;
2.3.2 Solicitation in any form by any player via the chat, begging, pestering and causing nuisance of any sort;
2.3.3 discussion regarding the purchase or sale of Chips, or solicitation for the same;
2.3.4 flooding of chat window with recurring messages, or any sort of persistent or constant unwanted communication;
2.3.5 advertising or promotional activity, or attempt to promote products or services of third-parties;
2.3.6 malicious or defamatory content about the Platform or any of its other Users; and
2.3.7 revelation of personal details like email address, contact numbers, postal address, etc.
3. Consequences of Fair Play Violations
3.1 All games on the Platform are monitored for Fair Play Violations, and detection of a Fair Play Violation may lead to the immediate freezing of the User accounts involved, including the funds therein, until the matter is resolved.
3.2 Users found engaging in Fair Play Violation may be permanently barred from accessing the Platform, their accounts blocked, and any funds therein frozen in accordance with this Policy. This includes the Users who profited from the Fair Play Violation, and Users who aided or colluded in committing the Fair Play Violation.
3.3 In the event, We believe or suspect that You have been engaged in Fair Play Violation:
3.3.1 we will notify You such breach/attempt to breach; and
3.3.2 We shall have the right to investigate and monitor Your Gamezy Account and Your gameplay for such period of time as in Our reasonable opinion may be needed to investigate such breach. The balance funds of both users guilty of Fair Play Violations may be frozen for a period of such 180 days. Such investigation will cover all the games that the concerned User has played since the time of registering on the Platform.
3.3.3 In the event, We, upon investigation are of the view that Your Gamezy Account is or has been engaged in Fair Play Violations, then to the extent of the amount which is involved in such Fair Play Violations, We may:
forfeit the fund in Your Gamezy Account;
repatriate the amount to the Generally Affected User or individuals who have been affected by the Fair Play Violation; and
levy a Fair Play violation fee of an amount up to an amount of INR 10,000 per instance of such Fair Play Violation. You agree that such fee is not in the form of a penalty but is a reasonable and justifiable cost, manpower, technology, loss of reputation that Gamezy as a platform may suffer due to Your violation of the Fair Play Policy. The fair play violation fee may be debited from the funds in Your Gamezy Account or in the event the balance in Your Gamezy account is not sufficient to cover the fee, we may raise a claim for such fee against You in accordance with applicable laws.
3.4 Gameskraft reserves the right to review any game of any User at any point in time, and if any games are found to be in violation of this Policy, then strict penalty and actions will be taken as per this Policy. There is no time limitation on the detection and actioning of a Fair Play Violation on the Platform.
4. Redressal for Genuinely Affected Users
4.1 For the purpose of this Policy, a "Genuinely Affected User" shall mean a User who is subject to a direct monetary impact during a game on account of a Fair Play Violation by one or more of the other players in that game.
4.2 Genuinely Affected Users may, after review and confirmation by the requisite risk management team of Gameskraft, be eligible for a refund of their Buy-in Fee for the affected game. Such refunds will be in accordance with the other policies governing the use of the Platform, including those relating to game cancellation and refunds in Clause 6 below.
4.3 Genuinely Affected Users must report a Fair Play Violation within 72 (seventy-two) hours of the occurrence, via email to support@gamezy.com or via the help section provided on the Platform. Gamesraft's risk management team shall investigate the reported game and respond with the investigation within 72 hours. The necessary action for valid cases will be taken as per the game cancellation and refund policy.
5.1 Gameskraft has implemented a 24x7 monitoring mechanism, involving a team of experts and proprietary technology, to review games played on the Platform for Fair Play Violations. However, we also rely on Users to report Fair Play Violations, in order to ensure swift redressal of such matters, and accordingly You are encouraged to contact Us if you suspect a Fair Play Violation during any game You participate in on the Platform.
5.2 Gameskraft and its partners make use of the Random Number Generator (RNG) algorithm in the game play on the Platform, which has been accredited by a renowned testing lab (iTech Labs, which is IEC/ISO 17025 certified). Accordingly, international standards of randomisation are implemented on the Platform to ensure the fairness of each game, including seating, shuffling, etc.
5.3 Gameskraft implements anti-collusion measures before and after each game on the Platform which include but are not limited to randomized seating at tables and use of anti-fraud algorithms.
5.4 Information about the playing cards is always encrypted, thereby preventing any third party from viewing the same.
5.5 Advanced techniques have been employed to assess whether two (2) or more players on the table are colluding during the game. In case, the system finds out about any such unethical practice happening on the table then an auto-alert is raised instantly.
5.6 An alert is automatically raised if there are any signs of Fair Play Violation.
6. Game Cancellation and Refund Policy
6.1 In the event, a Rummy Cash Game is canceled due to a technical error or any other reason solely attributable to the Platform, or on account of the detection of a Fair Play Violation, the following settlement policy shall be followed:
6.1.1 Points Rummy Cash Games
In the event of cancellation of a Points Rummy Cash Game, at any stage:
The buy-in amount paid by all Genuinely Affected Users shall be refunded; and
the buy-in amount paid by Users indulging in the Fair Play Violation shall be forfeit, and a penalty may be levied on such Users in accordance with this Policy.
6.1.2 Pool Rummy Cash Games, Deals Rummy Cash Games, and Tournament Games
In the event of cancellation of a Pool Rummy Cash Game, Deals Rummy Cash Game, or a Tournament Game, at any stage:
the Buy-in Fee paid by all Genuinely Affected Users shall be refunded, including the rebuy amount if any; and
the entry amount and rebuy amount, if any, paid by Users indulging in the Fair Play Violation shall be forfeit, and a penalty may be levied on such Users in accordance with this Policy.
6.2 In the event, a Poker Tournament is canceled due to a technical error or any other reason solely attributable to the Platform, or on account of the detection of a Fair Play Violation, the following settlement policy shall be followed:
6.2.1 Rollback in Poker Tournaments
In the event of cancellation of a Poker Tournament prior to late registration ending:
the buy-in amount of all Genuinely Affected Users shall be refunded and;
For the purposes of this section late registration shall mean the limited time period for which registration of Users is permitted after a Poker Tournament has commenced.
6.2.2 Rollforward in Poker Tournaments
(1) In the event of cancellation of a Poker Tournament post late registration ending but prior to any User being in the money:
The tournament fees of all Genuinely Affected Users, shall be refunded;
the remaining winnings are distributed amongst the Genuinely Affected Users based on the following formula:
fifty percent of the winnings are distributed evenly amongst the Genuinely Affected Users and;
fifty percent of the winnings are disturbed amongst the Genuinely Affected Users based on the chip count
the tournament fees paid by Users indulging in the Fair Play Violation shall be forfeit, and a penalty may be levied on such Users in accordance with this Policy.
(2) In the event of cancellation of a Poker Tournament post any User being in the money:
The tournament fees of all Genuinely Affected Users, shall be refunded and;
every Genuinely Affected User shall receive the minimum guaranteed winnings and;
the remaining winnings are distributed amongst the Genuinely Affected Users based on the chip count
(3) For the purposes of this section "in the money" means an instance where any User is guaranteed winnings before the conclusion of a Poker tournament.
6.3 Refunds on account of such cancellations, where reviewed and confirmed by the Gameskraft team as merited in accordance with this Policy, will be made within 14 (fourteen) days of receipt of such request.
Note: This Policy is subject to periodic review by Gameskraft that may result in updates, changes, and modifications, and thus You are advised to review this Policy periodically. The date on which this Policy was last updated is May 3, 2022.


Bits of Splendor: Project Hope Crafting for A Cause Event- A Thank you & Photos!
Project Hope Crafting for A Cause Event- A Thank you & Photos!
I can not believe that we are at the end of the week already. It has been a busy one for us. A really good, busy one. So many new firsts for our family. So many things happening and changing. I literally feel like my heart could just burst wide open with all of the gratitude and happiness I feel.
It took me a few days to sort of "decompress" from the Project Hope event. It was overwhelming for me. In the best way. Completely humbling. Totally blessed. Beyond words.
I told this story at the event. God reminded me of it as I laid in bed the night before. I was remembering the night we had Faith. The night I get the memory box that sparked Project Hope. The lovie bear....I've always remembered the story as picking up the lovie bear and telling myself that I would have another little girl one day and it would be hers. And that is still the story. But moments before that. Before holding the memory box in my hands, I held Faith. My sweet girl. Lifeless in this world, yet full of it in heaven. And rather than feeling anger or sadness, I felt a strong determination rise up in me. I felt something stirring in me even then, to do something. To make her name known. Her and her sister Grace. I didn't want there short lives to be in vain, but to live on somehow. I wanted to do something to carry on their names.
Last Friday night, I stood on stage in front of a group of about 60 women all there for one purpose and one cause. Lovie bears filled a basket, boxes waiting to be decorated...and I realized...I was doing it. The something I was determined to do. It was happening.
Something that started as a small seed in a hospital room, has grown beyond what I imagined. And in my heart I know it is really just the beginning.
Thank you to each person who made this happen. Every donation, every bit of time, gifts and talents that were given...thank you.
Will there be more to Project Hope? Yes. This is only just the tip of what I believe God has in store for it. I've always known and felt that it would be bigger than I could even try to imagine. So I don't even try. Last year when we sat in a school cafeteria with 15 women, I knew it was where Project Hope was supposed to be. And this year as I was surrounded by 60 women, fun decor, and amazing food, I knew it was where Project Hope was supposed to be.
One step after the other. Making much of Him where ever we are.
And in the end it is about that one mom, grasping for hope in moments where she feels none at all. My prayer is that though she won't hear our words, or see our faces, or feel our arms around her...she will feel His presence as she opens that box.
I will continue to take donations for memory boxes, so if you still want to donate any of the items we use for them, please feel free to contact and send them at any time. We will be doing this again...and again.
On that note, I have been asked about getting Project Hope into other cities. That is most definitely in the plans and I will keep you updated on it as it unfolds.
from Sandra Bludau on Vimeo.
I would also like to give a thanks to these special people & businesses who donated to Project Hope. Thank you for the seeing the heart of this cause and joining it. Thank you for allowing me to to tell my girls' stories. To let there names be known, not just for their short precious lives but for the hope that their mom carried and wants so badly to share.
The food, door prizes, craft supplies, decor---all of it was perfection and all made possible because of you who donated in one way or another.
Girls, there are just no words for what you mean to me.
Love the slide show! Can't wait for what's next!
Wow, I am so happy for you that a dream, and with such sadness, you began this journey only to find, love, the Lord, and a community so supportive. Congratulations! I hope to be a part of your next Project Hope even in some way! Blessings to you and your family.
This looked like it was amazing! I wish I could have been a part of it...maybe next one =) I am so happy for you!
I whell up in tears over and over each time I go back to the many occasions to be involved in this event with you. I am so honored to share just a little part of me to such an amazing cause. Thank you Laura. Really thank you for listening to God, and actually following his instruction. Its so unselfish to do what your doing and to open up something so hard and personal to Glorify God. Your an amazing person and your nor your precious baby girls will ever be forgotten.
Love the blog, Love the video, Love you Laura !!!!
it gave me chills to see how much was contributed for your cause Laura. He is no doubt using your dream and vision in a huge way! thankful for you!!! :) love you friend!!


Worst Things You Can Say To An Aircraft Mechanic?
Thought it might be fun to share some war stories on absurd or even funny requests you have received over the years as a aircraft maintenance professional. Use the Add Comment link below to post your favorites.
Many years ago, I was doing an extensive annual on a Cessna 340. The owner came in the shop - said he had a trip scheduled in it for that afternoon. All of the landing gear is removed and sitting on the floor. All of the exhaust (both sides) was on the floor. All of the seats, carpet, etc. was on the floor. When he asked me that question, I turned around, looked at the pile of airplane parts. All I could do was laugh. His face got red, he then stormed out of the shop slamming doors. From then on, I never saw him in the shop, he just called and politely asked if the airplane was ready to go.
On this particular aircraft the engine was directly over the baggage compartment. After the last flight of the day the pilot had written up "Oil Leaking into Baggage Compartment". I had directed a technician working for me to "Clear the Squawks" on that aircraft.
Later in the shift I would review the logbook entries to ensure they were completed correctly. The Technicians signoff was simply "Prosealed Baggage Compartment". Sure-enough, not a lick of effort towards actually stopping the oil leak at the source, but one of the finest proseal jobs I had ever seen!!!
HAHAHA. I liked the mechanics approach but I suspect he looked for the oil leak, found it to be so horrendous only an engine overhaul could repair and did the next best thing! I applaud his customer satisfaction skills and creativity.
I had a discussion with a customer on a needed fix. We had been talking for the last few years that this repair coming due. After going round and round with me about not wanting to spend the money he replied "Come on Mike, you know me, I WOULD NEVER SUE YOU!!" I calmly replied that he was absolutely correct that he would not sue me. I informed him that he would be DEAD. Of course his wife and kids are going to sue me. They need the money that he spent on his airplane.
Do I really need it? Isn't it done yet? Your repair on the entertainment system created a noise in the landing gear (or two completely unrelated systems).
OMG! That is hilarious. Out of sight, out of mind, I guess.
The FAA is coming today.
"then so will your Log Books be!"
We had a customer who purchased a piston engine for his experimental engine. I got a call from the owner who told me on the test flight the engine quit during power reduction on final approach. He'd landed safely but was extremely upset with us. During the discussion he informed me that he had installed the engine, performed a full power test run and then gone for his test flight.
With our paperwork we supply a notice that before flight the engine needs final fuel system, oil pressure and other parameter verification. He hadn't set up his idle speed and mixture rise so it was excessively rich and caused the engine to quit.
"We don't really need to do that right? It's just cosmetic."
"We don't need to fly the aircraft with stall warning working... we can feel if it's going to stall when it starts buffering."
"Just bend the wing back to where it was and return it to service."
"We thought we would remove the engine with the cherry picker - but sorry we destroyed the fuel line with the attaching chain." Flight School Owner and his asistant both not A&P's. They did it after I had left work to go home and reported it the next day.
On finding the elevator travel limits set backwards, "Well I guess that goes to show you it won't kill anyone." Said by a technician annoyed with the higher standards I brought to the company. As if that's the lesson we needed to learn from that.