text:
\section{Ethical Considerations}
	This work does not raise any ethical issues.

\end{document}

\section{Background and Motivation}\label{sec:bkgnd}

\subsection{The Status Quo of Edge Servers}

Edge servers provide computing and storage resources in proximity to end-users and devices.
They can be deployed as a micro data center in each city, or sunk into cellular stations and buildings/houses~\cite{aws-wavelength}.
With the increasing demands from ultra-low-delay applications like AR/VR and autonomous driving, edge computing is expected to play a critical role in future IT infrastructures.

The current edge server architecture still inherits the legacy of cloud computing that has been established for tens of years.
Generally speaking, a server is organized as a many-core CPU plus a series of domain-specific accelerators (GPUs, TPUs, FPGAs, etc.).
The largest edge resource providers worldwide, including Azure, AWS, and Alibaba, all follow this tradition~\cite{aws-ec2-types,ali-ecs,azure-instance-types}.

However, there are two notable differences between the edge and the cloud.
(1) \textbf{Electricity supply.} Data centers, either centralized or distributed, are extremely power-hungry~\cite{deng2014harnessing}.
Therefore, data centers are typically built close to a green electricity supply such as hydro power.
Instead, edge server deployment is limited to certain spots where the power supply is much more constrained and costly~\cite{asplos22-cooledge}.
(2) \textbf{Workload dynamics.} A recent empirical study shows that edge servers undergo much more dynamic workloads~\cite{ens}.
This is mainly attributed to the unique applications served by edges that are mostly user-oriented.
Such high dynamics complicate hardware and software design, e.g., how to scale energy consumption with workloads.

The traditional cloud server architecture is not designed for the above characteristics.
While our research community has realized this issue and proposed various software-level optimizations~\cite{lee2021greendimm,manousakis2015coolprovision,atc20-serverless-trace}, we advocate it's time to reconsider the architecture of edge servers.
More specifically, in this work, we explore if it's feasible and beneficial to organize many low-end SoCs as an edge server.

\input{fig-arch-spec}
\subsection{A glance at our SoC-Cluster\xspace}\label{sec:glance}

The design space is huge in materializing the concept of an SoC-Cluster\xspace into a server machine.
In this work, we use specific server architecture that we have built and commercialized with success.
This type of SoC-Cluster\xspace has been densely deployed in the wild, as will be discussed in $\S$\ref{sec:soc-wild}, and so is representative of the SoC-Cluster\xspace status quo.
For simplicity, in the rest of the paper, we still refer to this particular server as SoC-Cluster\xspace.
In this subsection, we look into its architecture, internal network/disk performance, power consumption, and running results with a list of micro-benchmarks.

\noindent \textbf{Server architecture.}
Figure~\ref{fig:arch} shows the architecture of the SoC-Cluster\xspace used in our measurements.
The major component of the server is a pool of 60 Qualcomm Snapdragon 865 SoCs~\cite{qcomm865-hardware}.
Each SoC contains an Octa-core CPU, an Adreno 650 GPU, a Hexagon 695 digital signal processor (DSP), and a 12 GB LPDDR5 DRAM.
The detailed hardware/OS specifications per SoC are summarized in Table~\ref{tab:spec}.
The server is organized as 12 PCBs where each PCB integrates 5 SoCs.
Each PCB has its dedicated power supply and network capabilities (served as a switch) for SoCs attached to it.
Network interface bandwidth between SoC and PCB is constrained to 1 Gbps.
Plugging PCB into SoC-Cluster\xspace will establish a physical connection with the Ethernet Switch Board (ESB) in SoC-Cluster\xspace.
The network capacity between each PCB and ESB is also 1 Gbps.
The ESB is responsible for exposing all SoCs for external access through its single RJ45 interface (1 Gbps) or dual SPF+ interfaces (2$\times$10 Gbps).
In our case, we use an external network switch (10 Gbps) to connect the external machine and SoC-Cluster\xspace through their SPF+ interfaces to ensure our benchmarks will not be bounded by network throughput.
SoC-Cluster\xspace also uses a Baseboard Management Controller (BMC) to manage and monitor the server status (e.g., power, temperature, and fans).

\input{fig-network-disk-power}

\noindent \textbf{Network capacity.}
We use \texttt{ping} to measure the network latency and \texttt{iPerf3} to measure the network throughput between SoCs as well as SoCs in SoC-Cluster\xspace and the external machine.
As shown in Figure~\ref{fig:network-rtt}, the RTT between SoCs, either in the same PCB or across PCBs, is around 0.44ms, which is a little bit higher than the RTT between an SoC and the external machine (0.32ms).
It is likely attributed to the defective implementation of network interfaces on PCBs.
Figure~\ref{fig:network-throughput} shows the TCP and UDP throughput.
We observe that the bandwidth between intra-PCB and inter-PCB SoCs is almost identical, which is bounded by the underlying network interfaces of SoCs.
The average bandwidth is quite stable around 935 Mbps and 744 Mbps with TCP and UDP, respectively.
In summary, the network throughput of each SoC is lower than the 25/40/100 GbE networks in data centers today.
To fully release its power for applications that need cross-SoC orchestration, we anticipate an enhanced network design at both the hardware and software levels.

\noindent \textbf{Disk I/O.}
Like mobile devices, each SoC in SoC-Cluster\xspace is attached to a 256 GB Sk-Hynix flash storage (UFS 3.1),
so the total storage capacity of SoC-Cluster\xspace is 15.36 TB.
The disk I/O speed could be crucial for data-intensive applications~\cite{spark,apple-FoundationDB}, so we compare that of a single SoC with HDD (Seagate ST6000~\cite{seagate-hdd}) and SSD (Samsung MZ7LH960~\cite{samsung-ssd}), commonly used in traditional edge servers.
All tests were performed using \texttt{fio}~\cite{fio} in synchronous mode.
The block is 1 MB for sequential read/write and 4 KB for random read/write.
As shown in Figure~\ref{fig:flash}, the sequential read speed of SoC-Cluster\xspace is 1,733 MiB/s, which is 1.76$\times$ and 4.1$\times$ faster than SSD and HDD, respectively.
For random read/write, SoC-Cluster\xspace shows similar performance levels to the current enterprise SSD but significantly outperforms HDD in edge servers.
In summary, SoC-Cluster\xspace inherits the high-performance disk from mobile devices because mobile apps' launching performance heavily relies on the disk I/O speed~\cite{mobisys12-fast-launch,ubicomp13-practical,mobisys15-appdelay,atc18-fasttrack}.
However, unlike traditional servers where the disks can be plugged in/out flexibly, the flash storage is integrated into each SoC, making expansion or repair difficult.

\noindent \textbf{Idle power consumption.}
While we will quantify the detailed energy efficiency of SoC-Cluster\xspace for specific applications in later sections, here we report the idle power consumption as well as its breakdown for SoC-Cluster\xspace and the traditional edge server.
By manually turning on and off a single SoC and PCB in SoC-Cluster\xspace, we can get the power breakdown result.
The overall power consumption of SoC-Cluster\xspace is read from its BMC's API (through PMBus protocol).
We read CPU and RAM power on our edge server using \texttt{turbostat}, which leverages the Intel RAPL system interfaces~\cite{intel-rapl}.
The GPU power is read through the \texttt{nvidia-smi} command.
We only consider the power contributed by these three types of hardware, which is sufficient to draw a conclusion.
The results are in Figure~\ref{fig:power}.
The average power consumption of SoC-Cluster\xspace in idle mode is 153 Watts, which is 2.7$\times$ smaller than the 8$\times$GPU server.
Even without the GPUs, the traditional server consumes more idle power than SoC-Cluster\xspace.
This is mainly because mobile SoCs are designed with more advanced transistor technology for low-power scenarios.
For SoC-Cluster\xspace, the SoCs contribute most of the idle-time power consumption (76.2\%).
It should be noted that the power consumption of all PCBs excludes that of the SoCs.

\input{tab-microbenchmark}
\noindent \textbf{Micro-benchmarks.}
We use a cross-platform benchmark, Geekbench 5~\cite{geekbench}, to perform a list of micro-benchmarks on both SoC-Cluster\xspace and the traditional edge server.
The results in Table~\ref{tab:microbenchmark} provide the following major observations.
First, the per-core performance of SoC-Cluster\xspace is close to the traditional Intel Xeon CPU on both computation-intensive and I/O-intensive workloads.
Second, from the whole server's perspective, the large number of SoCs makes SoC-Cluster\xspace much more powerful than traditional CPU servers, e.g., 12.6$\times$ CPU cores, 16.1$\times$ PDF rendering speed, and 12.5$\times$ speech recognition throughput.
SoC-Cluster\xspace's workload throughput can linearly scale with the number of integrated SoCs, as each SoC has its own memory and storage.
For a monolithic multi-core server, however, the performance gain from increasing the core number could be diminished by the memory/disk I/O speed.

These benchmarks are CPU-only and do not reveal the end-to-end performance of typical edge workloads.
The major contribution of this work, as will be presented below, is a comprehensive measurement study on two typical edge applications.

\input{sec-wild}
\section{Implications and Discussion}\label{sec:challenges}

\noindent \textbf{Potential killer apps on SoC-Cluster\xspace.}
This work shows the superior performance of SoC-Cluster\xspace for two critical workloads: video transcoding and DL serving.
Beyond that, we envision that SoC-Cluster\xspace can efficiently serve much richer applications than we have explored.
For instance, SoC-Cluster\xspace is endowed with large storage space and a high I/O speed, as presented in $\S$\ref{sec:glance}, making it a suitable platform for database systems with compatible design patterns~\cite{mobisys19-flash}.
In addition, the SoC-level workload scheduling granularity makes a good case for serverless workloads~\cite{serverless-berkeley}, which often execute very briefly during a session~\cite{atc20-serverless-trace}.
Besides that, DL training~\cite{google-fl}, which is notoriously power-hungry and can benefit from high hardware-level parallelism, is also suitable for SoC-Cluster\xspace with inter-SoC cooperation.

\noindent \textbf{Hardware design.}
The SoC-Cluster\xspace we experimented with is specifically designed for cloud gaming.
According to our observations, its hardware must be improved or even re-designed to embrace more general workloads.

$\bullet$ \textit{Network infrastructure and topology.}
We show in $\S$\ref{sec:video-trans} that an overall 20 Gbps network capacity is mostly enough to support video transcoding on all SoCs.
But with the SoC integrated more densely or its computing continuously improving, the network will likely bottleneck and leave the SoC hardware under-utilized.
More importantly, workloads such as DL inference/training or HPC demand frequent, high-volume data exchanges across SoCs within a server.
The current network infrastructure is not equipped for those workloads due to the high RTT and low bandwidth, especially the per-PCB network capacity (1 Gbps).
SoC-Cluster\xspace should incorporate recent progress from network research/industry~\cite{infiniband,programmable-switch-asic,sivaraman2020network}.

$\bullet$ \textit{Hardware reliability.}
Unlike data-center hardware, mobile SoCs are not designed to operate at full speed and 24/7, but SoC-Cluster\xspace requires them to do so.
Moreover, the corruption of one SoC module (e.g., flash) makes the whole SoC unusable.
As such, fault tolerance is critical.
On the one hand, it is good to have a better PCB design to support pluggable SoCs.
On the other, a software-level failure handler is required to quickly detect and respond to SoC failure.

\noindent \textbf{Software stacks.}
While the software is quite ample to support different kinds of workloads on mobile SoCs, we lack middleware to bridge the gap when SoCs are organized as a cluster instead of as a single smartphone chip.

$\bullet$ \textit{Operating system.}
Mobile OSs are designed and optimized for interactive scenarios rather than server workloads.
For instance, most Android framework components are not needed, such as the telephony manager.
Although it is viable to run Linux or even Windows~\cite{windows-snapdragon} on ARM SoCs, simply replacing Android with other OSs like Ubuntu makes SoC-Cluster\xspace no longer compatible with native apps (like cloud gaming) or unable to leverage certain hardware accelerators such as GPU and hardware codec because their drivers are vendor-specific and proprietary~\cite{wu2013impact}.
To get the most from both, the right way seems to be to revise the original Android OS to fit edge workloads.

$\bullet$ \textit{Intelligent scheduling.}
Workload management and scheduling is a hot topic in data centers to improve resource utilization and load balance~\cite{osdi21-pollux,osdi20-heter-sched,osdi20-antman,nsdi20-themis}.
Workload scheduling on SoC-Cluster\xspace is also urgently required but could give rise to unique challenges.
For example, each SoC can serve multiple workloads simultaneously with its heterogeneous processors.
This adds another dimension of flexibility (and thus complexity) in designing a workload hardware placement algorithm.
Furthermore, the asymmetric SoC processor (e.g., ARM big.LITTLE CPU) needs to be considered to achieve load balance.

$\bullet$ \textit{SoC virtualization.}
Virtualization is the key to the success of cloud computing by offering a more flexible business model to service providers.
In SoC-Cluster\xspace, each SoC consists of eight CPU cores and many co-processors and therefore can be virtualized into smaller units.
We are aware of a few engineering efforts~\cite{chen2015lightweight,song2021towards,anbox,xu2015condroid} to build containers on SoC, but their performance and capability are limited.
Two unique challenges need to be addressed to virtualize SoCs: first, the virtualization runtime must be lightweight to minimize hardware resource waste; second, since each SoC has very few resources, virtualization might lead to high resource fragmentation with an inefficient bin-packing strategy.
\section{Conclusions}\label{sec:conclusions}

In this work, we investigated the possibility and implications of a new form of edge server, i.e., an SoC-Cluster\xspace consisting of many mobile SoCs.
Through extensive benchmarks on two killer edge workloads (i.e., video transcoding and DL serving), we show that SoC-Cluster\xspace achieves significant energy savings as compared to conventional edge servers, as well as demonstrating its limitations.
The observations paint a promising future for SoC-Cluster\xspace in edge scenarios and also guide us to further improve it, in terms of both hardware and software.
\section{Methodology and Benchmark}\label{sec:design}

In this work, we perform an application-driven measurement study to demystify the performance of SoC-Cluster\xspace and traditional edge servers.
This section elaborates on how we set up the experiments for a fair comparison with best efforts.

\noindent \textbf{Applications.}
We select two modern and computation-intensive applications.
(1) \textit{Video transcoding}~\cite{video-tr} converts the format (FPS, resolution, etc.) of a given video stream, widely adopted in applications such as online conferences and live streaming.
It is reported to be a dominant workload at the edge~\cite{ens}.
We identify two scenarios for this workload (live streaming transcoding and archive transcoding) and illustrate their characteristics in $\S$\ref{sec:video-trans}.
(2) \textit{DL serving} is the vital building block of intelligent applications like AR/VR and autonomous driving~\cite{tf-serving,torch-serving,nv-serving}.
A DL serving system takes a stream of input data sent from end devices and executes it with a specified DL model.
Academia has invested tremendous effort into optimizing DL serving performance~\cite{atc17-clipper,osdi20-serving,atc19-mark,fb-inference}.

In summary, the above applications cover the interests of both industry and academia, representing the major workloads on edges now and in the near future.
They are both resource-hungry, indicating that, if SoC-Cluster\xspace can serve those applications well, it's likely that it can serve others well too.

\input{tab-metrics-new}
\noindent \textbf{Metrics.}
Table~\ref{tab:metrics} summarizes the metrics used in this paper, which are divided into two categories: application performance and comparison metrics.
The application performance metrics are mainly about throughput (processed samples/videos per second), latency (model inference time), and power consumption while serving the workloads.
To make a relatively fair comparison between SoC-Cluster and the traditional edge server, we further judiciously select three condition constraints by considering the following aspects:
\begin{itemize}
	\item \textit{Energy.}
	As mentioned previously, energy has become a killing constraint for edge sites.
	Thus, we use throughput per Energy (TpE) to compare application energy efficiency.
	Besides the energy efficiency under a given (often full) load, we also care about the energy proportionality under various load levels since edge servers experience high load variations.
	An ideal edge server should be able to proportionally scale its power consumption with the load to minimize wasted energy.
    \item \textit{Rack size.}
    Edge sites often have space constraints (e.g., deployed in cell sites).
    Therefore, delivering higher throughput within a given rack size is important.
    We normalize the application throughput to the server rack size (TpS) to reveal the per-rack unit performance.
    \item \textit{Cost.}
    We perform a total cost of ownership (TCO) analysis to show the relationship between application performance and monthly TCO, which might help edge operators to make purchase/scheduling choices clearly.
    Section~\ref{sec:tco-analysis} presents our TCO analysis results.
\end{itemize}

\noindent \textbf{Hardware.}
The SoC-Cluster\xspace server used in measurements was introduced in $\S$\ref{sec:glance}.
For comparison, we used a traditional server with an Intel Xeon Gold CPU (4.0 GHz and 40 physical cores), 8$\times$ NVIDIA A40 GPUs, and 768 GB DRAM.
The OS was Ubuntu 18.04 LTS.
Server hardware specifications are summarized in Table~\ref{tab:spec}.
We confirmed that this type of server is widely used in the edge sites deployed by our collaborator.
Additionally, we use a more high-end NVIDIA A100 GPU from the Google Cloud Platform in our DL serving experiments for a more comprehensive comparison.
Both the NVIDIA A40/A100 and the Intel CPU were released in the same year (2020) as the Qualcomm Snapdragon 865 SoC that makes up our SoC-Cluster\xspace.
We did not use an NVIDIA A100 GPU for video transcoding experiments due to the lack of support for NVENC on NVIDIA A100 as of September 2022~\cite{nv-support-matrix}.
In $\S$\ref{sec:soc-evol}, we also experiment with four more SoC models to understand the evolution of SoC performance over time.

\noindent \textbf{Benchmark suite.}
One challenge of our measurement study is the software stack selection.
Given the inherently different hardware architectures and their heterogeneity, we found there is rarely software that is compatible with each processor (i.e., SoC CPU/GPU/DSP, Intel CPU, and NVIDIA GPU).
If such software exists, its performance could be far from the state of the art.
To obtain meaningful results, we consulted our industry partners and also tested with common software.
We then selected the best-performing option for each workload and hardware type.

For video transcoding, we used FFmpeg (v4.4)~\cite{ffmpeg} with libx264~\cite{x264} and NVDEC/ENC~\cite{nvcodec} support.
We cross-compiled FFmpeg to SoC-Cluster\xspace with ARMv8 NEON acceleration.
One exception is that FFmpeg has poor support for the hardware codec of Qualcomm SoCs.
Thus, we used a popular open-source Android library, LiTr~\cite{litr}.
We built our benchmark on vbench~\cite{vbench}, a widely used benchmark tool for cloud video transcoding.

For DL serving, we used TFLite~\cite{tflite} on SoC-Cluster\xspace, TVM~\cite{tvm} for Intel CPU, and TensorRT~\cite{tensorrt} for NVIDIA GPU.
We selected one medium-sized DNN (ResNet-50~\cite{he2015}) and three large DNNs (ResNet-152~\cite{he2015}, YOLOv5x~\cite{glenn_jocher_2022_6222936}, and BERT~\cite{bert-tfhub}), which are representative for DL serving workloads.
All software was carefully tuned and orchestrated into the benchmark suite for automatic, flexible testing on both SoC-Cluster\xspace and the traditional edge server.

\noindent \textbf{Measuring power consumption.}
Power consumption is measured through the software-level APIs, as described in $\S$\ref{sec:glance}.
Note that the reported power consumption of SoC-Cluster\xspace obtained through its BMC's API includes not only SoCs but also the PCBs and cooling devices (i.e., fans).
The workload power consumption report excludes idle power consumption, as measured in $\S$\ref{sec:glance}.
By default, we always carry out our experiments with hardware fully loaded, e.g., using batched DL serving or many transcoding processes of live video streams.
Only when testing the energy proportionality of different hardware do we vary the load levels.

\noindent \textbf{Setups.}
For Intel CPU experiments, we split the 80 cores (80 hardware threads~\cite{hyperthreading}) into 10 separate 8-core docker containers.
We selected 8 cores for two reasons.
(1) Based on previous edge workloads~\cite{ens}, 8 is the median number of vCPU cores for edge IaaS VMs and is enough for serving most edge services.
(2) The SoC's embedded CPU also contains 8 cores, making the comparison more straightforward.
Additionally, to avoid the docker start-up overhead, we kept the container process in daemon and invoked test commands inside the container.
Furthermore, to mitigate power fluctuations during experiments:
(1) for live streaming transcoding, we simultaneously started the maximum number of streams supported by each hardware, ensuring that no stream degraded its transcoding FPS below the origin stream's;
(2) for archive transcoding, we repeatedly transcoded the same video ten times;
(3) for DL serving, we set the DL inference iteration to 1,000 for each test and then got the average power consumption for each frame.
\section{Deep Learning Serving}\label{sec:dl-serving}

In this section, we report the DL serving performance with various hardware types and models.
Besides models in floating-point format (FP32), we also tested models in INT8, which is widely used in DL serving workloads to speed up inference.

\subsection{Inference throughput}\label{sec:dl-throughput}
Inference throughput indicates the maximal workload (and thus users/applications) that can be served by a server.
It is measured as the number of samples processed by the whole server (60 SoCs, 8 A40/A100 GPUs, etc.).
We used two different batch sizes (BS = 1 or 64) for the NVIDIA GPU, which affects hardware utilization.
For other hardware, we only set the batch size to 1 as that is enough to fully utilize the hardware capacity.
Here, we used per-rack unit throughput (TpS) as the space-constrained comparison metric between diverse hardware types.
The results are shown in Figure~\ref{fig:dl-tps}.
\input{fig-dl-tps}

Our key observation is that \textit{SoC-Cluster\xspace, and especially its embedded GPUs, provides much higher throughput than CPU-only servers but less throughput than multi-GPU servers per rack unit.}
SoC-Cluster\xspace's embedded CPUs/GPUs can process 386/1,648 samples with ResNet-50 and 67/321 samples with ResNet-152 (both FP32 format) per second and rack unit.
This throughput is 1.11$\times$--8.22$\times$ more than that on the Intel CPU.
However, compared to NVIDIA A40 and A100 GPUs, the throughput of SoC GPUs is 3.31$\times$--5.41$\times$ and 6.51$\times$--10.51$\times$ smaller on ResNet-50 and ResNet-152, respectively.
In other words, 60 embedded GPUs provide a processing capacity of as much as 0.9$\times$ the NVIDIA A40, 0.46$\times$ the NVIDIA A100, or 325 Intel CPU cores on average.
When a smaller batch size is allowed (e.g., to guarantee low latency), the throughput of the NVIDIA A100 GPU is reduced by 432\% on average because the hardware resources cannot be fully utilized.
The gap between SoC-Cluster\xspace and NVIDIA GPUs is much smaller on the BERT model, mainly because TensorRT is not yet as well optimized for NLP models as for widely used CNNs.

The inference throughput of SoC-Cluster\xspace on INT8-quantized models (Fig.~\ref{fig:dl-tps}e and Fig.~\ref{fig:dl-tps}f) is even more impressive.
For instance, embedded DSPs can collectively process 1,467 samples per second and rack unit with ResNet-152, which is 12.02$\times$ more than the Intel CPU.
Its performance is even close to the NVIDIA A40 GPU and NVIDIA A100 GPU (BS=1).
Credit for this lies with the powerful SIMD co-processor in the Qualcomm Hexagon DSP, which is designed for efficient vector operations in a fixed-point format~\cite{qcomm-dsp}.
It is claimed that the Hexagon 698 (Snapdragon 865) can provide 15 trillion TOPS~\cite{qcomm865-hardware} and the number goes up to 26 trillion on the Hexagon 780 (Snapdragon 888)~\cite{qcomm888-hardware}, which was released just one year later.

\subsection{Inference latency}\label{sec:dl-latency}
\input{fig-dl-latency}
Inference latency is also critical to delay-sensitive workloads and thus the corresponding user experience.
As above, we tested this on NVIDIA GPUs with different batch sizes to trade off latency and energy efficiency but only used batch size 1 on other hardware as further increasing the batch size only incurred higher latency without benefiting energy efficiency.
The results are shown in Figure~\ref{fig:dl-latency}.

We make the following major observations.
(1) SoC-Cluster\xspace's embedded GPUs deliver 2.37$\times$--4.77$\times$ lower latency than its embedded CPUs and are comparable to the Intel CPU with eight cores used.
(2) The lowest latency provided by the NVIDIA GPU with batch size 1 is much lower than other processors due to its high hardware-level parallelism and well-optimized software (i.e., TensorRT).
However, if larger batch sizes are used, the latency on the NVIDIA GPU increases significantly and could be close to or even higher than SoC-Cluster\xspace, e.g., inference using the NVIDIA A40 GPU (BS=64) on YOLOv5x (FP32) and BERT (FP32).
(3) On a medium-sized DNN -- ResNet-50, embedded GPUs/DSPs are often good enough to deliver satisfactory inference latency, i.e., 32.7 ms in FP32 format and 8.8 ms in INT8 format.
On those models, using the NVIDIA GPU only provides a trivial increase in speed (e.g., about 8 ms on a quantized ResNet-50) but incurs much higher energy consumption, as we will discuss below.
(4) On large DNNs such as ResNet-152, the inference latency of SoC-Cluster\xspace is 20.4 ms--444.8 ms, which is unbearable for real-time applications.
This can potentially be addressed by a cooperative inference framework among multiple SoCs.

\input{fig-dl-tpe}
\input{fig-dl-power-efficiency}
\subsection{Energy efficiency}\label{sec:dl-energy}
We used throughput per energy (TpE) (e.g., the number of samples processed per Joule) as the metric to test the energy efficiency for each hardware type.
A server with higher energy efficiency can process more samples with the same electricity unit.
The results are illustrated in Figure~\ref{fig:dl-tpe}.
Our key observation is that \textit{SoC-Cluster\xspace is much more energy efficient than the Intel CPU and mid-end NVIDIA GPU, and is comparable to the high-end NVIDIA GPU.}
Especially on ResNet-50, embedded GPUs can process about 19 frames per second and Joule, which is 6.96$\times$ more than the Intel CPU, 1.75$\times$ more than the NVIDIA A40 (BS=64), and 1.13$\times$ more than the NVIDIA A100 (BS=64).
The benefit of SoC-Cluster\xspace is even more significant in quantized models.
Taking ResNet-152 (INT8) as an example, the energy efficiency of embedded DSPs is 42$\times$ higher than the Intel CPU and 1.5$\times$ higher than the NVIDIA A100 (BS=64).
This is because mobile DSPs are designed for low-power data processing running at $\le$500MHz.

The above energy efficiency is mainly reported when the servers are fully loaded.
We also measured energy efficiency when workloads varied, as in $\S$\ref{sec:video-energy}.
Figure~\ref{fig:dl-energy-scale} illustrates this scalability of energy efficiency on different hardware types.
We only used the embedded GPU and NVIDIA A100 for comparison based on their high energy efficiency on SoC-Cluster\xspace and the traditional edge server, respectively.
For ResNet-50, SoC-Cluster\xspace shows a significant energy benefit when the workload is lightweight, e.g., 5.71$\times$ more energy efficient than the NVIDIA A100 GPU on average with only five samples coming per second.
This is mainly because SoC-Cluster\xspace provides a much more fine-grained scheduling unit (i.e., per SoC) to process incoming requests.
When incoming data can be sufficiently processed by only ten SoCs, we can keep the other 50 SoCs in a low-power state -- a free lunch originally provided for mobile devices~\cite{android-doze} -- or even turned off.
Instead, datacenter-level GPUs have much coarser granularity to scale its energy consumption with dynamic workloads.
Figure~\ref{fig:dl-total-power} further presents the total power consumption times to the server idle power under various load levels.
When workloads are light (e.g., only a few inputs per second on ResNet-50), the NVIDIA A100 GPU shows a power spike compared with its idle power.
Instead, SoC-Cluster\xspace always linearly increases its total power consumption.
Considering the huge gap in the static power draws of the two servers when idle, SoC-Cluster\xspace might be more suitable for serving and scaling its energy consumption with various load levels.

\subsection{Summary}
The embedded GPUs and DSPs on SoC-Cluster\xspace deliver superior energy efficiency than traditional server-level CPUs (up to 17.86$\times$) or GPUs (up to 6.49$\times$) in FP32 and INT8 formats, respectively.
The inference latency of SoC-Cluster\xspace on medium-sized DNNs like ResNet-50 is satisfactory to meet the requirements of typical edge applications, but more advanced software that can orchestrate many SoCs is urgently demanded to serve large DNNs like YOLOv5x efficiently.

\section{Introduction}\label{sec:intro}

Energy efficiency has been recognized as a crucial criterion for data centers~\cite{google-energy-proportional}.
It is reported that 76.8 TWh of electricity was consumed by EU data centers in 2018, which accounted for 2.7\% of the total electricity usage within the EU.
This power consumption is estimated to rise to 98.52 TWh by 2030~\cite{eu-dc-power}.
Such high energy usage also burdens the cooling infrastructure and causes high bills for those who maintain the data centers~\cite{asplos22-cooledge,osdi20-thunderbolt}.

Edge clouds, which provide in-proximity computing resources to users/devices, are emerging as the new critical infrastructure for people's daily life.
According to Gartner, around 75\% of enterprise-generated data will be processed at the edge by 2025~\cite{edge-workloads}.
The energy issue will likely be aggravated on edges for a few reasons.
First, the power supply to edges is more constrained and costly, as edge servers tend to be deployed closer to the population, while cloud data centers are often placed in cherry-picked locations such as those close to hydro power supplies~\cite{nature-dcwater}.
Edge servers are also space-restricted and have one magnitude higher power density, putting pressure on their cooling mechanisms~\cite{asplos22-cooledge}.
Furthermore, edge workloads vary more than clouds, owing to the unique edge application characteristics~\cite{ens}.
Conventional servers can hardly scale their energy consumption with dynamic workloads in a proportional manner, as demonstrated both by prior work~\cite{google-energy-proportional,jiang2017energy} and our experiments (explained below).

A fundamental path towards green data centers is to improve the energy efficiency of every single cloud/edge server.
Following this principle, most prior efforts reside at the software stack~\cite{lee2021greendimm,liu2013arbitrating,liu2012renewable,ran2019deepee,zhou2015carbon,manousakis2015coolprovision,zhang2021flex}.
At the very least, hardware re-design sees more profits~\cite{DBLP:conf/asplos/RanganathanSCDG21} but is hard to deploy as most cloud infrastructure is already well established.
However, we sense the opportunity with edge: its infrastructure is still being finalized and we are now on the eve of its inauguration~\cite{ens}.

In this work, we advocate for a new form of edge server, namely SoC-Cluster\xspace, which consists of tens or hundreds of mobile SoCs like Qualcomm Snapdragon series, as a vital complement to the existing edge infrastructure.
The underlying rationale is that mobile SoCs are designed to be energy efficient by using a simpler instruction set (mostly ARM) and smaller transistors than traditional high-end processors.
For instance, the Qualcomm Snapdragon 888 SoC released in 2021 is based on 5nm technology, while Xeon is still using $\geq$10nm.
Moreover, each SoC can be independently managed (e.g., turned on/off and dynamic voltage/frequency scaling) to adapt to variational workloads, which is much more flexible than a monolithic powerful processor like NVIDIA GPU.

Apart from the low-power vantage, a few more incentives lead us to explore SoC-Cluster\xspace on edges.
First, mobile SoCs can run mobile OSs/apps seamlessly, therefore supporting computation/code offloading from mobile devices.
This characteristic makes them an ideal platform for \textit{cloud gaming}~\cite{amazon-luna,nvidia-geforce,google-stadia,xcloud-games,fb-cg}, which enables wimpy or low-battery smartphones to run resource-consuming mobile games anytime and anywhere.
Furthermore, mobile SoCs are highly scalable: each SoC has a standard CPU that can serve general workloads but also heterogeneous co-processors like GPU, DSP, and NPU that accelerate domain-specific workloads.
A monolithic server sees diminished performance worsen with more processors as the memory/disk I/O could bottleneck; an SoC-Cluster\xspace can linearly scale up its compute capacity with the number of SoCs integrated.
Last but not least, mobile SoC is still fast evolving~\cite{qcomm-evol}, and brings together the wisdom from many leading chip companies.
Building a server on those SoCs takes such free lunch.
Furthermore, the software stacks on mobile SoCs and OSs are mature enough and highly optimized, e.g., deep learning inference/training~\cite{mnn,tflite}, multimedia data processing~\cite{mediacodec}, or even containers~\cite{andrus2011cells,redroid}.

There have been very few, scattered thoughts on organizing mobile SoCs as servers.
\cite{rajovic2013supercomputing} conducts an early analysis of using mobile SoCs for HPC.
Some work focuses on reusing decommissioned mobile devices to reduce e-waste~\cite{shahrad2017towards,switzer2021architecture}.
Other studies focus on using SoCs for very specific applications like parallel computing~\cite{busching2012droidcluster}, key-value storage~\cite{andersen2009fawn}, web search~\cite{janapa2010web}, and video transcoding~\cite{liu2016greening}.
That work either contributes only theoretical analysis, carries out experiments with small-scale toy implementation consisting of 2--8 smartphones/RPIs, or evaluates ``wimpy'' workloads~\cite{lang2010wimpy}.
Instead, we seek to materialize the ``SoC-as-server'' concept in a more realistic and industry-level context.

\textbf{Hardware prototyping and commercialization.}
We have developed a concrete SoC-Cluster\xspace server, which integrates 60 Qualcomm Snapdragon 865 SoCs into a 2U rack.
The SoCs are attached to 12 independent PCBs with dedicated power and network supplies.
The detailed specifications of our SoC-Cluster\xspace are discussed in $\S$\ref{sec:glance}.
Within two years, we have manufactured more than 10,000 such SoC-Clusters\xspace, most of which go to edge service providers.
Those deployed SoC-Clusters\xspace are mainly used to serve cloud gaming workloads.
However, according to monitored traces ($\S$\ref{sec:soc-wild}), utilization of those deployed SoC-Clusters\xspace varies widely and is low on average.
Apparently the potential of those SoC-Clusters\xspace is far from being fully realized.
To fill the gap, the very first but critical step is to understand whether SoC-Clusters\xspace can efficiently serve other applications other than mobile cloud gaming.

\textbf{Measurement methodology.}
Therefore, we present a first-of-its-kind measurement study to quantitatively disclose \textit{how efficiently SoC-Cluster\xspace can support typical edge workloads} on our COTS SoC-Cluster\xspace machine.
The study is application-driven, and we focus on two types of modern and computation-intensive workloads: video transcoding and deep learning (DL) serving.
The former is the de-facto dominant workload on edges~\cite{ens}, supporting applications like live streaming, online conferences, and content archives.
The latter is the key building block of many intelligent applications like AR/VR and autonomous driving.
It is also one of the hottest research topics in the edge computing field~\cite{qvr,DBLP:conf/mmsys/ShiGHJ19,zhang2021emp}.
In the foreseeable future, those two applications will be the major hardware resource consumers of edge servers, making it critical to demystify their performance on SoC-Cluster\xspace.
For comparison, we used a typical edge server comprised of dual Intel Xeon Gold 5128R CPUs and 8 NVIDIA A40 GPUs.
To expand the results to more SoC models, we also tested on smartphones with five different high-end Qualcomm SoCs.

\textbf{Benchmark suite.} We built an automatic benchmark suite to test application performance on both SoC-Cluster\xspace and the conventional edge server.
The benchmark suite leverages state-of-the-art libraries for each application.
For video transcoding, we used FFmpeg~\cite{ffmpeg} to process six videos judiciously selected from vbench~\cite{vbench} with disparate characteristics.
For DL serving, it employed TFLite~\cite{tflite} (SoC-Cluster\xspace), TVM~\cite{tvm} (Intel CPU), and TensorRT~\cite{tensorrt} (NVIDIA GPU).
The NN models used were ResNet-50, ResNet-152~\cite{he2015}, YOLOv5x~\cite{glenn_jocher_2022_6222936}, and BERT~\cite{bert-tfhub}.
The reason we sometimes used different software stacks on different hardware platforms is that there is no single software that can perform well on heterogeneous processors.
The benchmark reports comprehensive metrics (throughput, latency, energy consumption) under various constraints (physical size, electricity, cost).
It also reports application-specific metrics such as video quality and bitrate.

\textbf{Key findings} are summarized below.

(1)
SoC-Cluster\xspace, and especially its co-processors (GPU, DSP, and hardware codec), shows superior energy efficiency than conventional servers.
In DL serving, the per-Joule throughput on SoC-Cluster\xspace is up to 42$\times$ and 2.3$\times$ higher than Intel CPU and NVIDIA GPU, respectively.
This means that SoC-Cluster\xspace consumes less energy to process each sample.
In live streaming transcoding, the per-stream energy saving is even up to 17.1$\times$/13$\times$ respectively.
SoC-Cluster\xspace also proportionally scales its energy consumption with dynamic loads.
For instance, when the number of live video streams decreases from 20 to 5, energy efficiency incurs negligible degradation on SoC-Cluster\xspace but decreases by an average of 3.3$\times$ on the NVIDIA GPU.

(2)
The throughput of SoC-Cluster\xspace per rack unit is noticeably higher than conventional servers in video transcoding services but lower in DL serving.
The maximal number of live video streams supported by SoC-Cluster\xspace is 60--900 depending on video characteristics, which equals 102 Intel CPU cores or 14 NVIDIA A40 GPUs on average.
Note that our SoC-Cluster\xspace fits into a 2U-rack case, which can typically only hold 4--8 GPUs.
In DL serving, SoC-Cluster\xspace delivers a throughput of 321 FPS on the ResNet-152 FP32 model and 5,870 FPS on the INT8 model, which equals 84/191 Intel CPU cores, 0.75/0.79 NVIDIA A40 GPU, or 0.38/0.48 NVIDIA A100 GPU.

(3)
Lacking software support for cross-SoC collaboration, SoC-Cluster\xspace struggles to handle delay-critical workloads.
For a medium-sized DNN model (ResNet-50), the inference latency of SoC-Cluster\xspace is low enough to meet the requirements of most edge applications (8.8ms after quantization).
On large models like YOLOv5x, however, the latency can be up to hundreds of milliseconds.
It urgently demands a DL library to enable collaborative inference on multiple SoCs.

(4)
Based on the total cost of ownership analysis, SoC-Cluster\xspace delivers higher throughput per cost than the traditional edge server for live streaming transcoding.
But for DL workloads, NVIDIA GPU shows significantly higher throughput per cost than SoC-Cluster\xspace.
Such a result could negatively impact the intention of purchasing new SoC-Clusters\xspace to serve DL workloads.
However, we advocate that it's still beneficial to migrate parts of DL workloads to the deployed, under-utilized SoC-Clusters\xspace for higher energy efficiency on edges.

(5)
Through a longitudinal study, we find mobile SoC has shown a tremendous performance increase in the past five years.
Specifically, the mobile CPU performance evolution is slowing, but the capacity of mobile co-processors (Adreno GPU and hardware codec) is still fast growing and their benefits over CPU are increasing.
The improvement of embedded GPU over time is even more profound than high-end NVIDIA GPU in DL serving.
To achieve sustained performance evolution on SoC-Cluster\xspace, it is critical to improve the existing software stack to fully harness mobile co-processors.

In a nutshell, SoC-Cluster\xspace has pros and cons, which also vary across different workloads.
The most promising workload is live streaming transcoding, for which SoC-Cluster\xspace delivers not only higher energy efficiency but also higher throughput and cost reduction.
For archive video processing and DL serving, SoC-Cluster\xspace sometimes shows attractive improvements in energy efficiency but has lower throughput and cost efficiency than data-center-level GPUs.
Nevertheless, we believe our results point to a promising future for SoC-Cluster\xspace for:
(i) video transcoding services such as online meeting (Skype and Zoom) and live streaming (Twitch and TikTok) applications; and
(2) general-purpose edge platforms whose workloads cannot always be accelerated by GPUs.
In such scenarios, the huge number of CPU cores in SoC-Cluster\xspace ($\sim$500 in 2U) is much more powerful than conventional CPU servers.
This is also demonstrated through our micro-benchmarks in $\S$\ref{sec:glance}.

\textbf{Contributions.}
In this work, we discuss the rationales and possibilities of organizing mobile SoCs as edge servers.
We present our first-of-its-kind hardware prototyping of this concept and how it has been commercially deployed on a large scale by edge service providers.
To disclose the performance of this SoC-Cluster\xspace, we built a benchmark suite that leverages state-of-the-art libraries for two killer edge workloads, i.e., video transcoding and DL serving.
We then performed a comprehensive measurement study and compared SoC-Cluster\xspace with a traditional edge server (Intel CPU and NVIDIA GPU) in terms of physical server size, energy efficiency, and monetary cost.
The results reveal both advantages and disadvantages of SoC-Cluster\xspace, especially its high energy efficiency, and provide valuable guidance to fully unleash its power in edge scenarios.

The remainder of the paper is organized as follows.
We provide the motivation for this paper and the very first revelation of SoC-Cluster\xspace in $\S$\ref{sec:bkgnd}.
The measurement methodology and metrics are illustrated in $\S$\ref{sec:design}.
The video transcoding and DL serving performance are shown in $\S$\ref{sec:video-trans} and $\S$\ref{sec:dl-serving}, respectively.
We give the total cost of ownership analysis in $\S$\ref{sec:tco-analysis}.
$\S$\ref{sec:soc-evol} further presents the SoC performance evolution over the years.
We then show the implications and discussion in $\S$\ref{sec:challenges}, related work in $\S$\ref{sec:related}, and our conclusions in $\S$\ref{sec:conclusions}.

\section{Related Work}\label{sec:related}

\noindent \textbf{Grouping mobile SoCs as servers.}
We are not the first to try to conceptualize a server consisting of tiny SoCs.
There have been attempts~\cite{rajovic2013supercomputing,rajovic2014tibidabo} to investigate whether mobile SoCs can provide sufficient performance and reduce costs for HPC.
To reduce e-waste, Shahrad et al.~\cite{shahrad2017towards} built computation nodes with used smartphones and analyzed server design, but they didn't evaluate real workloads.
Switzer et al. used only five smartphones to build a junkyard data center~\cite{switzer2021architecture} with carbon concerns.
Some work uses IoT/mobile SoCs to support specific applications like video transcoding~\cite{liu2016greening}, key-value storage~\cite{andersen2009fawn}, web search~\cite{janapa2010web}, and parallel computing~\cite{busching2012droidcluster}.
However, they mostly focus on specific app types and lack a performance comparison with traditional servers.
In contrast to the above work, we used a COTS SoC-Cluster\xspace, performed an application-driven measurement on it, and presented detailed performance metrics to show its capabilities for modern, computation-intensive edge workloads.

\noindent \textbf{DL model serving}
has become a pivotal module in modern intelligent applications and has received extensive attention from both academia and industry~\cite{aws-sagemaker,azure-ml,DBLP:conf/nsdi/CrankshawWZFGS17,atc21-infaas,osdi20-serving,tvm}.
Existing work has focused on optimizing DL performance on traditional data-center-level CPUs and GPUs.
Meanwhile, on-device DL is rising towards low inference delay and user privacy preservation~\cite{mobicom20-spinn,DBLP:conf/asplos/KangHGRMMT17,DBLP:conf/mobisys/LiuLZNLD18,DBLP:conf/mobisys/WuFTSLDXZ20,deeptype,deepcache} powered by the continuously improving SoC hardware.
This trend has inspired us to treat DL serving as a killer workload to be supported by SoC-Cluster\xspace.

\noindent \textbf{Video transcoding}
is the fundamental cloud service in our digital world to power cloud video infrastructures~\cite{netflix,youtube-infra,sosp17-fb-video} or applications like AR/VR~\cite{DBLP:conf/mmsys/ShiGHJ19,DBLP:conf/mobisys/ShiGJ19,qvr}, campus TV~\cite{liu2016greening}, or video analytics~\cite{DBLP:conf/ieeesec/XiaoXZZJ21,haynes2021vss,eurosys19-vstore}.
The previous measurement also shows that video-related applications contribute to most of the workloads at the edge~\cite{ens}.
For low-cost video transcoding, VideoCoreCluster~\cite{liu2016greening} used Raspberry Pi to build a video transcoder cluster.
Google presented its new hardware-accelerated video transcoding block in data centers~\cite{DBLP:conf/asplos/RanganathanSCDG21}.
In this work, we show that SoC-Cluster\xspace, and especially its hardware codec, can handle video transcoding with high throughput and low energy consumption.

\noindent \textbf{Energy-efficient cloud/edge.}
Energy efficiency has been recognized as a crucial factor in data centers~\cite{deng2014harnessing}.
Various techniques have been explored in a move towards green data centers, including workload scheduling and management~\cite{lee2021greendimm,liu2013arbitrating,liu2012renewable,ran2019deepee,zhou2015carbon}, resource under-provisioning~\cite{manousakis2015coolprovision,zhang2021flex}, greening the data-center network~\cite{nsdi10-elastictree,sigcomm12-green,guo2020aggreflow}, etc.
In contrast to these software-level approaches, we propose a re-design of servers to fundamentally boost energy efficiency.
As the edge infrastructure is still at a preliminary stage, we deem such a radical measure possible.

\noindent \textbf{Measurements at the edge.}
Measurement efforts have been made from centralized clouds~\cite{DBLP:conf/www/HaqRD17,DBLP:conf/sigcomm/SpringMA03,DBLP:conf/imc/DengPFMH18,DBLP:conf/sigcomm/LiNCGM19} and geo-distributed clouds~\cite{DBLP:conf/sigcomm/JinRAJPSCK19,DBLP:conf/hotnets/MohanCZBWK20,imc21-short-rtt}, and towards decentralized edge clouds~\cite{ens}.
They mostly focus on workload scheduling, network routing/topology, or billing models.
Some prior work built benchmarks for specific applications such as video transcoding~\cite{vbench}.
We inherited some of the spirit of previous work (e.g., the metrics and settings) to build our first-of-its-kind benchmark suite for SoC-Cluster\xspace.
\section{SoC Longitudinal Study}\label{sec:soc-evol}

Previous experiments are performed on the SoC-Cluster\xspace consisting of one specific SoC model, i.e., the Qualcomm Snapdragon 865 released in 2020.
To expand the observations to more hardware types, and more importantly to understand the SoC performance evolution over time, we performed a longitudinal study on five SoCs released in different years.
We selected five smartphones equipped with different high-end SoCs (Qualcomm Snapdragon 8xx series) released from 2016 to 2021 in Table~\ref{tab:soc-models}.
We repeated our throughput experiments for two targeted workloads on those SoCs.
\input{tab-dev}
\input{fig-soc-evolution}

First, we measured the DL serving throughput using ResNet-50 and YOLOv5x.
The experimental settings were the same as $\S$\ref{sec:dl-serving}.
Based on the results in Figure~\ref{fig:soc-evol-dl}, embedded GPUs see a significant performance boost from the Snapdragon 835 (2017) to the Snapdragon 888 (2021), e.g., 3.25$\times$ higher throughput on YOLOv5x.
Embedded CPUs are also gradually improving but not as significantly as GPUs, e.g., 2.52$\times$ higher throughput on YOLOv5x.
As a consequence, the performance gap between embedded CPUs and GPUs is growing quickly.
In the same period (2017--2020), the throughput of the Qualcomm Snapdragon (835 vs. 865) on YOLOv5x improved by 2.88$\times$, while server-end NVIDIA GPUs (V100 vs. A100) only increased by 2.56$\times$ according to our extra experiments.

For video transcoding workloads, we represent throughput using the average FPS while transcoding a fixed-duration video for simplicity.
Our key observation is that video transcoding tasks also reflect the gradual improvement in DL serving SoC performance, especially for the hardware codec.
For live streaming transcoding using embedded CPUs, the throughput for V4 on the Snapdragon 865 was 1.42$\times$, 1.82$\times$, and even 2.3$\times$ higher than the 855, 845, and 835, respectively.
Such an improvement is much more impressive on SoCs' hardware codec.
The live streaming transcoding throughput of the Snapdragon 865 was 3.8$\times$ and 3.24$\times$ higher than the 835 for V4 and V5, respectively.
The performance improvements of the hardware codec not only come from the evolution of hardware but also the enhancement of software~\cite{android11-dec,android12-enc}.

\textbf{Summary.}
\textit{
Mobile SoCs have shown tremendous performance improvements in the past five years.
With Moore's Law coming to an end, CPU performance evolution is slowing down, but the capacity of co-processors on mobile SoCs (GPU and hardware codec) is still fast growing and their benefits over CPU are increasing.
To achieve a sustained performance evolution for SoC-Cluster\xspace, it is pivotal to leverage these co-processors.
}

\section{Total Cost of Ownership Analysis}\label{sec:tco-analysis}
Cost is another critical dimension to evaluate the validity of new hardware.
In this section, we conduct a TCO analysis on SoC-Cluster\xspace and traditional edge servers.
The TCO consists of two parts: the capital expenditure to purchase the servers (CapEx, with a breakdown of each hardware component) and the operational expenditure (OpEx).
We use the retailing purchase cost as the total capital expenditure.
For the OpEx, we only consider the electricity cost, as in prior work~\cite{lang2010wimpy}.
Besides the traditional edge server that mainly consists of an Intel CPU and 8 NVIDIA GPUs, we additionally add a ``virtual server'' by removing all 8 NVIDIA GPUs.
We use throughput per cost (TpC) as the normalized performance.

\input{tab-expense}

\textbf{Capital expenditure.}
As shown in Table~\ref{tab:tco}, the GPUs almost dominate the total CapEx in a traditional edge server.
For a CPU-only server, the CapEx is amortized into different hardware components.
For SoC-Cluster\xspace, 60 SoCs and 12 PCBs constitute nearly 87\% of the CapEx.
In summary, SoC-Cluster\xspace has a lower CapEx than the traditional edge server with 8 NVIDIA GPUs but costs about 2.8$\times$ more than a non-GPU edge server.

\textbf{Operational expenditure.}
We did not use complex OpEx models~\cite{shahrad2017towards} as we observed that the amortized CapEx always dominates the TCO, as we describe later.
This was also observed in a previous study from Google~\cite{google-dccomputer}.
As a result, we only report the electricity cost here.
The monthly electricity cost is calculated by the monthly power consumption (kWh) multiplied by the electricity unit cost (\$/kWh).
The monthly power consumption of all workloads relates to their average power usage.
For example, performing live streaming transcoding when fully loading all 8 NVIDIA A40 GPUs incurs 1,251 Watts of power consumption on average.
By assuming servers' average power consumption is 50\% of their peak power, monthly power consumption is calculated as $1231W*50\%*24h*30/1000=443kWh$.
For the electricity unit cost, we referred to the U.S. industrial average electricity price over one year (from August 2021 to July 2022)~\cite{elec-price}.
Thus, the monthly electricity cost directly related to computation is $\$0.0786/kWh*443kWh\approx\$35$.
The PUE (Power Usage Effectiveness) overhead, which is determined by the PUE value, also contributes to the monthly electricity cost~\cite{google-dccomputer}.
The PUE value reflects the ratio of total building power consumption to IT infrastructure power consumption.
We used a slightly higher PUE value (2.0) at the edge, compared to 1.5 at cloud data centers~\cite{google-dccomputer}.
The overall monthly electricity cost is $\$35+\$35*(2.0-1)=\$70$.

\input{tab-workloads-tpc-vertical}
In line with prior work~\cite{lang2010wimpy}, we categorized the monthly TCO into the following:
\begin{itemize}
    \item \textbf{Total CapEx amortized to 36 months.}
    By assuming a 3-year lifetime of the whole server, as in previous work~\cite{lang2010wimpy,janapa2010web,google-dccomputer}, we amortized the CapEx of each server to 36 months, as shown in Table~\ref{tab:tco}.
    \item \textbf{Monthly operational expenditure} mainly refers to the electricity cost.

    Notably, the monthly OpEx is much less than the amortized CapEx (e.g., \$70 vs. \$1,340 for the traditional edge server).
\end{itemize}

We sum the above two expenditure figures to get the monthly TCO.
We then normalize the application throughput (measured in previous sections) to the monthly TCO to present the TpC metric in Table~\ref{tab:tco-vertical}.

For live streaming transcoding, SoC-Cluster\xspace is much more cost-efficient than the traditional server with GPUs, e.g., 4.19$\times$ higher than the Intel CPU and 2.33$\times$ higher than the NVIDIA GPU.
Moreover, the Intel CPU in the non-GPU server shows a higher TpC than NVIDIA GPUs.
For archive transcoding, SoC-Cluster\xspace is less cost-efficient than the traditional edge server due to its low throughput on a single SoC and relatively high CapEx.
In most cases, processing archive transcoding using the NVIDIA GPU provides higher TpC than using other hardware.
Finally, DL serving favors data-center GPUs, whose cost efficiency is much higher than SoC-Clusters\xspace.
This is mainly attributed to the fact that NVIDIA GPUs can be fully loaded with batched DL requests and therefore deliver very high throughput.

\textbf{Summary.}
\textit{
	The capital expenditure of SoC-Cluster\xspace is close to an 8-GPU server and much higher than a CPU server.
	Under the same utilization, SoC-Cluster\xspace reduces the operational expenditure by saving electricity, but the total cost is still dominated by the capital expenditure.
    Regarding concrete workloads, SoC-Cluster\xspace delivers much higher cost efficiency than both traditional CPU and GPU servers on live streaming transcoding, but it underperforms on archive video transcoding (huge computing capacity required) and DL serving (highly optimized workloads) compared to the NVIDIA GPU.

}

\section{Video Transcoding}\label{sec:video-trans}
\input{tab-video-task}

In this section, we compare the video transcoding performance in two scenarios: live streaming transcoding and archive transcoding.
The former takes video streams with a constant frame rate, widely used in live streaming and online conferences.
The latter processes a video clip from file storage, often used as an uploading stage before delivering it to the video content provider~\cite{youtube-upload,vbench}.
The computing complexity of video transcoding leaves a lot of differences in transcoding parameters between live streaming and archive transcoding.
Following the prior video benchmark~\cite{vbench}, we kept the same target bitrate for live streaming transcoding and the same output video quality for archive transcoding.
The metadata of videos used in this section are shown in Table~\ref{tab:transcoding-task}.

\input{fig-video-on-throughput}

\subsection{Transcoding throughput}\label{sec:video-throughput}
For live streaming transcoding, since the video streams are ingested at a constant frame rate, we use the maximum video stream number that can be supported by each hardware to indicate the transcoding throughput.
For archive transcoding, a video clip with a fixed duration needs to be processed as fast as possible, therefore the throughput is calculated by how many frames can be processed per second.
The results of live streaming and archive transcoding are illustrated in Figures~\ref{fig:video-on-tps} and ~\ref{fig:video-off-tps}, respectively.
We make the following key observations.

\textit{(1) For live streaming transcoding, embedded CPUs always provide higher transcoding throughput than both the Intel CPU and the NVIDIA GPU.}
For example, SoC-Cluster\xspace can transcode 390 V1 streams simultaneously, which is 6.29$\times$ more than the Intel CPU and 2.64$\times$ more than the NVIDIA GPU per rack unit on average, respectively.
The advantage of SoC-Cluster\xspace over the Intel CPU comes from its denser CPU cores.
More specifically, the Octa-core CPU (four big cores and four LITTLE cores) of a single SoC can support timely transcoding of nine low-complexity 1080p video (V4) streams and three high-complexity 1080p video (V5) streams, which is slightly fewer than eight Intel CPU cores (14 and 5).
However, the total CPU cores of an SoC-Cluster\xspace is 6$\times$ more than our Intel CPU server.
The throughput of the NVIDIA GPU is always bounded by its hardware encoder, as observed, leaving most of its general-purpose processor under-utilized.
Indeed, improving video transcoding performance is not a primary design goal of mainstream GPUs~\cite{nvenc-performance}.

\textit{(2) For archive transcoding, embedded CPUs also outperform the Intel CPU but underperform against the NVIDIA GPU for high-complexity videos.}
Among all videos, embedded CPUs provide 4.14$\times$--8.22$\times$ higher throughput compared with the Intel CPU.
For low-complexity videos (low resolution or low entropy) that may not exhaust hardware resources, i.e., V1, V2, and V4, embedded CPUs can also perform better than the NVIDIA GPU by 1.02$\times$, 3.61$\times$, and 2.85$\times$, respectively.
However, on more complex videos, the NVIDIA GPU shows a much higher transcoding throughput.
The reasons are twofold.
First, the increasing video complexity demands more computations from the encoder paid to the hard searching problems~\cite{DBLP:journals/pieee/SullivanW05}, which is well optimized on the NVIDIA GPU with its high clock frequency.
Second, due to the different processing capacities of the embedded CPU's big.LITTLE cores, simply increasing the encoding threads to leverage all cores causes a synchronization overhead and cannot accelerate the transcoding speed.
This calls for a heterogeneity-aware video transcoder that can fully utilize the asymmetric embedded CPU cores to accelerate archive video transcoding.

\subsection{Energy efficiency}\label{sec:video-energy}
For live streaming transcoding, energy efficiency is measured as how many live video streams can be supported per Watt (streams/W).
For archive video transcoding, energy efficiency is measured as how many frames can be processed per Joule (frames/J).
The following results are performed within the hardware processing capabilities (e.g., without 8-core constraint on the Intel CPU).
Results are shown in Figures~\ref{fig:video-on-tpe} (live streaming transcoding) and ~\ref{fig:video-off-tpe} (archive transcoding).
Our key observation is that \textit{SoC-Cluster\xspace shows significant energy savings in all live streaming and some archive video transcoding tasks as compared to the Intel CPU and the NVIDIA GPU.}

\input{fig-video-tpe}
\input{fig-video-power-efficiency}

For live streaming transcoding, embedded CPUs in SoC-Cluster\xspace are 2.58$\times$--3.21$\times$ more energy efficient than the Intel CPU, and 1.83$\times$--4.53$\times$ more energy efficient than the NVIDIA A40 across different videos.
In archive transcoding, embedded CPUs are always more energy efficient than the Intel CPU, but the comparison with the NVIDIA GPU varies across videos.
More specifically, the NVIDIA GPU is less energy efficient on V2 and V4.
We find the common feature of V2 and V4 is that they have low entropy (e.g., low motion or infrequent scene transitions) and therefore require fewer encoding computations.
For those videos, we observe that the NVIDIA GPU stays in a high-power mode (high clock frequency) while embedded CPUs can complete such tasks with only a little CPU usage.
The same conclusion -- that SoC-Cluster\xspace shows higher energy efficiency on low-complexity videos -- can also be proven in the live streaming scenario: the relaxed video quality constraint makes it consume less power than the Intel CPU and NVIDIA GPU.

We also show how energy efficiency scales with dynamic workloads at the edge, which is represented by various live video stream numbers in Figure~\ref{fig:video-power-efficiency}.
We used two 1080p videos with diverse scene complexity and present the relationship between per-Watt throughput (TpE) and processed stream numbers simultaneously.
Results of the two videos show the same trend:
embedded CPUs and the Intel CPU show nearly constant energy efficiency for launching each transcoding stream, indicating a linear power increase when the workload increases.
However, the NVIDIA GPU can only process 0.018 live video streams per Watt for transcoding only one video (V4), which is 14.9$\times$ less than the Intel CPU and 40.8$\times$ less than embedded CPUs.
With the increasing stream number, the GPU's energy efficiency gradually increases but is still lower than that of embedded CPUs.
The fine-grained usage control on embedded CPU cores or SoCs shows better energy scalability when serving dynamic video transcoding workloads on SoC-Cluster\xspace.

The traditional edge server always consumed more energy than SoC-Cluster\xspace, as shown in Figure~\ref{fig:video-total-power}, mainly due to its high static power draw.
The NVIDIA GPU also showed a power consumption spike with light incoming workloads, but its power consumption increased slowly and might be less than performing the same workloads on the Intel CPU.
Taking V5 as an example, the total power consumption of the traditional edge server reached 467 Watts when there was one transcoding stream using the NVIDIA GPU, while only increasing to 424 Watts using the Intel CPU.
The gradually increasing processed stream number makes the server power consumption using the Intel CPU more than that of using the NVIDIA GPU (e.g., more than 9 V5 streams).
The above results also provide insightful scheduling trade-offs when there are only traditional edge servers to serve video transcoding workloads.

\subsection{Hardware-accelerated video transcoding on mobile SoCs}\label{sec:video-bitrate}
Mobile SoCs are equipped with hardware codecs that handle most of the video encoding/decoding on smartphones, e.g., the videos captured from cameras.
We only tested their performance for live streaming transcoding, because the APIs exposed by Android MediaCodec~\cite{mediacodec} cannot control the video quality for a fair comparison in archive transcoding.
We used LiTr~\cite{litr} to perform hardware-accelerated video transcoding instead, as FFmpeg only supports hardware decoding but not encoding for Android~\cite{ffmpeg-hwaccel}.
We compare the performance and transcoding behavior of the hardware codec with embedded CPU in terms of the throughput per rack unit (TpS), energy efficiency (TpE), output bitrate, and Peak Signal-to-Noise Ratio (PSNR)~\cite{psnr} below.

Our key observation is that \textit{the hardware codec of the mobile SoC provides a further significant improvement over its CPU.}
As Figure~\ref{fig:video-mediacodec-tps} shows, using the hardware codec increases the maximal number of supported live video streams per rack unit by 1.07$\times$--3$\times$.
The increase in energy efficiency is even more impressive, as shown in Figure~\ref{fig:video-mediacodec-tpe}.
For videos with low-complexity scenes (V1, V2, and V4), hardware transcoders in SoC-Cluster\xspace could support on average 2.5$\times$ more streams per Watt than embedded CPUs.
When transcoding high-entropy and high-resolution videos (i.e., V3, V5, and V6), embedded CPUs will consume more power for encoding, while delegating transcoding workloads to the hardware codec can significantly improve energy efficiency by 4.7$\times$--5.5$\times$.

\input{fig-mediacodec}
\input{fig-bitrate-quality}

To demystify whether the hardware codec in SoC-Cluster\xspace could handle live streaming transcoding workloads or not, we also present the output video bitrate, which is one of the most critical metrics affecting user experience.
As shown in Figure~\ref{fig:video-mediacodec-bitrate}, we use red dash lines to represent the target bitrate for each transcoding task.
The key observation is that, \textit{in most cases, the hardware codec can meet the bitrate constraint, but it's hard for it to meet a relatively low bitrate cap.}
For example, setting the target bitrate to 90.5 Kbps for V2 will make the encoder create a higher bitrate output (even higher than the origin video stream).
Such unexpected behavior goes against the intention of most transcoding services to compress a video stream.
The same behaviors were confirmed by our additional experiments on other videos by setting ultra-low bitrates.
This is possibly a design trade-off decided by the mobile SoC vendor for energy efficiency and chip size.

\subsection{Transcoding quality}\label{sec:video-psnr}
Transcoding quality is how the output video is perceived by consumers.
The live streaming transcoding experiments were performed with a fixed bitrate target.
However, due to the nuances at both the software (libx264 vs. MediaCodec vs. NVENC) and hardware (CPU vs. GPU vs. ASIC) levels, their video quality could differ observably even under the same bitrate constraint.
We saved the live streaming transcoding output to files during previous experiments and used PSNR to represent the video quality (higher is better).

As shown in Figure~\ref{fig:video-enc-quality},
\textit{the software encoder using embedded CPUs can preserve almost the same video quality as the Intel CPU and NVIDIA GPU, while videos generated by SoC-Cluster\xspace's hardware codec have sightly poorer quality than others.}
The general-purpose computing unit (i.e., the embedded CPU and Intel CPU) and software encoders with the same transcoding configurations always result in the same video quality.
However, videos generated by MediaCodec have about 1.35\%--14.77\% lower PSNR values compared with those generated by libx264 using embedded CPUs.
This is caused by the loose quality and bitrate requirements of mobile encoders~\cite{DBLP:conf/asplos/RanganathanSCDG21}.
To further investigate what bitrate should be set to achieve the same video quality using MediaCodec, we manually tuned the target bitrate and then compared it with the origin video to get the PSNR value.
From our additional experiment, we find that simply loosening the target bitrate constraint using MediaCodec still couldn't meet the same video quality as libx264.
As such, if the quality loss incurred by MediaCodec is not bearable, it's better to choose embedded CPUs for video transcoding.

\subsection{Network bound analysis}
If the SoC-Cluster\xspace is fully occupied with video transcoding workloads, will the network capacity become the bottleneck?
Our analysis in Table~\ref{tab:transcoding-task} says not.
Among the six videos tested, network usage will slightly exceed the network capacity of PCB (1 Gbps) only when fully leveraging the embedded CPUs and hardware codecs together to transcode V5 video.
In practice, it is hard to fully load the embedded CPU and hardware codec simultaneously due to the software delegation daemon process or thermal control.
Considering the whole SoC-Cluster\xspace, under no circumstances will the ESB (20 Gbps) bottleneck.
However, with a denser SoC integration in a future-generation SoC-Cluster\xspace, the network could be a limiting factor and needs to be enhanced.

\subsection{Summary}
Embedded CPUs in SoC-Cluster\xspace provide both higher throughput per rack unit and greater energy efficiency than the traditional Intel CPU (up to 7.7$\times$ and 3.2$\times$ respectively) and NVIDIA GPU (up to 6.08$\times$ and 4.53$\times$ respectively) we used in experiments for live streaming transcoding.
The fully fledged software stack makes it possible to seamlessly migrate current transcoding services to SoC-Cluster\xspace.
Although the hardware codecs of SoC-Cluster\xspace deliver even higher throughput and energy efficiency than embedded CPUs,
they sacrifice a portion of video quality (1.35\%--14.77\%) and cannot guarantee a relatively low bitrate.
The software stack of the mobile SoC's hardware codec is also not readily integrated into current transcoding infrastructure due to a lack of flexibility.
Nonetheless, the embedded CPUs show high and solid potential for general video transcoding services, and the hardware codecs provide even higher profits for a relatively narrower scope of video transcoding scenarios.
\subsection{SoC-Clusters in the wild}\label{sec:soc-wild}

\input{fig-usage-variation}
The SoC-Cluster\xspace used in this work has been on sale for more than a year.
Up to September 2022, more than 10,000 had been shipped and deployed in the wild.
The majority of customers are edge service providers.
From collaborating with a leading worldwide edge service provider, we know that \textit{deployed SoC-Clusters\xspace are mostly used to serve one particular application: cloud gaming}, as Android-native games like Genshin Impact~\cite{genshin} can be seamlessly supported by the mobile SoC without any revision.
The cloud gaming deployed on edges enables wimpy or low-power devices to run resource-hungry games with a good user experience.
Developers are now able to access the resources of those SoC-Clusters\xspace through either a virtualized or native context, just like traditional cloud services.
In total, our shipped SoC-Clusters\xspace are estimated to serve millions of game sessions per day.

However, according to the runtime traces collected from our customers' deployed SoC-Clusters\xspace, we conclude that \textit{those servers experience workloads with relatively low utilization and high dynamics.}
In general, the resource usage of all deployed SoC-Clusters\xspace is below 20\%.
Figure~\ref{fig:usage-var} shows the temporal network traffic of one specific SoC-Cluster\xspace randomly sampled from real-world edge sites.
As observed, the gap between its highest and lowest outbound traffic is up to 25$\times$.
Such an observation is consistent with a recent empirical study on large-scale edge clouds~\cite{ens}.
The reason is that edge applications are mostly user-centric and thus highly relate to user activities.
Motivated by this observation, this work explores how efficiently SoC-Cluster\xspace can serve more general workloads beyond cloud gaming.

redpajama_set_name: RedPajamaArXiv


Video Resolution - 4K@60Hz (4:2:0), 4K @30Hz (4:4:4), 8–bit, full HD, 3D Deep Color across all lengths.
High Data Transfer Rate - Up to 10.2Gbps.
Embedded Audio - PCM 8-channel, Dolby Digital True HD and DTS–HD Master Audio.
High–Quality Connectors - 24K gold-plated, corrosion-resistant and best possible connectivity.
No External Power - Powers the active units via the HDMI connector.
Small Bending Radius - Only 6mm (0.24").
Jacket Construction - Low smoke &halogen free.


The Science Numeracy and Academic Proficiency (SNAP) Centre is a free, drop-in academic support centre dedicated to first-year students.
All first-year Bachelor of Science students are welcome to access the SNAP Centre for help with their first-year mathematics, physics, biology and chemistry coursework (and possibly for other courses, upon request).
What academic support does the SNAP Centre offer?
The SNAP Centre offers first-year students free drop-in access to tutors who provide academic support, including assistance with studying and assignments, for specific/targeted first-year Science courses. SNAP Centre tutors are all former first-year students themselves, and they have the experience and knowledge to help you succeed.
We have also developed 12 different, short "mini numeracy workshops" (approx. 15-20 minutes in length) to provide assistance in areas of basic mathematical tools. These numeracy workshops cover topics that will be used frequently in first-year science courses and that students may have forgotten from their high school courses. They can be downloaded below and are designed to be done either at home on-your-own or with a tutor in the SNAP Centre. Students are free to ask tutors to schedule a time to run through a workshop with them, or ask for assistance on specific things from a workshop they've worked through on their own.
Who are the SNAP Centre tutors?
SNAP Centre tutors are all upper-year Saint Mary's undergraduate science students who have demonstrated expertise in mathematics, chemistry, biology and/or physics. All tutors are Dean's List Science students who have achieved a high degree of academic success; Dean's List students are those with an annual average of A- or higher (GPA > 3.70). At any given time during the Centre's hours, two tutors will be present.
The SNAP Winter 2019 Schedule provides the names of the tutors on duty during each 30-minute time slot of the week. It also indicates which subjects the tutors are comfortable answering questions in, with B meaning biology, C chemistry, M math, and P physics. Tutors with more than one subject letter have their strongest subject listed first).
If you have any questions about the SNAP Centre, please speak to one of the tutors on duty or contact Dr. Lori Francis, Associate Dean of Science - Student Affairs (Acting).


text:
Home Education Latest Shrewsbury school budget proposal closes gap a bit
Latest Shrewsbury school budget proposal closes gap a bit
By Joan Goodchild Community Reporter
Shrewsbury – Shrewsbury School Superintendent Dr. Joseph Sawyer presented a revised district budget for fiscal year (FY) 2012 at the March 9 School Committee meeting. Although the proposal is a 2.98 percent increase over last year, the $48.5 million proposal is still $2.1 million more than the $46.3 million recommendation put forth by Town Manager Daniel Morgado.
Last month, Sawyer presented a potential budget scenario of $48.9 million, which is a 3.9 percent increase over FY 2011's budget of $47.1 million.
Prior to the economic downturn, Shrewsbury schools saw budget increases in the 6 to 7 percent range each year, Sawyer said.
The revised proposal contained many adjustments, but Sawyer cautioned that the document is never "static" because of changes that occur over the course of a year that cannot always be predicted. He laid out four priorities when crafting the latest proposal; including maintaining current personnel and programming levels, responding to ongoing mandates, making strategic investments to mitigate long-term cost drivers, and preventing further erosion of resources, especially textbooks, curriculum materials and technology.
Cost increases in the budget proposal include $35,000 for professional development, $207,197 for transportation, $210,000 for textbook and curriculum materials, and $400,000 for technology investments. While a recommendation for technology funding had been slimmed down from an initial recommendation of $455,000, Sawyer noted the money was desperately needed simply to move the district forward with regard to current standards. He informed the board the district was replacing computers at a rate of one every 15 years and that roughly two-thirds of the district's computer hardware cannot run current operating systems.
"We'se making the best use of all these machines as we can, but we'se fallen behind the curve in terms of our replacement plan," he said.
The personnel increases include salaries for a half-day kindergarten teacher and an aide for the classroom. Kindergarten classroom student levels are higher than where the district would like them to be currently, said Sawyer, who noted classes are around 20 or 21 students per class. Other personnel additions include a special education support secretary, a special education technology specialist and an advanced math coach at Oak Middle School, among others.
School Committee members thanked Sawyer and his administration for their hard work on the budget revisions, but expressed frustration at the seemingly ongoing budget issues that continue to plague the town each year.
"What's the end game?" asked member Mark Murray. "If you give people a plan, a vision, a little hope, they can suck it up for a little while. But if it's the same thing year in, year out, it gets old. What's the plan?"
"I think that's a question we should own," member B. Dale Magee said in response to Murray's comments. "Although we can's be guaranteed a certain amount of money every year for the school department, we have to argue it out each year. To me it gets a little easier if we have a blueprint for each area."
"It is a responsible budget that will meet the needs of students," said member Erin Canzano. "It's good enough, but it's not great. But it will keep us moving in the right direction and our direction is very good."
Previous articleRita A. Millett, 78
Next articleAthletes of the Week – Nickole Soto and Kevin Boucher
Fay School to host reading and literacy workshop

redpajama_set_name: RedPajamaCommonCrawl


Another Realtor was showing buyers a development in Charlottesville and an agent was onsite holding an open house. As the buyers looked around, the Buyer's Agent (BA) and the sellers' agent (SA) got to talking.
BA: This road in front of the development:Â It goes into a neighborhood, correct?
SA: Yes, but you don't want to take them that way. It looks much better the way you came in.
BA: I understand, but I want them to see what's around this property, going both ways.
SA: If you go back the way you came, you can show them how close the grocery store is.
BA: Thanks, but I can do both.
SA: You should really go out this way – it's much better.
When the Buyers and the Buyers' Agent got in the car – the wife asked, "Which way are we leaving?" To which the Buyer's Agent replied – "this way – I want to make sure that you see what's around the house. Then, I'll show you how close this is to grocery store, restaurant, etc." Turns out, the wife had heard the Realtors' conversation.
1 – Your Buyer Agent is supposed to look out for your best interests.
2 – Had this been a Dual Agency situation, do you think the buyers would have been more or less likely to see the "other" areas?
3 – Realtors don't have an obligation to assess an area's safety level, but they do have an obligation to be the "source of the source" – provide references by which the clients can determine for themselves.
4 – Recognize two things – not all Realtors are focused solely on "making the sale"; but some are.
I have been a dual agent on more than one occasion. Its not my favorite position to be in, but I certainly would have driven the couple both ways.
Suppose the agent in this situation was both a buyer agent for this couple as well as the listing agent for this owner of this partucular house.
Does he have an obligation to recommend designated agency?
At what point might it be appropriate for him to recommend that designated agency come into play?
1. Does he show it to them first?
4. Does he wait until an offer has been prepared?
If he does it after any of the five time periods above, is he acting as a dual agent himself until designated agency is invoked?
At what point should the "designated agent" prefer to get involved?
Who showed the potential buyers the other way to access the property?
2. Since there is no current way to check the blatant conflict of interest, anyone who can't use his common sense and figure that out dual agency, ed note: name removedeven the practice of using a RE agency to market new homes–are all bad ideas, the buyer deserves the screwing he'll probably get.
3. If I'm buying a house I'm damn sure going to explore the areas around the neighborhood ON MY OWN, including but not limited to looking at all the roads in and out. Anyone who needs a Realtor to figure that out had better hope their Realtor cares more about their clients' well-being than 3%. Common sense = an oxymoron.
Mark – I agree with almost everything you say. The concept of singla agent Dual Agency is a legacy that needs to go away.
I'd be curious to know also if any companies out there, big or small, offer bonuses to their agents for selling one of their own listings, whether new or resale.
A lot of buyers just don't understand the value of agency, and don't get the opportunity to learn from the first agent they meet what buyer's agency is.
All the agents wanted my business and Default Position.
What do we have to do to rid ourselves of Single Agent Dual Agency?


18/04/2013 · Today I´m Gonna show you how to convert file or folder in (Rar) or in (ZIP) 1.Choose the folder you want to convert and right click on the folder and click add the file... or add the "*.rar"... Here's how to open RAR files with some quick access software. We will walk you through the process, step by step. We will walk you through the process, step by step.
It can convert from around 45 different types including 7-Zip, LZH, CAB, ZIP, ARJ, ACE, RAR, TAR, UHARC and ISO. It converts to around 20 different formats of the most popular types, and also supports WinRAR and WinACE if you own them.... It can convert from around 45 different types including 7-Zip, LZH, CAB, ZIP, ARJ, ACE, RAR, TAR, UHARC and ISO. It converts to around 20 different formats of the most popular types, and also supports WinRAR and WinACE if you own them.
But when I downloaded some new hair and clothes they downloaded in a .rar file and they won't show up in my game So I figued out that they need to be a .package file in order to show up in my game so how do you convert .rar into .package ?
To create an RAR files directly from the WinRAR program, simply use the Add feature within the software. How To Open a RAR File Open the archive by double-clicking its icon.


text:
Michelle Chan is an Australian diplomat who served as Ambassador to Myanmar from May 2008 until January 2011.  Chan served as National Security Adviser to former Prime Minister Scott Morrison.  In April 2020, Chan was promoted to deputy secretary in the Office of National Intelligence. She is currently the Deputy Secretary of the Australian Department of Foreign Affairs and Trade.

Chan earned a Bachelor of Arts (Jurisprudence) and Bachelor of Laws from the University of Adelaide, and a Master of Arts and Master of Laws (International Law) from the Australian National University.

References

Australian women ambassadors
Ambassadors of Australia to Myanmar
Australian National University alumni
Adelaide Law School alumni
Living people
Year of birth missing (living people)
Morrison Government

redpajama_set_name: RedPajamaWikipedia


text:
Magazine Last Word
Postcapitalism? Better to make up your own mind...
Written by Robert McCaffrey Editorial Director, Global Cement Magazine
I've recently read a book called 'Postcapitalism,' by Paul Mason, which was subtitled 'A guide to our future.' The blurb on the front, written by Irvine Welsh, stated that it was 'The most important book about our economy and society to be published in my lifetime.' Having completed the book, I'm disinclined to believe the veracity of Mr Welsh's statement. However, the book did have some interesting points to make, which might impact on everyone involved in the global cement and building materials industry.
How many bird species are there at your cement plant? Join the Global Cement Bird Watch!
I was recently visiting the Sant Vincenç dels Horts cement plant of Cementos Molins S.A., on the field trip of the Global CemFuels Conference in nearby Barcelona, when I noticed a small dark bird perching on a stump, just between the preheater tower and the main petcoke storage building. The bird bobbed up and down a few times and then flew into the nearest tree. It reminded me of a wagtail, only it was much darker. I didn't get a great look at it. Then, at the end of the tour, we congregated at the striking concrete-built headquarters building, and I noticed a tiny bird singing for all it was worth in the coniferous tree just outside the entrance. It was hard enough to even spot it, let alone identify it and it soon flew off. However, seeing these two birds in such an industrial location made me think.
Stealing from the future/Being kind to your future self
How much debt is the right amount? My guess is that everyone reading this column will have a slightly different answer. Some might say, 'Neither a borrower nor a lender be,' or words to that effect, suggesting that their level of debt is zero. Some will no doubt have too much debt - where the repayments are not affordable. Most readers will have some level of debt (on a mortgage, car payments, credit cards etc) and will find that the level of repayments is currently affordable. Hopefully no readers are subject to the depradations of usury: the practice of making unethical or immoral monetary loans that unfairly enrich the lender1.
How 20 years have flown by - here's looking towards 2037!
So, 'Global Cement' is 20 years old. The fore-runner of this magazine - Asian Cement and Construction Materials Magazine - was first published in January 1997. The magazine was later amalgamated with other titles to become GCL: Global Cement and Lime Magazine and finally became simply Global Cement Magazine in October 2006. The original magazine was created in an attic on a computer with a then-huge memory of 1Gb. Each issue took up 500Mb of space, and so had to be separately archived each month, to make room for the next month's issue. We used huge SyQuest disk drives to send data to the printer. We scanned in real photographs using a scanner that cost around Euro/$/£3000 - something you could buy now for less than £300. When that first issue was published, I was a young man of 29, with a full head of hair, and was yet to meet my wife.
Artificial intelligence (AI) is here. It's probably in your pocket right now, or perhaps on your table at home. It might be in the next call centre that you telephone, or behind a pop-up 'Would you like to chat?' screen next time you shop for something online.
Conferences >>
Global Ash
Global CemBoards
Global CemFuels
Global CemProcess
Global FutureCem
Global Gypsum
Global GypSupply
Global Insulation
Global Slag
Global Synthetic Gypsum
Global Well Cem
Magazine >>
Register for free copies
Websites >>
Pro Global Media
Social >>
© 2021 Pro Global Media Ltd. All rights reserved.


The only Pre-Romanesque Asturian church with a bell tower.
Pre-Romanesque church probably built in the time of King Alfonso II. Destroyed in 1936, its ruins were identified in 1954 by Joaquín Manzanares, subsequently undergoing a controversial reconstruction in 1958.
The present church has a single nave, triple apse and narthex at the west end. In the north and south walls there are two compartments that communicate with the interior of the church.
In front of the church, in its southwest corner, there is a rebuilt bell tower.
The exterior of the church is decorated with brick arches and rebuilt latticework. However, there are original elements in the church, such as the triple-arched alfiz that illuminates the hidden chamber located above the central chapel.
The interior of the church has a wooden ceiling, except for the chapel of the nave, which is covered with a vault. Like other Pre-Romanesque churches, it was decorated with murals, some traces of which remain above the arch on the south side.


About the Author: Amir Mir, a former editor of Weekly Independent, is one of Pakistan`s foremost investigative journalists whose insightful writings on terrorism, Islamic militant groups and their links with the country`s military and intelligence establishment have repeatedly drawn the ire of the establishment. He started his journalistic career in 1988 with the English daily The Frontier Post, Lahore, and has worked with numerous leading newspaper organisations in Pakistan and abroad. He has earlier authored two books on the subject of militant Islam and terrorism. His first book, ``The True Face of Jehadis`` (Mashal Books/Roli Books) was published in 2006; and his second book, ``The Fluttering Flag of Jehad`` (Mashal Books) was published in 2008.


text:
Judge rules gruyere cheese name not owned by Europe's Gruyere region
Robert Besser
A U.S. federal judge has ruled that Gruyere cheese does not have to come from the Gruyere region of Europe to be sold under the name "Gruyere"
After the federal Trademark Trials and Appeals Board denied an application for trademark protection, a consortium of Swiss and French cheesemakers sued at U.S. District Court in Virginia
However, the U.S. Dairy Export Council and other groups opposed the trademark protection, noting that American consumers understand the gruyere name to be generic
RICHMOND, Virginia: A U.S. federal judge has ruled that Gruyere cheese does not have to come from the Gruyere region of Europe to be sold under the name "Gruyere."
After the federal Trademark Trials and Appeals Board denied an application for trademark protection, a consortium of Swiss and French cheesemakers from the region around the town of Gruyeres, Switzerland, sued at U.S. District Court in Virginia.
The consortium stressed that gruyere has been produced in the region since the early 12th century, and cheese made outside the region cannot truly be called the same name, similar to the argument that champagne can only be used in naming sparkling wines from the Champagne region of France.
However, the U.S. Dairy Export Council and other groups opposed the trademark protection, noting that American consumers understand the gruyere name to be generic.
Last week, District Judge T.S. Ellis ruled against the European consortium, deciding that American consumers do not associate the gruyere name with cheese made specifically from that region, adding that while similar trademark protections have been granted to Roquefort cheese and Cognac brandy, Ellis said the same case cannot be made for gruyere.
He also highlighted that the Food and Drug Administration regulates the use of the gruyere name and no requirements specify its place of origin.
The ruling has been appealed by the consortium.
According to Shawna Morris, senior vice president for trade policy with the U.S. Dairy Export Council, the legal battle over gruyere is one in a growing number of efforts in Europe to gain international trade protection for various products, including gorgonzola, asiago and feta cheese.
The consortium has not replied to an email seeking comment, but in court papers its lawyers argued that Swiss and French gruyere is "painstakingly made from local, natural ingredients using traditional methods that assure the connection between the geographic region and the quality and characteristics of the final product."
Allowing others to use the gruyere name would confuse American consumers, they claimed.
Get a daily dose of Chicago Chronicle news through our daily email, its complimentary and keeps you fully up to date with world and business news as well.
Publish news of your business, community or sports group, personnel appointments, major event and more by submitting a news release to Chicago Chronicle.
Fair in Chicago/Midway
5G cellular could be 'catastrophic' in aviation: US airline officials
CHICAGO, Illinois: According to U.S. airline officials, the introduction of a new 5G service could cause U.S. air traffic to ...
Ryan Hartman scored twice and Marcus Foligno and Calen Addison each tallied a goal and assist to boost the visiting ...
Giannis Antetokounmpo, Khris Middleton and Bobby Portis all notched double-doubles as the host Milwaukee Bucks beat the Chicago Bulls 94-90 ...
The Phoenix Mercury will name Las Vegas Aces assistant coach Vanessa Nygaard as their next head coach, The Athletic and ...
Both teams will be without some key personnel when Florida hosts Vanderbilt in a battle of mid-pack Southeastern Conference teams ...
India in touch with US authorities over loss of four Indian nations near Canada-US border
Washington [US], January 21 (ANI): Indian ambassador to the US, Taranjit Singh Sandhu, on Friday, informed that they are in ...
© Copyright 1999-2022 Chicago Chronicle. All rights reserved.


Home > Blog > Happy Fall Session!
Our Fall session began yesterday, with new group classes, a new promo, more classes to choose from… lots of new stuff to get you to pick up your running shoes and turn off Netflix. It's a great time to get moving—make the most of it!


Pass along to any friends or family members who may be going through this milestone.
According to the author of 3 Day Potty Training, Lora Jensen, kids are ready to be potty trained much earlier than you think. Lora, who is also a mother of six, says that kids can be trained to use the bathroom as early as 22-months-old.
She goes on to add that children are very curious and can respond to the right cues.
In an interview with ABC 6, Lora provided three tricks to help with the potty training process.
When you're potty training a child, make sure to focus all of your attention on the task at hand.
That means getting rid of any and all distractions, staying home and making potty training your full-time job, so to speak.
You can put down your phone, turn off the TV, and even set aside your Fitbit for three days. In the grand scheme of things, three days is just that—a short blip in your life.
"It really means staying home and doing nothing. Just focusing all your time on that child for three days," Jensen says.
When you dedicate 100 percent of your attention to getting your child potty trained it will be a lot easier and less stressful in the long run. In her e-book, Jensen suggests making potty training a special bonding time for your child—make this a nurturing experience, not a stressful one.
So, you work full time. What does this mean for you? Consider taking a Friday or a Monday off so that you can tack an extra day onto your weekend.
Another crucial component of successful potty training is teaching your kids to take control. They need to tell you when they have to go.
Up until potty training starts, kids have always had a built in bathroom so there was never reason for them to say anything to you. Reminding children that they need to go to the bathroom isn't always effective. Instead, give them the reins and encourage them to tell you when they need to go themselves.
Self-awareness empowers them to make the decision themselves. Jensen also suggests praising your kids and giving them small rewards each time they use the toilet.
Jensen also shares tips in her book for getting your daycare provider on board with the training and tips for traveling, swim days and public restrooms.
"If the child feels that they have control over the process, they're not going to fight it," Jensen says.
Get rid of all the diapers and pull-ups. This prevents any excuses and helps the parents focus and the kids quickly learn.
If you want some real proof, look at what this reader posted on Lora's Facebook page.
"Everyone is always shocked that my kids are fully potty trained by the time they are two years old and in such a fast manner. I'm your biggest fan," the reader wrote (you can view her full message below).
Though Jensen's method has plenty of fans, it's definitely not a guarantee. But, even if it can shorten the amount of time it takes for your child to be fully potty trained, it's worth a shot, right?
Jensen says that even if you've already started potty training, her method will cut down the time your training takes.
"I cannot guarantee that your child will be potty trained at the conclusion of the 3 days, but together we can make a valiant go of it," Jensen says on her website.