text:
Ministry of Domestic Trade and Consumer Affairs (Malaysia)
The Ministry of Domestic Trade and Consumer Affairs (Malay: Kementerian Perdagangan Dalam Negeri dan Hal Ehwal Pengguna), abbreviated KPDNHEP, is a ministry of the Government of Malaysia that is responsible for domestic trade, co-operatives, consumerism, franchise, companies, intellectual property, competition, controlled goods, price control, pyramid scheme, consumer rights, trader.
Kementerian Perdagangan Dalam Negeri dan Hal Ehwal Pengguna
(KPDNHEP)
Coat of arms of Malaysia
27 October 1990 (30 years ago) (1990-10-27)
Preceding agencies
Ministry of Domestic Trade, Co-Operation And Consumerms
Ministry of Entrepreneur and Co-operatives Development
No. 13, Persiaran Perdana, Precinct 2, Federal Government Administrative Centre, 62623 Putrajaya
MYR 634,350,000 (2017)
Minister responsible
Alexander Nanta Linggi, Minister of Domestic Trade and Consumer Affairs
Deputy Minister responsible
Rosol Wahid, Deputy Minister of Domestic Trade and Consumer Affairs
Ministry executives
Hasnol Zam Zam Ahamd, Secretary-General
Alauddin Sidal, Deputy Secretary-General (Domestic Trade)
Vacant, Deputy Secretary-General (Consumerism and Management)
www.kpdnhep.gov.my
Ministry of Domestic Trade and Consumer Affairs on Facebook
1.1 Federal agencies
2 Key legislation
3 Policy Priorities of the Government of the Day
4 Programmes
OrganisationEdit
Minister of Domestic Trade, Co-operatives and Consumerism
Under the Authority of Minister
Competition Appeal Tribunal
Deputy Minister of Domestic Trade, Co-operatives and Consumerism
Under the Authority of Secretary-General
Policy and Strategic Planning Division
Enforcement Division
Corporate Communication Unit
Petroleum Vehicles Subsidy Management Division
Internal Audit Division
Integrity Division
Delivery Management Office
Key Performance Indicator Unit
Ministry of Domestic Trade, Co-operatives and Consumerism State Offices
Deputy Secretary-General (Domestic Trade)
Franchise Development Division
Domestic Trade Division
Services Industry Division
Business Development Division
Co-operative Development Division
Deputy Secretary-General (Consumerism and Management)
Consumerism Movement Division
Research and Policy Division
Administration Services and Finance Division
Tribunal for Consumer Claims
Consumerism Standards Division
Information Management Division
Account Division
Federal agenciesEdit
Co-operative College of Malaysia, or Maktab Koperasi Malaysia (MKM). (Official site)
Companies Commission of Malaysia, Suruhanjaya Syarikat Malaysia (SSM). (Official site)
Malaysia Co-operative Societies Commission, or Suruhanjaya Koperasi Malaysia (SKM). (Official site)
Malaysia Competition Commission (MyCC), or Suruhanjaya Persaingan Malaysia. (Official site)
Intellectual Property Corporation of Malaysia (MyIPO), Perbadanan Harta Intelek Malaysia. (Official site)
Perbadanan Nasional Berhad (PNS). (Official site)
Bank Kerjasama Rakyat Malaysia Berhad (Bank Rakyat). (Official site)
Key legislationEdit
The Ministry of Domestic Trade, Co-operatives and Consumerism is responsible for administration of several key Acts:[1]
Ministry of Domestic Trade, Co-operatives and Consumerism
Weights and Measures Act 1972 [Act 71]
Control of Supplies Act 1961 [Act 122]
Petroleum Development Act 1974 [Act 144]
Hire-Purchase Act 1967 [Act 212]
Petroleum (Safety Measures) Act 1984 [Act 302]
Exclusive Economic Zone Act 1984 [Act 311]
Direct Sales and Anti-Pyramid Scheme Act 1993 [Act 500]
Franchise Act 1998 [Act 590]
Consumer Protection Act 1999 [Act 599]
Optical Discs Act 2000 [Act 606]
Electronic Commerce Act 2006 [Act 658]
Price Control and Anti-Profiteering Act 2011 [Act 723]
Trade Descriptions Act 2011 [Act 730]
Companies Commission of Malaysia
Kootu Funds (Prohibition) Act 1971 [Act 28]
Companies Act 1965 [Act 125]
Registration of Businesses Act 1956 [Act 197]
Companies Commission of Malaysia Act 2001 [Act 614]
Langkawi International Yacht Registry Act 2003 [Act 630]
Limited Liability Partnerships Act 2012 [Act 743]
Intellectual Property Corporation of Malaysia
Trade Marks Act 1976 [Act 175]
Patents Act 1983 [Act 291]
Copyright Act 1987 [Act 332]
Industrial Designs Act 1996 [Act 552]
Layout-Designs of Integrated Circuits Act 2000 [Act 601]
Geographical Indications Act 2000 [Act 602]
Intellectual Property Corporation of Malaysia Act 2002 [Act 617]
Co-operative College of Malaysia
Co-operative College (Incorporation) Act 1968 [Act 437]
Malaysia Co-operative Societies Commission
Co-operative Societies Act 1993 [Act 502]
Malaysia Co-operative Societies Commission Act 2007 [Act 665]
Bank Kerjasama Rakyat Malaysia Berhad
Bank Kerjasama Rakyat Malaysia Berhad (Special Provisions) Act 1978 [Act 202]
Malaysia Competition Commission
Competition Act 2010 [Act 712]
Competition Commission Act 2010 [Act 713]
Policy Priorities of the Government of the DayEdit
National Intellectual Property Policy
Fair Trade Practices Policy
National Consumer Policy
National Co-operatives Policy
ProgrammesEdit
1Malaysia Pengguna Bijak
Beli Barangan Buatan Malaysia
1Malaysia 1Harga
Friends of KPDNKK
Antipencatutan
Kedai Rakyat 1Malaysia
Menu Rakyat 1Malaysia
Pemasaran Produk Usahawan IKS di Pasar Raya
Legal FrameworkEdit
The Federal Constitution allows Parliament to make laws related to trade, commerce and industry that include:
production, supply and distribution of goods; price control and food control; adulteration of foodstuffs and other goods;
imports into, and exports from, the Federation;
incorporation, regulation and winding up of corporations other than municipal corporations (but including the municipal corporation of the federal capital); regulation of foreign corporations; bounties on production in or export from the Federation;
insurance, including compulsory insurance;
patents; designs, inventions; trade marks and mercantile marks; copyrights;
establishment of standards of weights and measures;
establishment of standards of quality of goods manufactured in or exported from the Federation;
auctions and auctioneers;
industries; regulation of industrial undertakings;
subject to item 2(c) in the State List: Development of mineral resources; mines, mining, minerals and mineral ores; oils and oilfields; purchase, sale, import and export of minerals and mineral ores; petroleum products; regulation of labour and safety in mines and oilfields;
factories; boilers and machinery; dangerous trades; and
dangerous and inflammable substances.
Minister of Domestic Trade and Consumer Affairs (Malaysia)
Deputy Minister of Domestic Trade and Consumer Affairs
^ http://www.kpdnkk.gov.my/kpdnkkv3/index.php?option=com_content&view=article&id=64:&catid=89:&Itemid=101&ml=1&lang=en Acts
Wikimedia Commons has media related to Ministry of Domestic Trade and Consumer Affairs (Malaysia).
Retrieved from "https://en.wikipedia.org/w/index.php?title=Ministry_of_Domestic_Trade_and_Consumer_Affairs_(Malaysia)&oldid=953988947"
This page was last edited on 30 April 2020, at 02:53 (UTC).


text:
FORECAST | Blowing & Drifting Snow Continues
LIVE BLOG | Winter storm hits Northeast Ohio: Real-time updates, closings and more
Cleveland, OH »
Education Station
A Turning Point
GO-HIO
Leon Bibb
LeBron James Family Foundation
Go! 4:30-7 a.m.
It's About You
What's New: 5 p.m.
What Matters Most: 6 p.m.
Front Row: 7 p.m.
What's Next: 11 p.m.
Official: White House didn't want to tell seniors not to fly
The Centers for Disease Control and Prevention submitted a plan to recommend elderly and physically fragile Americans not to fly.
Author: Associated Press
Published: 12:13 AM EST March 8, 2020
Updated: 12:13 AM EST March 8, 2020
NEW YORK — A federal official says the White House overruled health officials who wanted to recommend that elderly and physically fragile Americans to be advised not to fly on commercial airlines because of the new coronavirus.
The Centers for Disease Control and Prevention submitted the plan this week as a way of trying to control the virus, but White House officials ordered the air travel recommendation to be removed.
That's according to a federal official with direct knowledge of the plan who did not have authorization to talk about the matter and spoke to The Associated Press on condition of anonymity.
Vice President Mike Pence's press secretary Katie Miller said that "it was never a recommendation to the Task Force" and called the AP story "complete fiction."
RELATED: Italy quarantines north including Milan, Venice over virus
RELATED: Coronavirus live updates: Italy quarantine includes Venice and Milan
WKYC would like to send you push notifications about the latest news and weather.

redpajama_set_name: RedPajamaCommonCrawl


text:
\section{Introduction}
Intelligent edge devices with rich sensors (e.g., billions of mobile phones and IoT devices)\footnote{\url{https://www.statista.com/statistics/330695/number-of-smartphone-users-worldwide/}} have been ubiquitous in our daily lives. These devices keep collecting \emph{new} and \emph{sensitive} data through the sensor every day while being expected to provide high-quality and customized services without sacrificing privacy\footnote{\url{https://ec.europa.eu/info/law/law-topic/data-protection_en}}.
These pose new challenges to efficient AI systems that could not only run inference but also continually fine-tune the pre-trained models on newly collected data (i.e., on-device learning).

Though on-device learning can enable many appealing applications, it is an extremely challenging problem. First, edge devices are \emph{memory-constrained}. For example, a Raspberry Pi 1 Model A only has 256MB of memory, which is sufficient for inference, but by far insufficient for training (Figure~\ref{fig:memory_training_versus_inference} left), even using a lightweight neural network architecture (MobileNetV2 \cite{sandler2018mobilenetv2}). Furthermore, the memory is shared by various on-device applications (e.g., other deep learning models) and the operating system. A single application may only be allocated a small fraction of the total memory, which makes this challenge more critical.
Second, edge devices are \emph{energy-constrained}. DRAM access consumes two orders of magnitude more energy than on-chip SRAM access. The large memory footprint of activations cannot fit into the limited on-chip SRAM, thus has to access DRAM. For instance, the training memory of MobileNetV2, under batch size 16, is close to 1GB, which is by far larger than the SRAM size of an AMD EPYC CPU\footnote{\url{https://www.amd.com/en/products/cpu/amd-epyc-7302}} (Figure~\ref{fig:memory_training_versus_inference} left), not to mention lower-end edge platforms. If the training memory can fit on-chip SRAM, it will drastically improve the speed and energy efficiency.

There is plenty of efficient inference techniques that reduce the number of trainable parameters and the computation FLOPs \cite{howard2017mobilenets,sandler2018mobilenetv2,zhang2018shufflenet,han2015learning,han2016deep,tan2018mnasnet,howard2019searching,wu2018fbnet,cai2019automl,cai2020once}, however, parameter-efficient or FLOPs-efficient techniques do not directly save the training memory. It is the activation that bottlenecks the training memory, not the parameters. For example, Figure~\ref{fig:memory_training_versus_inference} (right) compares ResNet-50 and MobileNetV2-1.4. In terms of parameter size, MobileNetV2-1.4 is 4.3$\times$ smaller than ResNet-50. However, for training activation size, MobileNetV2-1.4 is almost the same as ResNet-50 (only 1.1$\times$ smaller), leading to little memory reduction.
It is essential to reduce the size of intermediate activations required by back-propagation, which is the key memory bottleneck for efficient on-device training.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/training_vs_inference.pdf}
    \caption{\textit{Left}: The memory footprint required by training is much larger than inference. \textit{Right}: Memory cost comparison between ResNet-50 and MobileNetV2-1.4 under batch size 16. Recent advances in efficient model design only reduce the size of parameters, but the activation size, which is the main bottleneck for training, does not improve much.}
    \vspace{-10pt}
    \label{fig:memory_training_versus_inference}
\end{figure}

In this paper, we propose \emph{Tiny-Transfer-Learning\xspace} (TinyTL\xspace) to address these challenges. By analyzing the memory footprint during the backward pass, we notice that the intermediate activations (the main bottleneck) are only needed when updating the weights, not the biases (Eq.~\ref{eq:linear_backward}). Inspired by this finding, we propose to freeze the weights of the pre-trained feature extractor and only update the biases to reduce the memory footprint (Figure~\ref{fig:lite_residual_learning}b). To compensate for the capacity loss, we introduce a memory-efficient bias module, called \emph{lite residual module}, which improves the model capacity by refining the intermediate feature maps of the feature extractor (Figure~\ref{fig:lite_residual_learning}c). Meanwhile, we aggressively shrink the resolution and width of the lite residual module to have a small memory overhead (only 3.8\%). Extensive experiments on 9 image classification datasets with the same pre-trained model (ProxylessNAS-Mobile \cite{cai2019proxylessnas}) demonstrate the effectiveness of TinyTL\xspace compared to previous transfer learning methods. Further, combined with a pre-trained once-for-all network \cite{cai2020once}, TinyTL\xspace can select a specialized sub-network as the feature extractor for each transfer dataset (i.e., feature extractor adaptation): given a more difficult dataset, a larger sub-network is selected, and vice versa. TinyTL achieves the same level of (or even higher) accuracy compared to fine-tuning the full Inception-V3 while reducing the training memory footprint by up to \textbf{12.9$\times$}.
Our contributions can be summarized as follows:
\begin{itemize}[leftmargin=*]
\setlength\itemsep{0pt}
\item We propose TinyTL\xspace, a novel transfer learning method to reduce the training memory footprint by an order of magnitude for efficient on-device learning. We systematically analyze the memory of training and find the bottleneck comes from updating the weights, not biases (assume ReLU activation).

\item We also introduce the \textit{lite residual module}, a memory-efficient bias module to improve the model capacity with little memory overhead.

\item Extensive experiments on transfer learning tasks show that our method is highly memory-efficient and effective. It reduces the training memory footprint by up to 12.9$\times$ without sacrificing accuracy.

\end{itemize}

\section{Related Work}

\paragraph{Efficient Inference Techniques.}
Improving the inference efficiency of deep neural networks on resource-constrained edge devices has recently drawn extensive attention. Starting from \cite{han2015learning,gong2014compressing,han2016deep,denton2014exploiting,vanhoucke2011improving}, one line of research focuses on compressing pre-trained neural networks, including i) network pruning that removes less-important units \cite{han2015learning,frankle2018the} or channels \cite{liu2017learning,he2017channel}; ii) network quantization that reduces the bitwidth of parameters \cite{han2016deep,courbariaux2015binaryconnect} or activations \cite{jacob2018quantization,wang2019haq}. However, these techniques cannot handle the training phase, as they rely on a well-trained model on the target task as the starting point.

Another line of research focuses on lightweight neural architectures by either manual design \cite{iandola2016squeezenet,howard2017mobilenets,sandler2018mobilenetv2,zhang2018shufflenet,huang2018condensenet} or neural architecture search \cite{tan2018mnasnet,cai2019proxylessnas,wu2018fbnet,cai2018efficient}.
These lightweight neural networks provide highly competitive accuracy \cite{tan2019efficientnet,cai2020once} while significantly improving inference efficiency.
However, concerning the training memory efficiency, key bottlenecks are not solved: \textbf{the training memory is dominated by activations, not parameters} (Figure~\ref{fig:memory_training_versus_inference}).

There are also some non-deep learning methods \cite{kumar2017resource,gupta2017protonn,edgeml03} that are designed for efficient inference on edge devices. These methods are suitable for handling simple tasks like MNIST. However, for more complicated tasks, we still need the representation capacity of deep neural networks.

\paragraph{Memory Footprint Reduction.}
Researchers have been seeking ways to reduce the training memory footprint. One typical approach is to re-compute discarded activations during backward~\cite{gruslys2016memory,chen2016training}. This approach reduces memory usage at the cost of a large computation overhead. Thus it is not preferred for edge devices. Layer-wise training \cite{greff2016highway} can also reduce the memory footprint compared to end-to-end training. However, it cannot achieve the same level of accuracy as end-to-end training. Another representative approach is through activation pruning \cite{liu2019dynamic}, which builds a dynamic sparse computation graph to prune activations during training. Similarly, \cite{wang2018training} proposes to reduce the bitwidth of training activations by introducing new reduced-precision floating-point formats. Besides reducing the training memory cost, there are some techniques that focus on reducing the peak inference memory cost, such as RNNPool \cite{saha2020rnnpool} and MemNet \cite{liu2019memnet}.
Our method is orthogonal to these techniques and can be combined to further reduce the memory footprint.

\paragraph{Transfer Learning.}
Neural networks pre-trained on large-scale datasets (e.g., ImageNet \cite{deng2009imagenet}) are widely used as a fixed feature extractor for transfer learning, then only the last layer needs to be fine-tuned \cite{chatfield2014return,donahue2014decaf,gan2015devnet,sharif2014cnn}.
This approach does not require to store the intermediate activations of the feature extractor, and thus is memory-efficient. However, the capacity of this approach is limited, resulting in poor accuracy, especially on datasets \cite{maji2013fine,krause20133d}  whose distribution is far from ImageNet (e.g., only 45.9\% Aircraft top1 accuracy achieved by Inception-V3 \cite{mudrakarta2018k}). Alternatively, fine-tuning the full network can achieve better accuracy \cite{kornblith2019better,cui2018large}. But it requires a vast memory footprint and hence is not friendly for training on edge devices. Recently, \cite{mudrakarta2019k,frankle2020training} propose to only update parameters of the batch normalization (BN) \cite{ioffe2015batch} layers, which greatly reduces the number of trainable parameters. \textbf{Unfortunately, parameter-efficiency doesn't translate to memory-efficiency.} It still requires a large amount of memory (e.g., 326MB under batch size 8) to store the input activations of the BN layers (Table~\ref{tab:transfer_results_under_different_backbones}). Additionally, the accuracy of this approach is still much worse than fine-tuning the full network (70.7\% v.s. 85.5\%; Table~\ref{tab:transfer_results_under_different_backbones}). People can also partially fine-tune some layers, but how many layers to select is still ad hoc. This paper provides a systematic approach to save memory without losing accuracy.

\section{Tiny Transfer Learning}

\begin{figure}[t]
    \centering
    \includegraphics[width=0.8\linewidth]{figures/tinyTL_framework.pdf}
    \caption{TinyTL\xspace overview (``C'' denotes the width and ``R'' denote the resolution). Conventional transfer learning relies on fine-tuning the weights to adapt the model (Fig.a), which requires a large amount of activation memory (in blue) for back-propagation. TinyTL\xspace reduces the memory usage by fixing the weights (Fig.b) while only fine-tuning the bias. (Fig.c) exploit \textit{lite residual learning} to compensate for the capacity loss, using group convolution and avoiding inverted bottleneck to achieve high arithmetic intensity and small memory footprint. The skip connection remains unchanged (omitted for simplicity).
    }\label{fig:lite_residual_learning}
\end{figure}

\subsection{Understanding the Memory Footprint of Back-propagation}\label{sec:backward_analysis}
Without loss of generality, we consider a neural network $\mathcal{M}\xspace$ that consists of a sequence of layers:
\begin{equation}
    \mathcal{M}\xspace(\cdot) = \mathcal{F}\xspace_{\mathbf{w}\xspace_n}(\mathcal{F}\xspace_{\mathbf{w}\xspace_{n-1}}(\cdots \mathcal{F}\xspace_{\mathbf{w}\xspace_2}(\mathcal{F}\xspace_{\mathbf{w}\xspace_1}(\cdot)) \cdots)),
\end{equation}
where $\mathbf{w}\xspace_i$ denotes the parameters of the $i^{th}$ layer. Let $\mathbf{a}\xspace_i$ and $\mathbf{a}\xspace_{i+1}$ be the input and output activations of the $i^{th}$ layer, respectively, and $\mathcal{L}\xspace$ be the loss. In the backward pass, given $\frac{\partial \loss}{\partial \act_{i+1}}\xspace$, there are two goals for the $i^{th}$ layer: computing $\frac{\partial \loss}{\partial \act_i}\xspace$ and $\frac{\partial \loss}{\partial \weight_i}\xspace$.

Assuming the $i^{th}$ layer is a linear layer whose forward process is given as: $\mathbf{a}\xspace_{i+1} = \mathbf{a}\xspace_i\mathbf{W} + \mathbf{b}$, then its backward process under batch size 1 is
\begin{equation}
\label{eq:linear_backward}
    \frac{\partial \loss}{\partial \act_i}\xspace = \frac{\partial \mathcal{L}\xspace}{\partial \mathbf{a}\xspace_{i+1}} \frac{\partial \mathbf{a}\xspace_{i+1}}{\partial \mathbf{a}\xspace_i} = \frac{\partial \mathcal{L}\xspace}{\partial \mathbf{a}\xspace_{i+1}} \mathbf{W}^T, \qquad
    \frac{\partial \mathcal{L}\xspace}{\partial \mathbf{W}} = {\color{mydarkred} \mathbf{a}\xspace_i^T} \frac{\partial \mathcal{L}\xspace}{\partial \mathbf{a}\xspace_{i+1}}, \qquad
    \frac{\partial \mathcal{L}\xspace}{\partial \mathbf{b}} = \frac{\partial \mathcal{L}\xspace}{\partial \mathbf{a}\xspace_{i + 1}}.
\end{equation}
According to Eq.~(\ref{eq:linear_backward}), the intermediate activations (i.e., $\{\mathbf{a}\xspace_i\}$) that dominate the memory footprint are only required to compute the gradient of the weights (i.e., $\frac{\partial \mathcal{L}\xspace}{\partial \mathbf{W}}$), not the bias. If we only update the bias, training memory can be greatly saved. This property is also applicable to convolution layers and normalization layers (e.g., batch normalization \cite{ioffe2015batch}, group normalization\cite{wu2018group}, etc) since they can be considered as special types of linear layers.

Regarding non-linear activation layers (e.g., ReLU, sigmoid, h-swish), sigmoid and h-swish require to store $\mathbf{a}\xspace_i$ to compute $\frac{\partial \loss}{\partial \act_i}\xspace$ (Table~\ref{tab:act_func_backward}), hence they are not memory-efficient. Activation layers that build upon them are also not memory-efficient consequently, such as tanh, swish \cite{ramachandran2017searching}, etc. In contrast, ReLU and other ReLU-styled activation layers (e.g.,
LeakyReLU \cite{xu2015empirical}) only requires to store a binary mask representing whether the value is smaller than 0, which is 32$\times$ smaller than storing $\mathbf{a}\xspace_i$.

\begin{table}[h!]
\centering
\caption{Detailed forward and backward processes of non-linear activation layers. $|\mathbf{a}\xspace_i|$ denotes the number of elements of $\mathbf{a}\xspace_i$. ``$\circ$'' denotes the element-wise product. $(\mathbf{1}_{\mathbf{a}\xspace_i \geq 0})_j = 0$ if $(\mathbf{a}\xspace_i)_j < 0$ and $(\mathbf{1}_{\mathbf{a}\xspace_i \geq 0})_j = 1$ otherwise. $\text{ReLU6}(\mathbf{a}\xspace_i) = \text{min}(6, \text{max}(0, \mathbf{a}\xspace_i))$.}
\vspace{10pt}
\resizebox{1\linewidth}{!}{
\begin{tabular}{l | c | c | c }
    \toprule
    Layer Type & Forward & Backward & Memory Cost \\
    \midrule
    ReLU & $\mathbf{a}\xspace_{i+1} = \text{max}(0, \mathbf{a}\xspace_i)$ & $\frac{\partial \loss}{\partial \act_i}\xspace = \frac{\partial \loss}{\partial \act_{i+1}}\xspace \circ \mathbf{1}_{\mathbf{a}\xspace_i \geq 0}$ & $|\mathbf{a}\xspace_i|$ bits\\
    sigmoid & $\mathbf{a}\xspace_{i+1} = \sigma(\mathbf{a}\xspace_i) = \frac{1}{1 + \exp(-\mathbf{a}\xspace_i)}$ & $\frac{\partial \loss}{\partial \act_i}\xspace = \frac{\partial \loss}{\partial \act_{i+1}}\xspace \circ \sigma(\mathbf{a}\xspace_i) \circ (1 - \sigma(\mathbf{a}\xspace_i))$ & 32 $|\mathbf{a}\xspace_i|$ bits \\
    h-swish \cite{howard2019searching} & $\mathbf{a}\xspace_{i+1} = \mathbf{a}\xspace_i \circ \frac{\text{ReLU6}(\mathbf{a}\xspace_i + 3)}{6}$ & $\frac{\partial \loss}{\partial \act_i}\xspace = \frac{\partial \loss}{\partial \act_{i+1}}\xspace \circ (\frac{\text{ReLU6}(\mathbf{a}\xspace_i + 3)}{6} + \mathbf{a}\xspace_i \circ \frac{\mathbf{1}_{-3 \leq \mathbf{a}\xspace_i \leq 3}}{6})$ & 32 $|\mathbf{a}\xspace_i|$ bits \\
    \bottomrule
\end{tabular}
}
\label{tab:act_func_backward}
\end{table}

\subsection{Lite Residual Learning}\label{sec:lite_residual}
Based on the memory footprint analysis, one possible solution of reducing the memory cost is to freeze the weights of the pre-trained feature extractor while only update the biases (Figure~\ref{fig:lite_residual_learning}b). However, only updating biases has limited adaptation capacity. Therefore, we introduce lite residual learning that exploits a new class of generalized memory-efficient bias modules to refine the intermediate feature maps (Figure~\ref{fig:lite_residual_learning}c).

Formally, a layer with frozen weights and learnable biases can be represented as:
\begin{equation}
    \mathbf{a}\xspace_{i+1} = \textcolor{frozoncolor}{\mathcal{F}\xspace_{\mathbf{W}}(\mathbf{a}\xspace_i)} + \mathbf{b}.
\end{equation}
To improve the model capacity while keeping a small memory footprint, we propose to add a lite residual module that generates a residual feature map to refine the output:
\begin{equation}
\label{eq:lrl_formula}
    \mathbf{a}\xspace_{i+1} = \textcolor{frozoncolor}{\mathcal{F}\xspace_{\mathbf{W}}(\mathbf{a}\xspace_i)} + \mathbf{b} + \mathcal{F}\xspace_{\mathbf{w}\xspace_r}(\mathbf{a}\xspace_i' = \text{reduce}(\mathbf{a}\xspace_i)),
\end{equation}
where $\mathbf{a}\xspace_i' = \text{reduce}(\mathbf{a}\xspace_i)$ is the reduced activation. According to Eq.~(\ref{eq:linear_backward}), learning these lite residual modules only requires to store the reduced activations \{$\mathbf{a}\xspace_i'$\} rather than the full activations \{$\mathbf{a}\xspace_i$\}.

\paragraph{Implementation (Figure~\ref{fig:lite_residual_learning}c).}
We apply Eq.~(\ref{eq:lrl_formula}) to mobile inverted bottleneck blocks (MB-block) \cite{sandler2018mobilenetv2}. The key principle is to keep the activation small. Following this principle, we explore two design dimensions to reduce the activation size:
\begin{itemize}[leftmargin=*]

\item \textbf{Width.} The widely-used inverted bottleneck requires a huge number of channels (6$\times$) to compensate for the small capacity of a depthwise convolution, which is parameter-efficient but highly  activation-inefficient. Even worse, converting 1$\times$ channels to 6$\times$ channels back and forth requires two $1\times1$ projection layers, which doubles the total activation to 12$\times$. Depthwise convolution also has a very low arithmetic intensity (its OPs/Byte is less than 4\% of $1\times1$ convolution's OPs/Byte if with 256 channels), thus highly memory in-efficient with little reuse.
To solve these limitations, our lite residual module employs the group convolution that has much higher arithmetic intensity than depthwise convolution, providing a good trade-off between FLOPs and memory. That also removes the 1$\times1$ projection layer, reducing the total channel number by $\frac{6 \times 2 + 1}{1 + 1} = 6.5\times$.

\item \textbf{Resolution.} The activation size grows quadratically with the resolution. Therefore, we shrink the resolution in the lite residual module by employing a $2 \times 2$ average pooling to downsample the input feature map. The output of the lite residual module is then upsampled to match the size of the main branch's output feature map via bilinear upsampling.
Combining resolution and width optimizations, the activation of our lite residual module is roughly $2^2\times6.5=26\times$ smaller than the inverted bottleneck.
\end{itemize}

\subsection{Discussions}\label{sec:method_discussion}

\paragraph{Normalization Layers.} As discussed in Section~\ref{sec:backward_analysis}, TinyTL\xspace flexibly supports different normalization layers, including batch normalization (BN), group normalization (GN), layer normalization (LN), and so on. In particular, BN is the most widely used one in vision tasks. However, BN requires a large batch size to have accurate running statistics estimation during training, which is not suitable for on-device learning where we want a small training batch size to reduce the memory footprint. Moreover, the data may come in a streaming fashion in on-device learning, which requires a training batch size of 1. In contrast to BN, GN can handle a small training batch size as the running statistics in GN are computed independently for different inputs. In our experiments, GN with a small training batch size (e.g., 8) performs slightly worse than BN with a large training batch size (e.g., 256). However, as we target at on-device learning, we choose GN in our models.

\paragraph{Feature Extractor Adaptation.}\label{sec:feature_adaptation}
TinyTL\xspace can be applied to different backbone neural networks, such as MobileNetV2 \cite{sandler2018mobilenetv2}, ProxylessNASNets \cite{cai2019proxylessnas}, EfficientNets \cite{tan2019efficientnet}, etc. However, since the weights of the feature extractor are frozen in TinyTL\xspace, we find using the same backbone neural network for all transfer tasks is sub-optimal. Therefore, we choose the backbone of TinyTL\xspace using a pre-trained once-for-all network \cite{cai2020once} to adaptively select the specialized feature extractor that best fits the target transfer dataset. Specifically, a once-for-all network is a special kind of neural network that is sparsely activated, from which many different sub-networks can be derived without retraining by sparsely activating parts of the model according to the architecture configuration (i.e., depth, width, kernel size, resolution), while the weights are shared. This allows us to efficiently evaluate the effectiveness of a backbone neural network on the target transfer dataset without the expensive pre-training process. Further details of the feature extractor adaptation process are provided in Appendix A.

\section{Experiments}
\subsection{Setups}\label{sec:exp_setup}

\paragraph{Datasets.} Following the common practice \cite{cui2018large,mudrakarta2019k,kornblith2019better}, we use ImageNet \cite{deng2009imagenet} as the pre-training dataset, and then transfer the models to 8 downstream object classification tasks, including Cars \cite{krause20133d}, Flowers \cite{nilsback2008automated}, Aircraft \cite{maji2013fine}, CUB \cite{wah2011caltech}, Pets \cite{parkhi2012cats}, Food \cite{bossard2014food}, CIFAR10 \cite{krizhevsky2009learning}, and CIFAR100 \cite{krizhevsky2009learning}. Besides object classification, we also evaluate our TinyTL\xspace on human facial attribute classification tasks, where CelebA \cite{liu2018large} is the transfer dataset and VGGFace2 \cite{cao2018vggface2} is the pre-training dataset.

\paragraph{Model Architecture.} To justify the effectiveness of TinyTL\xspace, we first apply TinyTL\xspace and previous transfer learning methods to the same backbone neural network, ProxylessNAS-Mobile \cite{cai2019proxylessnas}. For each MB-block in ProxylessNAS-Mobile, we insert a lite residual module as described in Section~\ref{sec:lite_residual} and Figure~\ref{fig:lite_residual_learning} (c). The group number is 2, and the kernel size is 5. We use the ReLU activation since it is more memory-efficient according to Section~\ref{sec:backward_analysis}. We replace all BN layers with GN layers to better support small training batch sizes. We set the number of channels per group to 8 for all GN layers. Following \cite{kolesnikov2019big}, we apply weight standardization \cite{weightstandardization} to convolution layers that are followed by GN.

For feature extractor adaptation, we build the once-for-all network using the MobileNetV2 design space \cite{cai2019proxylessnas,cai2020once} that contains five stages with a gradually decreased resolution, and each stage consists of a sequence of MB-blocks. In the stage-level, it supports elastic depth (i.e., 2, 3, 4). In the block-level, it supports elastic kernel size (i.e., 3, 5, 7) and elastic width expansion ratio (i.e., 3, 4, 6). Similarly, for each MB-block in the once-for-all network, we insert a lite residual module that supports elastic group number (i.e., 2, 4) and elastic kernel size (i.e., 3, 5).

\begin{table}[!t]
\centering
\caption{Comparison between TinyTL\xspace and conventional transfer learning methods (training memory footprint is calculated assuming the batch size is 8 and the classifier head for Flowers is used). For object classification datasets, we report the top1 accuracy (\%) while for CelebA we report the average top1 accuracy (\%) over 40 facial attribute classification tasks. `B' represents Bias while `L' represents LiteResidual. \textit{FT-Last} represents only the last layer is fine-tuned. \textit{FT-Norm+Last} represents normalization layers and the last layer are fine-tuned. \textit{FT-Full} represents the full network is fine-tuned.
The backbone neural network is ProxylessNAS-Mobile, and the resolution is 224 except for `TinyTL-L+B@320' whose resolution is 320. TinyTL\xspace consistently outperforms \textit{FT-Last} and \textit{FT-Norm+Last} by a large margin with a similar or lower training memory footprint. By increasing the resolution to 320, TinyTL\xspace can reach the same level of accuracy as \textit{FT-Full} while being 6$\times$ memory efficient.}
\label{tab:tinyTL_proxylessnas_mobile_results}
\vspace{10pt}
\resizebox{1\linewidth}{!}{
\begin{tabular}{l | c | c c c c c c c c | c }
\toprule
\multirow{2}{*}{Method} & Train. & \multirow{2}{*}{Flowers} & \multirow{2}{*}{Cars} & \multirow{2}{*}{CUB} & \multirow{2}{*}{Food} & \multirow{2}{*}{Pets} & \multirow{2}{*}{Aircraft} & \multirow{2}{*}{CIFAR10} & \multirow{2}{*}{CIFAR100} & \multirow{2}{*}{CelebA} \\
& Mem. & & & & & & & & & \\
\midrule
FT-Last & \textbf{31MB} & 90.1 & 50.9 & 73.3 & 68.7 & 91.3 & 44.9 & 85.9 & 68.8 & 88.7 \\
\midrule
TinyTL-B & \textbf{32MB} & 93.5 & 73.4 & 75.3 & 75.5 & 92.1 & 63.2 & 93.7 & 78.8 & 90.4 \\
TinyTL-L & \textbf{37MB} & 95.3 & 84.2 & 76.8 & 79.2 & 91.7 & 76.4 & 96.1 & 80.9 & 91.2 \\
TinyTL-L+B & \textbf{37MB} & 95.5 & 85.0  & 77.1 & 79.7 & 91.8 & 75.4 & 95.9 & 81.4  & 91.2 \\
TinyTL-L+B@320 & 65MB & 96.8 & 88.8 & 81.0 & 82.9 & 92.9 & 82.3 & 96.1 & 81.5  & - \\
\midrule
FT-Norm+Last & 192MB & 94.3 & 77.9 & 76.3 & 77.0 & 92.2 & 68.1 & 94.8 & 80.2 & 90.4 \\
FT-Full & 391MB & 96.8 & 90.2 & 81.0 & 84.6 & 93.0 & 86.0 & 97.1 & 84.1 & 91.4 \\
\bottomrule
\end{tabular}
}
\end{table}

\paragraph{Training Details.} We freeze the memory-heavy modules (weights of the feature extractor) and only update memory-efficient modules (bias, lite residual, classifier head) during transfer learning. The models are fine-tuned for 50 epochs using the Adam optimizer \cite{kingma2014adam} with batch size 8 on a single GPU. The initial learning rate is tuned for each dataset while cosine schedule \cite{loshchilov2016sgdr} is adopted for learning rate decay. We apply 8bits weight quantization \cite{han2016deep} on the frozen weights to reduce the parameter size, which causes a negligible accuracy drop in our experiments. For all compared methods, we also assume the 8bits weight quantization is applied if eligible when calculating their training memory footprint. Additionally, as PyTorch does not support explicit fine-grained memory management, we use the theoretically calculated training memory footprint for comparison in our experiments. For simplicity, we assume the batch size is 8 for all compared methods throughout the experiment section.

\subsection{Main Results}

\paragraph{Effectiveness of TinyTL\xspace.} Table~\ref{tab:tinyTL_proxylessnas_mobile_results} reports the comparison between TinyTL\xspace and previous transfer learning methods including: i) fine-tuning the last linear layer \cite{chatfield2014return,donahue2014decaf,sharif2014cnn} (referred to as \textit{FT-Last}); ii) fine-tuning the normalization layers (e.g., BN, GN) and the last linear layer \cite{mudrakarta2018k} (referred to as \textit{FT-Norm+Last}) ; iii) fine-tuning the full network \cite{kornblith2019better,cui2018large} (referred to as \textit{FT-Full}). We also study several variants of TinyTL\xspace including: i) TinyTL\xspace-B that fine-tunes biases and the last linear layer; ii) TinyTL\xspace-L that fine-tunes lite residual modules and the last linear layer; iii) TinyTL\xspace-L+B that fine-tunes lite residual modules, biases, and the last linear layer. All compared methods use the same pre-trained model but fine-tune different parts of the model as discussed above. We report the average accuracy across five runs.

Compared to \textit{FT-Last}, TinyTL\xspace maintains a similar training memory footprint while improving the top1 accuracy by a significant margin. In particular, TinyTL\xspace-L+B improves the top1 accuracy by \textbf{34.1\%} on Cars, by \textbf{30.5\%} on Aircraft, by \textbf{12.6\%} on CIFAR100, by \textbf{11.0\%} on Food, etc. It shows the improved adaptation capacity of our method over \textit{FT-Last}.
Compared to \textit{FT-Norm+Last}, TinyTL\xspace-L+B improves the training memory efficiency by \textbf{5.2$\times$} while providing up to \textbf{7.3\%} higher top1 accuracy, which shows that our method is not only more memory-efficient but also more effective than \textit{FT-Norm+Last}. Compared to \textit{FT-Full}, TinyTL\xspace-L+B@320 can achieve the same level of accuracy while providing \textbf{6.0$\times$} training memory saving.

Regarding the comparison between different variants of TinyTL\xspace, both TinyTL\xspace-L and TinyTL\xspace-L+B have clearly better accuracy than TinyTL\xspace-B while incurring little memory overhead. It shows that the lite residual modules are essential in TinyTL\xspace. Besides, we find that TinyTL\xspace-L+B is slightly better than TinyTL\xspace-L on most of the datasets while maintaining the same memory footprint. Therefore, we choose TinyTL\xspace-L+B as the default.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/tinyTL_proxylessnas_mobile_results.pdf}
    \caption{Top1 accuracy results of different transfer learning methods under varied resolutions using the same pre-trained neural network (ProxylessNAS-Mobile). With the same level of accuracy, TinyTL\xspace achieves 3.9-6.5$\times$ memory saving compared to fine-tuning the full network.}\label{fig:tinyTL_proxylessnas_mobile_results}
\end{figure}

\begin{table}[t]
\centering
\caption{Comparison with previous transfer learning results under different backbone neural networks. `I-V3' is Inception-V3; `N-A' is NASNet-A Mobile; `M2-1.4' is MobileNetV2-1.4; `R-50' is ResNet-50; `PM' is ProxylessNAS-Mobile; `FA' represents feature extractor adaptation. $^\dag$ indicates the last two layers are updated besides biases and lite residual modules in TinyTL\xspace. TinyTL+FA reduces the training memory by \textbf{7.5-12.9$\times$} without sacrificing accuracy compared to fine-tuning the widely used Inception-V3.}
\label{tab:transfer_results_under_different_backbones}
\vspace{5pt}
\resizebox{1\linewidth}{!}{
\begin{tabular}{l l | c c | c c c c c c c c }
\toprule
\multirow{2}{*}{Method} & \multirow{2}{*}{Net} & Train. & Reduce
& \multirow{2}{*}{Flowers} & \multirow{2}{*}{Cars} & \multirow{2}{*}{CUB} & \multirow{2}{*}{Food} & \multirow{2}{*}{Pets} & \multirow{2}{*}{Aircraft} & \multirow{2}{*}{CIFAR10} & \multirow{2}{*}{CIFAR100} \\
& & mem. & ratio \\
\midrule
\multirow{4}{*}{FT-Full}
& I-V3 \cite{cui2018large} & 850MB & 1.0$\times$ & 96.3 & 91.3 & 82.8 & 88.7 & - & 85.5 & - & - \\
& R-50 \cite{kornblith2019better} & 802MB & 1.1$\times$ & 97.5 & 91.7 & - & 87.8 & 92.5 & 86.6 & 96.8 & 84.5 \\
& M2-1.4 \cite{kornblith2019better} & 644MB & 1.3$\times$ & 97.5 & 91.8 & - & 87.7 & 91.0 & 86.8 & 96.1 & 82.5 \\
& N-A \cite{kornblith2019better} & 566MB & 1.5$\times$ & 96.8 & 88.5 & - & 85.5 & 89.4 & 72.8 & 96.8 & 83.9 \\
\midrule
FT-Norm+Last & I-V3 \cite{mudrakarta2018k} & 326MB & 2.6$\times$ & 90.4 & 81.0 &  - & - & - & 70.7 & - & - \\
\midrule
FT-Last & I-V3 \cite{mudrakarta2018k} & 94MB & 9.0$\times$ & 84.5 & 55.0 & - & - & - & 45.9 & - & - \\
\midrule
\multirow{4}{*}{TinyTL\xspace}
& PM@320 & \textbf{65MB} & \textbf{13.1$\times$} & 96.8 & 88.8 & 81.0 & 82.9 & 92.9 & 82.3 & 96.1 & 81.5 \\
& FA@256 & \textbf{66MB} & \textbf{12.9$\times$} & 96.8 & 89.6 & 80.8 & 83.4 & 93.0 & 82.4 & 96.8 & 82.7 \\
& FA@352 & 114MB & 7.5$\times$ & 97.4 & 90.7 & 82.4 & 85.0 & 93.4 & 84.8 & - & - \\
& FA@352$^\dag$ & 116MB & 7.3$\times$ & - & 91.5 & - & 86.0 & - & 85.4 & - & - \\
\bottomrule
\end{tabular}
}
\end{table}

Figure~\ref{fig:tinyTL_proxylessnas_mobile_results} demonstrates the results under different input resolutions. We can observe that simply reducing the input resolution will result in significant accuracy drops for \textit{FT-Full}. In contrast, TinyTL\xspace can reduce the memory footprint by \textbf{3.9-6.5$\times$} while having the same or even higher accuracy compared to fine-tuning the full network.

\paragraph{Combining TinyTL\xspace and Feature Extractor Adaptation.} Table~\ref{tab:transfer_results_under_different_backbones} summarizes the results of TinyTL\xspace and previously reported transfer learning results, where different backbone neural networks are used as the feature extractor. Combined with feature extractor adaptation, TinyTL\xspace achieves \textbf{7.5-12.9$\times$} memory saving compared to fine-tuning the full Inception-V3, reducing from 850MB to 66-114MB while providing the same level of accuracy. Additionally, we try updating the last two layers besides biases and lite residual modules (indicated by $^\dag$), which results in 2MB of extra training memory footprint. This slightly improves the accuracy performances, from 90.7\% to 91.5\% on Cars, from 85.0\% to 86.0\% on Food, and from 84.8\% to 85.4\% on Aircraft.

\subsection{Ablation Studies and Discussions}

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/vs_activation_pruning.pdf}
    \caption{Compared with the dynamic activation pruning \cite{liu2019dynamic},
    TinyTL\xspace saves the memory more effectively.}\label{fig:vs_activation_compression}
\end{figure}

\paragraph{Comparison with Dynamic Activation Pruning.} The comparison between TinyTL\xspace and dynamic activation pruning \cite{liu2019dynamic} is summarized in Figure~\ref{fig:vs_activation_compression}. TinyTL\xspace is more effective because it re-designed the transfer learning framework (lite residual module, feature extractor adaptation) rather than prune an existing architecture. The transfer accuracy drops quickly when the pruning ratio increases beyond 50\% (only 2$\times$ memory saving). In contrast, TinyTL\xspace can achieve much higher memory reduction without loss of accuracy.

\paragraph{Initialization for Lite Residual Modules.} By default, we use the pre-trained weights on the pre-training dataset to initialize the lite residual modules. It requires to have lite residual modules during both the pre-training phase and transfer learning phase. When applying TinyTL\xspace to existing pre-trained neural networks that do not have lite residual modules during the pre-training phase, we need to use another initialization strategy for the lite residual modules during transfer learning. To verify the effectiveness of TinyTL\xspace under this setting, we also evaluate the performances of TinyTL\xspace when using random weights \cite{he2016deep} to initialize the lite residual modules except for the scaling parameter of the final normalization layer in each lite residual module. These scaling parameters are initialized with zeros.

Table~\ref{tab:tinyTL_with_random_lite_residual} reports the summarized results. We find using the pre-trained weights to initialize the lite residual modules consistently outperforms using random weights. Besides, we also find that using TinyTL\xspace-RandomL+B still provides highly competitive results on Cars, Food, Aircraft, CIFAR10, CIFAR100, and CelebA. Therefore, if having the budget, it is better to use pre-trained weights to initialize the lite residual modules. If not, TinyTL\xspace can still be applied and provides competitive results on datasets whose distribution is far from the pre-training dataset.

\begin{table}[!t]
\centering
\caption{Results of TinyTL\xspace under different initialization strategies for lite residual modules. TinyTL\xspace-L+B adds lite residual modules starting from the pre-training phase and uses the pre-trained weights to initialize the lite residual modules during transfer learning. In contrast, TinyTL\xspace-RandomL+B uses random weights to initialize the lite residual modules. Using random weights for initialization hurts the performances of TinyTL\xspace. But on datasets whose distribution is far from the pre-training dataset, TinyTL\xspace-RandomL+B still provides competitive results.}
\label{tab:tinyTL_with_random_lite_residual}
\vspace{10pt}
\resizebox{1\linewidth}{!}{
\begin{tabular}{l | c | c c c c c c c c | c }
\toprule
\multirow{2}{*}{Method} & Train. & \multirow{2}{*}{Flowers} & \multirow{2}{*}{Cars} & \multirow{2}{*}{CUB} & \multirow{2}{*}{Food} & \multirow{2}{*}{Pets} & \multirow{2}{*}{Aircraft} & \multirow{2}{*}{CIFAR10} & \multirow{2}{*}{CIFAR100} & \multirow{2}{*}{CelebA} \\
& Mem. & & & & & & & & & \\
\midrule
FT-Last & \textbf{31MB} & 90.1 & 50.9 & 73.3 & 68.7 & 91.3 & 44.9 & 85.9 & 68.8 & 88.7 \\
\midrule
TinyTL-RandomL+B & \textbf{37MB} & 88.0 & 82.4 & 72.9 & 79.3 & 84.3 & 73.6 & 95.7 & 81.4 & 91.2 \\
TinyTL-L+B & \textbf{37MB} & 95.5 & 85.0  & 77.1 & 79.7 & 91.8 & 75.4 & 95.9 & 81.4  & 91.2 \\
\midrule
FT-Norm+Last & 192MB & 94.3 & 77.9 & 76.3 & 77.0 & 92.2 & 68.1 & 94.8 & 80.2 & 90.4 \\
FT-Full & 391MB & 96.8 & 90.2 & 81.0 & 84.6 & 93.0 & 86.0 & 97.1 & 84.1 & 91.4 \\
\bottomrule
\end{tabular}
}
\end{table}

\paragraph{Results of TinyTL under Batch Size 1.} Figure~\ref{fig:tinytl_batch1_results} demonstrates the results of TinyTL when using a training batch size of 1. We tune the initial learning rate for each dataset while keeping the other training settings unchanged. As our model employs group normalization rather than batch normalization (Section~\ref{sec:method_discussion}), we observe little/no loss of accuracy than training with batch size 8. Meanwhile, the training memory footprint is further reduced to around 16MB, a typical L3 cache size. This makes it much easier to train on the cache (SRAM), which can greatly reduce energy consumption than DRAM training.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/tinytl_batch1_results.pdf}
    \caption{Results of TinyTL when trained with batch size 1. It further reduces the training memory footprint to around 16MB (typical L3 cache size), making it possible to train on the cache (SRAM) instead of DRAM. }\label{fig:tinytl_batch1_results}
\end{figure}
\section{Conclusion}
We proposed Tiny-Transfer-Learning\xspace (TinyTL\xspace) for memory-efficient on-device learning that aims to adapt pre-trained models to newly collected data on edge devices. Unlike previous methods that focus on reducing the number of parameters or FLOPs, TinyTL\xspace directly optimizes the training memory footprint by fixing the memory-heavy modules (i.e., weights) while learning memory-efficient bias modules. We further introduce lite residual modules that significantly improve the adaptation capacity of the model with little memory overhead. Extensive experiments on benchmark datasets consistently show the effectiveness and memory-efficiency of TinyTL\xspace, paving the way for efficient on-device machine learning.

\section*{Broader Impact}
The proposed efficient on-device learning technique greatly reduces the training memory footprint of deep neural networks, enabling adapting pre-trained models to new data locally on edge devices without leaking them to the cloud. It
can democratize AI to people in the rural areas where the Internet is unavailable or the network condition is poor. They can not only inference but also fine-tune AI models on their local devices without connections to the cloud servers. This can also benefit privacy-sensitive AI applications, such as health care, smart home,
and so on.

\section*{Acknowledgements}
We thank MIT-IBM Watson AI Lab, NSF CAREER Award \#1943349 and NSF Award \#2028888 for supporting this research. We thank MIT Satori cluster for providing the computation resource.
\section{Details of Feature Extractor Adaptation}\label{appendix:feature_extractor_adaptation_details}

Conventional transfer learning chooses the feature extractor according to the pre-training accuracy (e.g., ImageNet accuracy) and uses the same one for all transfer tasks \cite{cui2018large,mudrakarta2019k}. However, we find this approach sub-optimal since different target tasks may need very different feature extractors, and high pre-training accuracy does not guarantee good transferability of the pre-trained weights. This is especially critical in our case where the weights are frozen.

\begin{figure}[h]
    \centering
    \includegraphics[width=\linewidth]{figures/feature_extractor_adaptation.pdf}
    \caption{Transfer learning performances of various ImageNet pre-trained models with the last linear layer trained. The relative accuracy order between different pre-trained models changes significantly among ImageNet and the transfer learning datasets.}\label{fig:discrete_op_selection}
\end{figure}

Figure~\ref{fig:discrete_op_selection} shows the top1 accuracy of various widely used ImageNet pre-trained models on three transfer datasets by only learning the last layer, which reflects the transferability of their pre-trained weights.
The relative order between different pre-trained models is not consistent with their ImageNet accuracy on all three datasets. This result indicates that the ImageNet accuracy is not a good proxy for transferability. Besides, we also find that the same pre-trained model can have very different rankings on different tasks. For instance, Inception-V3 gives poor accuracy on Flowers but provides top results on the other two datasets.

Therefore, we need to specialize the feature extractor to best match the target dataset. In this work, we achieve this by using a pre-trained once-for-all network \cite{cai2020once} that comprises many different sub-networks. Specifically, given a pre-trained once-for-all network on ImageNet, we fine-tune it on the target transfer dataset with the weights of the main branches (i.e., MB-blocks) frozen and the other parameters (i.e., biases, lite residual modules, classifier head) updated via gradient descent. In this phase, we randomly sample one sub-network in each training step. The peak memory cost of this phase is 61MB under resolution 224, which is reached when the largest sub-network is sampled. Regarding the computation cost, the average MAC (forward \& backward)\footnote{The training MAC of a sampled sub-network is roughly 2$\times$ larger than its inference MAC, rather than 3$\times$, since we do not need to update the weights of the main branches.} of sampled sub-nets is (776M + 2510M) / 2 = 1643M per sample, where 776M is the training MAC of the smallest sub-network and 2510M is the training MAC of the largest sub-network. Therefore, the total MAC of this phase is 1643M $\times$ 2040 $\times$ 0.8 $\times$ 50 = 134T on Flowers, where 2040 is the number of total training samples, 0.8 means the once-for-all network is fine-tuned on 80\% of the training samples (the remaining 20\% is reserved for search), and 50 is the number of training epochs.

Based on the fine-tuned once-for-all network, we collect 500 [sub-net, accuracy] pairs on the validation set (20\% randomly sampled training data) and train an accuracy predictor\footnote{Details of the accuracy predictor is provided in Appendix~\ref{appendix:acc_predictor_details}.} using the collected data \cite{cai2020once}. We employ evolutionary search \cite{guo2019single} based on the accuracy predictor to find the sub-network that best matches the target transfer dataset. No back-propagation on the once-for-all network is required in this step, thus incurs no additional memory overhead. The primary computation cost of this phase comes from collecting 500 [sub-net, accuracy] pairs required to train the accuracy predictor. It only involves the forward processes of sampled sub-nets, and no back-propagation is required. The average MAC (only forward) of sampled sub-nets is (355M + 1182M) / 2 = 768.5M per sample, where 355M is the inference MAC of the smallest sub-network and 1182M is the inference MAC of the largest sub-network. Therefore, the total MAC of this phase is 768.5M $\times$ 2040 $\times$ 0.2 $\times$ 500 = 157T on Flowers, where 2040 is the number of total training samples, 0.2 means the validation set consists of 20\% of the training samples, and 500 is the number of measured sub-nets.

Finally, we fine-tune the searched sub-network with the weights of the main branches frozen and the other parameters updated, using the full training set to get the final results. The memory cost of this phase is 66MB under resolution 256 on Flowers. The total MAC is 2190M $\times$ 2040 $\times$ 1.0 $\times$ 50 = 223T, on Flowers, where 2190M is the training MAC, 2040 is the number of total training samples, 1.0 means the full training set is used, and 50 is the number of training epochs.

\section{Details of the Accuracy Predictor}\label{appendix:acc_predictor_details}
The accuracy predictor is a three-layer feed-forward neural network with a hidden dimension of 400 and ReLU as the activation function for each layer. It takes the one-hot encoding of the sub-network's architecture as the input and outputs the predicted accuracy of the given sub-network. The inference MAC of this accuracy predictor is only 0.37M, which is 3-4 orders of magnitude smaller than the inference MAC of the CNN classification models. The memory footprint of this accuracy predictor is only 5KB. Therefore, both the computation overhead and the memory overhead of the accuracy predictor are negligible.

\section{Cost Details}
\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/cost_breakdown.pdf}
    \caption{On-device training cost on Pets. TinyTL\xspace requires 9.9$\times$ smaller memory cost (assuming using the same batch size) and 27$\times$ smaller computation cost compared to fine-tuning the full MobileNetV2-1.4 \cite{kornblith2019better} while having a better accuracy.}\label{fig:cost_breakdown}
\end{figure}

The on-device training cost of TinyTL\xspace and \textit{FT-Full} on Pets is summarized in Figure~\ref{fig:cost_breakdown} (left side), while the memory cost breakdown of TinyTL\xspace is provided in Figure~\ref{fig:cost_breakdown} (right side).
Compared to fine-tuning the full MobileNetV2-1.4, TinyTL\xspace not only greatly reduces the activation size but also reduces the parameter size (by applying weight quantization on the frozen parameters) and the training cost (by not updating weights of the feature extractor and using fewer training steps). Specifically,
TinyTL reduces the memory footprint by 9.9$\times$, and reduces the training computation by 27$\times$ without loss of accuracy. Therefore, TinyTL is not only more memory-efficient but also more computation-efficient.

redpajama_set_name: RedPajamaArXiv


text:
If you experience any difficulty in accessing content on Small Business Revolution, please contact us at 866.332.6127 or email us at accessibility@deluxe.com., and we will make every effort to assist you.
Articles & Tools
One Hundred Stories
Small Business Big Heart
SEASON 5 WATCH NOW
Story #43Drake MechanicalBoise, ID
It's just bred into my genes at this point. I don't know anything else.
It's rare to come across a small business that has been around for more than 100 years.
Since 1909, Drake Mechanical has been serving the plumbing, water, and mechanical needs of Idaho's Treasure Valley area. In a marketplace that has changed dramatically over the last century, Nick Drake is keeping the family business alive by learning to adapt while still holding onto the hard-working values instilled in him at an early age.
"If you want it, you've got to work for it." That is what Drake's grandfather and father used to repeat to him. By age 14, Drake was digging ditches, and as soon as he graduated high school he started his plumbing apprenticeship. Today, at age 35, he is president of Drake Mechanical and the fifth generation of Drakes to run the business.
Beyond the broad spectrum of work that they do, Drake Mechanical has established a reputation as an active member of the Boise community through charity involvement, and Drake himself is a member of the Good Samaritan Board that provides low-income housing and meals to veterans and disabled adults. Their hard work has paid off, as 60 percent of their business is repeat customers – who clearly trust the work that they do and rely on Drake Mechanical.
Being in business for more than 100 years, Drake Mechanical has weathered its fair share of economical storms – from the Great Depression, to the big recessions of the 80s and recent years. If a company isn't able to adapt and try new things, it won't survive. "We had to re-learn how to do business overnight," Drake says. Whereas the majority of their work used to be in construction, the 2008 recession forced them to shift their business to service and repair, which now makes up 90 percent of their business.
Through adapting along with their customer base and the industry as a whole, Drake Mechanical has managed to beat the odds and keep their family owned and operated business alive. Nick Drake is proud of that fact, and that he has 25 employees who are able to support their families – something he never forgets. "It's not just my kids I have to feed, I have to feed their kids and keep a roof over their head," he says.
Drake hopes that in the future their construction business will pick back up again, and he also hopes that one of his children becomes the sixth generation president of the company and carries on its legacy. "I want my kids to have something, if they want to be involved, great, if they don't want to, then [they'll] do what they enjoy."
Photos by Jared Moossy
Nick Drake's children play in the shop while their parents work in the office. Drake's daughter is now the same age that he was when he started going into the shop with his father, and she sometimes helps her mom out with paperwork in the shop. Drake's hope is that one of his children will want to carry on the Drake Mechanical legacy and become the sixth generation owner.
Nick Drake, owner of Drake Mechanical, inspects the duct work and HVAC system that his employees installed previously.
Detail of the duct work Drake Mechanical installed in the rafters of a remodel in downtown Boise, Idaho.
Nick Drake and Jerry Roberts look at the duct work the company installed in the rafters of a remodel in downtown Boise, Idaho. As an established part of the Boise community for over 100 years, Drake Mechanical has built up a loyal customer base, with 60 percent of their business coming from repeat customers.
Nick Drake and employees Jerry Roberts and Shayne Singleton park in downtown Boise, Idaho to inspect a previous job.
An employee at Drake Mechanical opens the back of a service van to load the day's supplies.
Nick Drake, owner of Drake Mechanical, now 35, took over ownership of the shop from his father just last year, but started going to the shop with him as early as age 10. He told his dad he wanted a bicycle and his dad told him, "If you want it, you have to work for it." These values, instilled in him at such a young age, are the backbone of Drake Mechanical and the reason it has stayed in business all these years.
Detail of the front door at Drake Mechanical, a fifth generation family-owned business that has been a staple for plumbing and mechanical work in the Boise, Idaho area for over 100 years.
One of many trucks in the Drake Mechanical fleet waits to depart for service calls in the city. During the recent recession, Drake Mechanical stayed alive by shifting their business from being mostly construction, to being 90 percent service and repair.
Drake Mechanical's Boise, Idaho location, where they've been since the mid 80s, is just a mile from the airport. Though it's not a large space, it's bigger than their first office - which was Drake's great grandfather's house.
Nick Drake, owner of Drake Mechanical, looks over some paperwork before signing off on the job.
Employees of Drake Mechanical attend their daily morning meeting to discuss the day's schedule. Employing 25 people is not something owner Nick Drake takes lightly. "It's not just my kids I have to feed, I have to feed their kids and keep a roof over their head."
In addition to plumbing and mechanical work, Drake Mechanical offers a suite of services including building sewer plants, central vacuums, heating and air conditioning, duct work, every facet that has to do with air/water, piping... "You name it, we do it." In their 100-plus years of being in business, Drake Mechanical has built a very loyal customer base, with 60 percent of their business coming from repeat customers.
Nick Drake, owner of Drake Mechanical, meets his employees before the day starts and they head out on service calls. Nick Drake is the fifth generation of Drakes to own the business, which has been a staple for plumbing and mechanical work in the Boise, Idaho area for over 100 years.
Proprietor: Nick Drake
5551 W Gowen Rd.	Boise, ID 83709
PH: 208-362-6200 / Website
« Reuben's Brews
Bobby J & Stuff Like That Band »
Independence. Originality. Belief in the bold experiment. Now more than ever, the passionate values that built America drive millions of our local businesses. Deluxe has been a proud champion of small business since 1915 and knows their impact is most apparent in small towns. To celebrate the incredible energy, drive and passion behind every successful small business, Deluxe took the Small Business Revolution to Main Street, awarding one small town a $500,000 revival and showcasing the transformation in an original online series. Learn more about our services and small business marketing solutions at www.deluxe.com.
Learn how we can help with ...
Promo & Apparel
Privacy Policy | Contest Rules | Do Not Sell My Personal Information | Website Accessibility Policy Copyright © 2021 Deluxe. All Rights Reserved.
_ _ Back


Stephanie Veltmann (Fashion Communication and Design '09) is a brand strategist, designer and principal of Bare Brand, a strategic creative agency. Stephanie is also the President of the Ryerson University Alumni Association Board of Directors.
I began working as a graphic designer in 2009, having recently graduated from Ryerson. This was not my first foray into an office environment but with school behind me, working a full week in the same four walls changed my perspective. It soon wasn't hard to notice productivity slipping to office distractions and courteous chit-chat, not to mention the tremendous cost to operate a physical space. This led me to realize remote work was the way of the future — to me it was an easy, simple solution.
Bailey Parnell (Radio and Television Arts '15, Communications and Culture '16 MA) is an award-winning digital marketer and the Founder & CEO of SkillsCamp, a soft skills training company. Bailey was named one of Canada's Top 100 Most Powerful Women. Her TEDx talk and research at Ryerson University focus on social media's impact on mental health.
Last year, I gave a TEDx talk about social media's impact on mental health, discussing concepts such as highlight reels, social currency, FOMO, and online harassment, which have led to lower self-esteem, anxiety, stress, and depression. I suggested small, one-time negative instances might not dramatically reduce your mental health, but when micro-aggressions repeatedly occur over time, you have a macro problem.
Jesica Cuaresma is a soon-to-be graduate of Ryerson's Business Management program. With a background in the arts — specifically visual arts and musical theatre, she hopes to integrate her creativity, along with her knowledge of marketing within the business world. Her passions include travel, fashion, photography and music.
As human beings living in the digital age, the internet is endlessly shifting the ways in which we communicate with one another.
Tanya Kostiw (MA, Literatures of Modernity '13) is the editor of the Alumni Blog and the Marketing and Communications Liaison for the Ryerson University Alumni Association's Board of Directors. She is the Associate Creative Director at Mayk Ideas.
Convocation is a momentous day — one that can be equal parts exhilarating and frightening. After all, it's the culmination of years of hard work and the start of a new chapter.
As Ryerson students prepare to take their first steps into post-graduation life, this month's Alumni Blog reflects on some key career-related lessons (for new grads and seasoned professionals, alike) that I have picked up since graduation.
Samantha Osaduke is a Health, Safety and Environment Coordinator at Ainsworth Inc. She has just under 3 years experience in Human Resources in various industries.
Changing careers can be daunting, and even more so after you've established yourself in a career for a few years. Most of your contacts are in your initial industry, and switching careers usually involves more continuing education on your own time with personal sacrifices, not to mention ensuring it doesn't interfere with your current career. Once you do have the education behind you, you need to make the move. I did just that.
Jonathan Chiriboga (Business Management '04, MBA '09) is a Digital Marketing & CRM Lead at McDonald's Canada. Jonathan shares his passion for marketing, leadership and personal branding at digitalopenconcept.com.
This month I'll be celebrating 13 years in the professional workforce. After six companies, two promotions and an MBA somewhere in the middle of all that, I constantly remind myself to run my own race.
Ron Alphonso (Retail Management '81, Business Management '84, MBA '09) is a Toronto-based mortgage specialist. He can be found at www.mortgagebrokerstore.com and on twitter @brokerstore.
I had sold my previous business and was looking to start a new one. I spent about 3 years searching for a business that could make money and have a good growth rate. This included going back to Ryerson to complete my MBA; I felt an MBA would give me a different point of view and help me evaluate my business opportunities.


text:
Peekskill Daily Voice serves Peekskill, NY
serves Brewster, Carmel-Kent, Cold Spring, Mahopac, Patterson & Putnam Valley
serves Buchanan, Cortlandt & Croton
Albert DeCrenza, 76, Of Peekskill
Nicole Zappone
More Articles Obituaries
Paula Rollins, Deputy Commissioner, Clerk In Westchester, Die...
Rockefeller Associate James Ford, Founder Of Acclaimed Stone ...
Lifelong Westchester Resident Alison Sullivan Dies At 26
PEEKSKILL, N.Y. -- Albert DeCrenza, a resident of Peekskill, died Monday, Feb. 1. He was 76.
He was born on Feb. 15, 1939 in Ossining,, to Alfonzo and Irene DeCrenza.
DeCrenza was a truck driver for Teamsters Local 456 in Elmsford, NY. He served in the U.S. Army from 1956-1959 and also served in the Army National Guard. He was a member of the Knights of Columbus Council 462, past President of the Sons of Italy in Ossining and was also a volunteer at Hudson Valley Hospital.
He is survived by his wife Cathleen DeCrenza, and his brothers; Salvatore Caiazza of Cortlandt Manor, Gary DeCrenza, Michael DeCrenza and Robert Longwell. Also survived by several nieces and nephews.
He was predeceased by a son, Albert Peter DeCrenza, Jr. and brothers; James Caputo and James Caiazza.
A Mass of Christian Burial will be celebrated on Friday, 10 a.m. at the Assumption Church. Interment will follow at Assumption Cemetery.
Susan Monaco named chief revenue officer at CareMount Health ...
Andi Gray: Combining brands after an acquisition
Peekskill Daily Voice!


text:
Mankato Stabbing Suspect Sought
Apr 21, 2021 @ 10:33am
A warrant has been issued for a Mankato man accused of stabbing one woman and injuring another.
Police say Travehl Jimmi Kelker, 32, fled the scene of a Mankato home Monday after he allegedly choked and stabbed a woman who had recently broken up with him.
Another woman who attempted to assist the victim was assaulted and threatened by Kelker, according to court documents.
One of the victims told police she'd called off her wedding with Kelker because of an unreported incident in which Kelker had strangled her. When she asked him to leave the apartment, he grabbed her by the throat and forced her against the wall, according to a criminal complaint filed in Blue Earth County Court Tuesday.
A second woman tried to intervene, but Kelker threatened to assault her, so she grabbed a kitchen knife to defend herself. Kelker let go of the victim, but produced his own knife, lunged at the woman, and threatened to kill her says the complaint.
Kelker stabbed his ex in the leg, and her hands were cut while trying to defend the woman, according to the charging document. He allegedly pushed past his ex and kicked the woman in her chest twice.
Kelker fled the apartment when the women called 911.
Responding police say the ex-girlfriend was visibly shaken by the assault, had blood on her hands, and a puncture-type stab wound that was dripping blood.
Both women were transported to a hospital.
Kelker faces felony counts of 2nd-degree assault with a dangerous weapon, threats of violence, and domestic assault.
Police say Kelker has two previous domestic assault violence-related convictions out of Ramsey County.
Source: southernminnesotanews.com


text:
[Jump to content] Quicklinks menu
Home: Heritage New Zealand
Currently consulting on
Archaeological authority
Notified proposals & reviews
Statements of general policy
Results for '{{searchNavItems.Name}}': {{ (flattenedTree | filter:searchNavItems).length }}
{{navItem.Name}}
Visit »
Places to Visit »
Wellington »
Old Government Buildings
Pompallier
Learning and discovery
Te Waimate Mission
Clendon House
Māngungu Mission
Alberton Summer Market Days
Alberton High Tea
Highwic
Melanesian Mission
Ewelme Cottage
The Waikato War
Driving tour sites
Pukekohe Church
Queen's Redoubt
Koheroa
Whangamarino
Rangiriri
Te Wheoros Redoubt
Ngaruawahia
Paterangi
Rangiaowhia
Kihikihi
Orakau
Alexandra Redoubt
Thames School of Mines
Hurworth
Old St Pauls
Funerals and memorials
Upper Hutt Blockhouse
Pencarrow Lighthouse
Turnbull House
Fyffe House
Cotons Cottage
Kate Sheppard House
Lyttelton Timeball
Hayes Engineering
Ophir Post Office
Totara Estate
Teatime Special
Clarks Mill
Matanaka
Tohu Whenua
Northland Tohu Whenua
Otago Tohu Whenua
West Coast Tohu Whenua
Pompallier Mission
Hayes Engineering Works
Mangungu Mission
Old St Paul's
In its heyday New Zealand's largest and grandest wooden building housed our entire public service. Government Buildings is an outstanding example of New Zealand's architectural heritage and one of the great wooden buildings of the world.
Image: Grant Sheehanexpand/collapse
The building was designed in the classically-derived Italian Renaissance revival style. Like many colonial buildings of the era, it was built to resemble a stone palace. This was to convey strength and stability in a young country undergoing rapid growth and change.
To avoid being seen as extravagant, the new government chose to build in timber including extensive use of kauri, and it remains probably the world's largest timber office building. The constant threat of fire meant that it also became one of New Zealand's first smoke-free buildings.
It was completed in 1876 - an important turning point in New Zealand's political history. This year saw the provincial governments abolished and a central government established. For 56 years, the building was the home of Ministers' offices, the Cabinet room and all Wellington-based civil servants.
It features include two staircases, eight vaults, 143 rooms, 126 fireplaces, 22 chimneys, two hydraulic lifts, 64 toilets, eight verandas and seven porticos. Government ministers used the building during Parliament's recess until 1921. The Executive Council met in the building until 1948.
Before long, the public service had rapidly expanded beyond the building's capacity. By 1990, the last of the public service departments had moved out, concluding 114 years of government service.
Limited restoration began in the early 1980s but by 1990 the building was empty. The government decided to restore the building and work began in 1994 under the management of the Department of Conservation. The work cost $25 million and was completed early in 1996. It is regarded as a landmark in government-initiated heritage conservation.
Today, the Law Faculty of Victoria University of Wellington occupies most of the building that has been at the heart of New Zealand's central government activities since 1865. Visitors may view the displays on the ground floor and the Cabinet room on the first floor. The grounds feature examples of New Zealand's flora, including many rare species.
Group guided tours
Explore some of the behind-the-scenes features of this stunning building on one of our summer guided tours every weekend from 16 January until 28 March. Numbers on each tour are limited to ensure everyone enjoys themselves so all tours must be pre-arranged. Minimum of five persons and maximum of 25. The cost is $15 per person.
Meet your tour guide, Mike Deavin at the main entrance about 10 minutes beforehand.
Sorry, there is no public parking available onsite at Old Government Buildings.
Summer WEEKEND Tours:
Saturdays and Sundays from 16 January - 28 March 2021.
10am - 11.30pm
Cost: $15 per person.
Tours require a minimum of 5 persons and maximum of 25. Pre-booking for groups is essential - please email information@heritage.org.nz with your name, contact phone number and the number of people in your party, or telephone (04) 472 4341.
Individual tours may be arranged on the day at reception at the main entrance.
New Zealand's largest and grandest wooden building.
Looking down the hanging staircase. Image: Grant Sheehan
Government Buildings at dusk. Image Sam O'Leary DOC
The former Cabinet Room can be visited today. Image: Grant Sheehan
The stunning kauri entranceway. Image: Grant Sheehan
A statue of Peter Fraser, the former Prime Minister greets visitors. Image: Grant Sheehan
A great example of the old-style birdcage lifts. Image: Grant Sheehan
The building once housed the entire public service. Image: Grant Sheehan
55 Lambton Quay
Email: information@heritage.org.nz
URL: http://www.visitheritage.org.nz
Monday to Friday (public display rooms only)
Saturday and Sunday 16 January - 28 March
9am - 5pm.
Summer guided tours: Weekends 16 January - 28 March 2021
$15 per person to join a guided or group tour. Group tours may be arranged on request. The building is privately leased to the Faculty of Law, Victoria University of Wellington.
Promote Heritage
© Heritage New Zealand 2021


text:
Chemical substances
Chemical substances fact sheets and frequently asked questions
Chemicals at a glance
Phosphoric Acid Derivatives Group - information sheet
Phenol, dimethyl-, phosphate (3:1) (trixylyl phosphate)
9-Octadecen-1-ol, (Z)-, phosphate (oleyl phosphate)
Phosphorous trichloride, reaction products with 1,1-biphenyl and 2,4-bis(1,1-dimethylethyl)phenol
CAS Registry Number 119345-01-6
About these substances
Human and ecological exposures
Key health and ecological effects (hazard)
Risk assessment outcomes
Screening assessment conclusions
Preventive actions and reducing risk
The Government of Canada conducted a science-based screening assessment under the Canadian Environmental Protection Act, 1999 (CEPA 1999) to address the potential for harm to Canadians and to the environment from 3 of the substances in the Phosphoric Acid Derivatives Group.
The risk posed by a substance is determined by considering both its hazardous properties (its potential to cause adverse human health or ecological effects) and the amount of exposure there is to people and the environment.
The ecological hazard and exposure potentials of these 3 substances were classified using the Ecological Risk Classification of Organic Substances (ERC) Approach.
As a result of the screening assessment, the Government concluded that these 3 substances are not harmful to human health or to the environment at levels of exposure considered in the assessment. One of the substances, trixylyl phosphate, is associated with human health effects of concern; however, the risk to Canadians is low at levels of exposure considered in the assessment.
The screening assessment summarized here focused on 3 of 6 substances referred to collectively as the Phosphoric Acid Derivatives Group, under the Chemicals Management Plan (CMP). These substances are trixylyl phosphate, oleyl phosphate, and the substance with CAS RN 119345-01-6.
Three other substances in the Phosphoric Acid Derivatives Group (CAS RNs 68604-99-9, 68952-35-2 and 111174-61-9) were determined to be of low concern to both human health and the environment through another approach. Conclusions for these substances are provided in the Final Screening Assessment for the Rapid Screening of Substances with Limited General Population Exposure.
These substances do not occur naturally in the environment.
According to information gathered by the Government, in Canada, trixylyl phosphate may be used as a flame retardant and in lubricants and greases. It may also be used in food packaging materials, in plasticizers, in hydraulic fluids and insulation for wires and cables.
Oleyl phosphate is used in cosmetics, such as permanent hair dye.
CAS RN 119345-01-6 is used in plastic and rubber materials and may also be used in food packaging materials.
Canadians may be exposed to trixylyl phosphate through the environment (for example, from its presence in house dust) and from certain products available to consumers, such as electronics and automotive fluids.
Exposure of Canadians to oleyl phosphate may occur from the use of certain cosmetics.
Canadians may be exposed to CAS RN 119345-01-6 mainly through their diet from its potential use in food packaging.
According to information considered under the ERC Approach, trixylyl phosphate was identified as having a moderate ecological exposure potential due to its long overall persistence (ability to stay in the environment over time) and moderate quantities of use, while oleyl phosphate and CAS RN 119345-01-6 were identified as having low ecological exposure potential.
To inform the health effects characterization for trixylyl phosphate, international reports of data were reviewed.
Trixylyl phosphate has been classified by the European Chemical Agency (ECHA) as a reproductive toxicant (may damage fertility) and is on the ECHA List of Substances of Very High Concern for reproductive toxicity. Effects on the reproductive organs and adrenal glands were considered to be the critical effects for the assessment of trixylyl phosphate.
No effects on human health have been identified for oleyl phosphate based on available information.
For CAS RN 119345-01-6, potential developmental effects in laboratory studies were the critical effects identified for the assessment.
According to information considered under the ERC Approach, trixylyl phosphate was identified as having a moderate ecological hazard potential, while oleyl phosphate was identified as having a high ecological hazard potential. This was based on their potential to cause adverse effects in aquatic food webs given their respective moderate and high potentials to bioaccumulate. CAS RN 119345-01-6 was identified as having a low ecological hazard potential.
The risk to human health from trixylyl phosphate and CAS RN 119345‑01‑6 was determined to be low, based upon a comparison of levels to which Canadians may be exposed to these substances, and the levels associated with health effects.
The risk to human health from oleyl phosphate was also determined to be low, based upon the information presented in the screening assessment.
Based upon the outcome of the ERC Approach, these 3 substances are considered unlikely to be causing ecological harm.
The Government of Canada published the Final Screening Assessment for the Phosphoric Acid Derivatives Group on December 19, 2020.
The Government concluded that trixylyl phosphate, oleyl phosphate, and CAS RN 119345-01-6 are not harmful to human health at levels of exposure considered in the assessment, and that these 3 substances are not entering the environment at levels that are harmful to the environment.
Although trixylyl phosphate is not considered to be harmful to human health at levels of exposure considered in the assessment, it is considered to have health effects of concern based upon its potential to cause effects on the reproductive system. There may be a potential risk if exposures were to increase.
Therefore, the Government published a notice of intent to apply Significant New Activity (SNAc) provisions of CEPA 1999 to trixylyl phosphate. This publication has a 60-day public comment period ending on February 17, 2021.
The SNAc provisions would require that the Government be notified of certain proposed new activities related to trixylyl phosphate, and that the new activity be assessed for potential risks to human health and the environment before being undertaken.
The notice outlines the proposed definition of a significant new activity in relation to trixylyl phosphate, as well as the information that would need to be provided to the Government for assessment before the new activity is undertaken.
These substances may be found in products available to consumers. Canadians should follow any safety warnings and directions related to the product and dispose of products responsibly.
The screening assessment for the Phosphoric Acid Derivatives Group focused on potential risks of exposure to the general population of Canada, rather than occupational exposure. Hazards related to chemicals used in the workplace are defined within the Workplace Hazardous Materials Information System. For information concerning workplace health and safety and what steps to take in the workplace, Canadians should consult their employer and/or the Occupational Health and Safety Regulator in their jurisdiction.
Please select all that apply:
Login error when trying to access an account (e.g. My Service Canada Account)
GC Key access
SecureKey Concierge (Banking Credential) access
Personal Access Code (PAC) problems or EI Access Code (AC) problems
Social Insurance Number (SIN) validation problems
Other login error not in this list
You will not receive a reply. For enquiries, contact us.


The Navarre Beach Area Chamber of Commerce will kick off its 11th Annual Free Summer Concert Series on Thursday, May 31, 2018. In 2017, "Tunes by the Dunes" (formerly Thursdays in the Park) relocated from Navarre Park to Navarre Beach Park's Sand Crab Pavilion, just east of the Navarre Beach Fishing Pier. Co-hosted by the Okaloosa Gas District, Navarre Press, Santa Rosa County Tourist Development Office, Chamber Foundation and Cat Courty 98.7, the summer concert series offers an exciting line-up of 14 FREE outdoor concerts every Thursday (7-9 pm), from May 31 to August 30, 2018.
The series will showcase a variety of local musicians with the Gulf of Mexico and sunset as the backdrop. Each night, from 7 to 9 pm you will find a different genre of music ranging from Jazz, Reggae, Southern Rock, Country, Oldies and much more. Spectators are encouraged to bring their beach blankets and lawn chairs for a night of free family entertainment. Come hungry as concessions will be provided by some of our Navarre Chamber member restaurants and a dance area will be provided for you to work off those extra calories.
As a reminder, smoking is prohibited and pets are not allowed. In the event of inclement weather, the Navarre Beach Area Chamber of Commerce will announce cancellations by 4:30 PM the day of concert. To ensure you have the most up-to-date information on Tunes by the Dunes, please like the Navarre Beach Area Chamber of Commerce on Facebook.


text:
Every Question You Have About Rome Answered
By Nathalie Alonso; Updated September 26, 2017
When in Rome: Making the most of your visit to the Italian capital
As the former nerve center of the Roman Empire and one of the most sacred sites in Catholicism, Rome's treasure trove of ancient monuments and legendary works of art might make it seem like a giant museum. Foodies will be delighted to know that the Italian capital's cuisine is equally rich. Any way you look at it, the Eternal City is a feast for the senses.
Q: What is there to do in Rome?
A: Art and architecture account for the bulk of the attractions in Rome, a city brimming with ancient wonders that evoke a once-mighty empire and churches filled with masterpieces by the most celebrated Italian painters and sculptors. Must-sees include the iconic amphitheater known as the Colosseum and the Roman Forum, once the political epicenter of the Roman empire. The Pantheon, a Roman temple-turned-church, is not to be missed. Rome's historic squares, which include the Piazza Navona and Piazza del Popolo, are great places to catch your breath and do some people-watching. Don't leave without tossing a coin into the famous Trevi Fountain; legend has it that doing so guarantees your return to Rome.
Everything You Need to Know Before Your Trip to Paris
Do's and Don'ts for Four Days in Rome
All Your Questions About Toronto Answered
The Colosseum: Many a gladiator battled for his life in the Colosseum, which dates to the first century. As you tour the ancient arena, note the different styles of arches on each of its three levels.
Q: Is the Vatican in Rome?
A: The seat of the Catholic Church, Vatican City occupies some 100 acres within the city of Rome. It is, however, a sovereign state – the smallest in the world. Its head of state is the pope. St. Peter's Basilica, the most important place of worship for Catholics, dominates Vatican City.
St. Peter's Basilica: There is no greater Catholic monument than St. Peter's Basilica, the largest church in the world, which encompasses 18,000 square yards. Raphael, Michelangelo and Bernini are just a few of the renowned Italian artists who left their mark on the basilica, which stands upon the tomb of Saint Peter. There is no charge to visit.
Q: Who painted the ceiling of the Sistine Chapel in Rome?
A: The ceiling of the Sistine Chapel in Vatican City, which is covered by a fresco that depicts stories from the Old Testament's Book of Genesis, was painted by one of the most famous artists of the Renaissance, Michelangelo, in the 1500s.
Q: Where can you see art by Caravaggio in Rome?
A: Works by Michelangelo Merisi, or Caravaggio, a key figure in the Baroque art movement, are found in churches and museums throughout Rome. The Church of San Luigi dei Francesi is home to three paintings by Caravaggio that depict Saint Matthew. They include "The Calling of St. Matthew," considered one of his greatest works. You can see other works by Caravaggio at the Capitoline Museums and the Vatican Museums.
The Vatican Museums: With a staggering 1,400 rooms, the Vatican Museums harbor one of thelargest collections of art in the world. It's impossible to see it allin a day, so you'll want to prioritize the highlights, which include the Sistine Chapel and rooms decorated by Raphael.
Q: What is the weather like in Rome, Italy?
A: Temperatures in Rome are, generally speaking, warm and mild. Summer days can be stifling, with average highs in the 80s from June through August, though evenings are pleasantly cooler. Winters are warm, with averages in the 50s. The rainiest months are November and February, while December and January tend to be the coldest.
Q: What is there to do in Trastevere, Rome?
A: Trastevere's name means "across the Tiber," which describes the neighborhood's location in relation to the heart of Rome. Once a working-class district, it now draws an international crowd with its collection of bars, restaurants and nightclubs. Here you can walk 20 minutes to the top of Janiculum Hill for a stunning view of the city. Churches worth perusing in Trastavere include the basilicas of Santa Maria and Santa Cecilia.
Q: What should I eat in Rome?
A: Because pizza and pasta are among the city's signature dishes, Rome is not the place to forgo carbs. Its pizza is distinguished by paper-thin crusts. When it comes to pasta, you might find that simple is better when you try cacio e pepe – spaghetti-like pasta served with a sauce that consists of pepper and cheese. Artichokes, prepared many different ways, are also a staple of Roman cuisine. More adventurous eaters might want to explore the side of the menu that features offal – animal parts like tripe and hearts – which is another defining element of Roman cuisine. For dessert, authentic gelato, or Italian ice cream, is the way to go.
Q: How do I get from Rome to Florence?
A: The Tuscan city of Florence, a major artistic center considered the birthplace of the Renaissance, is an hour and a half away from Rome by high-speed train. You have a choice of two companies: the state-run Trenitalia, which operates the Frecciarossa or Frecciargento with service between Rome and Florence, and the privately-owned Italo. Alitalia also offers flights between Rome and Florence that are just under an hour long.
All Your Questions About Stockholm Answered
All Your Questions About Istanbul Answered
All Your Macau Questions Answered
Do's and Don'ts for Three Days in Venice
Frommer's: When to Go in Rome
Vatican City State: State and Government
Vatican Museum: Sistine Chapel - Ceiling
Lonely Planet: A perfect day in Trastevere, Rome's favourite neighbourhood
Frommer's: Getting There in Florence
Frommer's: Neighborhoods in Brief in Rome
Fodor's Travel: The Colosseum Review
Italian Tourism Official Website: Itinerary: Caravaggio's Artworks in Rome
The Metropolitan Museum of Art: Caravaggio (Michelangelo Merisi) (1571–1610) and his Followers
Fodor's Travel Guides: Musei Vaticani Review
Business Insider: What to Eat in Rome
Frommer's: Restaurants in Rome
Fodor's Travel: Basilica di San Pietro
Frommer's: The Best Town Squares in Rome
Nathalie Alonso
Every Question You Have About Barcelona Answered


text:
Come From Away to Release Two-LP Vinyl Set of Cast Album Worldwide
The blue vinyl set drops May 15.
The cast of Come From Away on Broadway.
(© Matthew Murphy)
Molly Records and Junkyard Dog Productions have announced that Craft Recordings and Concord Theatricals will drop a two-LP, blue vinyl set of the original Broadway cast recording of Come From Away on Friday, May 15. Previously available as a limited-edition yellow vinyl in the United States, this edition, cut by Clint Holley at Well Made Music and pressed at Memphis Record Pressing, will be available worldwide.
Featuring a book, music, and lyrics by Irene Sankoff and David Hein, Come From Away is directed by Tony winner Christopher Ashley and choreographed by Kelly Devine. Set against the backdrop of the tragic events of September 11, 2001, the show follows what happens when 38 planes and 6,579 passengers are forced to land in Gander, Newfoundland, doubling the population of "one small town on the edge of the world."
The Grammy Award-nominated original Broadway cast recording features Petrina Bromley, Geno Carr, Tony Award nominee Jenn Colella, Joel Hatch, Rodney Hicks, Kendra Kassebaum, Tony Award nominee Chad Kimball, Lee MacDougall, Caesar Samayoa, Q. Smith, Astrid Van Wieren, Sharon Wheatley, Josh Breckenridge, Susan Dunstan, Tamika Lawrence and Tony LePage. David Lai, Ian Eisendrath, August Eriksmoen, Irene Sankoff and David Hein serve as album producers.


These groups aren't protected by a vehicle body in the same way car users are for example, and as they are harder for other drivers to see on the road, they are particularly vulnerable to accidents and injuries.
Taking that extra second to double check and take another look can mean the difference between an accident happening, or not. It's all about awareness of what other road users around you are doing. What's also certain is that the vast majority of them know nothing about driving a commercial vehicle so you've got to make allowances for them. It's all stuff you do every day, but an occasional reminder never does any harm.
Motorcyclists account for just 1% of road traffic in the UK but account for one in five of all road user deaths. According to THINK! they are roughly 38 times more likely to be killed in a road traffic accident than car occupants, per mile ridden.
Think bike! Take longer to look for motorcyclists. In nearly half of accidents involving motorbikes, drivers of other vehicles had failed to look properly when pulling out of a junction.
Check for bikes when changing lanes, turning and double-check that bikes aren't already passing you when you're planning to turn.
Don't underestimate speed. The Transport Research Laboratory suggests that even when a driver has seen a motorcyclist approaching the junction, they underestimate the speed of travel.
Keep your distance. Driving too close can intimidate less experienced motorcyclists.
Cycling has become an increasingly popular way to commute. Handy 'cycle lanes' enable riders to pass queuing traffic on the left at junctions. But this is one of the places where things can get nasty.
Pedestrians are amongst the most vulnerable road users. Always be aware of children, older and disabled people, particularly in crowded shopping streets and residential areas. According to RoSPA, 60 child pedestrians are killed or seriously injured every week and nearly half of all pedestrians killed are aged over 60.
More than half of all road accidents involving horses occur on minor roads. Horses are powerful animals and can be frightened easily. Their riders can be adults or children, and sometimes ride in double file to protect novice riders or nervous horses.
If you're pulling out into the middle lane on a motorway, use your wide angle mirrors to check there's nothing approaching from behind, especially when they are swapping from the fast lane to the middle lane. This is, apparently, when most motorway shunts occur.
Above all, always remember 'mirror, signal and then manoeuvre'. Making that final observation can make the difference between having an accident or avoiding one.


CoolAir exchange system for terminals and computers.
Access to financial market through the simple gaming interface. A fascinating and exciting process of trading with genuine excitement. The trading system TradeBox from the company CoolAir is a profitable investment!
TradeBox is a profitable investment!