text:
\section{Introduction}

\textbf{Motivation.} Self-supervised representation learning methods \cite{Chen2020, Caronb, Chenb, Chenf, Bardes} are the new norms for training large scale deep neural networks (DNNs). Their ability to exploit the underlying structure of the data from a large unlabelled corpus during pre-training makes them a preferred choice for transfer learning \cite{goyal2021self}. Pre-training not only helps with better transferability but also leads to better generalization on various downstream tasks compared to supervised pretraining. This ability, however, comes at the cost of substantially larger model size, computationally expensive training strategies (larger training times, large batch-sizes, etc.) \cite{chen2020big, goyal2021self}, and accordingly more expensive inference times. Such strategies prove effective in achieving \textit{state-of-the-art} (SOTA) results on several downstream tasks in computer vision but are impractical under resource (memory, power and compute) constrained \textit{industrial} settings, where major emphasize is given on deploying a lightweight model on edge devices.
\begin{figure*}[t]
  \centering
    \includegraphics[width=1.00\linewidth]{figure/main_figure.png}
    \vspace{-1.75\baselineskip}
    \caption{This figure illustrates different settings; \textbf{\textit{a.}} Shows the general setting for achieving a lightweight sub-network/small-network following different techniques. \textbf{\textit{b.}} More relevant to this work, illustrates the setting showcased in \cite{krishna2022dynamic} but modified as per our use case (i.e, instead of SimSiam\cite{chen2021exploring} we use VICReg objective). \textbf{\textit{c.}} \textit{This work}: we  learn a \textit{dense} encoder and some gates based on a  budget constraint $t_{d}$. With this gating mechanism the \textit{dense} encoder can be exploted as a \textit{lightweight} sub-network as well as \textit{dense} network. Here \textbf{\textit{b}} and \textbf{\textit{c}} are based on gating mechanisms introduced in \cite{veit2018convolutional, Li_2021_ICCV}. Figure best viewed in color.}
    \label{fig:main}
\end{figure*}

 A common practice to reduce this computational burden is to extract a lightweight sub-network\footnote{A network derived from another large model/network} from an off-the-shelf pre-trained model, or pre-training the model as a part of a multi-step training process and further compressing it by applying techniques like Knowledge distillation (KD) \cite{hinton2015distilling}, pruning \cite{frankle2018lottery}, dynamic computation (DC) \cite{veit2018convolutional}, etc. SSL based pre-training combined with KD\footnote{For clarification in KD a different lightweight network (not sub-network) is learnt.} \cite{tian2019contrastive, abbasi2020compress, fang2021seed}, DC \cite{krishna2022dynamic, 9878423}, or pruning \cite{caron2020pruning, chen2021lottery} also serve as effective ways to obtain a lightweight sub-network  for a given downstream task. This sequential learning procedure often involves fine-tuning a pre-trained self-supervised  model on a downstream task along with the corresponding training objective of KD, DC, or pruning with cross entropy (CE) loss.  This works, but obtaining a sub-network from large pre-trained models (e.g. large language models (LLMs)) via fine-tuning makes the overall process computationally more expensive and cumbersome. Furthermore, downstream tasks are diverse and vary widely; therefore, any change in the downstream task usually requires repeating the entire procedure multiple times, making it highly inefficient and less transferable.

\textbf{Research Questions.} These limitations compel us to ask the following question: $\bullet$ \textit{``Can we unify the learning of a lightweight sub-network along with a dense network from scratch and in a completely self-supervised fashion?"} In other words, by  the end of a pre-training procedure one has access to a \textit{dense} and a \textit{lightweight} sub-network simultaneously, saving not only time obtaining the \textit{sub-network} after pre-training, but also giving flexibility to exploit both the \textit{dense} and the \textit{lightweight} network as per the use case. Instead of fine-tuning a large model to get a lightweight network one can directly work on  lightweight sub-network directly after pre-training. Having said that, our focus is on faster inference however we intend to achieve this by not adding too much computational overhead during training. This leads us to reformulate the research question to: $\bullet$ ``\textit{Can we learn an encoder (function) that could serve the dual purpose of being used as a dense and lightweight network with minimal additional overhead?}'' Our goal is to simultaneously learn a \textit{dense} and a \textit{lightweight} model under a single training procedure. In Figure~\ref{fig:main}c we demonstrate the dual setting of obtaining a dense and a lightweight network (derived from dense). Mathematically, given an encoder $f_{\theta}$ and gating functions $\mathbf{\textbf{W}}$ then a sub-network $\mathcal{S}(\mathbf{x}) = f_{\theta}(x,W\odot \theta)$, where $\mathbf{W}$ can be interpreted as learning a mask at different levels in the network (Figure \ref{fig:modified} in Appendix \ref{channel_selection} ). For obtaining $\mathbf{W}$ we follow \underline{dynamic channel selection} (DCS) \cite{veit2018convolutional, Li_2021_ICCV} to induce sparsity while maintaining network topology. Figure~\ref{fig:main}a depicts the traditional setting alternating between pre-training and fine-tuning while Figure~\ref{fig:main}b depicts the setting recently introduced in \cite{krishna2022dynamic} using dynamic channel selection along with self-supervision. Also from now on, lightweight, sub-network or gated network refer to same thing.

The lightweight network can be interpreted as \textit{student} while dense network as the \textit{teacher} where the \underline{\textit{teacher} is as bad as (initially un-trained) the \textit{student}}.  Note that in this work we do not follow the vocabulary of teacher-student networks. Instead we restrict the terminology to lightweight (Gated) and a dense network, where encoder and gates\footnote{encoder with gates $\equiv$ sub-network depicted very clearly in Figure \ref{fig:main}c} are randomly initialised and trained jointly from scratch, with an aim that they co-evolve during training. It is, however, important to mention that in this work we are not proposing any new KD objective  or KD induced learning algorithm nor any new DC objective neither any pruning based learning. Instead we provide a novel \underline{perspective} on exploiting the synergies that exist between self-supervised representation learning and dynamic computing.
In addition to the teacher-student paradigm, this work also takes inspiration from SDCLR \cite{jiang2021self} and APPRENTICE \cite{mishra2017apprentice}, which exploit a joint student-teacher KD based approach for very specific applications.

To this end we focus on joint embedding \underline{symmetric}-siamese architecture specfically VICReg \cite{Bardes} architecture amalgamated with dynamic channel selection \cite{veit2018convolutional,Li_2021_ICCV}. Below we outline our main contributions:

\begin{itemize}
\item We present a novel perspective of unifying the learning of \textit{dense} and \textit{lightweight} networks by exploiting a symmetric joint embedding architecture of the SSL paradigm.

\item We demonstrate that a single encoder can be exploited as a dense as well as a lightweight network, we show in Table~\ref{table1} and Table~\ref{table2} that a single base encoder can serve this \underline{dual} purpose. This not only reduces computational overhead during training but also gives enough flexibility to use a single network and exploit it as per its requirement.

\item We exhaustively, through experiments, demonstrate that this unification preserves feature quality across different experimental settings and gives on-par performance when compared with strict baselines (Section \ref{result}).
\end{itemize}

\section{Related Works}
\subsection{Self-supervised Visual Representation Learning} SSL has recently matched the performance of supervised learning on several computer vision benchmarks \cite{Chen2020, Caronb, Chenb, Chenf, grill2020bootstrap, hjelm2018learning, wu2018unsupervised, asano2019self}. At the core of these approaches is the concept of learning joint embedding representations realised via a Siamese \cite{1467314} architecture through instance discrimination.
Most of the design choices of SSL methods are formulated in a way to avoid \textit{feature collapse}\footnote{dimensional collapse, information collapse}.
Broadly, these SSL methods can be divided into contrastive and non-contrastive techniques.
Contrastive learning \cite{wu2018unsupervised, oord2018representation, Henaff2020, Chen2020, he2020momentum, tian2020makes,9226466} based approaches intuitively try to bring similar instances closer in the embedding space while contrasting them with negative samples. Non-contrastive techniques includes clustering methods \cite{caron2018deep,asano2019self}, which alternate between cluster assignment and predicting clusters as pseudo labels, which make them computationally cumbersome as they require running a clustering step over the entire dataset. SwAV \cite{Caronb} mitigates this computationally expensive step by making the clustering completely online.
Approaches like BYOL \cite{grill2020bootstrap}, OBoW \cite{gidaris2020online}, and SimSiam \cite{chen2021exploring} avoid using contrastive samples or clustering, exploiting tricks like a ``momentum encoder" or applying a ``stop-gradient" operator in one branch \cite{chen2021exploring}. However, why and how these techniques avoid collapse is still unclear and an open research area. A more principled way of learning representations while avoiding feature collapse relies on information maximization; examples include: Barlow Twins \cite{Zbontar2021}, W-MSE \cite{Ermolov}, and VICReg \cite{Bardes}.  In this work we exploit one of the key property of VICReg i.e, \textit{independent} branch regularization which works better in scenarios where representation in each branch has different statistics, so the amount of regularization required for each branch may vary a lot, though this property seems to be under utilized in the SSL domain.

\subsection{Dynamic Computation (DC)}
DC is an effective, adaptive, and resource efficient computing mechanism to reduce  model complexity by skipping unimportant parts of  network depending on the input.
Several authors including \cite{figurnov2017spatially,mcgill2017deciding,huang2018multi,wu2020dynamic,wang2018skipnet} have proposed adding decision branches to different layers of convolutional neural networks (CNN) for learning early exiting strategies leading to faster inference. BlockDrop \cite{Wu_2018_CVPR} and SpotTune \cite{guo2019spottune} learn a policy network to adaptively route the inference path through fine-tuned or pre-trained layers.
ConvNet-AIG \cite{veit2018convolutional}, \cite{herrmann2020channel} introduced a network that adaptively selects specific layers of importance to execute depending on the input image by specifying a target rate of each layer. GaterNet \cite{chen2019you} introduced a network to generate input-dependent binary gates to select filters in the backbone network. CGNet \cite{hua2019channel} exploits spatial diversity to allocate different computation for different activations while DGNet \cite{Li_2021_ICCV} proposed a dual gating mechanism to induce sparsity along spatial and channel dimensions.
Furthermore, dynamic channel pruning methods have also been devised such as feature boosting and suppression (FBS) \cite{gao2019dynamic} to dynamically amplify and suppress output channels computed by CNN layers. Other works learn sparsity through a three-stage pipeline: \textit{pretrain-prune-finetune} as in \cite{tiwari2020chipnet} or use pre-trained models. See \cite{hoefler2021sparsity} for a more detailed explanation of sparsity, pruning, and dynamic computing.
In this work we use the dynamic channel selection mechanism proposed in \cite{veit2018convolutional,Li_2021_ICCV}.

\subsection{Self-supervised Dynamic Computation and Beyond}
Most of the works on dynamic computation have been mostly confined to supervised learning. Recently, \cite{krishna2022dynamic} used SimSiam \cite{Chenf} as a self-supervised objective combined with a dynamic channel gating (DGNet) \cite{Li_2021_ICCV} mechanism \textit{trained from scratch}, and showed that comparable performance can be achieved under channel budget constraints. Likewise \cite{9878423} used a channel gating-based dynamic pruning (CGNet) \cite{hua2019channel} augmented with contrastive learning to achieve inference speed-ups without substantial loss of performance.
Similarly, \cite{caron2020pruning} showed that the performance of the pruned sub-network is maintained on the pretext task, which is further maintained after being transferred to a downstream task. In a similar line of work, \cite{chen2021lottery} used iterative magnitude pruning (IMP) to obtain a winning ticket \cite{frankle2018lottery} for a pre-trained task (self-supervised objective) and evaluated its performance on various downstream tasks. \cite{9856963} extended the work done in \cite{caron2020pruning} in a MoCo (pre-trained) setting
augmented with ADMM \cite{zhang2018systematic} for systematic pruning
A self-supervised \textit{loss} objective can serve as a tool for knowledge distillation (KD) \cite{hinton2015distilling} and model compression (MC). \cite{tian2019contrastive} used contrastive objective (along with supervised loss \textit{task specific distillation}) to train a student network from pre-trained network. Similar to \cite{tian2019contrastive} but in a completely self-supervised setting  \cite{abbasi2020compress, fang2021seed} minimizes $KL$-divergence between the distribution of similarities for teacher (pre-trained) and student networks. The authors in \cite{xu2020knowledge} used a two-step strategy to train teacher (learn with labels and then learn SSL head with fixed backbone) followed by learning student using KD loss. However, we follow a more simplisitc approach through unification of SSL (VICReg) and DCS where DCS  adds more flexibility as compared to pruning (or KD) by maintaining network topology and making the fine-tuning process convenient on various downstream tasks while for later approaches it's irreversible (network structure).

\section{Preliminaries and Setups}
\subsection{Method} \label{method}
\textbf{VICReg.} Based on the principle of \textit{information maximization}, VICReg \cite{Bardes} learns a joint embedding space governed by a loss objective which comprises of \textit{invariance} ($s$) (mean squared error (MSE)), \textit{variance} ($v$) and \textit{co-variance} ($c$) as shown in Equation \ref{eq1}. Assume an image dataset $\mathcal{D}= \{\mathbf{x}_{1},\cdot \cdot \cdot\cdot,\mathbf{x}_{|\mathcal{D}|}\}$, where $\mathbf{x}_{i} \in \mathbb{R}^{3 \times H \times W}$ with empirical probablity distribution $p(\mathbf{X})$ and a set of transformations $\mathcal{T}$ (see Appendix \ref{a1} for details). An anchor image $\mathbf{x}_{i} \sim p(\mathbf{X})$ (or a batch $\mathcal{B}$) and corresponding two transformations $t_{1}$ and $t_{2}$ are sampled from $\mathcal{T}$ to get $\mathbf{x}^{1}_{i} = t_{1}(\mathbf{x}_{i})$ and $\mathbf{x}^{2}_{i} = t_{2}(\mathbf{x}_{i})$, augmented views of $\mathbf{x}_{i}$. $\mathbf{x}_{1}$ and $\mathbf{x}_{2}$ are encoded through $f_{\theta}$ (ResNet-18 \cite{he2016deep} in this study)
to get \textit{representations} $f_{\theta}(\mathbf{x}^{1}_{i}))$ and $f_{\theta}(\mathbf{x}^{2}_{i}))$ respectively. Furthermore, these representations are mapped to an \textit{embedding} space via \textit{expander} $g_{\phi}$ where the final VICReg loss is applied between the embedding vectors $\mathbf{z}_{i}^{1} = g_{\phi}(f_{\theta}(\mathbf{x}^{1}_{i})$ and $\mathbf{z}_{i}^{2} = g_{\phi}(f_{\theta}(\mathbf{x}^{2}_{i})$ (or a batch of embedding vectors $\mathbf{Z^{1}} = [\mathbf{z}_{1}^{1},\cdot\cdot\cdot,\mathbf{z}_{|\mathcal{B}|}^{1}]$ and $\mathbf{Z^{2}} = [\mathbf{z}_{1}^{2},\cdot\cdot\cdot,\mathbf{z}_{|\mathcal{B}|}^{2}]$).
Formally the loss is defined as\footnote{Notations same as \cite{Bardes}}:
\begin{equation}\label{eq1}
    \begin{aligned}
        \mathcal{L}_{\text{VICReg}}(\mathbf{Z}^{1}, \mathbf{Z}^{2}) &={} \overbracket{\underbracket{\mu [v(\mathbf{Z}^{1})+ v(\mathbf{Z}^{2})]}_{\text{\textbf{V}ariance}}
    + \underbracket{\nu[c(\mathbf{Z}^{1}) + c(\mathbf{Z}^{2})]}_{\text{\textbf{C}o-Variance}}}^{\text{Regularization Term}}\\ &+{} \underbracket{\eta s(\mathbf{Z}^{1}, \mathbf{Z}^{2})}_{\text{\textbf{I}nvariance}},
    \end{aligned}
\end{equation}
where $\mu = 25$, $\nu =25$ and $\eta = 1.0$.
For a more elaborate description of different loss term please refer \cite{Bardes}.

\textbf{Dynamic Channel Gating.}
Channel selection (or conditional computation) adapts the network to different inputs while keeping the architecture fixed \cite{han2021dynamic, veit2018convolutional,Li_2021_ICCV}. Given an input $\mathbf{x}_{l-1}$ (output from a previous layer), the output from a static network (or a module: A block in ResNet) is given by $\mathbf{x}_{l} = f_{\boldsymbol{\theta}}(\mathbf{x}_{l-1}, \boldsymbol{\theta})$ while a network (module) equipped with dynamic computation will be represented by $\mathbf{y} = f_{\boldsymbol{\theta}}(\mathbf{x}_{l-1}, \hat{\boldsymbol{\theta}})$ such that $\hat{\boldsymbol{\theta}} = \mathcal{W}(\mathbf{x}_{l-1}, \boldsymbol{\theta}, \boldsymbol{\omega} )$ where $\boldsymbol{\omega} \in \{\mathbf{w_{1}, \mathbf{w}_{2}}\}$. $\mathcal{W}$ can be thought of as a lightweight policy network to decide the \textit{importance}\footnote{also known as saliencies/relevance/attention} vector for a given input feature map $\mathbf{x}_{l-1}$, which is crucial in order to avoid trivial solutions. The most common and effective technique to realize $\mathcal{W}$ is  via \textit{squeeze and excitation} blocks \cite{hu2018squeeze}. This usually requires obtaining a context vector $\mathbf{z} \in  \mathbb{R}^{C_{l-1}}$ via global average pooling to accumulate spatial information. Consequently, this context vector $\mathbf{z}$ is passed through $\mathcal{W}$ which is composed of:
\begin{equation}
    \mathcal{W}(\mathbf{x}_{l-1}, \boldsymbol{\theta}, \boldsymbol{\omega}) = \mathbf{w}_{2}(\texttt{norm}(\textbf{w}_{1}\textbf{z}))_{+},
\end{equation}
where $+$ is \texttt{ReLU} operation, \texttt{norm} denotes \texttt{batch norm} layer, $\mathbf{w}_{1} \in \mathbb{R}^{\frac{C_{l-1}}{r} \times C_{l} }, \mathbf{w}_{2} \in \mathbb{R}^{C_{l} \times \frac{C_{l-1}}{r}}$ and $r$ is defined as reduction rate (set to 4). $\mathbf{w_{1}}, \mathbf{w}_{2}$ are realised through convolutional layer. Finally, in order to achieve $\hat{\theta}$, binary selection is applied on $\mathcal{W}(\mathbf{x}_{l-1}, \boldsymbol{\theta}, \boldsymbol{\omega})$ (\texttt{torch.where($\mathcal{W}$}$\geq$ 0.5, 1, 0))  to get a binary mask $m \in \mathbb{R}^{C^{l}}$ such that $\hat{\theta} = \theta \odot m$. This discrete selection works perfectly during inference but breaks the computational graph during training. To make training feasible the Gumbel-softmax trick \cite{jang2016categorical} is adopted. The Gumbel-Trick has been widely used as a reparameterisation technique for the task of dynamic channel selection \cite{krishna2022dynamic, Li_2021_ICCV, herrmann2020channel, veit2018convolutional}. For more  clarity  refer Figure \ref{fig:channel_selection} in Appendix \ref{channel_selection}. In this work we closely follow the setting of DGNet \cite{Li_2021_ICCV} for channel selection  where sparsity is induced by setting a global target budget ($t_{d}$) to optimize a loss objective defined as:
 \begin{equation}\label{sparloss}
     \mathcal{L}_{G} = \lambda  \Bigg(\frac{\sum_{l=1}^{L}F^{R}_{l}}{\sum_{l=1}^{L}F_{l}^{O}} -t_{d}\Bigg)^{2},
 \end{equation}
 where $F_{l}^{R}$ is the average FLOPs over the batch along with FLOPs computation lightweight $\mathcal{W}$ network, while $F_{l}^{O}$ is the original FLOPs without a gating module with $\lambda =5$ \cite{Li_2021_ICCV} across all  datasets and training regimes. Only blocks with gating modules take part in FLOP computation as they are responsible for introducing sparsity in the network. Refer Figure~\ref{fig:channel_selection} in Appendix~\ref{channel_selection}.

\textbf{Final Loss.}  The overall training objective is defined as: $\mathcal{L} = \mathcal{L}_{\text{VICReg}} + \mathcal{L}_{G}.$

\subsection{Experimental Setup and Implementation Details }

\subsubsection{Pre-Training}\label{pretraining}

 We closely follow the implementation of VICReg \cite{Bardes} (as our self-supervised objective) suited with our computational constraints using solo-learn library \cite{da2022solo}, while for dynamic gating we follow DGNet \cite{Li_2021_ICCV} for inducing channel sparsity via gating mechanism with \underline{ResNet18} (modified basic blocks Figure \ref{fig:modified} in Appendix \ref{channel_selection}) (\textit{single} network) as the base encoder as depicted in Figure \ref{fig:main}c, also the two branches have \texttt{separate batch normalization} layer following \cite{yu2018slimmable}. The encoder is \textit{randomly} initialized and trained with SGD for 500 epochs with a batch size of 256 on 2 Nvidia 2080Ti GPUs, with a warmup start of 10 epochs following a cosine decay with a base learning rate of 0.3 using LARS optimizer \cite{you2017large}. Since we are using a very lightweight model as our gating network, there is no significant computational overhead during training. We report the inference speedup in terms of hardware-independent theoretical metric of FLOPs and not wall-clock time as we are not using any hardware accelerators to utilise sparsity during training. \underline{Note}: we do not perform any hyper-parameter search and most of the choices are based on  computational availability. Any scaling parameter involved in the loss term is derived from respective papers. A Pytorch-style pseudocode is also provided in Appendix \ref{pseudo}.
\begin{table*}[t]
 \caption{Linear Evaluation: Here we compare the performance of VICReg-Dual with two  baselines VICReg and VICReg-Gating (VICReg modified following \cite{krishna2022dynamic}) respectively. $\uparrow$/$\downarrow$  in \textcolor{orange}{orange} font is comparison with \textit{Baseline-1}, while \textcolor{blue}{blue} font is comparison with \textit{Baseline-2}. FLOPs R. denotes FLOP reduction. We report Top-1 accuracy averaged over 5 runs. Best viewed in color. \Bstrut}
 \small
 \centering

 \resizebox{1.00\linewidth}{!}{
 \begin{tabular}{l|cc|c|cc|ccc}

      \hline
      \hline
      \multicolumn{1}{l|}{\textbf{}} & \multicolumn{2}{c|}{\textbf{VICReg} } & \multicolumn{1}{l|}{} &\multicolumn{2}{c|}{\textbf{VICReg-Gating}}&
      \multicolumn{3}{c}{\textbf{VICReg-Dual}} \Tstrut\Bstrut\\

     \multicolumn{1}{l|}{\textbf{}} & \multicolumn{2}{c|}{ \textcolor{orange}{Baseline-1}} & \multicolumn{1}{l|}{} &\multicolumn{2}{c|}{\textcolor{blue}{Baseline-2}}&
      \multicolumn{3}{c}{\textit{this work}} \Tstrut\Bstrut\\
\multicolumn{1}{l|}{\textbf{}} & \multicolumn{2}{c|}{\cite{Bardes}} & \multicolumn{1}{l|}{} &\multicolumn{2}{c|}{\cite{krishna2022dynamic}}&
      \multicolumn{3}{c}{} \\
      \textbf{Dataset}& Dense &FLOPs &$\textbf{t}_{d}$(\%) & \multicolumn{1}{c}{Gated} &\multicolumn{1}{c|}{FLOPs R.} &\multicolumn{1}{c}{Dense $\uparrow$} &\multicolumn{1}{c}{Gated $\uparrow$ } &FLOPs R. $\uparrow$ \Bstrut \\
\hline

&&&10\%& 87.75 \tksmall{0.03} & 85.92\% &88.99 \tksmall{0.04} \decd{2.12} & 88.94 \tksmall{0.06} \decd{2.17} \ing{1.19} & 81.49\% \decg{4.43}  \Tstrut\Bstrut\\
&&&20\%& 88.00 \tksmall{0.04}  & 78.03\% &89.68 \tksmall{0.05} \decd{1.43} & 89.44 \tksmall{0.01} \decd{1.67} \ing{1.44} & 74.36\% \decg{3.67}\\
CIFAR-10&91.11 \tksmall{0.03}  &7.03E8 & 30\% & 89.49 \tksmall{0.04} & 69.27\% &90.38 \tksmall{0.04} \decd{0.73}  & 90.27 \tksmall{0.03} \decd{0.84} \ing{0.78} & 66.43\% \decg{2.84} \\
&&&40\%& 89.87 \tksmall{0.03}  & 61.10\% & 90.46 \tksmall{0.05} \decd{0.65} & 90.10 \tksmall{0.03} \decd{1.01} \ing{0.23} & 57.53\% \decg{3.57}\\
&&&50\%& 90.70 \tksmall{0.04} & 51.62\% & 90.20 \tksmall{0.02} \decd{0.91} & 90.40 \tksmall{0.06} \decd{0.71} \decg{0.30} & 49.02\% \decg{2.60} \Bstrut\\
\hline

&&&10\%& 82.48 \tksmall{0.15} & 82.85\% & 84.29 \tksmall{0.21} \decd{1.86} &83.29 \tksmall{0.05}  \decd{2.86} \ing{0.81} & 78.34\% \decg{4.51}\Tstrut\Bstrut\\
&&&20\%& 82.90 \tksmall{0.21} & 76.35\% & 84.52 \tksmall{0.12} \decd{1.63} &84.32 \tksmall{0.02} \decd{1.83} \ing{1.42} & 72.21\%\decg{4.14}\\
STL-10&86.15 {\tiny$\pm$ 0.10} &3.33E8 & 30\% &84.16 \tksmall{0.11} & 68.38\%& 84.90 \tksmall {0.05} \decd{1.25}  &84.85 \tksmall{0.04} \decd{1.30} \ing{0.69} & 65.24\% \decg{3.14}\\
&&&40\%& 84.90 \tksmall{0.10}& 58.89\% & 85.43 \tksmall{0.15} \decd{0.72} &84.90 \tksmall{0.07} \decd{1.25} \ing{0.00} &57.11\% \decg{1.78} \\
&&&50\%& 85.40 \tksmall{0.20} & 49.93\% & 85.75 \tksmall{0.02} \decd{0.40} & 85.72 \tksmall{0.02} \decd{0.43} \ing{0.32} & 48.41\% \decg{1.52} \Bstrut\\
\hline

&&&10\%& 63.12 \tksmall{0.09} & 84.82\% &65.21 \tksmall{0.06} \decd{0.65} &64.31 \tksmall{0.08} \decd{1.55} \ing{1.19} & 81.71\% \decg{3.11} \Tstrut\Bstrut \\
&&&20\%& 64.64 \tksmall{0.01} & 77.25\% & 66.10 \tksmall{0.05} \ind{0.24} &65.82 \tksmall{0.03} \decd{0.04} \ing{1.18} & 74.94\% \decg{2.31}\\
CIFAR-100&65.86 {\tiny$\pm$ 0.10} &7.03E8 & 30\% &65.41 \tksmall{0.09}& 68.68\%&65.90 \tksmall{0.10} \ind{0.04} &65.64 \tksmall{0.00} \decd{0.22} \ing{0.23} & 66.83\% \decg{1.85} \\
&&&40\%&65.71 \tksmall{0.04} & 60.60\% &67.07 \tksmall{0.16} \ind{1.21} &66.70 \tksmall{0.05} \ind{0.89} \ing{0.99} &57.99\% \decg{2.61}\\
&&&50\%& 65.75 \tksmall{0.12} & 50.04\% &66.41 \tksmall{0.05} \ind{0.55} & 66.40 \tksmall{0.14} \ind{0.54} \ing{0.65} &49.06\% \decg{0.98} \Bstrut\\
\hline

&&& 30\% &74.04 \tksmall{0.09}& 67.95\%&75.12 \tksmall{0.07} \decd{2.62} &75.04 \tksmall{0.10} \decd{2.17} \ing{1.00} & 64.98\%\decg{2.97}\Tstrut\Bstrut\\
ImageNet-100& 77.74 {\tiny$\pm$ 0.12} & 1.81E9&40\%& 75.61 \tksmall{0.04}& 59.55\% & 76.45 \tksmall{0.03} \decd{1.29} &76.13 \tksmall{0.20} \decd{1.61} \ing{1.00} & 56.41\%\decg{3.14}\\
&&&50\%& 75.83 \tksmall{0.07}& 50.11\% & 76.42 \tksmall{0.26} \decd{1.32} &76.24 \tksmall{0.12} \decd{1.51} \ing{0.41} & 47.69\%\decg{2.42} \Bstrut\\
 \hline
 \hline
 \end{tabular}
 \label{table1}
 }
 \end{table*}

\begin{table*}[t]
 \caption{Transfer Performance: dense and gated under VICReg-Dual is compared with the common dense \textit{baseline} of VICReg. $\uparrow/\downarrow$ represents increment/decrement in performance. We report Top-1 linear evaluation accuracy averaged over 5 runs.  \Bstrut}
 \small
 \centering

 \resizebox{1.00\linewidth}{!}{
 \begin{tabular}{ll|c|cc|cc|cc|cc|cc}
      \hline
      \multicolumn{2}{l|}{\textbf{Dataset}}  &\multicolumn{1}{c|}{\textbf{VICReg}} & \multicolumn{10}{c}{\textbf{VICReg-Dual}}
       \Tstrut\Bstrut\\

       \cline{4-13}
\multicolumn{2}{l|}{\textbf{}} & \multicolumn{1}{l|}{\cite{Bardes}} &\multicolumn{2}{c|}{10\%}&\multicolumn{2}{c|}{20\%}&\multicolumn{2}{c|}{30\%}&\multicolumn{2}{c|}{40\%}&\multicolumn{2}{c}{50\%} \Tstrut\\
      \multicolumn{2}{l|}{From $\rightarrow$ To} &Dense  &\multicolumn{1}{c}{Dense}& \multicolumn{1}{c|}{Gated} & \multicolumn{1}{c}{Dense}& \multicolumn{1}{c|}{Gated} &\multicolumn{1}{c}{Dense}& \multicolumn{1}{c|}{Gated} &\multicolumn{1}{c}{Dense}& \multicolumn{1}{c|}{Gated} &\multicolumn{1}{c}{Dense}& \multicolumn{1}{c}{Gated} \Bstrut \\
\hline

CIFAR-100 $\rightarrow$ STL-10&& 70.37&64.69 $\downarrow$&64.29 $\downarrow$ &65.02 $\downarrow$ &65.69 $\downarrow$ &65.63 $\downarrow$ &65.93 $\downarrow$ &66.73 $\downarrow$ &66.90 $\downarrow$ &66.52 $\downarrow$ &67.30 $\downarrow$  \Tstrut\\
CIFAR-100 $\rightarrow$ CIFAR-10&& 80.08&80.56 $\boldsymbol{\uparrow}$&79.92 $\downarrow$& 80.00 $\downarrow$ &79.95 $\downarrow$  &80.23 $\boldsymbol{\uparrow}$ &80.24 $\boldsymbol{\uparrow}$ & 80.79 $\boldsymbol{\uparrow}$ &80.65 $\boldsymbol{\uparrow}$ & 80.16 $\boldsymbol{\uparrow}$ &80.09 $\boldsymbol{\uparrow}$ \\
CIFAR-100 $\rightarrow$  ImageNet-100&& 39.45 & 35.88 $\downarrow$ & 36.31 $\downarrow$ & 38.56 $\downarrow$ & 38.97  $\downarrow$& 38.49 $\downarrow$ & 38.20 $\downarrow$ & 38.05 $\downarrow$ & 38.69 $\downarrow$ & 39.41 $\downarrow$ & 40.68 $\boldsymbol{\uparrow}$ \Bstrut\\
\hline
 ImageNet-100 $\rightarrow$ STL-10 &&72.80&-&-&-&-&65.86 $\downarrow$ &65.01 $\downarrow$ &66.96 $\downarrow$ &67.09 $\downarrow$ &67.74 $\downarrow$ &67.15 $\downarrow$   \Tstrut\Bstrut\\
ImageNet-100 $\rightarrow$ CIFAR-10&&55.12&-&-&-&-& 53.38 $\downarrow$ &49.41 $\downarrow$ &53.42 $\downarrow$ &50.14 $\downarrow$ &54.01 $\downarrow$ &49.80 $\downarrow$ \\
ImageNet-100 $\rightarrow$ CIFAR-100&& 31.42 & -& -&-&-&28.81 $\downarrow$ &25.55 $\downarrow$ &27.41 $\downarrow$ &25.24 $\downarrow$ &28.75 $\downarrow$ &25.14 $\downarrow$ \Bstrut\\

\hline
\end{tabular}
\label{table2}
}
\end{table*}
\subsubsection{Evaluation}\label{eval}

Pre-training and evaluation is carried on the train and validation data of CIFAR-10/100 \cite{krizhevsky2009learning}, ImageNet100 (taken from \href{https://www.kaggle.com/datasets/ambityga/imagenet100}{here}) and STL-10 \cite{coates2011analysis}. For pre-training with STL-10 we considered only the \textit{un-labelled} set. We pre-train for a target budget, $t_{d} = \{10\%, 20\%, 30\%, 40\%, 50\% \}$ for each of the datasets except for ImageNet-100 where $t_{d}$ is restricted to $\{30\%, 40\%, 50\% \}$ due to computational constraints.

For evaluating the learned representation we follow the standard practice of evaluating the trained encoder by freezing its weights and training a linear classifier on top of it. We trained a single linear layer for 100 epochs with a batch size of 512 on a single NVIDIA 2080Ti with a learning rate of 0.3 following step decay of 0.1 at 60$^{\text{th}}$ and 80$^{\text{th}}$ epochs. This strategy is followed across all  datasets. We report top-1 accuracy averaged over 5 runs. We also use $k$-nearest neighbours (kNN) as our evaluation metric with $k=1$ for ablations.

To exhaustively compare the performance of the dense and gated models we consider VICReg \cite{Bardes} as a SSL \underline{\textit{dense}} baseline  while VICReg augmented with sparsity loss as in \tkeqref{sparloss} (following \cite{krishna2022dynamic}) serves as a \underline{\textit{gated}} baseline.

\section{Results}\label{result}

\textbf{1. Quantitative assessment.} Table~\ref{table1} compares the performance of VICReg-Dual-Gating with the other two baselines for dense and gated. The lightweight gated network achieves improved performance across all datasets and target budgets ($t_{d}$) as compared to \textit{Baseline-2} (Figure \ref{fig:main}b), with a negligible drop at $t_{d}=50\%$ for CIFAR-10 only.
The improved gated performance can be attributed to the fact that both dense and gated models co-learn during training via weight sharing. However, the performance gain is compensated by a slightly smaller reduction in FLOPs as compared to \textit{Baseline-2}. Improved performance of gated network further closes the gap with a completely self-supervised dense model (\textit{Baseline-1}). With only 10\% of the target budget, the gated network achieves  accuracy on-par with the \textit{Baseline-1} and even surpasses the performance on CIFAR-100 for $t_{d}=40\%$. Performance is comparable for all the other datasets across different $t_{d}$.
In comparison to \textit{Baseline-1}, we observe a drop in performance of VICReg-Dual-Gating e.g., at $t_{d}=10\%$ across CIFAR-10 \decd{2.17\%}, STL-10 \decd{2.86\%}, CIFAR-100 \decd{1.55\%}, ImageNet-100 \decd{2.7\%} but at a significant reduction in number of FLOPs. This illustrates that even under severe budget constraint our model achieves comparable performance to \textit{Baseline1}. This drop further decreases on increasing $t_{d}$ to $50\%$.
\begin{figure*}[t]
    \centering
    \includegraphics[width=1.00\linewidth]{figure/umap.png}
    \vspace{-1.75\baselineskip}
    \caption{\textbf{Qualitative analysis:} The top row displays UMAP embeddings of the learned representations through the \textit{lightweight} network while the \textit{bottom} row displays the dense model over different target budgets $t_{d}$. This is compared with embeddings of VICReg (dense) trained without any sort of sparsity. Best viewed in color. Note: $\{0.1, 0.2, 0.3, 0.4, 0.5\}$ refers to $\{10\%, 20\%, 30\%, 40\%, 50\%\}$ $t_{d}$.}
    \label{fig:umap}
\end{figure*}

Another important aspect of our learning method is performance of the dense model. Ideally one would aim to achieve fewer fluctuations with varying $t_{d}$ with a performance equivalent to dense model as in \textit{Baseline-1}. However, we  find that the performance of the dense network (\textit{this work}) is slightly below the performance of the dense (\textit{Baseline-1}) for CIFAR-10/STL-10/ImageNet-100 while for CIFAR-100 the performance is better than the self-supervised dense module. This is interesting because what we achieve from this single experiment of 500 training epochs is a single \underline{base encoder} (dense encoder) and gates (gating modules) and their combination gives a gated \underline{lightweight} network  which is equivalent to running two separate experiments as reported in Table~\ref{table1} under VICReg and VICReg-Gating.

\textbf{2. Transfer Learning.} Table~\ref{table2}  compares the transfer performance of VICReg-Dual (dense and gated) with VICReg. This experiment gives further  insights into the quality of the learned representation in this joint setting. In general, there is a drop in performance for VICReg-Dual for both \textit{dense} and \textit{gated}, although the difference is not significant. However, for CIFAR-100$\rightarrow$CIFAR-10 \textit{dense} and \textit{gated} outperforms only \textit{dense} in VICReg at a very low target budget. Even in the case when the model is pre-trained on ImageNet-100, performance is comparable. This is encouraging as this new perspective still maintains good generalization and transferability.
\textit{Note} the encoder was frozen and was linearly evaluated following the protocol as in Section~\ref{eval}.

\textbf{3. Qualitative assessment.}
Figure \ref{fig:umap} shows uniform manifold approximation and projection (UMAP) \cite{mcinnes2018umap} embeddings of the learned representation ($f_{\theta}(\mathbf{x}) \in \mathbb{R}^{512}$) trained using dual-setting on the STL-10 dataset and compares it with VICReg \cite{Bardes}. The learned structure is similar to dense (VICReg) at a very low budget. Along with that the classes appear to be visually distinct, similar to the VICReg setting, and this is observed for both the dense and gated networks of VICReg-Dual.

\begin{table}[t!]
 \caption{Symmetric architecture: BT vs VICReg in dual setting (kNN-1 as evaluation metric).\Bstrut}
 \small
 \centering

 \resizebox{.5\textwidth}{!}{
 \begin{tabular}{l|c|c>{\columncolor[gray]{0.85}}cc|c>
 {\columncolor[gray]{0.85}}cc}

      \hline
      \hline
      \multicolumn{1}{l|}{\textbf{}} & \multicolumn{1}{l|}{} &
      \multicolumn{3}{c|}{\textbf{BT-Dual}}&
      \multicolumn{3}{c}{\textbf{VICReg-Dual}} \Tstrut\Bstrut\\

      \textbf{Dataset}&$\textbf{t}_{d}$ &\multicolumn{1}{c}{Dense} &\multicolumn{1}{c}{Gated} &FLOPs R. &\multicolumn{1}{c}{Dense} &\multicolumn{1}{c}{Gated} &FLOPs R. \Bstrut \\
\hline
&10\%&53.64 & 53.01 & 80.60\% &\textbf{54.92} & \underline{54.66} & \underline{81.71\%} \Tstrut\\
&20\%&54.33& 53.88 & 74.56\% & \textbf{55.97} & \underline{55.86} & \underline{74.94\%} \\
CIFAR-100&30\% & 55.03&54.89&66.09\%& \textbf{56.34} & \underline{55.53} & \underline{66.83\%}\\
&40\%& 55.26&55.04&57.34\% & \textbf{56.66} & \underline{56.44} & \underline{57.98\%}\\
&50\%& 55.63&55.39&47.64\% & \textbf{56.83} & \underline{57.25} & \underline{49.06\%}\\
\hline
&10\%&73.45& 73.49&76.74\%&\textbf{76.61}&\underline{76.45}& \underline{78.27\%} \Tstrut\\
&20\%&74.89&74.81&69.68\%&\textbf{77.45}&\underline{77.70}&\underline{72.16\%} \\
STL-10&30\%&75.51&76.02&64.12\%&\textbf{78.55}& \underline{78.80}& \underline{65.17\%}  \\
&40\%&76.90&75.93&57.14\%&\textbf{79.01}&\underline{78.85}&\underline{57.05\%} \\
&50\%&76.99&77.10&48.32\%&\textbf{79.69}&\underline{79.95}&\underline{48.39\%}\\
\hline
\hline
 \end{tabular}
 \label{table3}
}
 \end{table}

\begin{table}[t]
 \caption{Different encoder: Single vs Double base encoder (kNN-1 as evaluation metric). \Bstrut}

 \small
 \centering

 \resizebox{.5\textwidth}{!}{
 \begin{tabular}{l|l|c>{\columncolor[gray]{0.85}}cc|c>{\columncolor[gray]{0.85}}cc}

      \hline
      \hline
      \multicolumn{1}{l|}{\textbf{}} & \multicolumn{1}{l|}{} &
      \multicolumn{3}{c|}{\textbf{VICReg-Dual}}&
      \multicolumn{3}{c}{\textbf{VICReg-Dual}} \Tstrut\Bstrut\\
      \multicolumn{1}{l|}{\textbf{}} & \multicolumn{1}{l|}{} &
      \multicolumn{3}{c|}{(1$\times$ResNet-18)}&
      \multicolumn{3}{c}{(2$\times$ResNet-18)} \Bstrut\\

      \textbf{Dataset}&$\textbf{t}_{d}$(\%) &\multicolumn{1}{c}{Dense} &\multicolumn{1}{c}{Gated} &FLOPs R. &\multicolumn{1}{c}{Dense} &\multicolumn{1}{c}{Gated} &FLOPs R. \Bstrut \\
\hline
&10\%& \textbf{54.92} & \underline{54.66} & 81.71\%&53.85 & 52.44 &\underline{85.77\%} \Tstrut\\
&20\%& \textbf{55.97} & \underline{55.86} & 74.94\% &54.78 &54.30 & \underline{77.63\%}   \\
CIFAR-100&30\% & \textbf{56.34} & \underline{55.53} & 66.83\% & 55.65&55.52&\underline{70.07\%}\\
&40\%& \textbf{56.66} & \underline{56.44} & 57.98\% & 55.54 & 55.40& \underline{61.05\%}\\
&50\%& \textbf{56.83} & \underline{57.25} & 49.06\%&55.64 &55.71 &\underline{52.60\%} \Bstrut\\
\hline
&10\%&\textbf{76.61}&\underline{76.45}& 78.27\%&74.93 &74.89&\underline{82.48\%} \Tstrut\\
&20\%& \textbf{77.45}&\underline{77.70}&72.16\%&76.10&75.29&\underline{76.51\%}\\
STL-10&30\%&\textbf{78.55}& \underline{78.80}&65.17\%&76.83&76.71&\underline{69.26\%}  \\
&40\%&\textbf{79.01}&\underline{78.85}&57.05\%&77.73&77.08& \underline{60.00\%}  \\
&50\%&\textbf{79.69}&\underline{79.95}&48.39\%&77.74&78.00 &\underline{51.10\%}\\
\hline
\hline

 \end{tabular}
 \label{table4}
 }
 \end{table}

\begin{table}[t]
 \caption{Investigating role of MSE (kNN-1 as evlauation metric). \Bstrut}

 \small
 \centering

 \resizebox{.5\textwidth}{!}{
 \begin{tabular}{l|l|c>{\columncolor[gray]{0.85}}cc|c>{\columncolor[gray]{0.85}}cc}

      \hline
      \hline
      \multicolumn{1}{l|}{\textbf{}} & \multicolumn{1}{l|}{} &
      \multicolumn{3}{c|}{\textbf{VICReg-Dual}}&
      \multicolumn{3}{c}{\textbf{VICReg-Dual}} \Tstrut\Bstrut\\

      \multicolumn{1}{l|}{\textbf{}} & \multicolumn{1}{l|}{} &
      \multicolumn{3}{c|}{w/ Invariance}&
      \multicolumn{3}{c}{w/o Invariance} \Bstrut\\

      \textbf{Dataset}&$\textbf{t}_{d}$(\%) &\multicolumn{1}{c}{Dense} &\multicolumn{1}{c}{Gated} &FLOPs R. &\multicolumn{1}{c}{Dense} &\multicolumn{1}{c}{Gated} &FLOPs R. \Bstrut \\
\hline
&10\%& \textbf{54.92} & \underline{54.66} & 81.71\%&4.06 & 6.31 &\underline{93.77\%} \Tstrut\\
&20\%& \textbf{55.97} & \underline{55.86} & 74.94\% &3.72 &6.02 & \underline{86.45\%}   \\
CIFAR-100&30\% & \textbf{56.34} & \underline{55.53} & 66.83\% & 4.84&4.89&\underline{82.72\%}\\
&40\%& \textbf{56.66} & \underline{56.44} & 57.98\% & 4.11& 3.36&\underline{68.26\%}\\
&50\%& \textbf{56.83} & \underline{57.25} & 49.06\%&2.79 &4.15 &\underline{49.05\%} \Bstrut\\
\hline
&10\%&\textbf{76.61}&\underline{76.45}& 78.27\%&11.79 &18.02&\underline{90.71\%} \Tstrut\\
&20\%& \textbf{77.45}& \underline{77.70}&72.16\%&11.79&20.81&\underline{82.45\%}\\
STL-10&30\%&\textbf{78.55}&\underline{78.80}&65.17\%&12.25&17.89&\underline{74.96\%}  \\
&40\%&\textbf{79.01}&\underline{78.85}&57.05\%&11.59&17.25&\underline{66.51\%}  \\
&50\%&\textbf{79.69}&\underline{79.95}&48.39\%&12.81&15.72&\underline{51.91\%}\\
\hline
\hline

 \end{tabular}
 \label{co-evolve}
 }
 \end{table}

\begin{table*}
 \caption{Evaluation of different training settings (kNN-1 as evaluation metric).\Bstrut}

 \small
 \centering

 \begin{tabular}{l|l|c>{\columncolor[gray]{0.85}}cc|c>{\columncolor[gray]{0.85}}cc|c>{\columncolor[gray]{0.85}}cc}

      \hline
      \hline
      \multicolumn{1}{l|}{\textbf{}} & \multicolumn{1}{l|}{} &\multicolumn{3}{c|}{\textbf{VICReg-Dual} (Vanilla)}&
      \multicolumn{3}{c|}{$+$\text{Separate\_bn}}&
      \multicolumn{3}{c}{$+$\text{Warmup-epochs}} \Tstrut\Bstrut\\

      \textbf{Dataset}&$\textbf{t}_{d}$(\%) & Dense &\multicolumn{1}{c}{Gated}&FLOPs Red. &\multicolumn{1}{c}{Dense} &\multicolumn{1}{c}{Gated} &FLOPs Red. &\multicolumn{1}{c}{Dense} &\multicolumn{1}{c}{Gated} &FLOPs Red. \Bstrut \\
\hline
&10\%&54.05 & 55.14 & 81.57\% & 54.92&54.66&\underline{81.71\%}&\textbf{55.59}&\underline{55.38}&79.61\%  \\
&20\%& 54.81 & 55.71 & 73.98\%& \textbf{55.97} & \underline{55.86} & \underline{74.94\%} & 55.22 & 55.12 &72.39\% \\
CIFAR-100&30\% & 55.79 & \underline{56.33} & 66.27\% &\textbf{56.34}&55.53&\underline{66.83\%} & 55.64&55.67&66.19\% \\
&40\%& 56.11 & \underline{57.15} & 57.72\% &\textbf{56.66}&56.44&\underline{57.99\%}&55.87 & 55.80&57.36\% \\
&50\%& 56.01 & 56.57 & 49.03\%&\textbf{56.83}&\underline{57.25}&49.06\%&56.06&56.13&\underline{49.26\%}\\
\hline
&10\%&76.50 & 76.53 & \underline{78.70\%} & 76.49 & 76.25 & 78.34\% & \textbf{77.53} & \underline{77.44}& 73.20\% \\
&20\%& 77.01 & 76.91 & \underline{72.33\%} & 77.41 & 77.59 & 72.21\%& \textbf{78.04} & \underline{78.14} & 68.11\%\\
STL-10&30\%& 78.10 & 77.85& \underline{65.26\%}& 78.49 & 78.54 & 65.24\%&\textbf{78.79} & \underline{78.80} &62.62\% \\
&40\%& 78.70 & 78.71&56.84\%&79.14&78.84&\underline{57.11\%}&\textbf{79.35} &\underline{79.04}&55.36\%\\
&50\%&79.68 & 79.85&48.36\%&79.76&79.95&\underline{48.41\%}&\textbf{79.89} & \underline{80.00} & 47.35\%\\
\hline
\hline

 \end{tabular}
 \label{table5}
 \end{table*}

\section{Auxiliary Insights and Ablation Study}

For all experimental settings and studies from now on; models (and settings) were pre-trained for 500 epochs on CIFAR-100 and STL-10 datasets. Instead of using linear evaluation we use kNN as the evaluation criterion with $k=1$ to report \textit{top-1} accuracy. In results table from now on \textbf{bold} and \underline{underline} are the best performing attributes for dense and gated module respectively.

\textbf{1. Comparison with symmetric architecture Barlow Twins (BT).} VICReg is build upon the findings of BT \cite{Zbontar2021} and it is straightforward to also apply the dual setting in BT because it again minimizes the cross-correlation (regularization term) to identity $\mathcal{I}$ although the loss function is entirely mutual unlike in VICReg. In Table~\ref{table3} we compare the performance of BT augmented with \textit{our} setting. We observe that kNN performance for BT is low as compared to VICReg-Dual. The drop in performance could be attributed to the fact that VICReg applies independent regularisation and further regularised representations are matched through invariance loss (MSE).

\textbf{2. Training with different base encoder.} In this setting we train a model with two base encoders, one w/ gate (gated) and other w/o gate (dense) i.e, ($2 \times \text{ResNet-18}$) (results in Table~\ref{table4}). An interesting observation is that VICReg-Dual with a single base encoder performs  better than a more powerful setting with two base encoders (Table~\ref{table4}) although FLOPs Reduction (FLOPs R.) is more ($\uparrow$) in the setting of two different encoders. This is due to the fact that sparsity loss ($\mathcal{L}_{G}$) operates solely on the un-shared branch so there is no trade-off involved as in case of single base encoder which simultaneously tries to enforce sparsity and visual invariance.

\textbf{3. Role of \textit{mean squared error} in co-evolving.} It's a well know fact in SSL that these methods suffer from dimension collapse \cite{hua2021feature, Jing}. Training without any regualrization term or any trick \cite{grill2020bootstrap, chen2021exploring} would lead to dimensional collapse. Also, authors in \cite{kim2021comparing} showed that MSE serves as a better option for exact logit matching as compared to Kullback-Leibler (KL) divergence. So, in order to understand the role MSE we trained a model w/o \textit{invariance} loss. As depicted in Table \ref{co-evolve} we found that that there is huge performance drop if we get rid of invariance term. This implies invariance term plays a crucial role and seem to be an important factor not only for co-evolving but for self-supervision as well.

\textbf{4. Effect of batch statistics and warm-up period during training.} In this ablation study, we evaluate different training settings (different batch statistics and warm-up period) to find  best performance and computational trade-off as highlighted in Table~\ref{table5}. For clarification, here \underline{warm-up} refers to the number of epochs (10 epochs in this study) for which the model was trained without any sort of sparsity loss (no gating) which is different from the one used in optimization in Section~\ref{pretraining}. In Table~\ref{table5}, \textit{Vanilla}
refers to the setting where we introduce gates at different levels and sparsity loss to the original VICReg \cite{Bardes}, while \texttt{Separate\_BN} refers to extending \textit{Vanilla} setting by having different batch normalization layers as batch statistics would vary for the sparse and dense branch \cite{yu2018slimmable, jiang2021self}. Finally, this setting is extended by training for 10 epochs without any sparsity loss(and gates) under \texttt{Warm-up period} in Table \ref{table5}. Overall the setting with \texttt{Separate\_BN} offers best trade-off for accuracy and flops reduction.

 \section{Discussion and Conclusion}
In this work we proposed a novel perspective on unifying synergies between self-supervised learning and dynamic computing essential for industrial applications. We exploit adaptive computing to induce sparsity into symmetric branches of self-supervised models enabling both  branches to co-evolve with each other during training. In addition, this approach also allows simultaneous training of a dense and gated (sparse) sub-network from scratch with a target budget $t_d$ under a self-supervised training objective with minimal computational overhead via weight sharing therefore offering a good accuracy-efficiency trade-off for a given downstream application. As a result, our single base encoder offers enough flexibility to serve a dual purpose to reduce excessive computational overhead as illustrated in Table \ref{table1} and Table \ref{table2}.

However, there are limitations with this work.
Firstly, dense model seems to under-perform and  it's performance further fluctuates with varying $t_{d}$. However, we tried RotNet \cite{gidaris2018unsupervised} as a extra proxy loss for dense branch but it didn't yield good performance.
Secondly, we haven't imposed any constraint in the training objective that enforces uniform distribution of channel activations i.e. preservation of channel diversity during inference (could be a solution for our first limitation). Also, it will also be interesting to study channel activation distribution for different classes to further build an intuition about the approach. Thirdly, in future we would like to extend this setting to contrastive approaches and non-symmetric architectures as well.

We would like to \underline{reiterate} that our goal is to demonstrate that when self-supervised learning is combined with adaptive computing, it results in a flexible  dual purpose framework. This work is an initial attempt to  draw parallels and make an inter-connection between both of these fields. However, more steps need to be taken in this direction to build a better intuition behind these models and also help us understand their other attributes such as generalization and ability to transfer to other downstream tasks.

redpajama_set_name: RedPajamaArXiv


To all the Kizuki fans in Chicago, Kizuki Ramen & Izakaya has finally opened its second Chicago location inside the NEWCITY Shopping Center in Lincoln Park. Kizuki is a Japanese chain restaurant with locations all across the U.S., Japan, and Taiwan. When its first Chicago location in Wicker Park became a hit, Kizuki announced its second restaurant opening back in July 2016. The wait ended in January 2019 when doors finally opened to customers.
Located directly across from Saks Fifth Avenue and ArcLight Cinemas, Kizuki Ramen & Izakaya offers the perfect dining experience during an evening of luxury shopping and a movie. The trendy venue serves authentic Japanese noodles and small dishes, and offers a bar with hand-picked Japanese craft beer. Though on the smaller side, the cozy space is well lit with hand-painted paper lanterns lining the walls and offers a modern take on the traditionally intimate Japanese dining setting through semi-walled seating. A hip blend of upscale decor, western and Japanese pop, and classic dishes, the family-friendly atmosphere finds a comfortable balance between luxury and affordability, as well as Western and Eastern influences.
Aside from the atmosphere, Kizuki also creates a unique dichotomy between modernism and traditionalism in their dishes. The varied menu includes classic dishes, like the Shoyu or Tonkotsu Ramen if you're looking for that classic savory noodle broth. Or, if you're craving something lighter, you can try one of their creative fusions, such as the Yuzu Shio Ramen, for a clean citrus flavor. More than just a noodle bar, Kizuki also offers an array of rice Don bowls, appetizers, and topping choices.
Upon my first visit, I decided to order the Garlic Tonkotsu Shoyu Ramen, which promised an "extra rich" shoyu flavored pork broth. The broth affords a savory, garlic kick as promised, and the noodles are the perfect texture of chewy and elastic. The portion sizes are thankfully not on the stingy side and come with generous slices of chashu pork, a whole soft-boiled egg, and a handful of various toppings, such as crunchy menma, fresh green onions, and crispy fried onion flakes. The best part, in my opinion, was the molten egg, executed to soft-boiled perfection, which, when cut into, complements the wavy noodles with thick coat of yolky, golden goodness. If you're looking for some side dishes to cleanse your palette, I would suggest ordering the edamame for a simple salty snack (complementary if you like their Facebook page!) and the Japanese style pork buns to satisfy a meat-lover's cravings.
To top it off, we finished with the mochi dumplings and Japanese style ice-cream & waffles for dessert. When confronted with the dilemma of choosing between matcha and vanilla ice cream, the staff was kind enough to give us a portion of each flavor for the price of one (shoutout to my man Alex!). The mochi dumplings were coated in a mysterious sweet and salty caramel sauce that balanced out my sweet tooth, although I must say the name is slightly deceiving. I was expecting mochi "dumplings" with some kind of filling, but was disillusioned to find that there was nothing inside upon taking a bite.
In addition to the mouth-watering food, the amazing staff will cater to your needs with genuine pleasure and a candid humor that contributes to the enjoyable and entertaining dining experience. They are quick to your service and deliver your requests with a personal conviction to the kitchen staff to ensure that you leave full and happy. Our kind server Alex went out of his way to bend the rules for us, from splitting the check beyond the limit of two cards to telling the kitchen to give us extra dessert portions so we could try the multiple ice cream flavors.
Overall, Kizuki Ramen & Izakaya stands out to me as a favorite spot for Japanese comfort food at an affordable price. The new location opens a door to authentic, downtown Chicago Asian cuisine and will hopefully lead to more diverse restaurants opening in nearby locations.
Server Alex and I. Go leave him an amazing tip!


text:
NEW PRODUCTS! SureSine inverters, GenStar MPPT DC charging system CLICK HERE TO FIND OUT MORE
Morningstar designs solar charge controllers, inverters, and accessories for off-grid and grid-tied battery backup systems through its Professional and Essential Series. Browse our product types below.
ReadyRail Products & Accessories
New Integrated Series and DC Charging Components
English, Spanish, French, and German versions
Request product(s) for purchase
Establish ownership and preserve your warranty
String Calculator
Determine your optimal module configurations
Free programming software and monitoring
Solar Charge ControllersWith over 4 million products sold in over 100 countries since 1993 — functioning in some of the most extreme environments & mission-critical applications in the world — Morningstar Corporation is truly "the leading supplier of solar controllers and inverters." Morningstar's stable management along with the lowest employee turnover rate has led to our success. We continually strive to keep our quality and reliability at an industry-record high. Need to learn more about solar charge controllers? Visit our solar charge controllers FAQs.
Other Industrial/Commercial
RV-Caravans and Marine
Energy Storage Partner Program
Best Practices By Battery Chemistry
Morningstar Online Community
Morningstar Expands by Adding New Technology to Its Solar Line and New Talent to Its Team
Home » News » Morningstar News » Morningstar Expands by Adding New Technology to Its Solar Line and New Talent to Its Team
Newtown, PA, April 17, 2018 –(PR.com)– With 25 years of renewable energy leadership behind it, Morningstar Corporation is firmly focusing on the future of solar by adding new game-changing products and three new faces to its sales and marketing team, to extend the brand's reach into power conversion products and systems.
On the sales side, Morningstar welcomes the additions of Gabrielle LeRoy and John White to its sales engineering group. Gabrielle's credentials include a masters' degree in electrical engineering with emphasis on renewable energy technologies, along with numerous global certifications and authorizations. She's a hands-on solutions architect who has worked in developmental projects from Ecuador to Fiji, and her systems and training accomplishments across Europe, Africa, Asia and the Middle East will greatly increase Morningstar's presence in key international projects and markets. Ms. LeRoy is based near Grenoble, France.
In North America, John White brings his comprehensive experience in solar distribution and manufacturing to help manage Morningstar's customer network serving the best PV installers in the domestic residential, commercial and off-grid business. John's qualifications include senior sales representation to Fortune 500 companies, leading renewable energy market development from Silicon Valley to Hawaii, and active involvement in numerous industry groups including CALSEIA. Mr. White will be based in the greater Los Angeles area.
To more effectively expand into new markets, channels and technology and product categories, Morningstar is adding Mark Cerasuolo to its marketing team. Mark brings over 15 years of combined electrical and solar industry marketing experience between Leviton Manufacturing and OutBack Power, as well as several decades with consumer brand leaders including Bose Corporation and JBL/Harman International. Mr. Cerasuolo's industry activities range include the Washington State Clean Tech Alliance and the Consumer Electronics Association (now CTA). He will be joining Morningstar as marketing director at the company's Newtown, Pennsylvania offices.
"With our product portfolio dramatically expanding into control software, advanced technology inverters, and high-value components for selected global niche opportunities, we're very excited to build our 'team roster' with these three new hires," notes CEO and founder Lee Gordon. "Morningstar's employee-owned culture and strong, independent growth means that we're very selective in this area. With Gabrielle, John and Mark now coming on-board, we're confident we can look forward to another 25 years of outstanding success in both our core business and with our new targeted opportunities."
With well over 3 million products installed and sold since 1993 – in some of the most extreme environments and mission-critical applications in the world – Morningstar Corporation is truly "the world's leading supplier of solar controllers and inverters." The US-based privately owned company is headquartered in Newtown, Pennsylvania, with a dedicated engineering research and development center in Maryland and a global network of dedicated, best-in-class distribution and manufacturing partners.
Morningstar Products
Morningstar offers its products through a network of distribution partners.
Morningstar Corporation
8 Pheasant Run,
Newtown, PA 18940 USA
© Morningstar Corporation

redpajama_set_name: RedPajamaCommonCrawl


Yeah, as of this morning, there's now three damn mouths to feed haha.
Congrats! Don't sweat the trophies, enjoy the new baby.
[quote=Mufasa][quote=markdjdeans16]Massive thanks for event boss and within the space of time you had as we all know you are now happy father again so big shout out and all best for future [/quote] Yeah, as of this morning, there's now three damn mouths to feed haha. Bare with me guys while I get the rewards out and all that jazz. I'll bump the foam fingers shortly[/quote] @neo88 said he hoped the baby looked like the momma cause you was an oooogly mofo.. LMAO Hope all is well sir..
@neo88 said he hoped the baby looked like the momma cause you was an oooogly mofo..
Hope all is well sir..
[quote=Mufasa]Trophies and favors all around. Good game[/quote] Thanks Mu!


We investigate the role of person-specific and place-specific factors in the opioid epidemic by analyzing cross-county migration of disabled Medicare recipients and its relationship with prescription patterns associated with opioid abuse. We find that movement to a county with a 20 percent higher rate of opioid abuse (equivalent to a move from a 25th to 75th percentile county) increases rates of opioid abuse by 6 percent, suggesting that roughly 30 percent of the gap between these areas is due to place-specific factors. These effects are particularly pronounced for prior opioid users, who experience an increase in opioid abuse four times larger than the increase for opioid naives.


Vishwakarma Institute of Technology, Pune, is a private institute and occupies a place of pride amongst the premier technical institutes of the western region of India. The Institute is running 9 undergraduate, 8 post graduate & 6 Ph.D Programme which are affiliated to the University of Pune.
Vishwakarma Institute of Technology offers numerous courses under the fields of management and engineering. It includes Chemical Engineering, Computer Engineering, Electronics Engineering, Electronics and Telecommunication Engineering, Industrial and Production Engineering, Instrumentation Engineering, Information Technology, MCA and Mechanical Engineering.
Per year for girls fees is INR 128535 per annum. For open we have EBC. And all caste scholarships are available for 60% of students every year. Depending on your family condition All the national banks provide education loan.
Vishwakarma Institute of Technology, Pune, a highly commendable private institute, occupies a place of pride amongst the premier technical institutes of the western region of India. Established in the year 1983, financed and run by the Bansilal Ramnath Agrawal Charitable Trust, Pune. It is affiliated to the University of Pune . Within two decades, the institute marched towards the pinnacle of glory through its remarkable achievements and laurels in the field of engineering education of high calibre. According to Indian mythology, 'Vishwakarma' was a divine architect-engineer. It is believed that He fashioned this world with His rare engineering skill. The Trust adopted this name with a vision to develop engineers of high calibre, who could take up challenges of any type of engineering job and become successfull in the chosen career.
The campus boasts of lush green lawns that provide an ambience and a serene atmosphere of ancient Gurukuls far from the madding crowd. It has got 17.625 acres of land, out of which 7 acres are utilised for academic building area and available built up area for the entire institution is 29,039 sq. mts.


text:
MBBS From Al-Farabi Kazakh National University, How to Get Direct Admission in MBBS in Al-Farabi Kazakh National University Kazakhstan, Direct Admission in Al-Farabi Kazakh National University Kazakhstan, MBBS Admission in Al-Farabi Kazakh National University, Study in Al-Farabi Kazakh National University, Apply for MBBS Admission in Al-Farabi Kazakh National University, MBBS Fee Structure of Al-Farabi Kazakh National University
https://www.kaznu.kz/en/17311/page/, https://www.moksh16.com/kazakh-national-university/, https://www.jagvimal.com/university/al-farabi-kazakh-national-university, https://www.edufever.com/al-farabi-kazakh-national-university-kazakhstan/, https://www.topuniversities.com/universities/al-farabi-kazakh-national-university, https://www.eklavyaoverseas.com/al-farabi-kazakh-national-university/, https://omkarmedicom.com/al-farabi-kazakh-national-university/
About Al-Farabi Kazakh National University, Kazakhstan
Al-Farabi Kazakh National University was founded in 1934. It is the leading institution of the system of higher education of the Republic of Kazakhstan, and it is the oldest classical university in the republic.
Al-Farabi Kazakh National University Kazakhstan is one of the top Medical Colleges in Kazakhstan. The Al-Farabi University is also known as KAZNU Kazakhstan.
It has taken its name from the renowned philosopher and scholar named Al-Farabi. In the year 2001, this was recognized as the national university. Currently, more than 20,000 students study at the university and there are 2500 faculty members teaching at the university.
Why Study MBBS at Al-Farabi Kazakh National University?
Below are some of the benefits of studying MBBS at Al-Farabi Kazakh National University:
The students are not required to sit for any entrance examinations before securing admission to the Kazakhstan Medical University.
The university does not demand any kind of donations.
Both tuition fees and cost of accommodation are on the lower side for students who choose to study at this university.
The university offers top notch education facilities for the students.
The university offers a safe and engaging environment for all.
The infrastructure of the university is modern and highly advanced.
There is a safe and peaceful atmosphere which is good for studies at the university.
Students are provided with extracurricular activities such as sports facilities, music and other related leisure activities.
The faculties at the university are cooperative and qualified.
The medium of instruction in English for the international medical candidates.
The university has modern and advanced infrastructure.
The eligibility criteria to study MBBS at Al-Farabi Kazakh National University is as follows:
Students must have more than 60% marks in 12th standard.
Students must have qualified for the NEET-UG examination.
The age of a student or applicant must be more than 17 years and less than 25 years.
A student must have chemistry, physics, and Biology as the main subject in 12th standard.
Following is the admission process for applying to the MBBS course at Al-Farabi Kazakh National University:
Step 1: Students should apply online. They should fill the online application form of the university.
Following documents are required for enrolling in the university:
Copy of the transcript of grade 10, grade 11, grade 12
school-leaving certificate
valid passport
Step 2: The students will be informed about the dismissal and acceptance of the application once their documents are evaluated. The student will be informed after 1-2 working days, in the event that a student & application structure is acknowledged. The students will be given the affirmation inside a similar period.
Step 3: Applicants should pay enrollment expenses. Students can pay the fees through net banking or other payment methods, after receiving the confirmation letter.
Step 4: Applicants can then apply for the student visa.
Documents Required at the Time of Admission
The following documents are required at the time of admission to Al-Farabi Kazakh National University:
Passport (Minimum 18 months validity)
10 passport size Photographs
The certificate of Secondary school education (after 9th grade)
The Certificate of High school education (after the 11th grade)
Passport size photos – 4 (in the size 3×4)
Address reference
Identity card and a PIN
Medical inquiry on Form No. 086-Y
Characteristics from school
Bank receipt of University's payment of 1st-year Tuition fee (required for some Universities)
Physical Examination Record for Foreigners.
Al-Farabi Kazakh National University, Fee details:
Al-Farabi Kazakh National University 5 Years 6100 4700 24,900
1st year fee include: Tuition Fees, Hostel for 1 Year, Medical Insurance (1 years).
2nd year onwards: Medical Insurance 100 USD, Hostel 600USD
Mess 1000 USD per yearly
Al-Farabi Kazakh National University Kazakhstan- MedStudies


text:
CPRBlog
The Center for Progressive Reform
White Collar Crime and the Trump Administration
Rena Steinzor
Rena Steinzor is the Edward M. Robertson Professor of Law at the University of Maryland Francis King Carey School of Law, and a past president of the Center for Progressive Reform. She is the author of Why Not Jail? Industrial Catastrophes, Corporate Malfeasance, and Government Inaction.
Professor Steinzor has taught an environmental law survey course, seminars in risk assessments and critical issues in environmental law and science, administrative law, contracts, torts and counseling and negotiation. She has written in the areas of (1) regulatory dysfunction in agencies assigned to protect public health, worker and consumer safety, and the environment; (2) the role of centralized White House review on the protectiveness of regulation; (3) environmental federalism, including so-called "unfunded mandates" imposed on state and local governments by the federal government and the impact on public health of devolving authority and responsibility for solving environmental problems; (4) the implications of industry self-regulation on the protection of the environment and human health; (5) "market-based" alternatives to traditional regulation; and (6) political interference with regulatory science.
She is the editor, with Christopher Schroeder, of the CPR-sponsored book A New Progressive Agenda for Public Health and the Environment, published by Carolina Academic Press. She is also the editor, with Wendy Wagner, of the book Rescuing Science from Politics, published by Cambridge University Press in 2006. Her book, Mother Earth and Uncle Sam: How Pollution and Hollow Government Hurt Our Kids was published by the University of Texas Press in December 2007. With Professor Sidney Shapiro of Wake Forest Law School, she co-authored The People's Agents and the Battle to Protect the American Public: Special Interests, Government, and Threats to Health, Safety, and the Environment published by the University of Chicago Press in 2010.
Professor Steinzor began her legal career in 1976, and entered academia in January 1994. From 1987 through 1993, she was associated - first as "of counsel" and ultimately as the partner in charge of the environmental practice - at Spiegel & McDiarmid, a 45-lawyer, Washington, D.C. firm representing approximately 400 cities, counties, states, and public agencies in the energy, environmental, communications, and transportation fields. The practice counseled federal, state, and municipal clients regarding compliance with federal and state laws and regulations.
Prior to joining Spiegel & McDiarmid, Professor Steinzor served as Staff Counsel, Subcommittee on Commerce, Transportation, and Tourism of the Energy and Commerce Committee, U.S. House of Representatives (James J. Florio, Chairman). She was the primary staff person responsible for legislation that became the "Superfund Amendments and Reauthorization Act of 1986" (Public Law 99-499) and the "Asbestos Hazard Emergency Response Act" (Public Law 99-519). She also prepared legislation to reauthorize the Toxic Substances Control Act during the 98th Congress.
Professor Steinzor has testified before Congress on several occasions, most recently regarding the impact of health, safety, and environmental regulations on the economy.
University of Maryland Carey School of Law
410.706 0564
Cross-posted by permission from the Columbia Blue Sky Blog.
The Obama administration had a mixed record on white collar crime. On one hand, it extracted $4 billion and a guilty plea from BP in the wake of the Deepwater Horizon spill. On the other hand, it allowed HSBC, then the fourth largest bank in the world, to sign a deferred prosecution agreement (DPA) over charges of laundering money for a Mexican drug cartel and serving as a banker for illicit regimes in Burma, Cuba, Iran, Libya, and Sudan. The bank paid $1.256 billion in penalties, but because it never admitted its crimes and controlled such vast amounts of money, the payment looked more like a cost of doing business than punishment. In fact, the Obama administration Department of Justice (DOJ) under the leadership of Attorney General Eric Holder and Criminal Division chief Lanny Breuer used DPAs more often than did any other administration. Holder and Breuer repeatedly explained such settlements as necessary to avoid putting corporations out of business and inflicting collateral damage on innocent employees and shareholders. Holder told senators who questioned the HSBC settlement that he thought some institutions were, in effect, too big to jail. When the statute of limitations ran out in 2015 for indictments arising out of the 2008 financial crisis, DOJ's reputation as an even-handed prosecutor suffered major damage.
Holder's successor, Loretta Lynch, and her deputy, Sally Yates, moved decisively to change these perceptions. "Stung by years of criticism that it has coddled Wall Street criminals, the Justice Department issued new policies on Wednesday September 9, 2015 that prioritize the prosecution of individual employees — not just their companies — and put pressure on corporations to turn over evidence against their executives," the New York Times reported.
The Yates Memo set forth what some attacked as radical and unfair changes and others dismissed as business as usual. In a nutshell, Yates and a task force of career prosecutors mandated that corporations hoping to get credit for cooperating with the government in criminal cases turn over all the information available to them regarding illegal acts by individual employees. If prosecutors do not develop charges against individual corporate officials in any given case, they must justify the omission in the materials they forward up the DOJ decision-making chain for approval. The Yates Memo reasoned that "one of the most effective ways to combat corporate misconduct is by seeking accountability from the individuals who perpetrated the wrongdoing. Such accountability is important for several reasons: it deters future illegal activity, it incentivizes changes in corporate behavior, it ensures that the proper parties are held responsible for their actions, and it promotes the public's confidence in our justice system."
A handful of former prosecutors announced that the Yates Memo merely formalized long-standing practice. But it caused a small tsunami of alarm among many members of the defense bar, a handful of academics, and influential industry trade associations, including the Chamber of Commerce. They were still whinging when Donald Trump won the election and arrived in Washington, D.C. intent on shaking up the established order. Where did this commitment to change lead with respect to white-collar crime? Had they been asked, Trump voters would have expressed disgust that banks and other financial institutions ducked criminal liability for corralling, losing, and recouping hundreds of billions of dollars at the expense of the middle class. Regardless of his promises to "drain the swamp," President Trump populated his cabinet with Wall Street veterans. Even the architect of his special brand of nationalistic populism, Steve Bannon, is an alumnus of Goldman Sacks.
The New York Times has reported that Trump son-in-law Jared Kushner, whose power is waxing, "has pushed to overhaul the criminal justice system, a goal that Mr. Trump embraced as a candidate near the end of the campaign when he tried to siphon black voters away from Hillary Clinton. But Mr. Kushner is running into opposition from Attorney General Jeff Sessions, who favors toughening, not relaxing, mandatory minimum sentences." Like so many other issues undermining the Trump presidency, different factions are at war within the White House, making any predictions about the future of white-collar enforcement seem premature.
Sessions will engender substantial controversy if he focuses federal enforcement on street crime and neglects white-collar prosecutions. Mass incarceration has edged to the forefront of social and economic problems in the U.S. During the last Congress, an unusual right/left alliance pushed so-called "mass incarceration reform" quite far. A bill to modify mandatory minimum sentences was approved by the House of Representatives Judiciary Committee but the legislation stalled before it reached the House floor, in part because it increased the burden of proof for federal prosecutors in white-collar crime cases. In the Senate, a bipartisan group of seven senators (Cory Booker (D-N.J.), John Cornyn (R-Texas), Lindsey Graham (R-S.C.), Charles Grassley (R-IA), Patrick Leahy (D-Vt.), Mike Lee (R-Utah), Charles Schumer (D-N.Y.), and Sheldon Whitehouse (D-R.I.)) produced comparable legislation. It too never reached the floor because Senator Orrin Hatch (R-UT) demanded that the bill be amended to add tougher burdens on prosecutors.
The strange bedfellows nature of these coalitions is its greatest strength. Fiscal and libertarian conservatives (Koch Industries, Freedom Works, and Right on Crime) participate out of concern about prison spending, which is about $80 billion annually and unsustainable for many states. Liberals (the Leadership Conference Education Fund, the NAACP Legal Defense Fund, the ACLU, and the Center for American Progress) hope to rebuild communities by preventing lengthy prison terms that disproportionately punish people of color. Members of these groups may grow restless and push reform legislation without Trump administration support. Alternatively, despite the advice of Attorney General Sessions, the Trump administration may support these efforts. In either event, the Yates Memo is likely to remain on the DOJ website, if for no other reason than pulling it down would provoke a new round of criticism that the Trump administration is soft on white-collar crime.
If implemented aggressively, the Yates Memo represents a white collar reset of consequence. Observers who conclude that its content is nothing new may wish to curry favor with government prosecutors. Even discounting for the fact that they hope to strengthen client relations by railing against DOJ heavy-handedness, some of their criticisms are well-placed.
The most salient criticism of the Yates Memo is that compelling a company to help DOJ make cases against its individual managers will cause excessive stress within a corporation. Managers will turn on each other in a desperate effort to avoid identification as a potentially culpable party, and in the chaos, innocent employees may well have their constitutional rights trampled. Tying the outing of individual managers to the sweetness of the final deal may well lead defendants to try their cases rather than implicating top managers. According to federal prosecutors, by far the most secure safe harbor is a robust compliance assurance program that disciplines executives who resist its demands.
The dearth of criminal prosecutions in the financial services industry in the wake of the 2008 crash will continue to shadow federal prosecutors. The last time bankers organized a comparable scam—namely, the savings and loan crisis of the 1980s—the result was 1,000 felony convictions, including cases against 600 individuals and 300 institutions. Perceptions of unfairness exacerbate the anger that increasingly drives our national politics. Resentment of the yawning income gap between the richest 1 percent and the rest of the population, static wages, and the exodus of industrial jobs to poorer countries are themes that President Trump used quite effectively throughout the campaign.
Yet two additional realities challenge this analysis. Because most federal white-collar prosecutions involve the flouting of regulatory mandates, Trump's pledge to "make America great again" by eliminating 75 percent of rules now on the books seems unlikely to generate much productive investigation from members of the career civil service, especially because he wants to cut their budgets severely.
Second, the president's war on regulation may well produce the deconstruction of much of the government, as adviser Steve Bannon has claimed is a top priority. That commitment was whetted by early vetoes of a dozen or so major rules issued by the Obama administration.
Trump's ally, Jeb Hensarling (D-TX), chair of the House Financial Services Committee, has pledged to repeal the Dodd-Frank Wall Street Reform and Consumer Protection Act even as some bankers protested such a drastic step. The Trump "skinny budget" (so named because it lacks the detail the president will provide when he has more time in office) would put the Environmental Protection Agency (EPA), the poster child for egregious regulatory activities, on a starvation diet of 31 percent fewer resources than it has today. If the president accomplishes even half of those cuts, EPA enforcement will slow to a crawl.
President Trump selected former Oklahoma Attorney General Scott Pruitt to serve as EPA administrator. Pruitt made his national reputation by suing the agency on behalf of business interests 14 times. To make sure that his beleaguered staff got the message, Pruitt invited representatives of the coal industry to the agency to celebrate the announcement that he planned to cancel Obama administration efforts to combat climate change.
Interestingly, a recent article published by the Bureau of National Affairs quoted several former prosecutors, most of whom have joined the defense bar. All agreed that Trump's hostile attitude toward the EPA and other regulatory agencies would not have much of an effect on white-collar enforcement, because DOJ only prosecutes obviously aberrant and illegal behavior, and its approach to enforcement of environmental and worker safety laws has been consistent over several decades. Perhaps they are right. Or perhaps these confident predictions do not take the Trump administration at its word, a chronic problem in the Capitol these days
This post comes to us from Rena Steinzor, the Edward M. Robertson Professor at the University of Maryland Carey Law School. It is based on her recent article, "White Collar-Reset: The DOJ's Yates Memo and Its Potential to Protect Health, Safety, and the Environment," available here.
Read More by Rena Steinzor
CPR HOMEPAGE
More on CPR's Work & Scholars.
Frank Ackerman David Adelman Robert Adler Amro Ali William Andreen Mary Jane Angelo Jeremy Baker Shalanda H. Baker Melanie Benesh Samuel Boden Karrigan Bork Liz Borkowski Rebecca Bratspies Maxine Burkett William Buzbee Natalia Cabrera Jake Caldwell Alejandro Camacho M. Isabelle Chaudry Johnathan Clark Thomas Cluderay Robin Kundis Craig Carl Cranor Kerry Darragh Mariah Davis Maggie Dewane Holly Doremus Gilonne d'Origny David Driesen Michael C. Duff K.K. DuVivier John Echeverria Joel Eisen Ross Eisenbrey Kirsten Engel Lee Ewing Daniel Farber Joe Feller Adam Finkel Robert Fischman Liz Fisher Victor Flatt David Flores Alyson Flournoy Elena Franco Matthew Freeman Bill Funk Margaret Giblin Robert Glicksman Dale Goble Alina Gonzalez Catalina Gonzalez James Goodwin Elizabeth Grossman Brian Gumm Emily Hammond Thomas Harter Anne Havemann Sean B. Hecht Lisa Heinzerling Yee Huang Colin Hughes David Hunter Evan Isaacson Michele Janin Peter Jenkins Catherine Jones Shana Campbell Jones Bradley Karkkainen Alice Kaswan Melissa Kelly Erin Kesler Alexandra Klass Christine Klein John Knox Sarah Krakoff Douglas Kysar Sarah Lamdan Amanda Cohen Leiter John Leshy Richard E. Levy Clarissa Libertelli Melissa Lutrell Patrick MacRoy Lesley McAllister Martha McCluskey Thomas McGarity Nina Mendelson Rachel Micah-Jones Katrina Miller Darya Minovi Joel Mintz Celeste Monforton Richard Murphy Jennifer Nichols Catherine O'Neill Uma Outka Dave Owen Jarryd Page Eric Panicco Richard Parker Michael Patoka Richard Pierce, Jr. Lena Pons Melissa Powers Emily Ranson Laurie Ristino Dan Rohlf Jorge Roman-Romero Daniel Rosenberg Mollie Rosenzweig Noah Sachs Katlyn Schmitt Christopher Schroeder Isaac Shapiro Sidney Shapiro Matt Shudtz Aimee Simpson Minor Sinclair Amy Sinden Karen Sokol Ben Somberg Rena Steinzor Scott Stern Allison Stevens Kim Sudderth Steph Tai A. Dan Tarlock Joseph Tomain Katie Tracy Robert Verchick Nicholas Vidargas David Vladeck Wendy Wagner Hannah Wiseman Chris Wold Sandra Zellmer Michelle Zemil
Jan 2022 Dec 2021 Nov 2021 Oct 2021 Sep 2021 Aug 2021 Jul 2021 Jun 2021 May 2021 Apr 2021 Mar 2021 Feb 2021 Jan 2021 Dec 2020 Nov 2020 Oct 2020 Sep 2020 Aug 2020 Jul 2020 Jun 2020 May 2020 Apr 2020 Mar 2020 Feb 2020 Jan 2020 Dec 2019 Nov 2019 Oct 2019 Sep 2019 Aug 2019 Jul 2019 Jun 2019 May 2019 Apr 2019 Mar 2019 Feb 2019 Jan 2019 Dec 2018 Nov 2018 Oct 2018 Sep 2018 Aug 2018 Jul 2018 Jun 2018 May 2018 Apr 2018 Mar 2018 Feb 2018 Jan 2018 Dec 2017 Nov 2017 Oct 2017 Sep 2017 Aug 2017 Jul 2017 Jun 2017 May 2017 Apr 2017 Mar 2017 Feb 2017 Jan 2017 Dec 2016 Nov 2016 Oct 2016 Sep 2016 Aug 2016 Jul 2016 Jun 2016 May 2016 Apr 2016 Mar 2016 Feb 2016 Jan 2016 Dec 2015 Nov 2015 Oct 2015 Sep 2015 Aug 2015 Jul 2015 Jun 2015 May 2015 Apr 2015 Mar 2015 Feb 2015 Jan 2015 Dec 2014 Nov 2014 Oct 2014 Sep 2014 Aug 2014 Jul 2014 Jun 2014 May 2014 Apr 2014 Mar 2014 Feb 2014 Jan 2014 Dec 2013 Nov 2013 Oct 2013 Sep 2013 Aug 2013 Jul 2013 Jun 2013 May 2013 Apr 2013 Mar 2013 Feb 2013 Jan 2013 Dec 2012 Nov 2012 Oct 2012 Sep 2012 Aug 2012 Jul 2012 Jun 2012 May 2012 Apr 2012 Mar 2012 Feb 2012 Jan 2012 Dec 2011 Nov 2011 Oct 2011 Sep 2011 Aug 2011 Jul 2011 Jun 2011 May 2011 Apr 2011 Mar 2011 Feb 2011 Jan 2011 Dec 2010 Nov 2010 Oct 2010 Sep 2010 Aug 2010 Jul 2010 Jun 2010 May 2010 Apr 2010 Mar 2010 Feb 2010 Jan 2010 Dec 2009 Nov 2009 Oct 2009 Sep 2009 Aug 2009 Jul 2009 Jun 2009 May 2009 Apr 2009 Mar 2009 Feb 2009 Jan 2009 Dec 2008 Nov 2008 Oct 2008 Aug 2008 Jul 2008
Will the 30 x 30 Initiative Protect 30 Percent of Freshwater Resources by 2030?
States Should Act to Protect People and Our Environment from Unregulated Chemical Tanks
Democracy, Rulemaking, and Outpourings of Comments
CPR, Partners Call for Climate Justice Reforms to the Chemical Industry
Memphis Commercial Appeal Op-Ed: Supreme Court Turns to Science to Resolve Groundwater Dispute Between Mississippi and Tennessee
Aboveground Chemical Storage Tanks Threaten Our Communities. It's Time for EPA and States to Act.
1250 Connecticut Ave. NW, Suite 700
Washington, DC. 20036
info@progressivereform.org
Through the expertise of our 60+ scholars and staff, the Center for Progressive Reform creates powerful tools and strategies for policymakers and the public to advance policies and processes that ensure the health and well-being of people, their communities, and our environment.
Home Donate About Us Subscribe Email CPR


Landswehr are the heavily-armored, heavily armed knights of Almania. They disdain fighting from horseback, going into battle on foot. They wear greathelms with thin visors over their heads and green capes over their backs, often wielding flails and spiked clubs. They worship Heron, Lord of Justice and are in full communion with the Magisterium. However, their prime loyalty is to the Grand Duke. They are his greatest asset.


text:
Q: screen brightness changing issue in change plan setting using c# code I am trying to set the screen brightness using WmiSetBrightness method.
The brightness is changed in the setting and control panel but the didn't change the below location Control Panel\Hardware and Sound\Power Options inside change plan setting.
Please help me
how I can change the change plan setting using C# code ?

redpajama_set_name: RedPajamaStackExchange


Another Faux Fur look !!! Are you tired of them yet? When I am blessed with a not so cold winter day, I take full advantage and show a little skin !! I recently went to Primart not expecting to purchase anything and then, BOOM !! I ended up purchasing 3 bags that I am absolutely obsessed with. Every bag was at least 15 dollars or under! Whos happy? Dwenne is !!
This sweater is actually oversized, I tucked it into this faux leather skirt to make the outfit more put together. The missing piece to this look was obviously my faux fur coat, it completes the look perfectly !!


text:
TurtleBu
Discover Sri Lanka
A Sri Lanka Itinerary
A Portugal Itinerary
Your Sri Lanka Trip
Your Portugal Trip
Ancient Culture, Nature Diversity and Endless Ocean Views Our 15 Day Sri Lanka Itinerary
Interactive Sri Lanka Itinerary (Click on stop to know more)
It can be a challenge to plan the perfect Sri Lanka itinerary that gives you glimpse of everything this beautiful country has to offer during only 2 weeks. However, reflecting and focusing on what we really wanted to experience made it much easier. We planned our trip in a way that we could dive into the world of meditation and Buddhism, climb mountains, discover tea plantations, make wild encounters with elephants and finally have plenty of time to just relax.
Our planning started during another European winter blues – the rainy, cloudy and cold days were officially back – and once more we were desperate for some time to relax after hard months of work. Isn't this a very common worldwide feeling? Well certainly not for people living close to amazing beaches and having good weather all year round, we thought! But everything started with the same old "first world dilemma": Where could we go? Where is it not utterly commercial and crowded? Where is it not only warm but there is also more to do? Where…?
It was, of course, Sri Lanka: a country plenty of culture, groove, breathtaking landscapes, wildlife and with places almost untouched by man! It was a fairy tale journey that made us absolutely fell in love with this land and its people. Come and relive it with us by following this Sri Lanka itinerary.
Sri Lanka in 15 Days - our video itinerary
Arriving in Colombo Sri Lanka Itinerary: Days 1-2 (1 Night)
As most of the people we chose Sri Lanka's capital as our first destination since it is close to the main port of entry Bandaranaike International Airport. We made a super bargain with our flights from Europe to Colombo for just 391 Euros per Person with Etihad Airways. The planes and service could not have been more convenient. In addition, an 8 hour-layover in Abu Dhabi gave us the chance to visit the beautiful Grand Mosque. Arriving late, we took a driver from the airport to our hotel in the city center. Furthermore, there is also buses connecting the airport with the central bus station.
Colombo is a very vibrant and colorful unique city. As soon as you conquer the streets, you can already feel its pulse: sense the strong smells of local food and hear loud traditional music pumping out of speakers mixed with the noise of driving tuk-tuks! The capital has lots to see depending on your taste. Therefore, we decided to stay for one night and a full day. As a result we were able to fit several strolls around main and secret streets as well as to tick some of Colombo's top temples and sights. We didn't feel we missed anything. However, if you like to sit down in between and breathe more of the city's atmosphere, you will not be bored when extending your stay for a day. Check out how we did in the bustling capital of Colombo.
Sigiriya Rock and Polonnaruwa Sri Lanka Itinerary: Days 2-4 (2 Nights)
Location of Sigiriya
In the afternoon we made our way from the central bus station of Colombo towards the city of Dambulla. There are two types of buses running on this 150km route: local ones that require 7-8 hours for the journey or express mini buses that take about 5 hours. We decided for the second option being more comfortable, equipped with AC and only a bit more expensive. This bus did not have a strict timetable and the driver waited for approximately 1 hour until there was enough people. We took a tuk-tuk to Sigiriya, the next stop of our Sri Lanka itinerary. The journey took just 20 minutes from Dambulla.
Sigiriya is a tiny village, which like many in Sri Lanka consists of just one single street. Locals businesses are located on both sides of the road and the villagers spread throughout the territory. Most of the people commute here only to reach the top of Sigiriya Rock or Pidurangala – read our story about climbing Sigiriya Rock. However, the place is strategically well located if you want to explore the temple region. Right in center between Anuradhapura, Polonnaruwa and Dambulla, Sigiriya is great to stay overnight. It offers a mid range of local restaurants and small bars, where we could relax after a long day of sightseeing. Its pleasant atmosphere and budget rates make it a nicer place to stay for backpackers than the bigger towns around.
We climbed Sigiriya Rock early in the morning. At the exit we arranged a local tuk-tuk driver to take us to Polonnaruwa during the afternoon. The route crosses Mineriya National Park so that you get to see lots of wildlife on the way there. We even had to stop because of an Elephant blocking the road. The ruins and Buddhist temples of Polonnaruwa are definitely worth a visit to dive into Sri Lanka's culture. Read our experience in the Journal entry about Polonnaruwa's ancient royal city.
Dambulla, Kandy and Surroundings Sri Lanka Itinerary: Days 4-7 (3 Nights)
The previous day, we had asked our tuk-tuk driver to take us 90km from Sigiriya to Kandy. However, we could not miss a stopover in Dambulla to exprience the amazing Cave Temple and Golden Temple complex in the morning. Then, after a long tuk-tuk journey with constant rain splashing into the vehicle, streets with heavy congestion and air pollution lead the way into "The Holy City" of Kandy.
The place is famous for its religious temples and everybody interested in meditation and Buddhism will love it! Although there are numerous temples, we decided to stick to the famous 4 and an insider recommendation, which has become our Top Choice. You can check them all in our review of Sri Lanka's temples. Most noteworthy, there is much more than this: we could immerse ourselves in the gasping green mountain landscapes, Kandy's beautiful Botanical Gardens and learn the secrets about local tea production. Check out what we have done in Kandy!
Hill Country Train Ride and Ella Sri Lanka Itinerary: Days 7-8 (1 Night)
Much as we enjoyed Kandy, we were keen on continuing our adventure. For the next stop of the Sri Lanka itinerary, we hopped on Sri Lanka's most scenic train ride through the tea fields of the highlands. We caught the train down the road of our hotel at Peradeniya station. Seems like we had no idea how truly wonderful it would be! It was a breathtaking ride through remote places and with an average speed 20 km/h you can easily let your feet hang out the door and breathe the fresh air. That is why we decided to called our travel report "a train ride through heaven". Never a seven-hour journey passed so quickly bringing us to the village of Ella.
Ella is in our opinion the "coolest" place in Sri Lanka. Famous for its tea scenery and gasping landscapes it also brings together young people. From trendy wooden-made restaurants to hipster bars, in Ella you will encounter the most of backpackers counting on seconds to climb the highest mountain of Sri Lanka (Adams Peak) or dying for a walk and take a snapshot of the famous railway track. Ourselves, we enjoyed the most probably best food in Sri Lanka and an original Spa treatment. Read more about our experience in Ella.
Tree House Stay and Yala National Park Sri Lanka Itinerary: Days 8-9 (1 Night)
After enjoying a last rotti for lunch and taking advantage of the good coffee served in Ella, we jumped on the next bus towards Yala National Park. There are only few and irregular direct buses to Kataragama and Tissaharama. Consequently, we decided to take the main bus route towards Mirissa/ Galle and got off at Pannegamuwa junction. From there it was just a short tuk-tuk ride to where we were staying at Beddegama Ecopark. It was one unique experience to sleep in a tree house all made out of wood and palm leaves. First of all, there was this huge king size bed and a tempting living room area. But most noteworthy, we had a charming wooden bathroom where shower waters run from a piece of wood over your body.
Much as we loved to relax in our tree house, the purpose of our stay was Yala National park. Hence, we pre-booked a Safari via our accommodation, which turned out to be amazing. We were travelling exclusively and had a really savvy driver that made us see all kind of animals: elephants, alligators, monkeys, peacocks and even leopards (which seems to be a tough job). Read our detailed experience about Yala.
Tropical Paradise in Tangalle Sri Lanka Itinerary: Days 9-15 (6 Nights)
The Safari finished at lunch time. Therefore, we asked our driver to leave us at the central bus station of Kataragama. Buses from there towards Galle and Colombo along the South Coast run every 20-30 minutes. The bus directly dropped us in the center of Tangalle to commence the lush part of the journey.
Most of all, for the last 7 days of our adventure in Sri Lanka, we looked for and found heaven on earth, where beach, nature, ocean and peace are in perfect harmony. Most of all, this was what brought us to this island before we even landed! Seems like we were very lucky to find Palm Paradise Cabanas and Villas. It is a resort made of remote wooden beach cabanas facing the ocean with a touch of modern villas on top of the hill. Seems like this was just what we needed to refill our empty batteries.
Tangalle itself is relatively remote and quiet, probably with less backpackers and party crowds than Mirissa or Unawatuna. Hence, lots of calm and picturesque beaches make Tangalle coast line the perfect hide-away. If you want to know more, check out our experience at the beaches of Sri Lanka.
Day Trip to Galle and Unnawatuna Sri Lanka Itinerary: Day 12
While staying in Tangalle, we wanted to seize the opportunity to explore more of the South coast. Therefore, we took a direct bus to Galle stopping at the main street in front of our cabanas every few minutes. Leaving in the early morning, we arrived after two hours of typical Sri Lankan bus ride experience.
Being one of Sri Lanka's biggest cities, Galle's central bus station is nothing different from a normal western main station: Busy, dirty, bit dodgy. Yet Galle's "crown jewel" is the old town inside its impressive fortress walls. Almost you feel like suddenly landing on another continent – a super clean and European-style village. Probably this gem makes Galle the most beautiful city in the country. Due to its buildings remembering of old Dutch colonialist architecture, it looks like every corner in every street has been kindly and designed in detail! Check for our report on Galle, the Europe at the Indian Ocean.
Some of the island's most famous ocean spots are just a stone's throw away from Galle. Therefore, we seized the opportunity to discover more of it's surroundings. Taking a tuk-tuk to Jungle beach and exploring the forest around while walking to Unnawatuna was a great experience. Later we had dinner at the beach and enjoyed a great sunset before returning to Tangalle. Some tips about the area you can find in our journal about Sri Lankas beaches.
What we had done differently? Wrap-up Sri Lanka Itinerary
We loved our itinerary and between planning and re-planning bits and bobs it took us around 3 months. We hope that maybe our journey will make it easier for you to plan your own one. However, there is certain things that we would do differently when doing this trip again:
Stay more days in Ella! It is a really pleasant village with plenty of breathtaking landscapes to discover and hike
On the way between Kandy and Ella, we would probably stop in Nuwara Eliya to discover the tea fields
Travel the South coast! We loved Tangalle and consider it the nicest beaches we saw. However, instead of making all the way up to Colombo on one day, we should have made a stop in between. Seems that Galle would have been a great option to also explore its western beaches such Hikkaduwa. The day trip we made turned out to be quite stressful.
Find out More! All Turtle Journal Entries about the Sri Lanka Itinerary
The Real Sri Lanka… Festivals, Voluntary Work and Aryuveda
Transport of Sri Lanka… Taking it Slow
Sri Lanka's Food… Spicy and Delicious
Beaches…Sri Lanka's Pearl Gems!
Temples of Sri Lanka… Celebration of Diversity!
Galle… Europe at the Indian Ocean!
Yala… A God's Hand Painting!
Ella… Toddler of Sri Lanka!
Kandy to Ella… A Train Ride through Heaven!
Kandy… Jewel Crown of Sri Lanka
June 12, 2018	@ 12:49 pm
What an amazing itinerary!
© TurtleBu 2022