text:
\section{Introduction}

\def\figTask#1{
\begin{figure}[#1]
\tb{@{}cc@{}}{0.5}{
\tb{c}{0}{\imw{images/TeasersSSL.pdf}{0.65}}&
\tb{c}{0}{\imw{images/TeasersAL.pdf}{0.34}} \\
\\
{\bf a)} data-centric vs. model-centric SSL &
{\bf b)} continuous active learning\\
}\\
\iffalse
\tb{@{}ccc@{}}{0.5}{
\imw{images/dataSamplingSSL1.eps}{0.32}&
\imw{images/dataSamplingSSL2.eps}{0.32}&
\imw{images/dataSamplingAL.eps}{0.32}\\
\multicolumn{2}{c}{{\bf c)} unsupervised data selection in SSL}&
{\bf d)} supervised data selection in AL\\
}
\fi
\vspace{-3pt}
\caption{
We consider the novel task of unsupervised data selection for semi-supervised learning (SSL).
{\bf a)} Existing SSL methods focus on training the best model given labeled and unlabeled data, whereas we focus on optimizing labeled data selection.
{\bf b)} Existing AL methods learn a classifier based on random initial selection and alternate between training and instance selection, making its annotation pipeline both difficult to implement and inefficient under low shot settings.
\label{fig:task}
}
\end{figure}
}

\def\figTeaserOne#1{
    \begin{figure}[#1]
        \centering
        \begin{subfigure}{0.47\textwidth}
            \vspace{-2pt}
            \centering
            \includegraphics[width=0.9\textwidth]{images/TeaserPrevs.pdf}\vspace{-1pt}
            \caption{Model-centric semi-supervised learning}
        \end{subfigure}\hfill
        \begin{subfigure}{0.47\textwidth}
            \vspace{-2pt}
            \centering
            \includegraphics[width=0.9\linewidth]{images/TeaserOurs.pdf}\vspace{-1pt}
            \caption{Data-centric semi-supervised learning}
        \end{subfigure}
        \vspace{-7pt}
        \caption{Conceptual illustration of a) previous methods on semi-supervised learning, which are always \textit{model-centric} and improve the efficiency of leveraging both labeled and unlabeled data through novel models or algorithms, ignoring the power of data, and b) our \textit{data-centric} semi-supervised learning, which concentrates on obtaining the most informative labels to improve learning efficiency.}
        \label{fig:teaser}
    \end{figure}
}

\def\figTeaserTwo#1{
    \begin{figure}[#1]
        \centering
        \vspace{-8pt}
        \begin{tabular}[t]{l|c|l}
            \begin{subfigure}{0.22\textwidth}
                \centering
                \smallskip
                \includegraphics[width=0.95\linewidth]{images/teaserAcc.pdf}\vspace{-3pt}
                \caption{Accuracy}
                \label{fig:acc}
            \end{subfigure}
            &
            \begin{tabular}{c
                \smallskip
                \begin{subfigure}[t]{0.22\textwidth}
                    \centering
                    \includegraphics[width=1.05\textwidth]{images/samples_vs_num_cls_box.pdf}\vspace{-5pt}
                    \caption{Class coverage}
                    \label{fig:coverage}
                \end{subfigure}\\\vspace{-2pt}
                \begin{subfigure}[t]{0.22\textwidth}
                    \centering
                    \includegraphics[width=1.02\textwidth]{images/samples_vs_std_box.pdf}\vspace{-5pt}
                    \caption{Data balance}
                    \label{fig:balance}
                \end{subfigure}
            \end{tabular}
            &
            \begin{subfigure}{0.46\textwidth}
                \centering
                \smallskip
                \includegraphics[width=1.0\linewidth]{images/teaserNew.pdf}\vspace{2pt}
                \caption{Informativeness}
                \label{fig:representativeness}
            \end{subfigure}
        \end{tabular} \vspace{-1pt}
        \caption{We quantitatively and qualitatively compare our methods against previous works on four properties, i.e., (a) classification accuracies on downstream tasks, (b) class coverage, (c) semantic class distribution and (d) informativeness of queries, and drastically improve on all of them. FixMatch* and FixMatch in (a) denotes sampling with ``supervised" stratified sampler, random sampler, respectively. Different dot colors in (d) denote different classes. 20 instances are sampled from these 3 classes with various sampling strategies. Our selections consistently cover informative samples across the space. (better viewed in color and zoomed in)}
        \label{fig:teasertwo}
        \vspace{-2pt}
    \end{figure}
}

In many real-world applications, data are plenty but annotations are hard to get by. Semi-supervised learning (SSL), i.e., learning from a small set of labeled data and large-scale unlabeled data, thus becomes very relevant. We study the novel task of {\it unsupervised data selection} for SSL (\fig{task}).

\figTask{ht}

\figTeaserTwo{bt}

Existing SSL methods assume that labeled and unlabeled data are already given (\fig{task}{\bf{a}}) and what's left is to optimize the classifier by integrating information from both small labeled data and large unlabeled data.  The basic idea for SSL is that, while labeled data provides direct supervision on the classifier, unlabeled data helps regularize the classifier in terms of smoothness and sharpness, captured by consistent \citep{sajjadi2016regularization, tarvainen2017mean, xie2020unsupervised, lee2013pseudo, berthelot2019mixmatch, berthelot2019remixmatch, sohn2020fixmatch} and minimal entropy \citep{grandvalet2005semi, lee2013pseudo, berthelot2019mixmatch} predictions respectively.

In practice, unlabeled data are ever expanding and labeled data have to be curated within a certain annotation budget.  Selecting which instances to be annotated not only becomes a valid question, but can also play an important role in the model performance, fairness, robustness, safety, and scalability \citep{halevy2009unreasonable, carlini2019distribution, sohn2020fixmatch, mehrabi2021survey, sambasivan2021everyone}.  We shift from past {\it model-centric} SSL to {\it data-centric} SSL, focusing on improving SSL performance by optimizing data selection from large-scale unlabeled data for annotation (\fig{task}{\bf{a}}).

Two simple instance selection strategies are adopted for studying SSL on fully-labeled datasets: random sampling or stratified sampling by category \citep{berthelot2019remixmatch, sohn2020fixmatch}.  Each sampler has its own caveats:  Random sampling can fail to cover all semantic categories and lead to poorer performance and model instability, whereas stratified sampling is in fact impossible in reality: If we can sample data by their categories, we would already have the label of every instance!

Data selection for optimizing model training is not new; it is in fact the focus of active learning (AL), where the task is defined by an initial set of labeled data and the goal is to select an additionally labeled subset (\fig{task}{\bf{b}}) such that a model learned over the subset is competitive with that over the whole labeled dataset \citep{sener2017active,ducoffe2018adversarial,yoo2019learning}.

However, our data-centric SSL addresses an {\it unsupervised data selection} problem:  Given an SSL model, among many possible ways to select a small fixed number of instances in the sea of unlabeled data for labeling, which way would lead to the best SSL model performance, even though we have no labels and thus no idea what the downstream classification task would be?  This task is also in a stark contrast to supervised data selection for AL, which is conditioned on an initial labeled set and for the benefit of the particular supervised classification task.

We develop a \textbf{D}ata-\textbf{C}entric SSL (\textbf{DC-SSL}) approach based on the intuition that ideal instances to be labeled shall collectively have maximum diversity and coverage for downstream classification tasks, and individually have maximum information propagation utility for SSL.  It has three steps: \\
\textbf{1.} Unsupervised feature learning that maps  data into a maximally discriminative feature space. \\
\textbf{2.} Select top-$m$ representative instances for labeling per diversity, coverage, and information utility. \\
\textbf{3.} Apply an SSL method, e.g. FixMatch, on the selected labeled data and the rest unlabeled data.

Our result analysis shows several benefits over past model-centric SSL methods (\fig{teasertwo}):
\ol{itemize}{-1}{
\item {\bf On classification accuracy (\fig{acc}):}  Just by selecting the right data to label, we improve the classification accuracy of FixMatch by an astonishing 8\% and 14\% in CIFAR-10 (with 0.08\% labeled) and ImageNet-1k (with 0.2\% labeled).  These results are even better than from the practically infeasible stratified sampling which assumes perfectly balanced labeled instances.
\item {\bf On class coverage (\fig{coverage}) and data balance (\fig{balance}):} Our method covers more classes of CIFAR-10 with a much smaller labeling budget and a more balanced distribution over classes: Ours cover 9.5 out of 10 classes on average when only 20 samples are selected for labeling. The standard deviation on the number of samples per class is also greatly reduced, resulting in a more uniform distribution over classes.
\item{\bf On informativeness (\fig{representativeness}):} Our method successfully discovers most representative and diversified samples from the unlabeled data of ImageNet: The 20 selected instances tend to be located near density peaks; they are tightly connected to a large number of neighbors and spread out more uniformly in the feature space, allowing labeling information to propagate more effectively and widely.
However, random and stratified sampling pick less informative outliers far too often, resulting in low learning efficiency and sometimes even learning collapse.  These reasons underlie our better performance over stratified sampling, even though it is gauranteed to have balanced labeled data across classes.  Note that our least-20 samples have the lowest utility values, lying in sparse areas with the weakest propagation abilities.
}

Our work shows that a small compute spent on careful labeled data selection brings big annotation efficiency and model performance gain without changing the learning pipeline. Our  completely unsupervised data selection can be easily extended to other weakly supervised learning settings.
\section{Related Work}
\label{related_work}

\noindent \textbf{Active Learning} (AL) aims to select a small subset of labeled data  to achieve competitive performance over supervised learning on fully labeled data \citep{cohn1994improving, roy2001toward, bilgic2009link}.
\noindent \textbf{\textit{Conventional AL}} has three major camps \citep{settles2009active, ren2020survey}:
1) Membership query synthesis \citep{angluin1988queries} requests to generate a membership query in the form of any unlabeled instance in the input space, including the membership queries generated from scratch;
2) Stream-based selective sampling \citep{dagan1995committee, atlas1990training} selects one unlabeled instance at a time and uses informativeness to determine whether to query its label;
3) Pool-based active learning \citep{tong2001support, huang2010active, wei2015submodularity} generates queries in a greedy fashion by ranking unlabeled data.
In \noindent \textbf{\textit{Deep AL}},
Core-Set \citep{sener2017active} approaches data selection as a set cover problem and with a derived upper bound it is equivalent to the k-Center problem.
Adversarial-based approaches \citep{ducoffe2018adversarial} use adversarial samples to estimate distance from decision boundaries based on sensitivity to adversarial attacks.
Learning Loss for Active Learning \citep{yoo2019learning} makes use of a novel parametric module to predict target losses of unlabeled data and queries the instance with largest loss for label.
\textit{\textbf{Semi-supervised Active Learning}} (SSAL) combines AL with SSL. \cite{song2019combining} merges uncertainty-based metrics with MixMatch \citep{berthelot2019mixmatch}. \cite{gao2020consistency} merges consistency-based metrics with SSL.

AL/SSAL often rely on initial labeled data to learn the model and sampler, requiring multiple (e.g. 10) rounds of sequential annotation and significant modifications of existing annotation pipelines.  To our best knowledge, our work is the first \textit{unsupervised} sampling method on large-scale recognition datasets that requests annotation only \textit{once}, consistent with previous SSL pipelines.  See more comparisons in Sec.~\ref{al-comparison}.

\noindent \textbf{Self-supervised Learning} learns representations transferable to downstream tasks without annotations \citep{wu2018unsupervised, grill2020bootstrap}.  \textbf{\textit{Contrastive learning}} \citep{wu2018unsupervised, he2020momentum, chen2020simple, wang2021unsupervised} learns representations that map similar samples or different augmentations of the same instance close  and dissimilar  instances apart.
\textbf{\textit{Similarity-based}} methods \citep{grill2020bootstrap} learn representations without negative pairs by predicting the embedding of a target network with an online network.
\textbf{\textit{Feature learning with grouping}} \citep{yang2010image, xie2016unsupervised, caron2018deep, zhuang2019local, caron2020unsupervised, wang2021unsupervised} respects the natural grouping of data by exploiting clusters in the latent representation.
We study unlabeled data in such an unsupervisedly learned feature space, due to its high quality and low feature dimensions.

\textbf{Semi-supervised Learning} (SSL) integrates information from both small labeled data and large unlabeled data.
\textbf{\textit{Consistency based regularization}} \citep{sajjadi2016regularization, tarvainen2017mean, xie2020unsupervised} applies a consistency term to the final loss by imposing invariance on unlabeled data under augmentations.
\textbf{\textit{Pseudo-labeling}}, also known as self-training \citep{lee2013pseudo, berthelot2019mixmatch, berthelot2019remixmatch, sohn2020fixmatch}, relies on the model's high confidence predictions to produce pseudo-labels of unlabeled data and trains them jointly with labeled data.
FixMatch \citep{sohn2020fixmatch} integrates strong data augmentation \citep{cubuk2020randaugment}, pseudo-labels filtering \citep{liu2019deep} and temperature re-scaling to achieve the current SOTA on SSL.
It also explores training on the most representative samples ranked by \citep{carlini2019distribution}. However, \citep{carlini2019distribution} is a supervised method that requires the use of all data labels.
As a two-stage method, SimCLRv2 \citep{chen2020big} is a transfer learning method for SSL:  It applies contrastive learning on unlabeled data, followed by supervised learning on labeled data.
\textbf{\textit{Entropy-minimization}} \citep{grandvalet2005semi, lee2013pseudo, berthelot2019mixmatch} assumes that classification boundaries do not pass through high-density area of marginal distributions, enforcing confident predictions on unlabeled data.

These SSL methods choose data to label by random sampling which leads to poor performance, or by stratified sampling that selects the same number of samples per category, assuming unrealistically that labels are available on all the data.
Instead of focusing on novel models and algorithms, our method is \textit{data-centric}, focusing on choosing the right samples for most effective SSL.
\section{Data-centric Semi-supervised Learning}
\vspace{-3pt}

\label{method}
\def\figDensity#1{

    \begin{wrapfigure}{r}{0.5\textwidth}
    \centering
    \includegraphics[width=0.42\textwidth]{images/Density.pdf}
    \vspace{-6pt}
    \caption{Illustration of the graph for evaluating the local density of the purple vertex using a third-order approximation. The purple, magenta, and orange lines represent the first, second, and third level connections to the nearest $k$ neighbors ($k=4$), respectively.
    }
    \label{fig:density}
    \vspace{-30pt}
    \end{wrapfigure}
}

\def\figCombined#1{
    \begin{figure}[#1]
    \centering
    \begin{minipage}[t]{0.50\textwidth}
        \vspace{-10pt}
        \centering
        \includegraphics[width=0.95\textwidth]{images/MultiHop.pdf}\vspace{-1pt}
        \caption{
        Two-hop propagation considers the indirect connection between two vertices, and more accurately estimate the information dissemination ability of each vertex.
        The overall distance, i.e., $\text{cost}(\cdot)$, of propagating semantic information from vertex $A$ to vertices $B$, $C$ and vertex $D$ to vertices $E$, $F$ are illustrated, where the purple and magenta lines denote one-hop and two-hop propagation, respectively. Numbers shown here are distances between two vertices.

        }
        \label{fig:MultiHop}
    \end{minipage}\hfill
    \begin{minipage}[t]{0.46\textwidth}
        \vspace{-12pt}
        \centering
        \includegraphics[width=0.95\textwidth]{images/InfoProp.pdf}\vspace{-6pt}
        \caption{
        A case where local density alone gives a bad querying results.
        If we only sample three instances to be annotated from the region with the highest estimated local density, only the three samples in cluster $A$ will be queried due to their highest density. However, this may impair the ability of queries to capture multiple data distribution patterns and is less efficient in propagating semantic information to all unlabeled data.

        }
        \label{fig:InfoProp}
    \end{minipage}
    \vspace{-10pt}
    \end{figure}
}

\def\figRegularization#1{
    \begin{wrapfigure}[16]{r}{0.65\textwidth}
    \centering
    \begin{minipage}[t]{0.32\textwidth}
        \vspace{-18pt}
        \centering
        \includegraphics[width=0.99\textwidth]{images/wo_regularization.pdf}

    \end{minipage}\hfill
    \begin{minipage}[t]{0.32\textwidth}
        \vspace{-18pt}
        \centering
        \includegraphics[width=0.99\textwidth]{images/w_regularization.pdf}\vspace{-6pt}

    \end{minipage}
    \vspace{-16pt}
    \caption{Sampling with regularization (right) can produce more uniform and diversified queries. On the contrary, the queries acquired without regularization (left) are usually located in a small dense region and have high correlations, losing the overall representativeness. (better viewed in color)}
    \label{fig:Regularization}
    \end{wrapfigure}
}

Existing SSL methods are \textit{model-centric}, focusing on novel models and algorithms.  We instead explore \textit{Data-Centric} Semi-Supervised Learning (DC-SSL), focusing on selecting the most informative data to label for effective SSL.
DC-SSL has three steps: 1) Unsupervised feature learning; 2) Unsupervised sample selection for annotation; 3) Semi-supervised learning on selected labeled data and rest unlabeled data.
We also demonstrate how a large-scale model trained on general network-crawled text-image pairs accelerates our algorithm on novel datasets while improving the quality.

\vspace{-3pt}
\subsection{Problem Setup}
\vspace{-3pt}
Consider a dataset ${\mathbb{D}} = ({\bm{x}}_i, y_i)_{i=1}^n$ with $n$ images, and the labels $y_i$ are not known unless we request annotation.
Since there is no preview of $y_i$ for \textit{any} of the ${\bm{x}}_i$,
stratified sampling that results in balanced labeled data per class is impossible,
as commonly assumed in  \citep{chen2020big,berthelot2019remixmatch, sohn2020fixmatch}.
Given a limited labeling budget,
our goal is
to identify the top-$m$ representative samples ${\mathbb{A}} = ({\bm{x}}_j, y_j)_{j=1}^m$ from ${\mathbb{D}}$ to annotate, so as to improve SSL performance jointly trained on unlabeled data ${\mathbb{D}}$ and labeled data ${\mathbb{A}}$. Several popular SSL methods are utilized to verify the effectiveness of our data selection method while keeping the algorithms unchanged, since our focus is on the data rather than model.

\vspace{-3pt}
\subsection{Unsupervised Representation Learning}
\vspace{-3pt}
We apply unsupervised feature learning, contrastive learning in particular, to obtain a lower dimensional (128-$D$) representation that maximally maintains the individuality of each instance \citep{wu2018unsupervised,oord2018representation,he2020momentum,chen2020simple}.  Its quality is shown close to or even better than the supervisedly learned representations on many downstream tasks.
We learn a mapping function $f$ such that in the $f(x)$ feature space, instance $x_i$ is attracted to its augmented version $x_i'$ and repulsive to a different instance $x_j$, $j\!\neq\! i$.
We model $f$ by a convolutional neural network, mapping $x$ onto a $d$-dimensional hypersphere with $L^2$ normalization.
To make a fair comparison with previous arts \citep{sohn2020fixmatch, cai2021exponential}, we use MoCo v2 to learn representations on ImageNet with instance-centric contrastive loss. The feature spaces of CIFAR-10 images are extracted with CLD \citep{wang2021unsupervised}, which adds a instance-group contrastive loss, due to its improved feature quality.  See formulations and interpretations of MoCo v2 and CLD in Appendix \ref{appendix-representation-learning}.

\iffalse
We model $f$ by a convolutional neural network, mapping $x$ onto a $d$-dimensional hypersphere with $L^2$ normalization.
To make a fair comparison with previous arts \citep{cai2021exponential}, we use MoCo v2 \citep{chen2020improved} to learn representations on ImageNet with the instance-centric contrastive loss:\vspace{-2pt}
\begin{align}
    \begin{split}
    &C\left(f_i,f^+_i, f^-_{\neq i}\right) \!=\!
    -\log \frac%
    {\exp(<f_i,f_i^+>\!/T)}
    {\exp(<f_i,f_i^+>\!/T)+\sum\limits_{j\neq i}\exp(<f_i,f_j^->\!/T)}\\[-6pt]
    \end{split}
    \label{eqn:instance-contrast}
\end{align}
where $T$ is a regulating temperature.
Minimizing it can be viewed as maximizing mutual information (MI) lower bound between features of same instance \citep{hadsell2006dimensionality, oord2018representation}.

The feature spaces of CIFAR-10 images are extracted with CLD \citep{wang2021unsupervised}. The instance-group contrastive loss is added in symmetrical terms over views $x_i$ and $x_i'$:
\vspace{-2pt}
\begin{align}
    \begin{split}
    &L(f;T_I,T_G,\lambda) \!=\!\sum_{i} \!
      C(f_I(x_i),v_i,v_{\neq i};T_I) \!+\!
      C(f_I(x_i'),v_i,v_{\neq i};T_I) \\[-6pt]
    &\!+\!\lambda\sum_{i} \!
      C(f_G(x_i'), M_{\Gamma(i)}, M_{\neq\Gamma(i)};T_G) \!+\!
      C(f_G(x_i), M'_{\Gamma'(i)}, M'_{\Gamma'(i)};T_G)\\[-8pt]
    \end{split}
    \label{eqn:cld}
\end{align}

Cross-level discrimination of Eqn.~\ref{eqn:cld} (second term) can be understood as minimizing the cross entropy between hard clustering assignment based on $f_G(x_i)$ and soft assignment predicted from $f_G(x_i')$ in a different view, where $f_G$ ($f_I$) is instance (group) branch, and $M_{\Gamma(i)}$ denotes the cluster centroid of instance $x_i$ with a cluster id $\Gamma(i)$ \citep{wang2021unsupervised}. Empirically, we found that CLD has great feature quality on CIFAR-10 and better respects the underlying semantic structure of data.
\fi

\vspace{-3pt}
\subsection{Unsupervised Sample Selection for Annotation}
\vspace{-3pt}
We represent $({\bm{x}}_i)_{i=1}^n$ as a weighted graph $G=(V, E; W)$, where nodes are points in the feature space, and edges between nodes are attached with weights of pairwise feature similarity \citep{bondy1976graph,deo1975graph,chung1997spectral,shi2000normalized}:  $w_{ij} = 1/{\lVert f(x_i) - f(x_j) \rVert}$,
where $f(x_i)$ is the $L^2$ normalized feature of $V_i$, learned from e.g. MoCo \citep{he2020momentum}. While $w_{ij}$ is equivalent to cosine similarity $w_{ij} = \cos(V_i, V_j)$ in theory, we find that $1/{\lVert f(x_i) - f(x_j) \rVert}$ works better empirically than $\cos(V_i, V_j)$.  Please see an explanation in Appendix \ref{appendix-cosine-similarity}.

Given a labeling budget of $m$ instances, we aim to select $m$ instances that are diverse for coverage and informative for propagating label information.
We evaluate the utility of each instance in terms of information transport efficiency on a graph.

\textbf{Density estimation with $K$-NN}.
The most straightforward idea of sample selection for labeling is to select the well-connected node, which is most likely to spread semantic information to nearby nodes with the smallest cost. It corresponds to a density peak in the feature space.
We evaluate the density by $K$-Nearest Neighbor ($K$-NN) density estimation, whose key idea is to base estimation on a closest $K$ observations \citep{fix1989discriminatory, loftsgaarden1965nonparametric}. We also conduct ablation studies on other density estimation methods and find our customized $K$-NN best performing (\fig{Metrics}).

The basic $K$-nearest neighbour density ($K$-NN) estimate is constructed as follows \citep{fix1989discriminatory, loftsgaarden1965nonparametric, orava2011k}:\vspace{-5pt}
\begin{align}
    \hat{f}_{\text{KNN}} (V_i, K) = \frac{1}{n}\sum_{j=1}^K\frac{1}{A_d\cdot D(V_i, \hat{V}_j)}, \qquad \text{with } A_d=\frac{\pi^{d/2}}{\Gamma(\frac{d}{2} + 1)}
    \label{eqn:knn}
\end{align}
where $A_d$ is the volume of a unit $d$-dimensional ball, $\Gamma(x)$ the Gamma function, $R_k^d(V_i)$ the distance between $V_i$ and its $K$th nearest neighbor, $D(V_i, \hat{V}_j)$ the radius $R_k^d(V_i)$ of the ball centered at $V_i$:
\begin{align}
    B(V_i,R_k^d(V_i)) = \{\hat{V}_j: \lVert V_i-\hat{V}_j \lVert \leq R_k^d(V_i)\}, j\in \{1, ..., K\}.
    \label{eqn:ball}
\end{align}

\figCombined{t}

\textbf{Drawbacks of $K$-NN density estimation on self-supervised models.}
\textbf{1)} Only the $K$th nearest neighbor is referenced, while the relationships with all other $K\!-\!1$ points in the ball are not taken into consideration, rendering it sensitive to outliers.
\textbf{2)} Densest peaks capture representativeness in local small regions but ignores diversity in the global feature space (\fig{InfoProp}). Nearby density peaks also introduce redundancy along with deficient coverage.
\textbf{3)} Evaluating the utility of a node based on its direct neighbours ignores  indirect connections between nodes (\fig{MultiHop}). It is likely to miss vertices (false negatives) without direct connections with anchored vertices, even if such vertices are well connected to anchored vertices through other intermediate vertices.
\textbf{4)} Our unsupervisedly learned feature does not model semantics explicitly.  We propose an extended KNN estimator that can mitigate the aforementioned issues.

\textbf{Averaged distance function for robust density estimation.}  The basic $K-$NN estimator uses the distance to the $k$th nearest neighbor alone ($D(V_i, V_j)\!=\!R_k^d(V_i)$) for density estimation, which makes it highly sensitive to outliers.  We consider all $k$ neighbors instead:
\begin{align}
\hat{f}_{\text{KNN}} (V_i, k) = \frac{k}{n}\frac{1}{A_d\cdot \frac{1}{k}\sum_{j=1}^k\frac{1}{w_{ij}}}.
\end{align}
We compare it to alternative formulations and show its advantage in Sec. \ref{ablation}.

\textbf{Multi-hop propagation for measuring indirect connections between vertices.}
To model indirect connections, we formulate the utility of vertex $V_i$ with two-hop propagation:
\begin{align}
    U(V_i) = \hat{f}_{\text{KNN}} (V_i, K) + \sum_{\hat{V}_j\in B(V_i,R_k^d(V_i))}
    \cos(V_i, \hat{V}_j)\cdot\hat{f}_{\text{KNN}} (\hat{V}_j, K)

    \label{eqn:twohop}
\end{align}
Compared with a single-hop propagation for direct connections, the multi-hop propagation estimates the information dissemination ability, i.e., informativeness of each vertex, more accurately.
In \fig{MultiHop}, the one-hop cost of information propagation from $A$ to vertices ($B$, $C$) and $D$ to ($E$, $F$) is the same for the left and right graphs, although $C$ can be learned more effectively through $B$ in the left graph instead of directly from $A$.
The two-hop propagation can be naturally extended to multi-hop propagation by looking at higher order neighbors. In consideration of simplicity and efficiency, we only explore propagation of up to two-hop.

\textbf{$K$-Means clustering for diversity and coverage.} The instances to be selected for labeling shall not only be representative of its local neighbourhood, but also collectively capture the structure of the entire unlabeled data.
We perform $K$-Means clustering that partitions the $n$ instances into $m$ ($\leq\!n$) clusters ${\mathbb{S}}=\{S_1, S_2, ..., S_m\}$ in which each instance belongs to the cluster with the nearest cluster centroid $\mu$, serving as a prototype of the cluster \citep{lloyd1982least, forgy1965cluster}. $m$ is approximately equal to our annotation budget.
The objective is to find ${\mathbb{S}}$ that minimizes the within-cluster sum of squares (WCSS) \citep{kriegel2017black}:\vspace{-6pt}
\begin{align}
    \begin{split}
    &\argmin_{{\mathbb{S}}} = \sum_{i=1}^{m}\sum_{V\in S_i}\rVert V-\mu_i \rVert^2 = \argmin_{{\mathbb{S}}}\sum_{i=1}^m|S_i|\text{Var}(S_i)\\[-6pt]
    \end{split}
    \label{eqn:kmeans}
\end{align}
We then query one instance from each cluster according to their utility score evaluated according to Eqn.~\ref{eqn:twohop}. The $m$ centroids of $K$-Means clustering are initialized randomly and optimized with the EM algorithm \citep{mclachlan2007algorithm} until convergence.

\figRegularization{ht}
\textbf{Regularization with inter-cluster information passing channels.}
We observe that our instance sampler still selects samples very close to other selected samples in adjacent regions (\fig{Regularization}), especially when the densest area in a cluster is at a corner or boundary.

To avoid such repetitive sampling that reduces the total information carried in the labeled set, we apply an iterated regularization algorithm.
At each iteration, different clusters exchange information about other clusters' selected samples and gradually adjust their selections for better diversity and coverage.
Specifically, after the original sample selection, we compute a regularizer $\text{Reg}(V_i, t)$ for each sample $V_i$ based on the distance from each candidate to all members of the selected query set
$\hat{{\mathbb{V}}}^{t-1} = \{\hat{V}_1^{t-1},...,\hat{V}_m^{t-1}\}
$
at iteration $t\!-\!1$, except for the query in the same cluster $S_i$, with a tunable strength hyperparameter $\alpha$:
\begin{align}
\text{Reg}(V_i, t) = \sum_{\hat{V}_j^{t-1}\not\in S_i} \frac{1}{\rVert V_i - \hat{V}_j^{t-1} \rVert^\alpha}.
\end{align}
The regularizer is updated with an exponential moving average with momentum $m_{\text{reg}}$:
\begin{align}
\bar{\text{Reg}}(V_i, t) = m_{\text{reg}}\cdot\bar{\text{Reg}}(V_i, t-1) + (1-m_{\text{reg}})\cdot\text{Reg}(V_i, t).
\label{regularization}
\end{align}
The new samples are chosen by subtracting the regularizer for that sample multiplied by a hyperparameter $\lambda$.
$\lambda$ controls the trade-off between diversity and informativeness: A low $\lambda$ leads to samples close to each other and of low diversity; a high $\lambda$ leads to uniformly distributed samples of low informativeness.  After that, at iteration $t$, we select sample $i$ with maximum \textbf{regularized utility} $U'(V_i, t)$ in each cluster, formulated as:\vspace{-6pt}
\begin{align}
U'(V_i, t) = U(V_i) - \lambda\cdot\bar{\text{Reg}}(V_i, t).
\label{utility-reg}
\end{align}
The selection formed at the last iteration is our final choice. By adding a soft regularization, this formulation does not just push a data point selection away to a certain distance, but it considers tradeoff between diversity and representativeness and asks each selection to find another mode in the cluster which potentially brings new information if labeled, greatly improving the overall informativeness. We refer readers to Alg. \ref{alg:regularization} in appendix for pseudo-code of the regularization algorithm.

For measuring the regularizer on large datasets, calculating the distances between a sample and all queries in $\hat{{\mathbb{V}}}^{t-1}$ is no longer feasible.
Fortunately, only selected samples that are close to the current candidate make a big difference in $\text{Reg}(V_i, t)$. Therefore, we set a horizon $h$, defined as the number of nearest neighbor selections from $\hat{{\mathbb{V}}}^{t-1}$ to be considered, for large datasets. We found that $h$ ranging from $32$ to $128$ generally work well on $128$-dim feature on MoCo or CLD and $512$-dim feature on CLIP and changing $h$ does not make a difference.

\iffalse
Using this methods alone ignores the correlations between selected samples, resulting in labeled samples that cannot transfer semantic information about learned labels to a wider range of nodes in the network and redundant.

Selecting samples based on the informative of each node alone is not representative of the distribution of unlabeled datasets. On the other hand, considering only strategies that promote diversity in sampling may lead to increased labeling costs, as may be a considerable number of samples with low information content will consequently be selected.

We believe that this is because the density-based methods only considers the local density of each area, whereas the trade-off between global uniformity and local density of selected set of samples is completely ignored.
\fi

\noindent \textbf{Acceleration with general-domain multi-modal models.}
We further improve our method on two aspects: 1) We need to train a self-supervised model for each new dataset, which is time-consuming and could potentially delay data annotation in real-world applications. 2) Self-supervised methods do not model semantic information explicitly, and datasets with varying intra-class variance assume regions of various sizes and may be treated unexpectedly.  To address these issues, we resort to a large pretrained model that encodes semantics without manual human annotation.

We make use of publicly-available CLIP \citep{radford2021learning} models, a collection of models trained on Internet-crawled large-scale data with a wide general domain, and use CLIP's image model as feature extractor.
We observe that such substitution does not hurt and even helps the performance when annotation budget is large when compared to self-supervised models, demonstrating the effectiveness of making use of semantic information. Since we do not need to train models with target dataset, the whole sample selection process could complete in \textit{0.5 hours} on a commodity server with one GPU, which illustrates the possibility of our methods without delaying the schedule of human annotation or modifying the annotation pipeline to wait for self-supervised models and enables it for industry on real-world dataset collection.
More discussions on CLIP can be found in Appendix \ref{appendix-clip}.
\section{Experiments}
\label{experiment}

\def\tabImageNet#1{
    \begin{table}[#1]
    \centering
    \tablestyle{15pt}{0.85}
    \normalsize
    \tablefont{
    \begin{tabular}{l|l|l|l|l}
    \shline
    & \multicolumn{2}{c|}{Supervised Learning} & \multicolumn{2}{c}{Semi-supervised Learning} \\
    Sampling Methods &\multicolumn{1}{c}{1\%} & 0.20\%  &\multicolumn{1}{c}{1\%} & 0.20\% \\
    \shline

    Random                 & 23.5  & 6.0   & 58.8  & 34.3  \\
    DC-SSL with MoCo (Ours)   & 24.7 \color{Green}{$\uparrow$ 1.2} & 9.2 \color{Green}{$\uparrow$ 3.2} & 61.6  \color{Green}{$\uparrow$ 2.8} & \bf 48.6 \color{Green}{$\uparrow$ 14.3}  \\
    DC-SSL with CLIP (Ours)   & \bf 27.3 \color{Green}{$\uparrow$ 3.8} & \bf 9.7 \color{Green}{$\uparrow$ 3.7} & \bf 62.2 \color{Green}{$\uparrow$ 3.4} & 47.5 \color{Green}{$\uparrow$ 13.2} \\
    \textcolor{gray}{Stratified $\dagger$} & \textcolor{gray}{22.9}  & \textcolor{gray}{6.3} & \textcolor{gray}{60.9$^*$} & \textcolor{gray}{41.1} \\
    \shline
    \end{tabular} \vspace{-8pt}
    }
    \caption{ImageNet experiments with fully supervised training and SSL algorithm FixMatch \citep{sohn2020fixmatch} and Exponential Moving Average Normalization (EMAN) \citep{cai2021exponential}. * indicates results reported in EMAN. Note that $\dagger$ is not a fair comparison, as explained earlier.}
    \label{tab:imagenet}
    \vspace{-10pt}
    \end{table}
}

\def\tabCifar#1{
    \begin{table}[#1]
    \centering
    \tablestyle{14.5pt}{0.85}
    \normalsize
    \tablefontsmall{
    \begin{tabular}{l|l|l|c}
    \shline
                           & \multicolumn{2}{c|}{Accuracy (\%)} &\\
    Sample Selection                &  \multicolumn{1}{c}{Acc. (Last Epoch)}   & \multicolumn{1}{c|}{Acc. (Best Epoch)} & Num. of Labels \\
    \shline
    Random          & 82.94 $\pm$ 9.77 & 86.67 $\pm$ 4.71 & 40\\
    DC-SSL (Ours)      & \textbf{91.10} $\pm$ 0.95 \bf\small\color{Green}{$\uparrow$ 8.16}
                     & \textbf{91.27} $\pm$ 0.95 \bf\small\color{Green}{$\uparrow$ 4.60} & 40 \\
    \textcolor{gray}{Stratified $\dagger$*}         & \textcolor{gray}{88.61 $\pm$ 3.35} & - & 40 \\
    \hline
    Random          & 88.72 $\pm$ 4.61 & 91.32 $\pm$ 2.05 & 100\\
    DC-SSL (Ours)      & \textbf{93.22} $\pm$ 0.31 \bf\small\color{Green}{$\uparrow$ 4.50}
                     & \textbf{93.55} $\pm$ 0.33 \bf\small\color{Green}{$\uparrow$ 2.23} & 100 \\
    \textcolor{gray}{Stratified $\dagger$}         & \textcolor{gray}{90.19 $\pm$ 2.24} & \textcolor{gray}{91.62 $\pm$ 1.41} & 100 \\
    \hline
    Random          & 93.28 $\pm$ 0.04      &   93.69 $\pm$ 0.05   & 250 \\
    DC-SSL (Ours)      & \textbf{93.63} $\pm$ 0.14 \bf\small\color{Green}{$\uparrow$ 0.35}
                     &   \textbf{93.92} $\pm$ 0.15 \bf\small\color{Green}{$\uparrow$ 0.23} & 250 \\
    \textcolor{gray}{Stratified $\dagger$*}         & \textcolor{gray}{94.93 $\pm$ 0.33} &   -   & 250 \\
    \shline
    \end{tabular}\vspace{-10pt}
    }
    \caption{CIFAR-10 experiments with pseudo-label-based SSL method FixMatch \citep{sohn2020fixmatch}, with 3 different folds.
    $\dagger$: not a fair comparison with us because it assumes balanced labeled data available and leaks information about ground truth labels. * indicates results reported in FixMatch.
    Note that the 0.35\% improvement in 250 label task is \textit{not marginal}, since 250 labels to 4000 labels ($16\times$) only leads to around 0.7\% improvement as reported in FixMatch.}
    \label{tab:cifar-10-pseudo-label-based}
    \vspace{-5pt}
    \end{table}
}

\def\tabCifarWide#1{
    \begin{table}[#1]
    \centering
    \tablestyle{3pt}{0.9}
    \normalsize
    \vspace{-4pt}
    \tablefontsmall{
    \begin{tabular}{l|l|l|l}
    \shline
    & \multicolumn{3}{c}{Accuracy (\%) of Last Epoch (Best Epoch)} \\
    Sample Selection &  \multicolumn{1}{c}{40 labels}   & \multicolumn{1}{c}{100 labels} & \multicolumn{1}{c}{250 labels} \\
    \shline
    Random           & 82.94$\pm$9.77 (86.67$\pm$4.71) & 88.72$\pm$4.61 (91.32$\pm$2.05) & 93.28$\pm$0.04 (93.69$\pm$0.05) \\
    DC-SSL (Ours)    & \textbf{91.10}$\pm$0.95 (\textbf{91.27}$\pm$0.95) & \textbf{93.22}$\pm$0.31 (\textbf{93.55}$\pm$0.33) & \textbf{93.63}$\pm$0.14 (\textbf{93.92}$\pm$0.15) \\
    $\Delta$ (DC-SSL - Random) & \bf\color{Green}{$\uparrow$ 8.16} (\bf\color{Green}{$\uparrow$ 4.60}) & \bf\color{Green}{$\uparrow$ 4.50} (\bf\color{Green}{$\uparrow$ 2.23}) & \bf\color{Green}{$\uparrow$ 0.35} (\bf\color{Green}{$\uparrow$ 0.23}) \\
    \textcolor{gray}{Stratified $\dagger$*} & \textcolor{gray}{88.61$\pm$3.35} & \textcolor{gray}{90.19$\pm$2.24} & \textcolor{gray}{94.93 $\pm$ 0.33} \\
    \shline
    \end{tabular}\vspace{-2pt}
    }
    \caption{CIFAR-10 experiments with pseudo-label-based SSL method FixMatch \citep{sohn2020fixmatch}, with 3 different folds.
    $\dagger$: not a fair comparison with us because it assumes balanced labeled data available and leaks information about ground truth labels. * indicates results reported in FixMatch.
    Note that the 0.35\% improvement in 250 label task is \textit{not marginal}, since 250 labels to 4000 labels ($16\times$) only leads to around 0.7\% improvement as reported in FixMatch.}
    \label{tab:cifar-10-pseudo-label-based}

    \end{table}
}

\def\tabCifarActiveComparison#1{
    \begin{table}[#1]
    \centering
    \tablestyle{9.9pt}{0.87}
    \normalsize

    \tablefont{
    \begin{tabular}{l|l|c|c}
    \shline
    Sample Selection                &  Accuracy (\%) & \# Initial Labels* & Label Budget \\
    \shline
    Core-Set \citep{sener2017active} $\dagger$        & 48.33 $\pm$ 0.49 & 100 & 150 \\
    Core-Set $\dagger$      & 50.96 $\pm$ 0.45 & 100 & 200 \\
    CBSSAL \citep{gao2020consistency}       & 87.57 $\pm$ 0.31 & 100 & 150 \\
    \hline
    DC-SSL (Ours)      & 91.10 $\pm$ 0.95 ({\color{Green}{$\uparrow$ 3.53}}) & 0 & 40 \\
    DC-SSL (Ours)      & \textbf{93.22} $\pm$ 0.31 ({\bf\color{Green}{$\uparrow$ 5.65}}) & 0 & 100 \\
    \shline
    Core-Set $\dagger$        & 53.77 $\pm$ 0.49 & 100 & 250 \\
    Core-Set + SSL $\dagger$  & 88.75 $\pm$ 0.42 & 100 & 250 \\
    CBSSAL                    & 90.23 $\pm$ 0.39 & 100 & 250 \\
    MMA \citep{song2019combining}           & 91.69 $\pm$ 0.52                   & 250 & 500 \\
    MMA + k-Means & 91.46 $\pm$ 0.38             & 250 & 500 \\
    \hline
    DC-SSL (Ours)      & \textbf{93.63} $\pm$ 0.14 ({\bf\color{Green}{$\uparrow$ 2.17}}) & 0 & 250 \\
    \shline
    \end{tabular} \vspace{-4pt}
    }
    \caption{Comparison with active learning methods. Note that although MMA also experimented with k-Means, it does not benefit their performance. $\dagger$: from \cite{gao2020consistency}. *: We give all queries in one round and do not need initial labels. CBSSAL considers interacting with FixMatch for better performance, however DC-SSL can still yield better performance without any initial labels.}
    \label{tab:cifar-10-comparison-active-learning}
    \vspace{-2pt}
    \end{table}
}

\def\tabCifarSemi#1{
    \begin{table}[#1]
    \centering
    \tablestyle{8.7pt}{0.85}
    \normalsize

    \tablefont{
    \begin{tabular}{l|l|l|l|l}
    \shline

    Sampling Methods & \multicolumn{1}{c}{40 Labels} & \multicolumn{1}{c}{100 Labels} & \multicolumn{1}{c}{150 Labels} & \multicolumn{1}{c}{250 Labels} \\
    \shline
    Random & 60.8 $\pm$ 3.2 & 73.7 $\pm$ 2.5 & 76.2 $\pm$ 2.2 & 79.4 $\pm$ 1.7 \\
    DC-SSL (Ours) & \textbf{76.3} $\pm$ 1.6 \textbf{\color{Green}{$\uparrow$ 15.5}} &
     \textbf{79.0} $\pm$ 0.3 \textbf{\color{Green}{$\uparrow$ 5.3}} & \textbf{80.8} $\pm$ 0.5 \textbf{\color{Green}{$\uparrow$ 4.6}} & \textbf{82.1} $\pm$ 0.3 \textbf{\color{Green}{$\uparrow$ 2.7}} \\
    \textcolor{gray}{Stratified $\dagger$} & \textcolor{gray}{66.5 $\pm$ 1.6} &
     \textcolor{gray}{74.5 $\pm$ 0.8} & \textcolor{gray}{78.3 $\pm$ 1.1} & \textcolor{gray}{80.4 $\pm$ 0.8} \\
    \shline
    \end{tabular}\vspace{-8pt}
    }
    \caption{CIFAR-10 experiments with transfer-learning-based SSL method,

    as proposed in SimCLRv2 \citep{chen2020big}, with the mean and std of 5 different folds and 2 runs in each fold. $\dagger$: assumes prior label information and thus not a fair comparison.}
    \label{tab:cifar-10-simclr-style}
    \vspace{-8pt}
    \end{table}
}

\def\tabAblationCifar#1{
    \begin{table}[#1]
    \centering
    \begin{tabular}{l|l|l}
    \shline
    Methods          & FixMatch       & Transfer Learning \\ \hline
    Random           & 82.9 & 60.8 \\
    First Order      & 85.8 & 72.7 \\
    + Regularization & 90.4 & 76.0 \\
    + Second Order   & \textbf{90.7} & \textbf{76.3} \\
    \shline
    \end{tabular}\vspace{-8pt}
    \caption{Ablation study experiments on CIFAR10 with 40 samples. In both FixMatch and Transfer Learning settings, applying only first order label selection gives a large improvements (2.9\% and 11.9\%, respectively), and regularization as well as second order together give an additional improvement of 5.8\% on FixMatch and 3.6\% on Transfer Learning.}
    \label{tab:cifar-10-ablation}
    \end{table}
}

\def\figAblationCifar#1{
    \begin{figure}[#1]
    \centering
    \includegraphics[width=0.45\textwidth]{images/ablation_plot.pdf}
    \caption{Illustration of our methods with different ablations on CIFAR10 with 40 samples. In both Transfer Learning and FixMatch settings, applying only first order label selection gives a large improvements (11.9\% and 2.9\%, respectively), and regularization as well as second order together give an additional improvement of 3.6\% on Transfer Learning and 4.3\% on FixMatch.}
    \vspace{-5pt}
    \label{fig:ablation}
    \end{figure}
}

\def\figHyperparam#1{
    \begin{figure}[#1]
    \centering
    \includegraphics[width=0.45\textwidth]{images/hyperparam.pdf}
    \caption{Illustration of sampling with different levels of regularization parameter $w$. All values are averaged with 5 different seeds and mean accuracy is performed on transfer learning approach and is additionally averaged with 2 runs for each seed. $w$ balances representative and uniformity across the feature space. Larger $w$ indicate more uniform choices but potentially less representative samples, or vice versa.}
    \vspace{-5pt}
    \label{fig:hyperparam}
    \end{figure}
}

\def\figMetrics#1{
    \begin{figure}[#1]
    \centering
    \includegraphics[width=0.45\textwidth]{images/metrics.pdf}
    \caption{Illustration of the behavior of different metrics. \textit{kNN Max} refers to kNN density estimation method, which takes the max distance of all $k$-nearest-neighbors. \textit{Ball Counts} counts neighbors within a radius. First Order is our method, which takes the inverse of mean neighbor distance as the score. Reg is regularization and second order adds another term of propagation as described above. 1/dist means replacing the cosine similarity with 1/dist term, which illustrates the importance of using a similarity metric to connect the first order and the second order term.}
    \vspace{-5pt}
    \label{fig:Metrics}
    \end{figure}
}

\def\figAblationMetrics#1{
    \begin{figure}[#1]
    \centering
    \begin{minipage}[t]{0.45\textwidth}
        \vspace{0pt}
        \centering
        \includegraphics[width=0.92\textwidth]{images/ablation_plot.pdf}\vspace{-4pt}
        \caption{Illustration of our methods with different ablations on CIFAR-10 with 40 samples. In both SSL methods, applying 1-hop propagation gives a large improvements (11.9\% and 2.9\%, respectively), and regularization as well as 2-hop propagation together give an additional improvement of 3.6\% on Transfer Learning and 5.3\% on FixMatch.}
        \label{fig:ablation}

    \end{minipage}\hfill
    \begin{minipage}[t]{0.52\textwidth}
        \vspace{-5pt}
        \centering
        \includegraphics[width=0.78\textwidth]{images/metrics.pdf}\vspace{-8pt}
        \caption{Illustration of the behavior of different utility estimations.
            \textit{$K$NN Max} refers to vanilla $K$-NN density estimation that takes the max distance of $K$ nearest-neighbors.
            \textit{Ball Counts} counts neighbors within a radius.
            Our methods (with *):
            \textit{1-hop} is our method without regularization or second order term. \textit{1-hop+Reg} adds regularization and \textit{2-hop+Reg} adds a second order propagation.

        }
        \label{fig:Metrics}
        \vspace{-10pt}
    \end{minipage}
    \end{figure}
}

\def\figHyperparamCombined#1{
    \begin{figure}[#1]
    \centering
    \vspace{-1pt}
    \centering
    \includegraphics[width=0.96\textwidth]{images/hyperparam.pdf}\vspace{-5pt}
    \caption{Effect of different hyperparameters, $\lambda$ (Fig.a,b,c) and $K$ (Fig.d). $\lambda$ balances representative and uniformity across the feature space. Larger $\lambda$ indicate more uniform choices but potentially less representative samples, or vice versa. Larger $K$ indicates that we are taking more neighbors into account when estimating the representativeness, which reduces variance but may consider non-relevant samples as being represented. Due to this trade-off, considering more neighbors usually leads to better representativeness estimates, until $K$ is greater than its optimal choice, 400.}
    \vspace{-7pt}
    \label{fig:hyperparam}
    \end{figure}
}

In this section, we perform a comprehensive evaluation on the proposed Data-Centric Semi-Supervised Learning (DC-SSL) integrated with two mainstream SSL approaches: pseudo-label-based FixMatch \citep{sohn2020fixmatch} and approach based on transfer learning \citep{chen2020big}.

\subsection{CIFAR-10: Extremely Low-shot Settings}
\tabCifarWide{h}
\vspace{-1pt}
\tabCifarSemi{t}

\noindent\textbf{Setup.} We evaluate both approaches on low-shot settings from 40 samples to 250 samples in total (4 to 25 shots per class on average). We also re-evaluate FixMatch with random sampling as a fair comparison. Since our work is data-centric, we follow the same training recipe unless stated. Detailed setup is in Appendix \ref{appendix-setup-cifar}.

\noindent\textbf{Main Results.}  \noindent\textbf{\textit{Pseudo-label-based SSL.}}
Three charming properties of DC-SSL are observed in the CIFAR-10 experiments: \textit{efficiency, stability and counter-collapsing}.
In 40/100 labels task, collapsing happens since outliers could mislead the network into classifying a range of samples to a wrong class, which further generates wrong pseudo-labels and misleads the classifier.
Our method reduces this effect by selecting labels that are both representative and diverse, shown by reducing the difference between best and last epoch from 3.73\% to negligible in 40 samples task. In 250 samples task, although FixMatch already performs well, we are able to push the limit further by a small amount.
In 40 samples task, two of first 5 seeds of random selection has labels from fewer than 10 classes and to better understand the effect of our methods, we skip these two seeds in random selection, whereas it does not occur in our method, even when we sample 100 times with 100 seeds.
\noindent\textbf{\textit{Transfer-learning-based SSL.}}
Since transfer learning only considers labeled samples in fine-tuning stage, the quality of the trained classifier greatly depends on selected samples' quality when compared to FixMatch. Table.~\ref{tab:cifar-10-simclr-style} shows that our improvement is prominent especially when the number of selected samples is low. In 40 (250) labels case, we are able to achieve a 15.5\% (2.7\%) improvement.

\tabImageNet{t}
\subsection{ImageNet}

\noindent\textbf{{Setup.}} To further evaluate the effectiveness of our method on large-scale datasets with more classes. We perform evaluation of our method on ImageNet \citep{ILSVRC15} with about 1.28M images and 1k classes. We perform supervised learning (SL) and SSL on both 1\% labeled data (12820 samples) and 0.2\% (2911 samples). More detailed setup on ImageNet are in Appendix \ref{appendix-setup-imagenet}.

\noindent\textbf{{Main Results.}} As summarized in Table \ref{tab:imagenet}, samples selected from both  MoCo and CLIP models boost the performance of SL and SSL. In 1\% case, DC-SSL provides 1.2\% to 3.8\% (2.8\% to 3.4\%) gains in SL (SSL) setting. What is more interesting is the 0.2\% setting, where our method leads to an improvement ranging from 13.2\% to 14.3\%.
We found that samples selected from MoCo performs 1.1\% better than samples from CLIP in the FixMatch setting. This is, in part, due to mismatch between parameter initialization (MoCo) and the feature space used for the sampling process (CLIP).
However, for 1\% case, we find that this dependency disappears and samples with CLIP perform 0.6\% better than MoCo counterparts, which demonstrates the benefits of using a model with explicit semantic information during pretraining with sufficient general knowledge.

\textbf{Universality in Results with CLIP:} Since CLIP \textit{does not use ImageNet samples} in training and the downstream SSL task \textit{has never seen the CLIP model} in training either, we stress the importance of \textit{universality}: it means that 1) when a new dataset is collected, we could use a general model with knowledge of the target domain to select samples without waiting for self-supervised learning (see Sec.~\ref{run-time}); 2) different from active learning, where sample selection process is strictly coupled with training, our annotated dataset works universally rather than only with the model used to select it.

\subsection{Comparison with Active Learning and SSAL Methods}
\label{al-comparison}
\tabCifarActiveComparison{t}
Following previous deep AL/SSAL works, we mainly use CIFAR-10 for comparison. Our method is under disadvantage in comparison because we request annotation only once, but we still outperform previous AL/SSAL algorithms with higher label efficiency.
We first compare with AL. Since AL only utilizes labeled samples, even recent AL algorithm \cite{cho2021mcdal} requires at least 10,000 samples to achieve 90\% accuracy (250$\times$ more than ours). For fair comparison, we attempt to allocate 20 samples for initial learning, use AL to select another 20 samples, and utilize the 40 samples in SSL. After implementation of \cite{yoo2019learning}, we found that since $20$ is smaller than batch size $128$, AL model has about 20\% accuracy given 20 initial labeled samples and thus could not serve as a useful comparison.
We then compare with SSAL. Similar to our work, SSAL \citep{gao2020consistency, song2019combining} uses labeled samples with SSL methods. However, it still requires initial annotation from random sampling. Since random sampling is not efficient on low-shot selection (\fig{teasertwo} (b)), to cover all classes in initial random sampling, such methods typically require much more samples than DC-SSL as in Table \ref{tab:cifar-10-comparison-active-learning}. In contrast, our method could select samples efficiently even in low-shot. We believe that it is possible to further improve our work if we use DC-SSL for initial selection in SSAL and we leave it to future works due to the complexity of both algorithms.

\subsection{Ablation Study and Analysis}
\label{ablation}

\figAblationMetrics{ht}

\textbf{Component Analysis.} To systematically investigate the contributions of each component we propose, we evaluate our method with an extensive ablation study. We adhere to previous settings unless stated. Illustrated in \fig{ablation}, applying the vanilla version of our method based on first order scores achieves large improvements on both SSL methods, especially transfer learning approach, where the accuracy rises by 11.9\%. Adding regularization gives another 4.6\% and 3.3\% by contributing to sample diversity. In the end, 2-hop propagation leads to a marginal of 1.2\% and 0.3\% improvement on both methods. These contributions together lead to our final 8.2\% and 15.5\% improvements.

\textbf{Formulation Analysis.}
We also systematically evaluated variants of our formulations used in representativeness estimation, as presented in \fig{Metrics}, by comparing different ablations of our methods to other metrics. We compare against two common metrics: \textit{KNN Max} (vanilla $K$-NN)
and \textit{Ball Counts}, which uses neighbor counts within a radius. Although all three methods are much better than random selection, our metric leads to the best outcome.
More ablation studies on hyperparameters, such as weight of regularizer and number of neighbors, can be found in Appendix \ref{appendix-hyperparam}.

\textbf{Run Time Analysis.}
\label{run-time}
We only add a negligible delay to label selection compared to SSL methods.
On ImageNet, we introduce only less than 1 GPU hour overhead when compared to about 2300 GPU hours with the original FixMatch pipeline, which is in turn, much less than the time for labelling 1.28 million images with 1k categories. With CLIP, the data selection stage could start as soon as data is available, which enables informative labeling with very low delay. Compared to active learning which requires up to 10 rounds of sequential human annotations, we only request annotation \textit{once}, consistent to SSL pipelines. More discussions are in Appendix \ref{appendix-run-time}.

\section{Summary}
\label{summary}
Existing SSL methods are model-centric, focusing on models and algorithms that integrate both labeled and unlabeled data. Instead, our DC-SSL is the first work that turns to labeled data selection in SSL in an unsupervised way.  By simply optimizing what is fed into a model, we demonstrate significant gains on annotation efficiency, model stability and accuracy on all experimented benchmarks.  Our work is also in a stark contrast to supervised data selection for active learning.  Our completely unsupervised data selection can be easily extended to other weakly supervised learning settings.

\subsection*{Reproducibility Statement}
In order to make our results reproducible, we listed all used parameters and changes we applied for training in the Section~\ref{appendix-setup-cifar} and Section~\ref{appendix-setup-imagenet} of Appendix and Section 4.1, 4.2 and 4.3 of Experiment, and cited the corresponding papers on which our method is build on which provide publicly available Github repositories themselves. We are committed to reproducible results reported in the paper and public code release.

\section{Appendix}
\def\algRegularization#1{
\begin{algorithm}[#1]
\caption{The iterative regularization algorithm}\label{alg:regularization}
\begin{algorithmic}
\REQUIRE
\STATE $\{U(V_i)| V_i \in {\mathbb{V}}\}$: The unregularized utility for each vertex $V_i$
\STATE $\lambda$: weight for applying regularization
\STATE $m_{\text{reg}}$: momentum in exponential moving average \\
\STATE $l$: the number of iterations

\hspace*{-\algorithmicindent}\textbf{Procedure:} \\
\STATE {$\bar{\text{Reg}}(V_i, 0) \gets 0$, $\forall V_i \in {\mathbb{V}}$}
\STATE $\hat{{\mathbb{V}}}^0 \gets $ samples with largest $U(V_i)$ in each cluster
\FOR{$t=1$ \TO $l$}
    \FORALL{$V_i \in {\mathbb{V}}$}
        \STATE $\text{Reg}(V_i, t) \gets \sum_{\hat{V}_j^{t-1}\not\in S_i} \frac{1}{\rVert V_i - \hat{V}_j^{t-1} \rVert^\alpha}$
        \STATE {$\bar{\text{Reg}}(V_i, t) \gets m_{\text{reg}}\cdot\bar{\text{Reg}}(V_i, t-1) + (1-m_{\text{reg}})\cdot\text{Reg}(V_i, t)$}
        \STATE $U'(V_i, t) \gets U(V_i) - \lambda\cdot\bar{\text{Reg}}(V_i, t)$
    \ENDFOR
    \STATE $\hat{{\mathbb{V}}}^t \gets $ samples with largest $U'(V_i, t)$ in each cluster
\ENDFOR

\RETURN $\hat{{\mathbb{V}}}^{l}$
\end{algorithmic}
\end{algorithm}
}

\algRegularization{ht}
\subsection{Detailed Description for Regularization Algorithm}
\label{appendix-regularization}
We summarized the regularization algorithm in pseudo-code in Alg. \ref{alg:regularization}. In Alg. \ref{alg:regularization}, we first obtain $\hat{{\mathbb{V}}}^0$, the selection without regularization, and set the moving average regularizer $\hat{\text{Reg}}(V_i, 0)$ to 0 for every $V_i \in {\mathbb{V}}$; then in each iteration, we update $\hat{\text{Reg}}(V_i, t)$ with moving average from a closeness measurement to other previously selected samples, where $t$ is the index of current iteration. We re-select samples according to regularized utility at the end of each iteration, with $\lambda$ being a balancing factor. In the end, the selection from the last iteration is returned.

\subsection{Overview on Unsupervised Representation Learning}
\label{appendix-representation-learning}
In self-supervised learning stage, we aim to learn a mapping function $f$ such that in the $f(x)$ feature space, the positive instance $x_i'$ is attracted to instance $x_i$, meanwhile, the negative instance $x_j$ (with $j\!\neq\! i$) is repelled, and we model $f$ by a convolutional neural network, mapping $x$ onto a $d$-dimensional hypersphere with $L^2$ normalization.
To make a fair comparison with previous arts \citep{cai2021exponential}, we use MoCo v2 \citep{chen2020improved} to learn representations on ImageNet with the instance-centric contrastive loss:\vspace{-2pt}
\begin{align}
    \begin{split}
    &C\left(f_i,f^+_i, f^-_{\neq i}\right) \!=\!
    -\log \frac%
    {\exp(<f_i,f_i^+>\!/T)}
    {\exp(<f_i,f_i^+>\!/T)+\sum\limits_{j\neq i}\exp(<f_i,f_j^->\!/T)}\\[-6pt]
    \end{split}
    \label{eqn:instance-contrast}
\end{align}
where $T$ is a regulating temperature.
Minimizing it can be viewed as maximizing the mutual information (MI) lower bound between the features of the same instance \citep{hadsell2006dimensionality, oord2018representation}.
For experiments on ImageNet, the MoCo model pre-trained for 800 epochs is used for initializing SSL model, as in \citep{cai2021exponential}.

The feature spaces of CIFAR-10 data we work on are extracted with CLD \citep{wang2021unsupervised}. The instance-group contrastive loss is added in symmetrical terms over views $x_i$ and  $x_i'$:
\vspace{-2pt}
\begin{align}
    \begin{split}
    &L(f;T_I,T_G,\lambda) \!=\!\sum_{i} \!
      C(f_I(x_i),v_i,v_{\neq i};T_I) \!+\!
      C(f_I(x_i'),v_i,v_{\neq i};T_I) \\[-6pt]
    &\!+\!\lambda\sum_{i} \!
      C(f_G(x_i'), M_{\Gamma(i)}, M_{\neq\Gamma(i)};T_G) \!+\!
      C(f_G(x_i), M'_{\Gamma'(i)}, M'_{\Gamma'(i)};T_G)\\[-8pt]
    \end{split}
    \label{eqn:cld}
\end{align}

Cross-level discrimination of Eqn.~\ref{eqn:cld} (second term) can be understood as minimizing the cross entropy between hard clustering assignment based on $f_G(x_i)$ and soft assignment predicted from $f_G(x_i')$ in a different view, where $f_G$ ($f_I$) is instance (group) branch, and $M_{\Gamma(i)}$ denotes the cluster centroid of instance $x_i$ with a cluster id $\Gamma(i)$ \citep{wang2021unsupervised}. Empirically, we found that CLD has great feature quality on CIFAR-10 and better respects the underlying semantic structure of data. To be consistent with original FixMatch settings, our semi-supervised learner on CIFAR-10 is trained from scratch, without using pretrained weights.

\subsection{Using Euclidean Distance or Cosine Similarity?}
\label{appendix-cosine-similarity}
Because the features of all instances are projected to a unit hypersphere with L2 normalization, theoretically, maximizing the cosine similarity between two nodes is equivalent to maximizing the inverse of Euclidean distance between two nodes: $\argmax_{i, j}(\lVert f(x_i) - f(x_j) \rVert_2)^{-1} = \argmax_{i, j}(2 - 2\cos(f(x_i), f(x_j)))^{-1} = \argmax_{i, j}(\cos(f(x_i), f(x_j)))$. However, empirically, using maximizing the inverse of Euclidean distance $1/d(\cdot)$ as the objective function performs better than maximizing the cosine similarity $\cos(x)$. The reason is that, when two nodes are very close to each others, $1/d(\cdot)$ is more sensitive to the change of its Euclidean distance, whereas $\cos(\cdot)$ tends to be saturated. Therefore, the function $1/d(\cdot)$ has the desired property of non-saturating and can better focus on the distance difference with closest neighbors.

\subsection{General-domain Multi-modal Models}
\label{appendix-clip}
Although our method works well in both small and large scale datasets, as we will demonstrate in Section \ref{experiment}, there are still two interesting aspects that we would like to explore. First, in our approach, self-supervised models need to be re-trained for each new dataset, which is time-consuming and could potentially delay the schedule for data annotation in real-world industry. In addition, unsupervised models do not model semantic information explicitly, which may lead to confusion that could potentially be mitigated (e.g. datasets with varying intra-class variance will take regions of different sizes and may be treated differently in an unexpected way).

To address these issues, we put our focus on a large pretrained model that encodes semantic information. However, a large fully-annotated dataset, which is required to train a large supervised model, is very costly to obtain, and labelling the target dataset to train a supervised model defeats our purpose of requiring only a small amount of annotation on the target dataset.
Fortunately, the availability of large-scale text-image pairs online makes it possible to train a large-scale model that encodes images in the general domain with semantic information. In this paper, we make use of publicly-available CLIP \citep{radford2021learning} models, a large-scale collection of models trained on Internet-crawled data with a wide general domain and use CLIP's image model as feature extractor.

As showed in Section \ref{experiment}, using models trained on multi-modal datasets answers these two issues. Even through CLIP is never trained on our target dataset, nor does the categories in its training set exactly matches the dataset we are using, using it to select does not degrade our performance of sample selection and labeling pipeline. This indicates that the effectiveness of our label selection does not necessarily depend whether same pretrained model is used in the downstream. In addition, we observe that such substitution even helps with a slightly larger annotation budget, demonstrating the effectiveness of making use of semantic information. Since we only perform inference on the CLIP model, the whole sample selection process could complete in \textit{0.5 hours} on a commodity server using one GPU, indicating the possibility of our methods without delaying the schedule of human annotation or modifying the annotation pipeline and enables it to be used by industry on real-world dataset collection.

Note that although CLIP supports zero-shot inference by using text input (e.g. class names) to generate weights for classifier, it is not always possible to define a class with names or even know all the classes beforehand. Since we only make use of the image part of the CLIP model, we do not make use of prior text information (e.g. class descriptions) that are sometimes available in the real world. We leave better integration of our methods and zero-shot multi-modal models to future work.

\subsection{Hyperparameter Analysis}
\label{appendix-hyperparam}
We focus on two most important hyperparameter here: $\lambda$, the weight for regularization, and $k$, the number of neighbors we use for $k$NN. For hyperparam $\lambda$, we evaluated label selections with different $\lambda$ values used in regularization. In this series of experiments, we select $\lambda \in \{0, 0.1, 0.5, 1.0, 3.5, 7.0\}$, where 0 indicates no regularization and larger $\lambda$ indicates a stronger regularization. We then evaluate the mean accuracy from 10 runs, the standard deviation of classes, the difference with selection without regularization (i.e., $\lambda=0$), and mean first order metric. We observe that as $\lambda$ gets larger, we select more different samples compared to the case without regularization, and instance standard deviation gets lower since we are sampling more uniformly across the space. However, as a trade-off, we could not sample from area which has as high density as before because selecting samples from that area leads to selections that are close to each other, leading to high penalty. Here, uniformity and representativeness show a trade-off and the optimal choice is to balance each other at $\lambda=0.5$, which is demonstrated by the mean accuracy.
For hyperparam $k$, we find that although initially adding $k$ contributes to a better representation estimation by considering more neighbors, adding too much to $k$ makes the selection algorithm lose focus. We find that $k$ around $400$ is optimal for our scenario.

\figHyperparamCombined{t}

\subsection{Run Time Discussion}
\label{appendix-run-time}
CLD only takes about 4 hours to train on CIFAR-10 on a single GPU and sample selection with our method takes less than 10 minutes on CLD with one GPU. This takes significantly less GPU-time than FixMatch (120 GPU hours with 4 GPUs), which is, in turn, much less than the time for labelling the whole dataset of 50, 000 samples. On ImageNet, MoCo takes about 12 days with 8 GPUs to achieve 800 epochs \citep{he2020momentum}, our algorithm takes about an hour on one GPU to select samples for both 1\% and 0.2\% labels and in the end, FixMatch takes another 20 hours on 4 GPUs to train. Although it sounds like we are using a lot of compute time just to train a self-supervised learning model for selecting what samples to annotate, the fact is that FixMatch requires a self-supervised pretrained checkpoint to work well when the number of labeled samples is low, as demonstrated in \cite{cai2021exponential}. The only compute overhead introduced is the sample selection process, which is \textit{negligible} when compared to the other two stages. In addition, shown in our experiments, CLIP, as a model trained on a general and diverse image-text dataset, could also be used to select samples with comparable and sometimes even better samples to label, which indicates that the self-supervised training stage is not required in our method for sample selection.

\subsection{Experiment Setup For CIFAR-10}
\label{appendix-setup-cifar}
\noindent\textbf{Setup for FixMatch.}
To maintain consistency with the original FixMatch \citep{sohn2020fixmatch}, we evaluate FixMatch trained on CIFAR-10 with $2^{20}$ steps in total. To illustrate the ability of our algorithm to select informative samples, we evaluate both approaches on an extremely-low setting from 40 samples to 250 samples in total (4 shots to 25 shots per class on average).
Since the original FixMatch is evaluated with stratified sampling on CIFAR-10, we also retrain FixMatch with random sampling with the same number of samples in total as a fair comparison.
Unless otherwise stated, we train FixMatch with a learning rate of 0.03, and weight decay $10^{-3}$ on 4 Nvidia RTX 2080 Ti GPUs with batch size 64 for labeled samples and with $2^{20}$ steps in total. All experiments are conducted with same training and evaluation recipe for fair comparisons.

\noindent\textbf{Setup For Transfer Learning.}
In addition to pseudo-label based semi-supervised learning methods,
we also evaluate our algorithm on two-stage SSL based on transfer learning \citep{chen2020big} by fine-tuning the linear layer of a ResNet-18 pretrained with self-supervised learning algorithm CLD \citep{wang2021unsupervised}. Specifically, we fine-tune the linear layer on a ResNet-18 trained with CLD \citep{wang2021unsupervised}.
Since it is easy for the network to overfit the few-shot labeled samples, we freeze the backbone and fine-tune only the linear layer.
We use SGD with learning rate $0.01$, momentum $0.9$, and weight decay $10^{-4}$ for 5 epochs because longer training time will lead to overfitting.

\subsection{Experiment Setup for ImageNet}
\label{appendix-setup-imagenet}
To evaluate the effectiveness of our method on large-scale datasets with more classes, we perform evaluation of our method on ImageNet \citep{ILSVRC15} with approximately 1 million images and 1000 classes. We perform supervised learning and semi-supervised learning on both 1\% labeled data (12820 samples) and the extremely data-scarce setting 0.2\% (2911 samples). For each of these settings, we use either a MoCo-pretrained model with Exponential Moving Average Normalization (EMAN) \citep{cai2021exponential} or a CLIP ViT/16 model \citep{dosovitskiy2020image} to select samples to annotate. For ImageNet 1\%, we run $K$-Means clustering with 12900 clusters, which is slightly more than 12820 samples we are selecting, because we observe that there will sometimes be empty clusters.
For ImageNet 0.2\%, we use the same number of clusters as the number of samples that we will select. In the supervised learning setting, we use a batch size of 256, learning rate 0.01, and weight decay $10^{-3}$ and train a ResNet-50 for 1000 epochs, with learning rate decayed by 0.1 at 700, 800, and 900 epochs. Since supervised learning only uses labeled data, training for 1000 epochs with 1\% data takes about the same time to train the same network for 10 epochs with full data labeled. In the semi-supervised learning setting, we use FixMatch \cite{sohn2020fixmatch} with EMAN as our semi-supervised learning algorithm, and to maintain consistency with prior works, we use the same setting as in \cite{cai2021exponential} besides the selection of input labeled data, unless otherwise stated. Specifically we use a learning rate of 0.03 with weight decay $10^{-4}$and train a ResNet-50 for 50 epochs with a MoCo \citep{he2020momentum} model as pretrained model. We perform learning rate warmup for 5 epochs and decay the learning rate by 0.1 at 30 and 40 epochs. Note that we load MoCo model as the pretrained model for FixMatch for fair comparison so that the only difference between MoCo and CLIP setting is the sample selection.

\subsubsection*{Acknowledgments}

redpajama_set_name: RedPajamaArXiv


text:
Send Flowers for Robert
Robert J. "Bob" Cron
Robert J. "Bob" Cron, age 74 of Ft. Recovery, Ohio passed away on Tuesday, December 14, 2021, at the Gardens at Celina. He was born on Dec. 22, 1946 in Celina, Ohio to the late F. J. and Lucille K. (Buehler) Cron. In 1977 he met the love of his life Gay L. Garman, who died on Feb. 28, 2020. He is survived by his sister and brother-in-law Barbara (Tom) Bueltel of Clayton, brother and sister-in-law Charles (Nancy) Cron of St. Marys, several Nieces and Nephews and several close friends. He was also survived by his in-laws Hunter (Lora) Garman of Albany, IN, Dusty Garman of Findlay and Nina (Max) Hamrick of Willshire.
Bob graduated from St. Henry High School with the Class of 1964. He then proudly served his country, in the United States Army, during the Vietnam War from 1966 until 1968. He owned and operated the Cron Construction Company in Ft. Recovery and retired in 2011.
He was a member of the St. Henry Catholic Church in St. Henry. He was a member of the Democratic Party and a member of the Loyal Order of Moose Lodge #1473 of Celina.
Bob was a true outdoorsman and very involved in the Restoration of Rivers Project in Ohio and an advocate of the Ohio Wetlands Project. He enjoyed Fishing, Hunting and Camping.
A Catholic Prayer service will be held at 10:00 a.m. on Saturday, December 18, 2021 at the Lehman-Dzendzel Funeral Home in Celina, with Fr. Kenneth Schnipke C.PP.S. officiating. Burial will follow in the St. Henry Catholic Cemetery. Friends may call from 4-7 p.m. on Friday and an hour before services on Saturday at the Lehman-Dzendzel Funeral Home. Memorial contributions may be made to EverHeart Hospice 1350 N. Broadway St. Greenville, Ohio 45331. Condolences may be shared with the Cron Family on-line at LehmanDzendzelFH.com.
To send flowers to the family or plant a tree in memory of Robert J. "Bob" Cron, please visit our floral store.
Lehman-Dzendzel Funeral Home
901 Myers Rd.
Celina, OH 45822
EverHeart Hospice
230 W. Main St., Coldwater OH 45828

redpajama_set_name: RedPajamaCommonCrawl


The feud between developers and IT operations teams is likely to come to head with the launch this week of a new framework that gives developers a lot more control over how and when applications are deployed.
Ever since the advent of agile development methodologies and the rise of virtual machines, IT operations teams have been under the gun to keep pace with the number of new applications and updates that are being released with increased fury and frequency.
Now Electric Cloud, a provider of application deployment tools, is releasing ElectricDeploy, a framework that not only automates the deployment process, it allows developers to roll back an application to its last known working state in the event of a problem. According to Electric Cloud CTO Anders Wallgren, that not only gives developers more control over the IT environment, it allows developers an ability to recover from problems and issues in a way that has a minimal impact on the business.
That's critical, explains Wallgren, because up until now IT operations teams have not been able to effectively support the need for an IT infrastructure environment that is both flexible and dynamic enough to support agile application development. Applications developed using agile methodologies are not necessarily more error-prone than others. But because iterative updates now arrive at a mind-boggling rate, the number of opportunities to have an error in an application increases exponentially, says Wallgren.
In fact, a new survey of more than 730 applications development and operations professionals conducted by Electric Cloud indicates that 75 percent of application deployments are done manually or with little automation. At the same time, 70 percent of respondents stated that their deployments take more than one day, while 50 percent admitted missing release dates due to complexity and lack of coordination between their development and operations teams.
ElectricDeploy overcomes many of the problems these issues create, says Wallgren, by allowing developers to first model and test how their application will behave in a production environment, and then roll back if there are any unexpected issues.
Businesses today want to be able to roll out application updates with greater frequency, but existing IT operations processes are increasingly being seen as a bottleneck in achieving that goal. As a result, companies that specialize in "DevOps" technologies have been steadily introducing higher levels of abstraction and automation to give developers more control of IT infrastructure.
Obviously, automation of the application deployment process provides an opportunity to increase and reduce tensions between developers and the IT operations team. On the one hand, IT operations teams may resent the implied loss of control. On the other hand, many of them may come to appreciate not having to manually manage application rollouts that generally tend to be error-prone.
In either case, the business just wants everyone involved to get out of each other's way in order to derive business value as quickly as possible from new and existing applications that increasingly are becoming the lifeblood of the business. Anything that prolongs that process is ultimately going to be seen as an obstruction that needs to be reduced, if not outright eliminated, no matter how good the intentions of the IT operations team might actually be.