text:
\section*{Acknowledgments}
The authors would like to thank the remaining members of OLIVES and the anonymous reviewers for their feedback. In particular, we would like to thank Dr. Gukyeong Kwon for fruitful discussions within the preliminary phase of the project. This work was funded by a Ford-Gerogia Tech Alliance Project.

\bibliographystyle{plain}

\section{Introduction}

A core factor in the success of machine learning algorithms is the selection of suitable data. While several samples contain valuable information for a given task, other samples may be redundant with little additional information or anomalous with contradicting features. Active Learning~\cite{cohn1996active} is a paradigm in machine learning that tackles this issue by iteratively selecting the most informative samples from a large unlabelled data pool for annotation and training. Due to its intuitive practicality, active learning has already impacted multiple industrial sectors including manufacturing \cite{tong2001active}, robotics \cite{alrobotics}, recommender systems \cite{alrecommender}, medical imaging \cite{hoi2006batch}, and autonomous vehicles \cite{haussmann2020scalable}.

At the core of every active learning approach stands its method to select the next set of annotation samples - its acquisition function. Typically, the acquisition function establishes a ranking based on the \emph{importance} of unlabeled samples and interactively selects the next batch based on the prior ordering. Therefore, the acquisition functions typically differ by the manner in which \emph{importance} is defined and how data is selected based on the prior definition of importance. Overall, there are two simple intuitive approaches that form the basis of active learning acquisition functions. The first approach establishes the ordering/sampling method based on generalization difficulty \cite{roth2006margin, wang2014new, gal2017deep, houlsby2011bayesian}. For instance, the most "informative" samples could be underrepresented classes or difficult samples with overlapping class features. In contrast, the second approach orders samples according to dataset representation \cite{sener2017active, longtailcoreset, gissin2019discriminative}. Intuitively, this branch focuses on finding data subsets that best represent the entire dataset and emphasize data diversity within the acquisition batch.

Although both approaches follow fundamentally different principles, existing protocols employ similar mechanics to define importance within an active learning setting. Specifically, they define importance on the model representation directly. For instance, entropy sampling \cite{wang2014new} selects samples with the highest softmax entropy of the model and is therefore directly dependent on the logits or sample representation. As another example, coreset sampling \cite{sener2017active} maximizes diversity by constructing the coreset of the sample representations directly. Even though these definitions are intuitive, high representation dependence results in severe challenges that often inhibit the deployment of active learning.\\

The first of these challenges is a model's sensitivity to architecture, dataset, optimizer, and hyperparameter choices. For this reason, we find that existing protocols are not consistent over different setup constellations and performance can severely differ when any component is changed or slightly altered. For instance, our experiments show that several protocols outperform the random selection baseline with one architecture but underperform with another. In practice, a user must therefore pick the ``correct" protocol for the specific application and it is unclear if selecting with a random baseline would render higher results. The second challenge in using representation-dependent definition of informativeness is that model representations are sensitive to unknown inputs \cite{hendrycks2019robustness, Temel2017_NIPSW}. A model's performance is severely affected if it is exposed to distributions it was not trained on~\cite{kwon2019distorted, prabhushankar2021contrastive}. We refer to such samples as being out-of-distribution (OOD). As a toy example, consider a model that is exposed to aberrant samples during deployment (Figure~\ref{fig:ood-toy}). Two different protocols may result in similar in-distribution test performance but one strategy may select more aberrant training samples resulting in a more robust model representation. Even though out-of-distribution performance is inherently relevant for active learning, existing protocols do not consider performance on different distributions.

\begin{figure}[!h]
    \begin{center}

        \includegraphics[scale=0.37]{Fig/ood-toy.png}
    \end{center}
    \caption{Toy example of out-of-distribution performance in active learning. Both protocols result in similar test performance but severely differ when deployed on out-of-distribution samples.}
    \label{fig:ood-toy}
\end{figure}

In this paper, we approach this problem by viewing sample importance as shifts within the data manifold. Specifically, we define ``informativeness" by approximating the frequency in which a model ``forgets" unlabeled samples during training. By effectively counting decision boundary shifts we naturally alleviate the direct representation dependence. We find that our definition not only results in robust importance scores but further find that our scores are exceptionally accurate that can even result in bias towards anomalous samples if deployed directly as an active learning protocol. For this purpose, we combine our definition with an interactive mixture model and develop Gaussian Switch Sampling or \texttt{GauSS} in short. We validate our algorithm with exhaustive experiments and compare against popular protocols used in practical active learning pipelines. Overall, we find that \texttt{GauSS} is robust to setup changes, performs favorably in out-of-distribution settings, and achieves up to 5\% improvement in terms of accuracy.

\section{Defining Sample Information with Learning Dynamics}
Active learning consists of iteratively selecting a set of unlabeled samples for annotation. We refer to a single iteration as active learning round. In this section we formulate information content as learning difficulty within each active learning round. Intuitively, we define informative as the frequency in which a network ``forgets" unlabeled samples during training. We introduce active learning and neural network forgetting singularly and further analyze our definition in comparison to existing definitions of informativeness.
\subsection{Active Learning}
In active learning, the objective is to improve data efficiency iteratively by involving the model in the data annotation process. In this context, consider a dataset $D$ where a small subset $D_{train}$ represents the initial training data and $D_{pool}$ represents the unlabelled data pool for the selection process. The goal within each active learning round is to select a batch of $b$ samples  $X^* = \{x^*_1, ..., x^*_b\}$ that improves the model accuracy the most when added to the training data $D_{train}$. The selection of $X^*$ can be defined as

\begin{equation}
\label{eq:aquisition-function}
    X^* = \argmax_{x_1, ..., x_b \in D_{pool}} a(x_1, ..., x_b | f_w(x))
\end{equation}

where $a$ represents the acquisition function and $f_w$ is the fully trained deep model conditioned on the parameters $w$. In this context, we define first-order algorithms and second-order algorithms by the manner in which they are conditioned on $f_w$:

\begin{definition}[First- and Second-Order Active Learning]
We define an active learning algorithm as first-order when the acquisition function is conditioned on the fully trained neural network $f_w$ exclusively. Further, an algorithm is of second-order when the acquisition function is conditioned on dynamic fluctuations of $f_w$ caused by external interference (e.g. training).
\end{definition}

\subsection{Defining Importance with Forgetting Events}
\label{sec:learning-dynamics-in-active-learning}
In this paper, we define importance through the learning difficulty of different unlabeled samples. Specifically, we build upon \cite{toneva2018empirical} and measure information content with the frequency in which samples are "forgotten" during training. Intuitively, a sample is "forgotten" if it was classified correctly ("learnt") at time $t$ and subsequently misclassified ("forgotten") at a later time $t' > t$.\\

Without loss of generalizability, we consider recognition tasks where the objective is to predict a label $\tilde{y_i}$ of sample $x_i$ that corresponds to the ground truth annotation $y_i$. The accuracy of the model for sample $x_i$ at an arbitrary time $t$ can be defined as,

\begin{equation}
\label{eq:accuracy}
    acc^{t}_{i} = \mathbf{1}_{\tilde{y}^t_i = y_i}.
\end{equation}

In Equation~\ref{eq:accuracy}, $\mathbf{1}_{\tilde{y}^t_i = y_i}$ describes a binary variable that indicates whether the classification was correct at time $t$. Specifically, it reduces to one if $x_i$ was correctly predicted or zero if the sample was misclassified. Based on our notation, we define a sample as "forgotten" if the accuracy decreases within two subsequent time steps:

\begin{equation}
f_{i}^{t} = int( acc^{t}_{i} < acc^{t-1}_{i} ) \in [{1, 0}].
\end{equation}

Here, $f_i^t$ is a binary result of accuracy variables and reduces to one if the accuracy within subsequent optimization steps decreases (a switch from correctly to incorrectly classified). Similar to \cite{toneva2018empirical}, we define the binary event $f_i^t$ as a \emph{forgetting event} at time $t$. Based on these terminologies, we define the information content of each sample in a simple and intutive manner as follows:

\begin{definition}[Information in Active Learning]
Within the context of active learning, we quantify information by the amount of forgetting events that occur for a given sample each round. Specifically, we consider samples with the largest amount of forgetting events as the most informative while fewer forgetting events imply redundant information.
\end{definition}

In contrast to existing definitions, forgetting events are not based on the representation directly but are second-order artifacts of decision boundary shifts. We reason that this characteristic is favorable for distribution shifts and setup choices.

\subsection{Prediction Switches}
Even though forgetting events are simple to formulate and provide an intuitive measure for importance, the computation is not tractable for practical active learning protocols. Specifically Equation~\ref{eq:switch-events} requires the label annotation $y_i$ which is unavailable for the unlabeled set $D_{pool}$ by definition. For this purpose, we approximate forgetting events with prediction switches. A prediction switch occurs when the model output prediction $\tilde{y}$ changes within time intervals. Formally, we write
\begin{equation}
\label{eq:switch-events}
s^{t}_{i} = \mathbf{1}_{\tilde{y}^t_i != \tilde{y}^{t-1}_i}.
\end{equation}

Analogous to our previous definition, we call $s^t_i$ a \emph{switch event} at time $t$.

\section{Information Definitions in Active Learning}
In this section, we investigate the extent in which prediction switches differ to existing importance definitions. Specifically, we want to answer 1) how effectively do prediction switches estimate information content? and 2) how accurate are the importance scores when the representation is constrained?

In practical settings, sample representations are constrained by the size of the training set $D_{train}$ and can result in inaccurate importance quantification, especially when the definition is based on the representation directly. For this purpose, we conduct our analysis by artificially removing the constraint imposed by limited data and base our importance scores on fully trained model representations - i.e. the output of the acquisition function is based on representations derived from a model trained on both $D_{train}$ and $D_{pool}$. Even though this analysis puts our method at a clear disadvantage we note that our importance rankings are significantly more accurate than existing protocols.

\subsection{Accurate Importance Scores vs. Performance Trade-off}
In active learning, acquisition functions define sample importance by the information content added to the dataset. However, informative samples in practice are frequently irregular and can result in insufficient data representations when trained on informative samples exclusively. Using our cat/dog example classifier from the introduction, a training set with informative samples could exclusively contain rare cat/dog breeds that are underrepresented within the underlying distribution. A classifier trained on this constrained dataset will result in low generalization performance on a test set containing mostly well represented breeds. Within the context of active learning, acquisition functions that render importance scores that are ``too accurate" frequently result in significantly \emph{lower} performance than the random baseline or other acquisition functions with ``less accurate" importance scores.

In our analysis, we consider scenarios where the representation space is not constrained by limited data and therefore significantly improves the importance score accuracy of the compared strategies. For this reason, effective practical acquisition functions perform significantly lower than the random baseline and exhibit \emph{lower} generalization performance with \emph{higher} information content within the acquisition batch.

\subsection{Importance in Optimal Settings}
\label{sec:optimal-strat-lcs}
We investigate which acquisition functions provide the most informative samples when acquisition functions produce accurate measurements. For this purpose, we consider an optimal setting where we first train a model on the full CIFAR10 training data and derive the sample representations from the fully trained model. The following importance scores (e.g. softmax entropy) are subsequently calculated with the optimal sample representations. For our proposed definition, we count the switch events while training on the full training data and query the samples ordered from most to least switch events. In this section, we compare optimal settings only as the discussion on practical implementations with limited annotation access is thoroughly discussed in Section~\ref{sec:experiments}.

In all of our active learning experiments, we start with an initial training set $D_{train}$ of 128. In each round, we query 1024 additional samples with the repsective active learning strategy. For our active learning models we use a resnet-18, as well as a resnet-34 architecture \cite{he2016deep}. We optimize with the adam variant of SGD and a learning rate of $10^{-4}$. No augmentations are used for our active learning experiments.
For our optimal representations, we train a network on the full CIFAR-10 dataset for 200 epochs and extract the representations for our active learning experiments. For the fully trained model, we optimize with SGD and use a multi step learning rate scheduler for training. Specifically, our initial learning rate is 0.1 and we divide the learning rate by 5 in epochs 100, 125, 150, and 175 respectively.
We further use the augmentations random crop, random horizontal flip, and cutout. We choose this setup to derive competitive representations for the acquisition functions. We show the results of our analysis in Figure~\ref{fig:optimal-strats}. Specifically, we compare switch events to entropy sampling \cite{wang2014new}, random sampling, least confidence sampling \cite{wang2014new}, and coreset \cite{sener2017active}. We note that querying with optimal representations results in significantly lower generalization performance than the random baseline due to the high importance score accuracy. This is especially true for early rounds where only a limited amount of training data is available for the active learning model. Samples with a high information content are typically aberrant outliers that are disruptive to the representation space. Therefore, a more significant deviation from random sampling implies a higher information content. We note, that querying with switch events results in the most profound difference compared to the other strategies and implies the highest information content within each query. It is important to note that most existing active learning protocols outperform random sampling in practical settings - i.e. when the sample representations are inaccurate due to limited training data. In these cases, limited representation capabilities ``help" the acquisition function by introducing a random noise factor within the query. Several strategies even introduce targeted sampling techniques to counteract accurate information scores \cite{ash2019deep, haussmann2020scalable}. In GauSS we do this by sampling from a gaussian mixture model.

\begin{figure*} [ht!]

    \begin{center}

        \includegraphics[scale=0.45]{Fig/optimal-strats.png}
    \end{center}
    \caption{Different query strategies in with optimal sample representations. We compare random sampling, entropy sampling, least confidence sampling, coreset, as well as switch events.}
    \label{fig:optimal-strats}
\end{figure*}

\subsection{Approximation of Optimal Representations}
We further investigate the approximation quality of different importance definitions with respect to their optimal counter part. As described within the previous section, most existing strategies outperform the random baseline while their optimal counter part significantly underperforms in terms of generalization accuracy. We follow that accuracy alone is not a sufficient metric to capture the optimal approximation capabilities of an active learning protocol. For this purpose, we measure protocol similarity through information content within the training set of each active learning round. In this context, we evaluate the importance score for each sample within the training set and compare the distributions of the importance scores for each round through statistical difference metrics. If the samples contain similar information, the resulting importance scores render similar distributions and vice versa. Specifically, we perform the following steps for each active learning round: 1) We evaluate the importance score for every training sample. 2) We estimate the distribution by creating histograms over the importance scores for every acquisition function. 3) We evaluate the statistical difference between different acquisition function pairs (in our case we use a smoothed Kullback-Leibler divergence).
We show our results in Figure~\ref{fig:kl-dist}. The different rows show the active learning protocols while the columns show the optimal counter parts. We compare switch events against entropy sampling and least confidence sampling as these are the most competitive definitions in our previous analysis. Further, we use forgetting events as the optimal counterpart of our switch event protocol. We choose this setup, as switch events should not only approximate their optimal switch event counterpart but should ideally approximate our original importance definition based on sample forgetting, not prediction switches. In our experiments, we use the same active learning setup as the previous section for experiments with optimal representations; The non-optimal protocol setup is identical with the exception of the data representation source - i.e. we follow the normal protocol workflow and gather sample representations from the model trained on $D_{train}$. We further generate importance scores for the histograms with forgetting events and compare histograms by calculating the Kullback-Leibler divergence smoothed by a gaussian filter. We average our result over the first 20 rounds.

Overall, we note that switch events show the closest distribution similarity to all optimal strategies indicating the most accurate information content approximation within the round-wise training sets. We reason that switch events are not dependent on the model representation directly and are therefore more accurate in identifying the most informative batch regardless of the importance definition. This further manifests itself visually in the distribution histograms of individual rounds (Figure~\ref{fig:nideal-hist}). We see that switch events exhibit the visually most similar distribution to optimal forgetting events distribution than the other strategies. In particular, we note that the other strategies query more samples with less amounts of forgetting events than the switch event acquisition function which results in different distribution shapes. As a final note, we add that even though we compare distributions with forgetting events we observe a similar behavior when comparing against other importance definitions.

\begin{figure}[!h]

    \begin{center}

        \includegraphics[scale=0.35]{Fig/kl-dist-difference.png}
    \end{center}
    \caption{Average KL-Divergence between the forgetting events distributions over 20 active learning rounds. We compare query strategies with their capability to approximate optimal settings with different importance definition. Columns represent optimal settings and rows show their practical approximations. Low scores indicate a high similarity in protocol behavior.}
    \label{fig:kl-dist}
\end{figure}

\begin{figure*}[!ht]
    \begin{center}

        \includegraphics[scale=0.50]{Fig/nideal-hist.png}
    \end{center}
    \caption{Histograms for different definitions of information content in different rounds. The first row, samples data points with the highest forgetting events within an optimal setting - i.e. the importance scores are derived from a model trained on the fully labeled training set. The remaining rows are practical active learning scenarios where the importance scores are derived from a model with a constrained training pool. This includes entropy sampling, coreset, least confidence sampling and prediction switches.}
    \label{fig:nideal-hist}
\end{figure*}

\section{Conclusion}
\label{sec:conclusion}
In this paper, we introduced a grounded definition of information content for active learning protocols. In contrast to existing approaches, our definition is not based on the representation directly but defines importance with decision boundary shifts - or neural network ``forgetting". For practical usage, we approximate ``forgetting" with prediction switches and find that prediction switches produce accurate importance scores when compared to other definitions. Finally, we develop a novel acquisition function that samples interactively with a gaussian mixture model. We validate our algorithm empirically with exhaustive experiments and compare against popular protocols used in practical active learning pipelines. Overall, \texttt{GauSS} is robust to setup changes, performs favorably in out-of-distribution settings, and achieves up to 5\% improvement in terms of accuracy.
\section{Experiments}
\label{sec:experiments}
With our experiments we want to answer two central questions: 1) How well does \texttt{GauSS} perform in conventional active learning where the only representation constraint is imposed by $D_{train}$?, and 2) How does the active learning performance change in out-of-distribution scenarios? We feel that both 1) and 2) are realistic scenarios in practical deployment and represent a comprehensive study of the generalization capabilites of \texttt{GauSS}.

\subsection{Numerical Comparison}
\label{sec:nemerical evaluation}
In our experiments, we compare \texttt{GauSS} against four popular active learning protocols. Specifically, we compare against entropy sampling \cite{wang2014new}, coreset \cite{sener2017active}, active learning by learning (ALBL) \cite{hsu2015active}, least confidence sampling \cite{wang2014new}, and BatchBald \cite{batchbald}. We choose this constellation as it provides a large variety of importance definitions: Entropy sampling and least confidence sampling define sample importance with generalization difficulty while coreset maximizes the diversity of the training pool; active learning by learning represents a bandit style approach that interactively switches between coreset and least confidence during each round. It represents a fusion protocol that combines both generalization difficulty and data diversity. Finally, Batchbald is a recent bayesian approach that extends the popular BALD \cite{houlsby2011bayesian} method.

We consider three out-of-distribution settings as well as three in-distribution settings.
For our out-of-distribution analysis, we sample our training set from the CIFAR10 training set and measure accuracy on CIFAR10-C \cite{hendrycks2019robustness}, STL10 \cite{stl10}, and CINIC10 \cite{cinic10}.
We choose these benchmarks, as they contain examples of data corruption and domain shifts - both of which are common in practical machine learning pipelines: Both STL10 and CINIC10 represent examples of cross domain out-of-distribution. STL10 is a difficult dataset with 8000 test samples and nine out of ten overlapping classes with CIFAR10 (we only consider the overlapping classes for accuracy calculations). CINIC10 is significantly larger than CIFAR10 with a test set of 90000 images and fully overlapping classes with CIFAR10. Furthermore, CIFAR10-C is an artificially corrupted version of CIFAR10 with 20 different corruption types on five different severity levels. We consider all corruptions except ``labels", ``shot noise", and ``speckle noise" and test on level two as well as level five each round. We choose this setup as it includes realistic data corruptions on a wide variety of severity levels. Our in-distribution analysis considers the datasets CIFAR10, CINIC10, and CURE-TSR \cite{Temel2017_NIPSW}. Both CIFAR10 and CINIC10 are popular in-distribution benchmarks with increasing difficulty and dataset size (60000 images for CIFAR10 and 270000 images for CINIC10).
CURE-TSR represents the practically relevant application of traffic sign recognition with corrupted and uncorrupted training and test sets. The dataset is especially challenging as traffic signs occur at different frequencies resulting in a high class imbalance. Hence, an effective algorithm must perform on both well represented as well as underrepresented classes to improve generalization performance on the test set. In our experiments, ee consider the uncorrupted training and test set only.
For all datasets except CURE-TSR, we start with an initial training pool of 128 randomly chosen samples and iteratively query 1024 samples each round according to the respective protocols.
For BatchBald, we implement a bayesian model by applying monte-carlo droupout. Similar to the setup of \cite{batchbald}, we apply a dropout layer with a dropout probability of 0.4 before the final fully connected layer and sample 70 monte-carlo instances.
When experimenting with CURE-TSR, we start with an initial training pool of 32 samples and query an additional 32 samples each round. We choose this setup as it significantly increases the difficulty for traffic sign recognition. We further use three popular network architectures: Resnet-18 \cite{he2016deep}, resnet-34 \cite{he2016deep}, and densenet-121 \cite{densenet}. Similar to our previous experiments, we optimize with the Adam variant of SGD with a learning rate of $10^{-4}$ and no learning rate scheduler. No data augmentations are used and we use pre-trained model weights for our out-of-distribution experiments as well as our experiments on the CIFAR10 dataset. We reason that this represents a practical scenario where data is abundant in one domain but scarce in another. Finally, we retrain our model from scratch each round to prevent warm starting \cite{ash2020warm}. For all constellations, our results are averaged over 5 random seeds. Overall, this amounts to a total of 450 separate active learning experiments (we count all CIFAR10-C datasets as one experiment).

In several cases, the performance of active learning strategies differs across selection rounds. For instance, a strategy may show a strong accuracy increase for early rounds, and underperform in later rounds. Therefore, we quantify algorithm performance by summarizing the margin in which it outperforms (or underperforms) the random baseline. With random baseline, we refer to sampling random data points from the unlabeled pool each round.  First, we calculate the difference to the random baseline for each round by subtracting the random baseline accuracy curve from the accuracy curve of the respective strategy. Second, we combine the difference values by integrating over the subtraction curve across several active learning rounds.
In short, the metric amounts to the difference in accuracy area compared to the random baseline. For reference, we provide the absolute area of the random baseline in the first row of each architecture.
A negative difference (random accuracy is higher) contributes negatively to the integral and positive differences (comparison accuracy is higher) contributes positively to the integral. Hence, a negative integral indicates an overall higher random accuracy curve while a positive integral indicates an overall outperformance with respect to the random baseline. In addition, the higher the integral the more significant the positive difference and the larger the outperformance margin.
By quantifying performance with differences to the same reference point, we remove biases from absolute values in each round. For instance, the difference in early rounds contributes to the integral with the same magnitude as later rounds even though the absolute accuracy is higher in later stages.
For our results on CIFAR10-C, we average the difference curve accross all corruptions and levels. We show the integration results in Table~\ref{table:results-overall}.

We note, that \texttt{GauSS} shows the overall best performance over both in-distribution, and out-of-distribution benchmarks in nearly every experiment constellation. In particular, we note that \texttt{GauSS} shows an overall better performance than the random baseline in nearly every constellation. In contrast, the other strategies can outperform or underperform the random baseline. For instance, entropy sampling outperforms the random baseline in several cases (e.g. CIFAR10-C with resnet-34), but underperforms in other cases (e.g. CIFAR10 in-distribution test set with densenet-121). Moreover, we note that \texttt{GauSS} is more consistent accross different dataset and architecture choices. In all instances except a few outliers, \texttt{GauSS} outperforms the random baseline. In contrast, we find high fluctuations with competitive protocols. For instance, least confidence sampling significantly outperforms the random baseline on CIFAR10-C and CIFAR10 when using resnet-18, but underperforms when switching the architecture to densenet-121. We reason that \texttt{GauSS} queries more consistently as sample queries and model representation are not directly coupled. If the model representation severely suffers within a specific experiment configuration (e.g. when densenet-121 is used instead of resnet-18), the acquisition batch severely degenerates. In contrast \texttt{GauSS} does not query based on the representation exclusively. Therefore switching experiment components does not affect the overall protocol performance.
Finally, we discuss the behavior of \texttt{GauSS} in settings with high class imbalances on CURE-TSR. In our experiments, \texttt{GauSS} outperforms on two of the three architecture choices and consistently shows positive improvements over the random baseline (i.e. positive integral values). Similar to our previous observations, competitive strategies are strong in one setting but underperform the random baseline in another. For instance, entropy sampling underperforms the random baseline on resnet-18 but outperforms on resnet-34. We follow that \texttt{GauSS} proposes consistent and accurate data samples even when high imbalances are present.

In addition to Table~\ref{table:results-overall}, we provide absolute values of the different strategies (as well as the random baseline) in Table~\ref{table:results-absolute}. Specifically, we show accuracy values in rounds one, four, eight, and nine on CIFAR10, as well as its corrupted variant CIFAR10-C. We choose this constellation as it represents one early round, one intermediate round, and two late-stage rounds. Further, we conduct a t-test to determine the statistical significance of our results, where one refers to a statistically significant difference to \texttt{GauSS}, and zero to a statistically insignificant difference with \texttt{GauSS}.  We note, that \texttt{GauSS} matches or outperforms the competing strategies across all rounds: \texttt{GauSS} either outperforms or is statistically insignificant with respect to the outperforming strategy. In contrast, existing strategies are either competitive in early rounds or later rounds but rarely show consistent performance across the entire active learning experiment.

\begin{table*}[!ht]
	\centering
	\caption{Area under difference curve with reference to random sampling over several datasets, query strategies, and architectures. Positive, and higher numbers are better. Negative numbers imply that the approach has a lower accuracy curve than the random baseline. The spread represents the standard deviation. Random reference area shows the total area under the random baseline accuracy curve.}
	\label{table:results-overall}
	\begin{tabular}{ |p{1.9cm}|p{1.9cm}||p{1.6cm}|p{1.6cm}|p{1.6cm}|p{1.6cm}|p{1.8cm}|p{1.6cm}|}
		\hline
		\multicolumn{2}{|c||}{}   & \multicolumn{3}{|c|}{Out-Of-Distr.} & \multicolumn{3}{|c|}{In-Distr.} \\
		\hline

		\multicolumn{1}{|c|}{Architectures} & \multicolumn{1}{|c||}{Algorithms} & \multicolumn{1}{|c|}{CIFAR10-C}  & \multicolumn{1}{|c|}{CINIC10}  & \multicolumn{1}{|c|}{STL10} & \multicolumn{1}{|c|}{CIFAR10} & \multicolumn{1}{|c|}{CINIC10} & \multicolumn{1}{|c|}{CURE-TSR}\\
		\hline
        \multirow{6}{*}{ResNet-18}
                                    & Rand. Reference           & 649.53     & 666.06      & 399.29     & 831.19       & 636.35     & 972.48\\
                                    & ALBL \cite{hsu2015active}       & 10.19 ± 8.24     & 5.66 ± 8.08      & -4.55 ± 4.66     & 7.25 ± 6.86       & -2.54 ± 5.12     & 22.80 ± 1.48\\
                                    & Coreset \cite{sener2017active}  & -41.87 ± 2.02    & -35.07 ± 2.13    & -20.58 ± 3.14    & -41.04 ± 0.71    & -48.54 ± 3.11     & -38.60 ± 0.83            \\
                                    & Entropy \cite{wang2014new}      & 7.83 ± 1.03      & 8.58 ± 0.76      & 2.16 ± 3.79      & 0.45 ± 0.34      & -6.09 ± 3.44      & -25.02 ± 8.55            \\
                                    & L. Conf. \cite{wang2014new}     & 16.23 ± 1.17     & 13.47 ± 0.12     & 3.20 ± 1.42      & 9.93 ± 0.94      & -2.54 ± 1.22      & 5.74  ± 6.28              \\
                                    & BatchBald \cite{batchbald}      & -20.98 ± 2.40    & -19.93 ± 0.75    & -14.71 ± 0.01    & -17.49 ± 2.33    & -356.14 ± 5.30   & -2.56 ± 3.17                \\
                                    & \bf{GauSS}                & $\mathbf{17.76\pm1.61}$ & $\mathbf{13.72\pm0.13}$ & $\mathbf{7.17\pm4.48}$ & $\mathbf{17.77\pm0.89}$ & $\mathbf{5.23\pm0.47}$ &                                        $\mathbf{33.55\pm6.13}$ \\
                                                                        		\hline
        \multirow{6}{*}{DenseNet-121}
                                     & Rand. Reference           & 653.55     & 674.07      & 413.96     & 860.81       & 661.66     & 1018.35\\
                                     & ALBL \cite{hsu2015active}       & 7.69 ± 1.41      & 6.67 ± 2.45      & -8.56 ± 2.26     & 3.28 ± 3.08      & -4.58 ± 2.61     & 30.91 ± 9.30             \\
                                     & Coreset \cite{sener2017active}  & -23.40 ± 2.32    & -14.36 ± 0.70    & -24.98 ± 1.94    & -21.74 ± 2.89    & -12.72 ± 0.77    & 31.66 ± 6.83             \\
                                     & Entropy \cite{wang2014new}      & 1.77 ± 0.68      & 2.52 ± 2.75      & -10.25 ± 5.34    & -7.76 ± 4.20     & -23.17 ± 3.92    & 8.33 ± 6.34             \\
                                     & L. Conf. \cite{wang2014new}     & -0.20 ± 1.59     & 3.81 ± 1.89      & -11.48 ± 4.65    & -3.57 ± 4.17     & -17.19 ± 5.08    & 43.25 ± 8.85             \\
                                     & BatchBald \cite{batchbald}      & -8.85 ± 0.47     & -11.98 ± 0.01    & -1.15 ± 1.79     & -10.51 ± 0.64    & -383.24 ± 19.51  & $\mathbf{60.13\pm9.30}$               \\
                                     & \bf{GauSS}                      & $\mathbf{13.64\pm1.96}$ & $\mathbf{10.01\pm0.52}$ & $\mathbf{-1.28\pm0.64}$ & $\mathbf{11.46\pm0.18}$ & $\mathbf{4.52\pm2.61}$  &                                 53.25 ± 5.47 \\
                                                                        		\hline
        \multirow{6}{*}{ResNet-34}
                                     & Rand. Reference           & 705.04     & 708.21      & 427.31     & 893.57       & 687.10     & 1018.21\\
                                     & ALBL \cite{hsu2015active}       & 7.56 ± 0.09             & 9.67 ± 2.42     & 0.41 ± 4.06           & 13.96 ± 2.40     & -4.55 ± 3.78            & 28.63 ± 4.16             \\
                                     & Coreset \cite{sener2017active}  & -8.25 ± 0.17            & -7.18 ± 0.24    & -9.23 ± 4.28          & -5.21 ± 0.27     & -14.16 ± 3.70     & -9.40 ± 6.88             \\
                                     & Entropy \cite{wang2014new}      & 14.80 ± 0.16            & 15.31 ± 0.53    & $\mathbf{5.93\pm1.34}$& 12.51 ± 1.03     & -12.49 ± 3.61           & 10.01 ± 6.47             \\
                                     & L. Conf. \cite{wang2014new}     & 17.52 ± 1.21            & 16.60 ± 1.33    & 4.91 ± 5.17           & 15.99 ± 0.36     & -4.40 ± 0.83            & $\mathbf{47.01\pm3.83}$  \\
                                     & BatchBald \cite{batchbald}      & -25.90 ± 0.39           & -25.84 ± 0.93   & -11.85 ± 0.76         & -26.10 ± 1.49    & -391.38 ± 11.78  & 0.93 ± 3.13               \\
                                     & \bf{GauSS}                      & $\mathbf{18.14\pm0.96}$ & $\mathbf{16.58\pm1.17}$ & 2.12 ± 2.66        & $\mathbf{19.30\pm1.16}$ & $\mathbf{9.34\pm1.04}$ &                                 25.44 ± 9.62             \\
		\hline
	\end{tabular}
\end{table*}

\begin{table*}[!ht]
	\centering
	\caption{Accuracy values and binary statistical significance over active learning rounds one, four, eight, and nine on CIFAR10 and CIFAR10-C. A significance of one indicates a statistically significant difference to GauSS while zero indicates a statistically insignificant difference with \texttt{GauSS}. We show our results on ResNet-18 and DenseNet-121.}
	\label{table:results-absolute}
	\begin{tabular}{ |p{0.2cm}|p{1.9cm}|p{1.9cm}||p{1.0cm}|p{0.5cm}|p{1.0cm}|p{0.5cm}|p{1.0cm}|p{0.5cm}|p{1.0cm}|p{0.5cm}|}
		\hline
		\multicolumn{3}{|c||}{}   & \multicolumn{2}{|c|}{1152 Samples} & \multicolumn{2}{|c|}{4224 Samples} & \multicolumn{2}{|c|}{8320 Samples} & \multicolumn{2}{|c|}{9344 Samples} \\
		\hline

		 \multicolumn{1}{|c|}{} & \multicolumn{1}{|c|}{Dataset} & \multicolumn{1}{|c||}{Algorithms} & \multicolumn{1}{|c|}{Acc.}  & \multicolumn{1}{|c|}{Stat. Sign.}  & \multicolumn{1}{|c|}{Acc.}  & \multicolumn{1}{|c|}{Stat. Sign.}  & \multicolumn{1}{|c|}{Acc.}  & \multicolumn{1}{|c|}{Stat. Sign.}  & \multicolumn{1}{|c|}{Acc.}  & \multicolumn{1}{|c|}{Stat. Sign.} \\
		\hline
  \parbox[t]{2mm}{\multirow{6}{*}{\rotatebox[origin=c]{90}{ResNet-18}}}  	& \multirow{6}{*}{CIFAR10}
                                    & Random                          & 42.48 & 0   & 57.05 & 0   & 63.89 & 1   & 65.02 & 1   \\
                                    & & ALBL \cite{hsu2015active}       & 39.14 & 1   & 57.16 & 1   & 64.39 & 0   & 65.89 & 0   \\
                                    & & Coreset \cite{sener2017active}  & 38.00 & 1   & 52.07 & 1   & 61.96 & 1   & 63.21 & 1   \\
                                    & & Entropy \cite{wang2014new}      & 37.35 & 1   & 54.90 & 1   & 64.02 & 1   & 65.00 & 1   \\
                                    & & L. Conf. \cite{wang2014new}     & 38.90 & 1   & 56.11 & 1   & 64.29 & 0   & 65.47 & 0   \\
                                    & & BatchBald \cite{batchbald}      & \bf{43.36}  & 0  & \bf{57.25} & 0   & 63.20 & 1   & 64.10 & 1   \\
                                    & & \bf{GauSS}                      & 42.00 & Baseline & 57.06 & Baseline & \bf{64.76} & Baseline & \bf{66.11} & Baseline \\
		\hline
   \parbox[t]{2mm}{\multirow{6}{*}{\rotatebox[origin=c]{90}{ResNet-18}}}  	& \multirow{6}{*}{CIFAR10-C}
                                    & Random                          & \bf{33.27} & 0   & 44.19 & 0   & 49.49 & 1   & 50.72 & 1\\
                                    & & ALBL \cite{hsu2015active}       & 31.01 & 1   & \bf{44.64} & 0   & \bf{50.46} & 0   & 51.60 & 0\\
                                    & & Coreset \cite{sener2017active}  & 29.83 & 1   & 39.49 & 1   & 47.44 & 1   & 48.44 & 1\\
                                    & & Entropy \cite{wang2014new}      & 30.17 & 1   & 42.88 & 1   & 50.28 & 0   & 51.33 & 0\\
                                    & & L. Conf. \cite{wang2014new}     & 31.15 & 1   & 43.68 & 1   & 50.45 & 0   & 51.59 & 0\\
                                    & & BatchBald \cite{batchbald}      & 33.97 & 0   & 43.69 & 0   & 48.83 & 1   & 49.92 & 1\\
                                    & & \bf{GauSS}                      & 33.09 & Baseline & 44.43 & Baseline & 50.43 & Baseline & \bf{51.72} & Baseline\\
		\hline
  \parbox[t]{2mm}{\multirow{6}{*}{\rotatebox[origin=c]{90}{DenseNet-121}}}  	&
        \multirow{6}{*}{CIFAR10}
                                    & Random                          & \bf{44.17} & 0   & 59.27 & 0   & 66.73 & 0   & 67.63 & 1   \\
                                    & & ALBL \cite{hsu2015active}       & 38.53 & 1   & 59.34 & 0   & \bf{67.22} & 0   & 68.81 & 0   \\
                                    & & Coreset \cite{sener2017active}  & 36.13 & 1   & 58.08 & 1   & 65.89 & 1   & 67.30 & 1   \\
                                    & & Entropy \cite{wang2014new}      & 37.53 & 1   & 57.54 & 1   & 66.94 & 0   & 68.17 & 0   \\
                                    & & L. Conf. \cite{wang2014new}     & 39.06 & 1   & 58.07 & 1   & 66.62 & 0   & 67.62 & 1   \\
                                    & & BatchBald \cite{batchbald}      & 43.84 & 0   & \bf{60.45} & 0   & 66.17 & 0   & 66.84 & 1   \\
                                    & & \bf{GauSS}                      & 43.38 & Baseline & 59.53 & Baseline & 67.02 & Baseline & \bf{69.01} & Baseline \\
		\hline
        \parbox[t]{2mm}{\multirow{6}{*}{\rotatebox[origin=c]{90}{DenseNet-121}}}  	& \multirow{6}{*}{CIFAR10-C}
                                    & Random                          & \bf{33.21} & 0   & 44.74 & 0   & 50.31 & 1   & 51.42 & 1   \\
                                    & & ALBL \cite{hsu2015active}       & 29.53 & 1   & \bf{45.27} & 0   & 51.26 & 0   & 52.64 & 0   \\
                                    & & Coreset \cite{sener2017active}  & 26.94 & 1   & 43.25 & 1   & 49.52 & 1   & 50.55 & 1   \\
                                    & & Entropy \cite{wang2014new}      & 29.29 & 1   & 43.69 & 1   & 50.71 & 1   & 51.72 & 1   \\
                                    & & L. Conf. \cite{wang2014new}     & 30.07 & 1   & 44.18 & 0   & 50.19 & 1   & 51.63 & 1   \\
                                    & & BatchBald \cite{batchbald}      & 32.87 & 0   & 45.43 & 0   & 50.12 & 1   & 50.82 & 0   \\
                                    & & \bf{GauSS}                      & 33.06 & Baseline & 44.75 & Baseline & \bf{51.65} & Baseline & \bf{52.76} & Baseline \\
		\hline
	\end{tabular}
\end{table*}

\begin{figure*}[!h]
    \begin{center}

        \includegraphics[scale=0.38]{Fig/lcs-final.png}
    \end{center}
    \caption{Learning curves of in-distribution, as well as out-of-distribution experiments with the resnet-18, resnet-34, and densenet-121. Top: Out-of-distribution experiments with resnet-18 on brightness level two (CIFAR10-C), JPEG compression level five (CIFAR10-C), and in-distribution experiment on CINIC10 with resnet-34. Bottom: In-distribution experiments with CIFAR10 with desnsenet-121, CINIC10 with densenet-121, and CURE-TSR with resnet-18.}
    \label{fig:lcs-final}
\end{figure*}
\subsection{Learning Curves}
We further plot the learning curves of several experiment configurations in Figure~\ref{fig:lcs-final}. Specifically, we show the results of several configurations using a resnet-18 architecture on in-distribution, as well as out-of-distribution experiments. As out-of-distribution examples, we plot the CIFAR10-C corruptions brightness level two, JPEG compression level five, as well as the CINIC10 test set when training on CIFAR10. Further, we plot the in-distribution datasets CIFAR10, CININC10, and CURE-TSR. The learning curves, further confirm our observations from Table~\ref{table:results-overall}. \texttt{GauSS} qualitatively outperforms the other strategies over a large variety of in-distribution, and out-of-distribution experiments. We further note, that the performance of \texttt{GauSS} is especially strong in early rounds where the training set contains few samples. We reason that the representation is especially inaccurate for early active learning rounds and that resulting importance metric is severly affected for the competitive strategies. For later rounds, the model develops a more accurate representation space and we observe \texttt{GauSS} outperforming by a significantly smaller margin. A further explanation for the superior performance is the targeted sampling approach in \texttt{GauSS}. As previously discussed, representation noise can be beneficial in scenarios where accurate importance definitions result in difficult outlier selections that can potentially damage the model representation (see our experiments with optimal representations in Section~\ref{sec:optimal-strat-lcs}). This is especially true for the two protocols least confidence sampling, and entropy sampling that have a forcefully added random query noise due to the limited representation capabilities of the model. In contrast, \texttt{GauSS} adds targeted (or biased) noise that, in combination with accurate importance measurements, results in a more conclusive data representation when the representation is severely susceptible to outlier samples.

\begin{figure*}[!th]
    \begin{center}

        \includegraphics[scale=0.38]{Fig/CIFAR100-incremental.png}
    \end{center}
    \caption{Incremental analysis on CIFAR100. We perform active learning with a resnet-18 architecture on 5, 10, 20, and 50 of the 100 CIFAR100 classes respecitvely. We compare \texttt{GauSS} with coreset, entropy, least confidence, active-learning-by-learning, and batchbald.}
    \label{fig:lcs-incremental}
\end{figure*}
While we only show a small subset of learning curves, we note that this behavior is consistent across the other experiment constellations as well but are omitted due to space limitations. With 16 different CIFAR10-C corruptions at two levels as well as the remaining benchmarks this amounts to 555 different active learning curves. In summary, we can report an improvement of up to 5\% compared to the other strategies.

\subsection{Incremental Analysis}
Within this subsection, we investigate the performance of \texttt{GauSS} in settings with higher task complexity. For this purpose, we consider an incremental analysis where we perform active learning on a subset of the CIFAR100 dataset with 5, 10, 20, and 50 of the 100 classes respectively. In our experiments, we use a resnet-18 architecture and opt for a similar setup as our previous experiments on CIFAR10. In particular, we start with an initial training set size of 128 and query 1024 samples each round until the entire unlabeled pool is fully annotated. We show the resulting learning curves of the four data subsets in Figure~\ref{fig:lcs-incremental}. We note that settings with less classes contain fewer samples in the data pool and are therefore fully queried in less rounds.

Overall, \texttt{GauSS} shows favorable qualities with a higher number of tasks. While lower task quantities do not show distinguishable trends, \texttt{GauSS} performs well in complex settings with higher amounts of tasks. In particular, we note that \texttt{GauSS} performs well regardless of the active learning round. In contrast, the competitive strategies show a strong performance in early or later active learning rounds exclusively. As an example, entropy and least confidence sampling perform well in later rounds and underperform in early stages. On the other hand, Batchbald is particularly strong in early stages but is outperformed later stages where the data representations are more mature.

\section{Introduction}
A core factor in the success of machine learning algorithms is the selection of suitable data. While several samples contain valuable information for a given task, other samples may be redundant with little additional information or anomalous with contradicting features. Active Learning~\cite{cohn1996active, settles2009active} is a paradigm in machine learning that selects the most informative samples from a large unlabelled data pool for annotation and training. Due to its intuitive practicality, active learning has already impacted multiple industrial sectors including manufacturing \cite{tong2001active}, robotics \cite{alrobotics}, recommender systems \cite{alrecommender}, medical imaging \cite{hoi2006batch}, and autonomous vehicles \cite{haussmann2020scalable}.

At the core of every active learning approach stands its method to select the next set of annotation samples - its acquisition function. The function establishes a ranking based on \emph{information content} and selects interactively from the most informative samples. Therefore, the success of an acquisition function heavily relies on 1) the definition of high information content and on 2) the information content approximation when the model representations are constrained due to the lack of annotated training data. In other words, a successful acquisition function relies on \emph{what} it considers ``informative" and \emph{how} well the model produces importance scores. Hence, an effective definition is both accurate in defining sample importance and simple to approximate by a constrained model.

Intuitively, we would assume that an effective definition of informativeness would result in superior generalization performance. In our analysis, we find this assumption to be incorrect. In fact, definitions that produce accurate importance estimates can even hurt generalization performance for early active learning rounds when the model is heavily constrained. For a simple explanation, consider a toy classifier that distinguishes cats from dogs. The samples with the highest information content are frequently anomalous or subject to data noise. In our example, this could be an image where both a cat and a dog are present or the presence of a rare breed that significantly differs from the remaining data points. Within the context of active learning, training on highly informative (or anomalous) samples results in an inaccurate representation space and thus a lower generalization performance on the test set. We sketch such a scenario in Figure~\ref{fig:information-toy} and further show this behavior empirically in later sections. With acquisition functions that produce ``too accurate" information content scores, the resulting model overfits to outlier data and therefore results in lower performance. Even though several existing approaches do not consider this essential design argument, they can still be effective in practical scenarios. In particular, existing algorithms (willingly or unwillingly) introduce noise within the acquisition function that results in inaccurate information content estimates and hence better performance. However, existing approaches define sample importance on the model representation directly which is noisy in early rounds due to the lack of data. As an example, entropy sampling \cite{wang2014new} selects samples with the highest softmax entropy, a direct manifestation of the model representation.

\begin{figure}[!h]
    \begin{center}

        \includegraphics[scale=0.37]{Fig/information-toy.png}
    \end{center}
    \caption{Toy example of active learning with samples containing high/low information content. With a high information content outliers are selected for the training set $D_{train}$ and can result in inaccurate decision boundaries.}
    \label{fig:information-toy}
\end{figure}

Even though existing methods can be effective, relying on the representation directly has two hazardous implications: First, the acquisition function performance is highly dependent on experimental parameters (e.g. architecture) that directly influence the model representation. For this reason, several methods perform well for one parameter constellation but not in another. Second, the acquisition function does not consider model robustness to samples that do not originate from the training distribution (out-of-distribution). In particular, the model is incapable of reliably selecting informative samples that increase robustness due to the inherently unreliable model representations of outliers \cite{hendrycks2019robustness}.
We show a toy example of two different protocols exposed to out-of-distribution settings in Figure~\ref{fig:ood-toy}. In active learning round~N, we show two different protocols with similar test set performance. However, when exposed to out-of-distribution samples, the performance of both protocols significantly diverges and protocol~2 outperforms protocol~1 by a large margin. While papers on active learning generalization are ubiquitous, out-of-distribution active learning studies are significantly less common \cite{kothawade2021similar, benkert2022forgetful}. For this purpose, popular active learning protocols lack robustness qualities and performance is unpredictable, especially in out-of-distribution scenarios. In fact, our experiments in Section~\ref{sec:nemerical evaluation} show significant performance variations of popular active learning approaches in out-of-distribution settings.

\begin{figure}[!h]
    \begin{center}

        \includegraphics[scale=0.37]{Fig/ood-toy.png}
    \end{center}
    \caption{Toy example of out-of-distribution performance in active learning. Both protocols result in similar test performance but severely differ when deployed on out-of-distribution samples.}
    \label{fig:ood-toy}
\end{figure}

In this paper, we propose a grounded approach to defining information content in active learning. Specifically, we consider a second-order approach where ``informative" is defined by representation shifts rather than the representation directly. We consider a sample most informative if it was ``forgotten" the most during training. In practice, evaluating ``forgetting" requires annotations which are not present within the unlabeled pool. For this purpose, we approximate ``forgetting" as a switch within the sample prediction. This characteristic is especially important for selecting samples that make our model more robust to out-of-distribution samples. In a detailed analysis, we show that our practical ``switch" definition is both exceedingly accurate in assessing sample importance as well as simple to approximate by a constrained model. To showcase the quality of our approach, we show samples with a high information content in Figure~\ref{fig:examples} according to different definitions of information content. Specifically, we show highly informative samples as by our definition and two popular definitions in active learning literature. We see that our informative samples are exceedingly harder to distinguish or are ambiguous which implies a higher information content. For instance, our ``informative" samples are ambiguous images with complex shapes or ambiguous colors (e.g. the frog image in the top row; third image from the right side). In contrast, the other definitions consider clear images informative that are highly similar among each other. For instance, entropy contains five similar car images that are clearly distinguishable.

\begin{figure*}[!h]
    \begin{center}

        \includegraphics[scale=0.50]{Fig/example-images.png}
    \end{center}
    \caption{Examples of informative samples when using different definitions of information content. The examples are generated without limited data constraints. From top to bottom: Our definition, entropy sampling, least confidence sampling.}
    \label{fig:examples}
\end{figure*}

In order to provide favorable generalization performance, we further extend our method with an interactive sampling technique based on a gaussian mixture model. Our approach produces a targeted noise vector that successfully induces enough noise for model generalization while biasing the selection batch to highly informative samples that improve model robustness. We call our method \texttt{Gaussian Switch Sampling} or \texttt{GauSS} in short. We validate our algorithm with exhaustive experiments and compare against popular protocols used in practical active learning pipelines. Overall, we find that \texttt{GauSS} is robust to setup changes, performs favorably in out-of-distribution settings, and achieves up to 5\% improvement in terms of accuracy over popular existing strategies.
\section{Gaussian Switch Sampling}
Based on our initial analysis, we reason that switch events are effective for estimating information for unlabeled samples. Further, we reason that the second-order nature is potentially more robust to different training setups and out-of-distribution settings. However, we noted in our experiments that effective importance scores bias queries towards outliers that inhibit generalization capabilities. For this purpose, we force targeted random noise into our acquisition batch by probabilistic sampling with a bias towards higher switch events. Specifically, we model the switch event distribution with a two component gaussian mixture model and sample from the component with the higher switch event mean.

Building on the classification of \cite{toneva2018empirical} into forgettable and unforgettable data, we model each sample within the unlabeled sampling pool to be either less-switching $U$ or frequently switching $F$. The total amount of switch events $S_{total}$ can then be modeled as the sum of the switch events from both less switching $S_U$ and frequently switching samples $S_F$:

\begin{equation}
\label{eq:sevents-rv-modeling}
S_{total} = S_U + S_F
\end{equation}

When we query from the unlabeled pool, we would ideally target the most informative samples or, more specifically, the samples originating from class $F$.
For this reason, we fit a gaussian mixture model with two components (one for $U$ and one for $F$) to the switch event distribution and assign the less-switching distribution and frequently switching distribution to the gaussian mixture with the lower and higher mean respectively. During selection, we distinguish frequently switching samples from less switching samples with the separate gaussian components and sample the next acquisition batch from the frequenly switching distribution. Mathematically, we model the switch event distribution as

\begin{equation}
\label{eq:rv-modeling}
S_{total} \sim \omega_u N_u(\mu_u, \sigma_u) + \omega_f N_f(\mu_f, \sigma_f)
\end{equation}

where $\omega_{u/f}, \mu_{u/f}$, and $\sigma_{u/f}$ refer to the weight, mean, and  standard deviation of the singular distribution components $U$ and $F$ respectively. We subsequently establish the acquisition batch by sampling data points with probabilities from the second switch event distribution component $N_f$:

\begin{equation}
\label{eq:acq-rv-modeling}
X^* \sim N_f(\mu_f, \sigma_f)
\end{equation}

In Figure~\ref{fig:rv-toy}, we show a toy example of our approach. Specifically, we show a histogram over the switch events occurring for a given round and sketch the ideally fitted gaussian mixture components over the histogram distribution. In this case, we would sample from the right (green) gaussian component. We call our method \emph{Gaussian Switch Sampling} or \texttt{GauSS} in short.

\begin{figure}[!h]
    \begin{center}

        \includegraphics[scale=0.50]{Fig/rv-toy.png}
    \end{center}
    \caption{Toy example of gaussian switch sampling. We sketch the switch event distribution within a single active learning round, as well as two gaussian mixture components. The next batch is generated by sampling from the most switching component in green.}
    \label{fig:rv-toy}
\end{figure}
\section{Related Work}
\subsection{First Order and Second Order Active Learning}
In active learning, a strong research branch involves exploring different acquisition functions and their definitions of information content. In this context, the majority of existing approaches leverage the model representation directly to define the next acquisition batch. Due to the direct dependency, we refer to these approaches as first-order acquisition functions. To the best of our knowledge, second-order approaches remain largely unexplored.

Overall, active learning approaches differ in their definition of information content for samples within the unlabeled data pool. In this context, several approaches define information content with generalization difficulty. As an example, several approaches define sample importance using softmax probabilities \cite{wang2014new, roth2006margin} where information content is related to the output logits of the network. For instance, entropy sampling queries samples that have the highest softmax entropy while least confidence sampling queries samples with the lowest prediction probability (or confidence) score. Closely related to this concept, the authors in~\cite{schohn2000less, tong2001support} query samples based on decision boundary proximity. For instance, \cite{tong2001active} query samples with support vector machines. \cite{gal2017deep} define importance as model uncertainty and select the following sample batch based on Monte Carlo dropout. Finally \cite{Beluch_2018_CVPR} use an ensemble of classifiers to query the next set of samples. In contrast, other approaches define information content based on data representation within the dataset. Acquisition functions in this field are based on the assumption that representative samples best approximate the overall dataset structure and improve performance. In this context, a large group of approaches focus on constructing the core-set of the unlabeled data pool \cite{sener2017active, longtailcoreset}. Furthermore, \cite{gissin2019discriminative} use a discriminative approach, where the authors reformulate the active learning problem as an adversarial training problem. Specifically, they formulate active learning as a binary classification problem and select samples that minimize the differences between the labeled and unlabeled pool. Finally, there are several approaches that consider the combination of both data representation and generalization difficulty within their definition of information content. In several cases \cite{ash2019deep, haussmann2020scalable}, an ordering is established based on generalization difficulty and the representative component is introduced by sampling interactively from the ordering. \cite{batchbald} integrate both generalization difficulty and data representation by extending \cite{houlsby2011bayesian} to diverse batch acquisitions. Finally, \cite{baram2004online, hsu2015active} deploy "mix and match" meta-active learning approaches, that switch strategies each active learning round.

Even though several approaches are effective, they directly depend on the model representation of the data and are susceptible to setup shifts or out-of-distribution samples.

\subsection{Out-of-Distribution Analysis}
Within the field of deep learning, the importance of out-of-distribution samples within the context of robustness is well established \cite{hendrycks2019robustness}. In this context, several approaches involve measurements of uncertainty \cite{gal2015bayesian, kendall2017uncertainties, van2020uncertainty}, gradient representations \cite{kwon2019distorted, kwon2020backpropagated, kwon2020novelty, lee2021open, gradientsuncertainty, prabhushankar2020contrastive, prabhushankar2021contrastive, alregib2022explanatory}, or model calibration \cite{guo2017calibration, prabhushankar2021introspective}. For instance, \cite{van2020uncertainty} reject out-of-distribution samples by ranking samples uncertainty scores from a deterministic neural network. \cite{kwon2020backpropagated}, detect anomalies with by imposing a gradient constraint on the model that distinguishes inliers from outliers, \cite{guo2017calibration} introduce temperature scaling as an effective method to calibrate neural networks. With a few exceptions, out-of-distibution scenarios remain largely unexplored within the context of active learning with the scarce examples \cite{kothawade2021similar} being first-order approaches.

\subsection{Efficient Active Learning and Theory}
Apart from developing new active learning acquisition functions, there has been extensive research regarding efficient active learning as well as its practical implementation. \cite{coleman2019selection} improve computational efficiency by approximating the target model with a smaller proxy model. Further, \cite{haussmann2020scalable} consider scalability issues of active learning for object detection. In particular they investigate the implications of varying diverse and uncertain active learning components on large-scale data.

A substantial research direction is further dedicated to theoretical aspects of active learning. For instance, \cite{wang2018optimal, ting2018optimal, han2016local} consider subsampling in regression problems. Several works also focus on understanding the active learning process itself. In this context, \cite{farquhar2021statistical} investigate the implicit bias inherent within the active learning process.

In this work, we explore the capabilities of different protocols to assess information content. This represents a substantial contribution for designing efficient active learning paradigms as well as theoretic explanations for a acquisition function performance.

\subsection{Continual Learning}
In addition to active learning, our work is related to the field of continual learning and catastrophic forgetting \cite{chen2018lifelong, kirkpatrick2016overcoming, ritter2018online, toneva2018empirical}. In this context, significant research efforts involve reducing forgetting through model constraints \cite{aljundi2018memory, kirkpatrick2016overcoming}, loss exploration \cite{shi2021overcoming}, or augmentation \cite{benkert2021explaining, benkert2022example}. While the approaches effectively reduce forgetting, they do not consider (or estimate) information content in active learning settings

Most significantly, our work shares a strong connection with the study of \cite{toneva2018empirical} on forgetting events. Within the article, the authors quantify neural networks forgetting in the training set and empirically show dependencies between forgetting and several important machine learning topics such as dataset compression and label noise. Even though the paper provides an interesting framework for quantifying neural network forgetting, the quantification requires labels and is unsuitable for information content estimation in practical active learning settings. In contrast, our work expands the concepts of \cite{toneva2018empirical} to function without annotation and develop a grounded definition of information content for active learning.

redpajama_set_name: RedPajamaArXiv


August 1, nightly recap of the minor leagues.
The AZL Cubs had a double header on Wednesday, with each game being seven innings a piece. The pitching did a great job in the first one winning 4-0. The 43rd overall pick in this past years draft Pierce Johnson made his first appearance of the season. He started and pitched one scoreless inning. Carlos Martinez-Pumarino improved to 4-0 pitching four scoreless innings in relief. He gave up three hits, one walk, and struck out three, his ERA on the season is 0.93. Steve Perakslis secured the shutout by striking out five in two scoreless innings. Albert Almora was back in the lineup and leading off on Wednesday. He delivered going 3-for-4 with a run scored and an RBI. Jorge Soler was 2-for-4 with a run scored and three stolen bases. Ben Carhart had a double and an RBI.
The AZL Cubs had a good time of it in game two as well defeating the AZL Indians 7-2. Ben Carhart was 2-for-4 with twor runs scored and Yasiel Balaguert also had two runs with a run scored. Trevor Gretzky had one hit and an RBI and Neftali Rosario collected two RBIs without getting a basehit. Albert Almora didn't have a hit, but collected his sixth RBI of the season and Jorge Soler was 0-2, but walked twice and scored two runs. Brian Smith started for the Cubs and thew two perfect innings while striking out two. Loiger Padron improved his record to 3-0 with his win in relief. he gave up two runs, but struck out six in three innings of work. Alvido Jimenez closed out the game striking out three in two scoreless innings.
Boise was back at it and won again on Wednesday. After getting a big lead early, they gave up nine runs in the last four innings, but held on to win 14-10. There were a lot of good things happening offensively in this one. Xavier Batista was 3-for-5 with two doubles, a home run, and three RBIs. Jeimer Candelario and Dong-Yub Kim each had two hits and drove in three as well. Stephen Bruno hit his first home run of the season, a two run shot in the third inning. Daniel Vogelbach and Trey Martin each two hits with Martin scoring three runs. Lance Rymel had quite a nice night going 3-for-4 with an RBI. It was a tale of two games on the mound. Su-Min Jung started and pitched well allowing one run on three hits in three innings, while striking out three. Nathan Dorris got his first win of the season as he pitched the next two scoreless innings. Unfortunately Tyler Bremer and Willengton Cruz both struggled, but a win is a win.
Peoria won a crazy one on Wednesday scoring three in the top of the ninth to win 11-10. The day belonged to Dustin Geiger. Geiger was 3-for-5 with a double, two home runs and eight RBIs, yes eight RBIs. He stole the show early and finished it late with a two run bomb that turned out to be the game winner. Javier Baez had two more hits, including a double. He also scored two runs and drove one in. Zeke DeVoss and Pin-Chieh Chen each had two hits, with Chen scoring three. Zach Cates started and didn't have his best stuff, but came away with a no decision. Bryce Shafer got the win striking out four of the five outs he recorded. Austin Reed picked up his first save of the season pitching a scoreless ninth. Congrats to Geiger on just an ubelievable game.
Daytona scored one in the fifth and one in the seventh to win a pitchers dual 2-1. Lendy Castillo continued his comeback throwing four innings, giving up three hits and allowing an unearned run. He had four strikeouts in the start. Eduardo Figueroa got the win, his third of the season pitching three scoreless innings. Scott Weismann and Jeffrey Lorick combined to pitch the last two innings, with Lorick earning his second save of the season. Rubi Silva was 1-for-4, but drove in the two Cubs runs on Wednesday. Ronald Torreyes continued hitting the ball going 2-for-3. John Andreoli and Taylor Davis each had two hits in the contest. It was a good win for Daytona.
Because of rain Tennessee had another double header on Wednesday. The first one was a continuation of the night before. The Smokies got two in the fifth, winning 4-2. There were a couple bright spots on the mound. Dae-Eun Rhee started and took a no-decision pitching four solid innings only allowing one run on three hits. Trey McNutt got the win in relief pitching three innings allowing one run on two hits and Kevin Rhoderick recorded his eighth save of the season striking out three in two scoreless innings. Justin Bour was 1-for-4 with another RBI. Greg Rohan went 1-for-2 with an RBI and Nate Samson was 1-for-2 with a home run and two RBIs.
One was all Tennessee needed as Justin Bour was the hero hitting a walk off home run in the bottom of the seventh. He now has 12 home runs and 86 RBIs on the season. Dallas Beeler started and was tremendous allowing only one hit in five scoreless innings of work. Brian Schlitter picked up his second win in relief pitching the final two frames. Junior Lake, Nelson Perez, and Matt Cerda all had one hit a piece in the victory. Two wins to start the month is the right way to start it.
Iowa was able to score six, but an eight run inning was way too much to come back from losing 11-6. We'll start with the offense in this one. Brett Jackson was 3-for-5 with a double, triple, and a run scored. Josh Vitters hit his 16th home run of the season and had two RBIs. Greg Rohan, who was just promoted from AA was 3-for-4 with two home runs and three RBIs. James Adduci collected two more hits including a triple and an RBI. Ty'Relle Harris got the start and took the loss as he was not very sharp. Michael Bowden did a very nice job striking out six in three innings and recently acquired Jaye Chapman threw a scoreless inning in relief.
All around it was a nice night for the Cubs minor leaguers with the only loss coming in Iowa.
After a busy trade deadline here is what the Cubs minor league affiliates did on July 31.
The AZL Cubs got back on track after a day off beating the AZL Royals 8-2. Ben Carhart and Daniel Lockhart each had two, a run scored, and an RBI. Jacob Rogers was 1-for-4 with a run scored and two RBIs. Albert Almora was leading off tonight, but left after his first at-bat. He was replaced by Yasiel Balaguert who had a nice night going 2-for-4 with a run and an RBI. Jorge Soler did not have a hit, but he did drive in his seventh run of the season and he stole his third base. Carl Lang got the start and pitched an impressive four innings allowing no runs on four hits and struck out four. Jasvis Rakkar pitched one scoreless inning striking out one. Ethan Elias got the win, but did not have a great night giving up two runs on five hits in two innings. Daniel Adrian finished the game off striking out two in the last two frames. Obviously we are a bit concerned as to why Almora left the game. When we have an update we will let you know, but it was a nice night for the Cubs.
After winning a few 1-0 games the Boise Chiefs held on to win a shootout 10-9. It was a wild game as Boise had a big lead, but gave it up and had to win in walk-off fashion. The Cubs had a few nice nights at the plate highlighted by Daniel Vogelbach's first home run at Boise, a three run shot with two outs in the third. Stephen Bruno had three hits and two RBIs, raising his average to .324. Jeimer Candelario was the hero going 3-for-4 with a double and three RBIs including a walk-off double. Dong-Yub Kim and Trey Martin each had two hits on the night and Marco Hernandez connected for his third home run of the season. The Hawks pitching struggled most of the night, but the offense did the the job. Eduardo Orozco blew a save, but ended up getting his second win of the season. it is nice to see Boise maybe turning things around.
Peoria never got things going against Quad Cities losing 5-1. Michael Jensen got the start and pitched pretty good, although he took the loss. He pitched five innings surrendering two runs on three hits, while striking out six. Jeffry Antigua continued to impress pitching two more scoreless innings. Pete Levitt struggled in his one inning of work. There wasn't much excitement at the plate for the Chiefs on Tuesday. Javier Baez was one of the few bright spots going 2-for-4 with the Peoria's lone run scored. Bijan Rademacher also had two hits including his third double of the season. Nothing too exciting happening for Boise tonight.
Daytona won a close one on Tuesday beating Fort Myers 5-4. Daytona pounded out 14 hits so there were a lot of nice nights at the plate. It was good to see Timothy Saunders get back on track going 3-for-4 with double, runs scored, and an RBI. Ronald Torreyes continued his hot stretch of late adding two more hits, including his 19th double of the season and two RBIs. Matt Cerda, Rubi Silva, and John Andreoli each had two hits with Andreoli driving one run in. Austin Kirk started and got the win allowing four runs in five innings. Kirk was not as sharp as he has been in the past walking five batters, but he did strike out five. The relief was outstanding on Tuesday. Hunter Cervenka and Jeffrey Lorick combined to pitch three scoreless innings, striking out a batter each. AJ Morris closed the deal striking out two in one scoreless inning, notching his fourth save of the season.
Tennessee's game was suspended because of rain. The Smokies were winning 2-1 in the fourth and will be continued tomorrow as part of a double header.
Iowa dropped another to New Orleans losing Tuesday 5-3. Frankie De La Cruz started and took the loss giving up three runs in four innings. Jairo Asencio and Marcus Hatley did a nice job in relief both throwing a scoress inning. Dave Sappelt was the main bright spot for the Cubs offense going 3-for-4 with two doubles and two runs scored. Brett Jackson hit his 21st double of the season and drove in his 46th run as well. James Adduci had another hit and is hitting .375 since his promotion to Triple-A. As I have said recently Iowa just can't seem to get things going in the right direction.
The July 25th Chicago Cubs Minor League recap.
The most exciting team in the history of the Universe was off tonight, that's right no Almora or Soler tonight.
The Boise Hawks and Yakima Bears went toe-to-toe with each other and Yakima was the last one standing. The Boise Hawks were in comeback mode all night as they trailed 6-3, and 8-6 before tying the game each time. The Hawks scored twice in the bottom of the 8th to tie the game at eight but gave both runs right back in the top of the ninth and fell to Yakima 10-8. Felix Pena got the start for Boise and struggled. He gave up six runs, four earned on seven hits while walking none and striking out two in four innings of work. The loss however went to Matt Iannazzo who gave up four runs on seven hits in 1.1 innings of work. Daniel Vogelbach is at a new, higher level in the Cubs system but continues to pound the baseball. Vogelbach was 3-for-5 in the loss with a double and an RBI. Rock Shoulders, Dong-Yub Kim, and Gioskar Amaya all had two hits for Boise in the loss, Kim hit is fourth home run of the season.
Another game and another outstanding effort from Peoria Chiefs starter Joseph Zeller as he weathered a rough first inning to earn his second win of the season as the Chiefs hung on to beat Cedar Rapids 8-6. The Chiefs used an enormous seven run third inning to break this one open, but had to fight to hang on as Cedar Rapids scored twice in the 8th and 9th to make it close. Zeller went six strong innings giving up two runs, both unearned on seven hits while walking one and striking out two. He lowered his ERA to 1.32 on the season. Javier Baez was the only Chiefs player with multiple hits as he was 2-for-5 at the plate with an RBI and a run scored. Pin-Chieh Chen had the only extra base hit for the Chiefs as he was 1-for-3 on the night with a triple (8), two walks, an RBI, and a run scored.
Busy day for the Daytona Cubs as they had a double-header against the Brevard County Manatees, and game one didn't get the day off to a great start. P.J. Francescon had a rough outing, and has been shaky since his promotion to Daytona which is to be expected. It's not like Francescon has been terrible, but today he just didn't have it as he gave up six runs on seven hits over 3.2 innings while walking four without a strikeout. The Daytona offense couldn't keep up as Brevard County took game one 6-4. It was the Elieser Bonne show in game one for Daytona as the right fielder was 4-for-4 at the plate with two doubles (8,9), unfortunately no one was on base for him. Rafael Lopez had a solid game going 1-for-2 with a home run (1), two walks and two RBI's in the losing effort.
Game two was a pitching duel right from the get go, unfortunately for Daytona the result was the same, as Brevard County won the second half of the double-header 3-0. Frank Del Valle got the start for Daytona and was outstanding going five scoreless innings giving up only one hit while walking three and striking out four, but had to settle for a no decision. Jeffrey Lorick pitched a scoreless inning of relief before Scott Weismann got lit up in the 7th to take the loss. Matt Szczur had a solid game at the plate going 2-for-3 to raise his average to .292 on the season, he also stole his 38th base of the season. That was it as Daytona couldn't push across any runs and only managed three hits as this was a lost day all around.
The Smokies came out swinging against the Suns in a wild one and broke through late to register a 7-5 victory. Dae-Eun Rhee who has not been very good (understatement) all year, pitched well enough in this one to earn his seventh victory of the season against five losses. Rhee was the beneficiary of an offensive explosion that was led by Junior Lake. Lake was 2-for-5 with two home runs (6,7), a walk, and four RBI's. Lake wasn't alone in the destruction of Suns pitching as Nelson Perez went 2-for-5 as well with his fourth home run of the season, and two RBI's. Justin Bour continued his solid 2012 season going 3-for-4 with a double (29!) and an RBI, Bour is now hitting .297 on the campaign. Trey McNutt has had a down year, that includes his recent performances out of the pen, but he was excellent tonight going two strong innings giving up nothing and striking out three.
The Iowa Cubs jumped out to an early 3-0 lead against Oklahoma City, but if we have learned anything we know that there is no lead too big for Iowa to blow. It happened again tonight as OKC scored in the 3rd, 4th, and 5th to take a 7-4 lead on their way to a 10-8 victory. The I-Cubs made things somewhat interesting in the ninth inning by scoring four times but it was way too late. Chris Rusin got roughed up again as he gave up seven runs on ten hits in five innings of work to record his 7th loss of the season. James Adduci, Dave Sappelt, and Adrian Cardenas all had two hits in the loss. Adduci had a great game at the top of the order going 2-for-4 with a double (3), an RBI, a walk, and three runs scored.
July 5 Cubs minor league recap.
AZL Cubs played a very entertaining game on Thursday winning 12-11 in ten innings. After giving up ten runs in the first three innings and being down by five the Cubs came storming back to tie the game and eventually win. There were a lot of good things that happened on the offensive side of things. Leading off Timothy Saunders was 5-for-6 with a home run, double, three RBI's and four runs scored. Rafael Lopez was 3-for-4 with a double, two runs scored, and one batted in. Izaac Garsez also had three hits including a double, a run scored, and one batted in. Garrett Schlecht and Daniel Lockhart each had two hits and Lockhart had two RBI's and two runs scored. Justin Berg got the start and wasn't very good pitching only one inning and giving up six runs, three earned on four hits. Nathan Dorris pitched three scoreless innings and Steve Perakslis recorded the win pitching two innings. It was a wild game and it is nice to see the the Cubs come out on top.
Boise scored two in the ninth to win 7-6. Down by one going to the bottom of the ninth The Hawks were trailing 6-5. After a Dong-Yub Kim and Xavier Batista walk Darien Martin bunted them to second and third. Gioskar Amaya delivered a single to tie the game. After a fielders choice up stepped Stephen Bruno who came through with a single to win the game. Amaya also had three hits and scored two runs and drove in one. Bijan Rademacher was 2-for-3 including his first home run of his professional career. Marco Hernandez and Kim each had two hits and Hernandez had two RBI's. Bryce Shafer got the win in relief, despite giving up an unearned run in the top of the ninth. Starter Jose Arias pitched quite well, giving up one run on seven hits in 5.2 innings pitched. He was in line to get the win, but Michael Hamann struggled in relief. All in all it was a nice win for Boise on Friday.
Peoria scored two runs in the second and the pitching staff was able to make it stand winning 2-1. Zach Cates got the start, but did not factor in the decision as he was only able to pitch 4.2 innings. Zachary Rosscup came in relief and pitched three great innings striking out five and only allowing a hit. Luis Lira came in to close the game and recorded his fourth save of the season. Dustin Geiger started off where he left off hitting a solo home run in the second. After the homer, Javier Baez stepped to the plate and delivered a single. He then stole his 12 base of the season and advanced to third on a throwing error by the catcher. Wes Darvill drove him in on a ground out. Zeke DeVoss and Pin-Chieh Chen each had a hit and DeVoss stole his 21st base of the year.
Kyler Burke pitched ok on Thursday, but took the loss nonetheless. He gave up three runs, two earned on seven hits in five innings of work. Offensively John Andreoli had a really nice day. He was 3-for-4 and stole his 29th base of the season. Matt Cerda also reached base three times with a double and two walks. He also drove in two runs and scored one. Arismendy Alcantara led off and was 2-for-5 with an RBI. Brian Schlitter pitched an effective two innings in relief not allowing a run and giving up two hits while striking out one. His ERA is now down to 1.33.
The Tennessee Smokies game was postponed on Friday. It will be made up as a doubleheader tomorrow. Each game will be seven innings.
Iowa was shutout on Thursday 3-0. Brooks Raley got the start and took the loss pitching 6.2 innings and allowing three runs, one earned on five hits. Offensively there isn't much to talk about. Diory Hernandez was the only Cubs player to have a multi-hit game going 2-for-3. Alfredo Amezaga, Dave Sappelt, and Josh Vitters, and Brian Esposito all had one hit a piece on Friday. Alberto Cabrera and Ryan Rowland-Smith each pitched well in relief. Nothing too exciting about this one.
A look at the Cubs minor leaguers on Sunday, July 1.
The AZL Cubs held on to beat the AZL Athletics 9-7 on Sunday. Dan Vogelbach kept up his torrid pace going 4-for-5 with a triple, homer, two runs scored, and three RBI's. He is now hitting .471. Timothy Saunders had two more hits and and also drove in one run. His average now stands at .483. Carlos Escobar and Izaac Garsez each had two hits and a run scored on Sunday. Ben Carhart and Carlos Penalver each had an RBI in the victory. Nathan Dorris got the win pitching a scoreless inning in relief and Alvido Jimenez recorded his first save of the young season.
In a well pitched game Boise came up short losing 2-1 to Tri-City in ten innings. Rafael Diplan struggled with his comand walking three in .2 of an inning to take the loss. James Pugliese got the start and struck out five in four scoreless innings. Eduardo Orozco and Michael Heesch each pitched two scoreless inning on Sunday. Not much to talk about offensively in this one as Boise could only plate one. The lone RBI came off the bat of Dong-Yub Kim who was 2-for-4 with game-tying solo homer in the seventh. Jeimer Candelario and Chadd Krist continued their hot hitting as Candelario had two hits and Krist had three.
Starling Peralta pitched eight solid innings allowing three runs, two earned on four hits, but took the loss as Peoria lost to Kane County 7-2. As good as Peralta was Larry Suarez was not so good in relief giving up four runs in one inning of work. The offense didn't do much damage, but Taiwan Easterling was 3-for-3 and Dustin Geiger had two hits of his own. Chiefs got RBI's from Zeke DeVoss and Wes Darvill. Top prospect Javier Baez collected a hit and scored a run.
Matt Loosen and Daytona fell to Lakeland 5-3 on Sunday. Loosen started and gave up four runs, three earned in six innings pitched. A.J. Morris pitched well in relief striking out two in two innings of work. From an offensive standpoint Daytona couldn't get much done recording only four hits. Rebel Ridling was 2-for-3 with two doubles, a run scored, and an RBI. Matt Szczur also had a double, scored a run, and drove one in. The other Cubs hit came from Elieser Bonne and the other RBI came off the bat of John Andreoli.
Tennessee couldn't get much going at the plate and fell to Chattanooga 3-0. Nicholas Struck pitched well enough to win, but took the loss, giving up two runs, one earned in five innings pitched. Casey Harman and Casey Weathers each pitched in relief. Not too much excitement from the offense, but Justin Bour was 3-for-4 with two doubles. Nate Samson had two hits of his own and Jae-Hoon Ha and Juan Apodaca each had a hit on the night.
Iowa's frantic rally comes up short on Sunday dropping a high scoring game to Nashville 13-12. At one point the Cubs were down by nine runs headed to the sixth, but after scoring eight runs in three innings, they failed to score in the ninth. Offensively Iowa pounded out 16 hits and 12 runs. Adrian Cardenas and Josh Vitters each had three hits, including two doubles a piece, two runs scored, and an RBI. Blake Lalli also had three hits including a double, two runs scored, and three RBI's. Dave Sappelt hit his fourth home run of the year and Matt Tolbert drove in two runs and also two hits. Brett Jackson raised his average to .261 going 2-for-5 with a double, triple, two runs scored, and two RBI's. Frankie De La Cruz got the start and, but didn't last long giving up nine runs, seven earned in just 4.1 innings pitched. Jay Jackson threw another scoreless inning out of the bullpen and also struck two out. Michael Bowden struck out three in a scoreless inning.
A rough day for the Cubs minor leaguers dropping every game except the AZL.
Your nightly trip around the bases for June 28th, 2012.
AZL Cubs had a nice night winning 11-2. Jacob Rogers had a great night at the plate going 4-for-4 with two home runs, four runs scored, and four RBI's. Trey Martin had three hits and three RBI's on Thursday. Ben Carhart and Carlos Penalver each had two hits. Dan Vogelbach and Justin Marra each had a double and drove in a run. Carlos Martinez-Pumarino got the win pitching five innings in relief giving no runs on three hits while striking out five. Nathan Dorris and Ryan Hartman each pitched a scoreless inning in relief. Hope to see more nights like this from the Cubs as the summer goes forward.
We have been waiting, and Boise finally delivered beating Spokane 12-2. Chadd Krist was 4-for-5 with two doubles, three runs scored, and one driven in. Wilson Contreras also was 4-for-5 with a double, two runs scored, and three driven in. Stephen Bruno had two hits with a run scored and an RBI. Dong-Yub Kim hit his second home run of the season, a grand slam. Ian Dickson was the beneficiary of all those runs earning his first win of the season pitching five innings, giving up two runs, one earned on four hits, while striking out six. Eduardo Orozco, Rafael Diplan, and Su-Min Jung all pitched well in relief. All in all it was a very good night for the Hawks.
Gerardo Concepcion was roughed up early, giving up four runs in the first, but settled down as he pitched six inning giving up five runs on six hits. He took the loss and is now 2-6 with a 7.39 ERA. Javier Baez had a great night at the plate going 3-for-4 with two doubles, a run scored, and three stolen bases. Dustin Geiger, Taiwan Easterling, and Oliver Zapata all had RBI's on the night and Paul Hoilman had two hits. Larry Suarez also struggled in relief giving up two runs on four hits in two innings of work.
It was a great night for the Daytona offense beating Dunedin 14-4. Cubs got home runs from Ronald Torreyes, Arismendy Alcantara, and Rebel Ridling. Arismendy went 3-for-5 with three runs scored and four RBI's. Rubi Silva was also 3-for-5 including two doubles and drove in two. Lead-off hitter Matt Szczur was on base five times. He was 3-for-3 with two walks, two RBI's, three stolen bases, and three runs scored. Robert Whitenack got the start but didn't get the win pitching only four innings while giving up three runs, only one earned on six hits while striking out four. A.J. Morris got the win in relief pitching two scoreless innings while striking out one. Brian Schlitter threw another good outing in relief pitching a scoreless ninth.
Trey McNutt was not able to get out of the first inning on Friday pitching only .2 inning and giving up eight runs, four earned on six hits. Ty'relle Harris threw four quality innings of relief giving up two runs, one earned on five hits, while striking out two. The Smokies were pretty good at the plate scoring six runs on twelve hits. Jae-Hoon Ha, Junior Lake, Michael Brenly, and Elliot Soto all had two hits on the night. Nelson Perez and Brenly each hit a home run. The Smokies also had RBI's from Ha, Lake, Justin Bour, and James Adduci.
Chris Volstad pitched pretty good on Friday, but it wasn't enough for Iowa losing 3-1. Volstad gave up three runs, two earned on six hits in seven innings. He also struck out seven and walked zero. Matt Tolbert was great at the plate going 4-for-5 with a double, triple, and a run scored. Adrian Cardenas, Ty Wright, and Blake Lalli each had two hits. Josh Vitters was finally held hitless and Brett Jackson went 1-for-5. Jay Jackson struck out two and didn't allow a baserunner in one inning of work out of the bullpen.


Then download my 3-part baby sleep video series to discover how you can help your child sleep through the night.
View cart "Sleep Like an Angel – Sleep Secrets of the Mindful Mama and the Blissful Toddler – Twelve months to Four years" has been added to your cart.
I offer a variety of consultation services customized to your needs. You can view the options by going to Work With Kate.
I'm Kate Turner, founder of Sleep Like An Angel - The Gentle Baby Sleep Site. I'm an holistic baby sleep consultant, and author of the Sleep Like An Angel eBook series. I'm a mother and, just like you, I have struggled with the sleep deprivation and exhaustion that comes with having a baby.
In 2012, during my search to find a gentle baby sleep solution for my daughter, I ended up developing my own gentle and very successful sleep solution, currently being used by parents all over the world.
Join our Sleep Like An Angel community and let me help solve all your baby sleep dilemmas for good.
To Fix Them Right Now.
© 2016 Kate Turner - Sleep Like An Angel. All rights reserved.
8 Baby Sleep Mistakes That Are Keeping You Up At Night And How To Fix Them Right Now.
Learn how to help your baby sleep better right now. Download your FREE eBook here YES, PLEASE!


text:
Lion Ambassadors to highlight Biobehavioral Health Building for Campus Showcase
UNIVERSITY PARK, Pa. — The Penn State Lion Ambassadors will feature the Biobehavioral Health Building for their annual Campus Showcase event, "BBHealthyState."
"BBHealthyState" will take place from 1 to 4 p.m. Thursday, Sept. 22, at the Biobehavioral Health Building and HUB lawn. The event will promote overall health and wellness at Penn State in addition to highlighting the building.
"I'm very excited to feature BBH Building because I think that it's a unique and interesting building that people walk past every single day, but don't really know much about," said Catherine Tondi, project leader for the event. "I think it is also very important to promote health and wellness on campus because there are so many simple and fun ways to be active, so this event is a great way to share that with the community."
At the event, there will be a three-sided rock wall, free food, inflatables, tours of the building, obstacle courses, and giveaways for free gym memberships, Fitbits, and gift cards to Anthym and Rapid Transit. University Health Services, the BBH Society, Adventure Rec and other clubs and organizations will also be at the event. The event is free and open to the public.
"It's fun to feature a different building on campus each year because it gives students and the community the opportunity to experience a part of Penn State that they may not have been very familiar with before," said Katy Leslie, Lion Ambassadors executive vice president.
The Penn State Lion Ambassadors are the student alumni corps of the Penn State Alumni Association. The Lion Ambassadors communicate the University's history, personality and traditions to students, alumni and friends through tours and programming. Founded in 1982, the Lion Ambassadors serve as an important facet of student outreach for the Alumni Association. For more information, visit lionambassadors.com.
"BBHealthyState" will take place from 1:00-4:00 p.m. Thursday, Sept. 22, at the BBH Building and HUB lawn. The event will promote overall health and wellness at Penn State in addition to highlighting the building.
IMAGE: Penn State Alumni Association
Ashley Martin
aus20@psu.edu
Summer construction projects improve campus
Progress visible for Biobehavioral Health Building
Biobehavioral Health Building construction to begin Oct. 25
Activities and Traditions, Campus Life
BBHealthyState, BioBehavioral Health Building, Campus Showcase, Lion Ambassadors

redpajama_set_name: RedPajamaCommonCrawl


GENERAL ELEKTRIKS – Facing That Void on No Fat Clips!!!
The song is included in Cliquety Kliqk, the first album, released in Europe on Compost Records and in the US on Quannum Projects in August 2005.
La canzone è inclusa in Cliquety Kliqk, il primo album, uscito in Europa per Compost Records e negli Stati Uniti per Quannum Projects nell'agosto 2005.
DOWNLOAD: Scarica Facing That Void da Compost Media.
WATCH: Guarda Facing That Void sul sito dei General Elektriks.
MIRROR: Guarda Facing That Void sul sito di Arno Salters.
DOWNLOAD (LOW Q.): Scarica Facing that Void. Bassa qualità.
WATCH: Guarda Facing That Void su Quannum.
LINK: Visita il sito dei General Elektriks.
LINK: Visita il sito di Arno Salters.
Posted by DeK on Monday, June 26, 2006, at 12:37. Filed under Music Videos. Tagged Arno Salters, bullet time, martial arts, puppets, Tennessee Reid Norton. Follow any responses to this post with its comments RSS feed. You can post a comment or trackback from your blog.