text:
\section{Introduction}

After receiving paper reviews, authors may optionally submit a rebuttal to address the reviewers' comments, which will be limited to a {\bf one page} PDF file.
Please follow the steps and style guidelines outlined below for submitting your author response.

The author rebuttal is optional and, following similar guidelines to previous CVPR conferences, is meant to provide you with an opportunity to rebut factual errors or to supply additional information requested by the reviewers.
It is NOT intended to add new contributions (theorems, algorithms, experiments) that were absent in the original submission and NOT specifically requested by the reviewers.
You may optionally add a figure, graph, or proof to your rebuttal to better illustrate your answer to the reviewers' comments.

Per a passed 2018 PAMI-TC motion, reviewers should refrain from requesting significant additional experiments for the rebuttal or penalize for lack of additional experiments.
Authors should refrain from including new experimental results in the rebuttal, especially when not specifically requested to do so by the reviewers.
Authors may include figures with illustrations or comparison tables of results reported in the submission/supplemental material or in other papers.

Just like the original submission, the rebuttal must maintain anonymity and cannot include external links that reveal the author identity or circumvent the length restriction.
The rebuttal must comply with this template (the use of sections is not required, though it is recommended to structure the rebuttal for ease of reading).

\subsection{Response length}
Author responses must be no longer than 1 page in length including any references and figures.
Overlength responses will simply not be reviewed.
This includes responses where the margins and formatting are deemed to have been significantly altered from those laid down by this style guide.
Note that this \LaTeX\ guide already sets figure captions and references in a smaller font.

\section{Formatting your Response}

{\bf Make sure to update the paper title and paper ID in the appropriate place in the tex file.}

All text must be in a two-column format.
The total allowable size of the text area is $6\frac78$ inches (17.46 cm) wide by $8\frac78$ inches (22.54 cm) high.
Columns are to be $3\frac14$ inches (8.25 cm) wide, with a $\frac{5}{16}$ inch (0.8 cm) space between them.
The top margin should begin 1 inch (2.54 cm) from the top edge of the page.
The bottom margin should be $1\frac{1}{8}$ inches (2.86 cm) from the bottom edge of the page for $8.5 \times 11$-inch paper;
for A4 paper, approximately $1\frac{5}{8}$ inches (4.13 cm) from the bottom edge of the page.

Please number any displayed equations.
It is important for readers to be able to refer to any particular equation.

Wherever Times is specified, Times Roman may also be used.
Main text should be in 10-point Times, single-spaced.
Section headings should be in 10 or 12 point Times.
All paragraphs should be indented 1 pica (approx.~$\frac{1}{6}$ inch or 0.422 cm).
Figure and table captions should be 9-point Roman type as in \cref{fig:onecol}.

List and number all bibliographical references in 9-point Times, single-spaced,
at the end of your response.
When referenced in the text, enclose the citation number in square brackets, for example~\cite{Alpher05}.
Where appropriate, include the name(s) of editors of referenced books.

\begin{figure}[t]
  \centering
  \fbox{\rule{0pt}{0.5in} \rule{0.9\linewidth}{0pt}}
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
   \label{fig:onecol}
\end{figure}

To avoid ambiguities, it is best if the numbering for equations, figures, tables, and references in the author response does not overlap with that in the main paper (the reviewer may wonder if you talk about \cref{fig:onecol} in the author response or in the paper).
See \LaTeX\ template for a workaround.

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.
Please ensure that any point you wish to make is resolvable in a printed copy of the response.
Resize fonts in figures to match the font in the body text, and choose line widths which render effectively in print.
Readers (and reviewers), even of an electronic copy, may choose to print your response in order to read it.
You cannot insist that they do otherwise, and therefore must not assume that they can zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it is almost always best to use \verb+\includegraphics+, and to specify the  figure width as a multiple of the line width as in the example below
{\small\begin{verbatim}
   \usepackage{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.pdf}
\end{verbatim}
}

{\small
\bibliographystyle{ieee_fullname}

\section{Related Work}
\label{sec:related_work}
\noindent\textbf{Semi-supervised learning: }
Semi-supervised learning is the problem of leveraging a large corpus of unlabeled data along with labeled data to improve model performance. Berthelot \etal~\cite{Berthelot2019MixMatchAH, Berthelot2020ReMixMatchSL, SohnBCZZRCKL20} proposed a series of methods, each one simplifying and improving on the previous work, that have become a mainstay in semi-supervised learning. The latest in this line of work, FixMatch~\cite{SohnBCZZRCKL20}, generates pseudo-labels for weakly augmented unlabeled data and uses them as supervision for strongly augmented samples. We adopt a similar strategy of generating pseudo-labels for the unlabeled regions. Unlike image classification, object detection also has to deal with background regions and a naive application of such methods is sub-optimal.
All the above methods leverage predictions of a model, to train on unlabeled data. Such a pseudo-label generation can be harmful when the model makes a mistake. Pham \etal~\cite{PhamDXL21} proposed Meta Pseudo labels, which uses the signal from the student model's loss to improve the predictions of the teacher. In this work, we do not perform any such optimizations.
With the advent of self supervised learning, Chen \etal~\cite{ChenKSNH20} demonstrated that large models trained using self supervised learning can be good semi-supervised learners. Kim \etal~\cite{abs-2101-06480} improve the performance of FixMatch\cite{SohnBCZZRCKL20} by combining it with self-supervised Learning. Employing self-supervision offers an orthogonal direction for improvement and is out of scope of this work.
\\
\noindent\textbf{Semi-supervised object detection: }
Existing works on semi-supervised object detection, have focused mainly on consistency regularization~\cite{jeong2019consistency, tang2021proposal} or pseudo-labeling-based approaches~\cite{sohn2020simple, liu2021unbiased, xu2021end, yang2021interactive}.
The main idea behind these approaches is to perturb the images, or sometimes features, and apply a consistency regularization loss to enforce consistency between the predictions.
Among the pseudo-labeling based approaches, a standard practice is to train a teacher model on the available labeled data, and to use this teacher to predict labels on the unlabeled data to train the student model.  The teacher model's weights are updated in an exponential moving average fashion of the student's weights. Different methods apply different techniques and heuristics to improve the quality and stability of the generated pseudo-labels.
Our approach takes advantage of both these techniques. In contrast to these methods, our algorithm does not require an exhaustively labeled training set.

\begin{figure*}[!t]
\centering
\includegraphics[width=\linewidth]{latex/figures/saod_pipeline.pdf}
  \caption{Illustration of our algorithm for sparsely annotated object detection. Refer to Section~\ref{sec:approach} for more details.}
  \vspace{-1em}
\label{fig:pipeline}
\end{figure*}

\noindent\textbf{Sparsely annotated object detection: }
One of the initial works addressing this problem, by Niitani \etal~\cite{niitani2019sampling}, proposes utilizing logical relations between co-occurrence of objects and pseudo-labeling.
Yoon \etal~\cite{yoon2021semi} use object tracking to densely label objects across sparsely annotated frames along with single stage detectors to mitigate the negative effects of missing annotations.
Wu \etal~\cite{wu2018soft} propose a re-weighting approach where the gradients corresponding to region of interest are weighed as a function of overlap with ground truth instances.
Improving upon  the previous work, Zhang \etal~\cite{zhang2020solving} propose an automatic re-calibration strategy for single stage detectors where the negative branch is changed to take into account low confidence background predictions which might correspond to missing annotations.
Finally, one of the most recent works, Co-Mining~\cite{Wang2021CominingSL} uses a co-training strategy by using two views of an image and using predictions from one view and the ground truth to train the other over view and vice versa. While all the methods discuss thus far treat sparsely annotated object detection as a separate problem, we identify and pose it as a region based semi-supervised problem.\\
\vspace{-1.5em}
\section{Approach}\label{sec:approach}
\input{latex/approach_eqn}

\section{Experimental Evaluation}
\label{sec:experimental_evaluation}
\input{latex/experiments}

\input{latex/conclusion.tex}
\section{Acknowledgement}
This project was partially supported by DARPA SemaFor (HR001119S0085) and DARPA SAIL-ON (W911NF2020009), and Amazon Research Award to AS.
{\small
\bibliographystyle{ieeetr}

\section{Introduction}
\begin{figure}[!tbh]
    \centering
    \includegraphics[width=\linewidth]{latex/figures/Teaser_6.pdf}
    \caption{Semi-supervised object detection (SSOD) approaches assume availability of an exhaustively labeled set while sparsely annotated object detection (SAOD) approaches assume atleast one annotation per image. Our formulation of SAOD as semi-supervised object detection at region level makes the problem more general and removes any assumptions. Here FG and BG represent foreground and background respectively.}
    \label{fig:comp_teaser}\vspace{-1em}
    \vspace{-0.5em}
\end{figure}
Traditional object detection methods assume that the training dataset is exhaustively labeled. The performance of such detectors is sensitive to the quality of labeled data~\cite{Chadwick2019TrainingOD, wu2018soft, Li2020TowardsNO}. Noise in the data used for training object detectors can arise due to noisy class labels or incorrect/missing bounding boxes. In this work, we deal with the harder problem of missing class labels and bounding box annotations, i.e., presence of sparse annotations.
Sparsely annotated object detection (SAOD) is the problem of improving the robustness of object detection under the presence of missing annotations in the training data. This problem is of utmost importance in the current age, as obtaining crowd-sourced datasets~\cite{Deng2009ImageNetAL,Lin2014MicrosoftCC} can be expensive and laborious. The alternative is to use computer-assisted protocols to collect annotations~\cite{Kuznetsova2020TheOI}.  However, these methods suffer from noisy/missing labels~\cite{Kuznetsova2020TheOI}. Hence, it is imperative to adapt the current object detectors to address the sparsely annotated problem.

The crux of the issue is the way labels are assigned during training. The sparse annotations reduce the true positive samples and introduce false negatives into training, degrading the network performance~\cite{wu2018soft}. While this is an oversimplification, it gives a better idea of the problem being dealt. We build on a simple observation that SAOD is semi-supervised object detection (SSOD) at a region level.
However, the unlabeled data, which are regions containing foreground objects in our case, are not known a-priori and have to be identified.
Hence, a SSOD method cannot directly be applied for SAOD as current approaches~\cite{niitani2019sampling, yoon2021semi, wu2018soft, zhang2020solving, yang2020object, Wang2021CominingSL, Yan2021LearningFM, Yan2020UniversalLD, Zhang2020BeyondWS, Li2020ANL} assume a known unlabeled set along with an exhaustively labeled training set.
Similarly, recently proposed SAOD methods drop all images without a single annotation during training and cannot truly leverage the power of unlabeled data like SSOD methods.
We illustrate SSOD and SAOD in row 1 and 2 of Figure~\ref{fig:comp_teaser} respectively.
We posit that a sparsely annotated object detection method should be a good semi-supervised learner as the unlabeled images in SSOD can be treated as missing annotations for SAOD. We show this formulation in row 3 of Figure~\ref{fig:comp_teaser}.

Following popular self-supervised~\cite{He2020MomentumCF, pmlr-v119-chen20j, caron2020unsupervised, ChenH21, GrillSATRBDPGAP20}, semi-supervised~\cite{Ren2020UFO2AU, jeong2019consistency, sohn2020simple, tang2021proposal, xu2021end, liu2021unbiased, yang2021interactive} and sparsely annotated object detection~\cite{Wang2021CominingSL} literature, our proposed model operates on an image and its augmented counterpart. As previously mentioned, the problem then becomes one of identifying the unlabeled regions containing foreground objects (called unlabeled regions to distinguish from background regions), from the labeled ones and process each group differently.
For this purpose, we propose an end-to-end approach that automatically identifies the unlabeled regions.
Following successful semi-supervised methods~\cite{Berthelot2020ReMixMatchSL, SohnBCZZRCKL20, abs-2101-06480}, a consistency regularization loss is applied on the features of these unlabeled regions extracted from the original and augmented images.
While the available ground truth is used to supervise the training on the original image, a combination of its prediction on the original image, and the ground truth acts as supervision for the augmented counterpart.

For experimental evaluation, the standard practice is to artificially create sparsely annotated splits on detection datasets for training. A quick survey of recent SAOD methods~\cite{niitani2019sampling, wu2018soft, zhang2020solving, yang2020object, Wang2021CominingSL} reveals that there are at least five different ways to create such splits in practice. While each of them have their merits, it is difficult to compare methods, as approaches use only a subset of them. We evaluate our approach on all these splits and plan on releasing them to facilitate future research. Moreover, to demonstrate that our proposed method for SAOD can also perform competitively on SSOD, we report results on two standard splits of SSOD. Finally, we propose a novel setup for ``true" semi-supervised object detection where the dataset is not exhaustively labeled. We briefly summarize our contributions below.

\begin{itemize}
    \setlength\itemsep{0.1em}
    \item We formulate the problem of sparsely annotated object detection as a semi-supervised object detection at a region level.
    \item We demonstrate the effectiveness of this formulation by proposing an algorithm which achieves state-of-the-art performance on sparsely annotated object detection and competitive performance on semi-supervised object detection.

    \item We propose a new experimental setup that evaluates methods on sparsely annotated semi supervised learning. To the best of our knowledge, this is the first work which evaluates both semi-supervised and sparsely annotated methods on a common setup.
\end{itemize}

In Section~\ref{sec:related_work} we discuss some of the related works on SAOD and related fields. We describe our approach in detail in Section~\ref{sec:approach}. We describe our experimental setup and present results in Section~\ref{sec:experimental_evaluation} and conclude in Section~\ref{sec:conclusion} along with limitations and broader impact.

\section{Recall analysis}
To study the effect of missing annotations on the RPN, we compute the recall of the RPN trained using a sparsely annotated dataset. For a model trained using all the annotations, the recall of the RPN is $0.83$.
As we drop annotations progressively from 30\% to 70\% the recall drops to $0.79$.  This shows that the drop in recall due to missing annotations is not that significant. The reduction in performance is due to false signals given at the detector head due to missing annotations.

\section{Details of ablation dataset}
We perform all the ablation experiments on a randomly selected subset of the COCO 2017 trainval set~\cite{Lin2014MicrosoftCC}, referred to as COCO-mini. This is to avoid long training times during ablation experiments. To demonstrate that the conclusions made on this random subset holds for the larger COCO 2017 trainval set, we show the distribution of instances per class in Figure \ref{fig:annot_dist}. From the figure, one can see that the distribution of the randomly constructed set is not far from the complete training set.
\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{latex/figures/annot_dist.pdf}
    \caption{Fraction of images per class.}
    \label{fig:annot_dist}
\end{figure}

\section{Additional Experiments}
\subsection{Experiments on publicly released splits}
Authors of Co-mining~\cite{Wang2021CominingSL} create a split similar to Split-4 but using COCO 2017 trainval set to perform ablation experiments. To the best of our knowledge, this is the only split released publicly for SAOD. We compare the performance of our approach with Co-Mining, which uses a RetineNet architecture, on this split and report results in Table \ref{tab:cosplit}. We observe a similar trend as reported in the main paper and outperform Co-mining by $0.77$, $2.1$ and $3.01$ percentage points on the \textit{Easy}, \textit{Hard} and \textit{Extreme} subsets respectively.
\begin{figure*}[!t]
\centering
\includegraphics[width=\linewidth]{latex/figures/aug.pdf}
  \caption{Illustration of the effects of various augmentations used in this work.}
\label{fig:aug}
\end{figure*}

\begin{table*}[t]
\begin{minipage}[t]{0.45\linewidth}
\begin{minipage}[t]{\linewidth}
\setlength{\cmidrulewidth}{0.01em}
\renewcommand{\tabcolsep}{4pt}
\renewcommand{\arraystretch}{1.1}
\caption{Results on the splits released by authors of \cite{Wang2021CominingSL}.}\label{tab:cosplit}\vspace{-0.5em}
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}lcccc@{}}
\toprule
 \textbf{Method}& \textbf{Backbone} &\multicolumn{3}{c}{\textbf{Performance}} \\
 \cmidrule{3-5}
 & & \textit{Easy} & \textit{Hard} & \textit{Extreme}\\
 \midrule
 Co-mining~\cite{liu2021unbiased} & R50-FPN  & 37.60 & 34.40 & 23.10 \\
 Ours & R101  & \textbf{38.37} & \textbf{36.50} & \textbf{26.11}\\
\bottomrule
\end{tabular}
}
\end{minipage}\vspace{1em}

\begin{minipage}[t]{\linewidth}
\setlength{\cmidrulewidth}{0.01em}
\renewcommand{\tabcolsep}{4pt}
\renewcommand{\arraystretch}{1.1}
\centering
\footnotesize
\caption{Analysis on warm-up iteration for performing PPM}
\label{tab:warmup}\vspace{-1em}
 \resizebox{\linewidth}{!}{
\begin{tabular}{@{}lcccc@{}}
\toprule
 \textbf{Iteration}&  0 & 4000 & 9000 & 12000  \\
 \midrule
 AP & 18.50 & 18.68 & \textbf{18.82} & 18.69\\
    AP$_{50}$ & 35.08 & 35.52 & \textbf{35.98} & 35.68\\
\bottomrule
\end{tabular}
}
\end{minipage}
\end{minipage}\hfill
\begin{minipage}[t]{0.55\linewidth}
\setlength{\cmidrulewidth}{0.01em}
\renewcommand{\tabcolsep}{4pt}
\renewcommand{\arraystretch}{1.3}
\caption{Analysis of different augmentations used in this work.}\label{tab:aug}\vspace{-0.5em}
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}cccccc@{}}
\toprule
 \textbf{Contrast}& \textbf{Brightness} & \textbf{Saturation} & \textbf{Lighting}&\textbf{Bbox-Erase} & \textbf{Performance}  \\
 \midrule
 \xmark & \xmark & \xmark & \xmark & \xmark & 17.07\\
 \cmark & \xmark & \xmark & \xmark & \xmark & 17.42  \\
 \xmark & \cmark & \xmark &\xmark & \xmark & 17.36 \\
  \xmark & \xmark & \cmark & \xmark & \xmark & 17.03 \\
 \xmark & \xmark & \xmark &\cmark & \xmark & 16.69\\
  \cmark & \xmark & \xmark & \xmark & \cmark & 17.94  \\
  \midrule
 \cmark & \cmark & \xmark &\xmark & \xmark & 17.50 \\
  \cmark & \cmark & \cmark & \xmark & \xmark & 18.05  \\
  \cmark & \cmark & \cmark & \cmark & \xmark & 18.07  \\
 \cmark & \cmark & \cmark &\cmark & \cmark & 18.82 \\
\bottomrule
\end{tabular}
}
\end{minipage}\vspace{-1em}
\end{table*}

\subsection{Warm-up }
Since we rely on the RPN objectness score, we employ a warmup strategy which switches on the pseudo-positive mining during training. In Table \ref{tab:warmup} we start the pseudo positive mining at different iterations and report results. This experiment is performed on the COCO-mini ablation dataset. Our experiments show that for models which are trained for fewer iterations, starting at $9000$ iterations worked the best. For longer experiments we allow a warmup of $30000$ iterations.

\subsection{Augmentation}
The proposed approach process an input image and its augmented counterpart. For augmentation, a cascade of random contrast, brightness, saturation, lighting and bounding box erase are used. In Table \ref{tab:aug}, we analyze the effect of various augmentations on the performance of the model. This experiment was conducted on the COCO-mini ablation set. We observe that applying random saturation or random lighting alone are not as effective compared to other augmentations. Applying bounding box erase alone provides the most improvement. However, using random saturation and lighting along with random contrast and brightness improve the model. Finally, we achieve the best performance when we use all the augmentations. We randomly sample contrast, brightness, and saturation from $\left[0.5, 1.5\right]$. For lighting, a random scale of $1.2$ was used. For bounding box erase, we randomly erase an area of of $\left[0.4, 0.7\right]$ at an aspect ratio of $\left[0.3, 3.3\right]$. We show the effect of these augmentations on a sample image in Figure \ref{fig:aug}.
\vspace{-1em}
\section{Additional details of splits}
\noindent\textbf{Split-4: }The subsets of this splits are created as follows. For each image in the training set, the easy set randomly removes one annotation. The \textit{hard} set randomly removes half of the annotations while the \textit{extreme} set just retains one annotation per image. All the sets ensure each image consists of atleast one annotation. \\
\noindent\textbf{Split-5: } For each image in this construction, instances of labeled categories are exhaustively annotated while there could be categories with no annotation even if the instances of the category exist.

\begin{figure*}[!t]
    \centering
\begin{minipage}[t]{\linewidth}
 \centering
\includegraphics[width=0.8\linewidth]{latex/figures/PPM Positive.pdf}
\caption{Figure showing regions (white) identified by the PPM step and available ground truth (red) during training.}\label{fig:sup_ppm_pos}
\end{minipage}

\begin{minipage}[t]{\linewidth}
\vspace{1em}
\centering
\includegraphics[width=0.8\linewidth]{latex/figures/PPM negative.pdf}
    \caption{Failure cases of PPM step.}
    \label{fig:sup_ppm_neg}
\end{minipage}
\end{figure*}

\begin{figure*}[!tbh]
    \centering
    \includegraphics[width=\linewidth]{latex/figures/Detector Positive.pdf}
    \caption{ Qualitative results comparing the output of a "vanilla" model trained using available ground truths (top) to a model trained using our approach (bottom). Predictions with a class confidence score greater than 0.9 are shown.}
    \label{fig:det_pos}
\end{figure*}

\begin{figure*}[!tbh]
    \centering
    \includegraphics[width=\linewidth]{latex/figures/Detector Failure.pdf}
    \caption{Failure cases comparing the output of a "vanilla" model trained using available ground truths (top) to a model trained using our approach (bottom). Predictions with a class confidence score greater than 0.9 are shown.}
    \label{fig:det_neg}
\end{figure*}

\section{Results}
In this section, we show some more results of the regions identified by the PPM in Figure \ref{fig:sup_ppm_pos} and a few failure cases in Figure \ref{fig:sup_ppm_neg}. In Figures \ref{fig:sup_ppm_pos}, \ref{fig:sup_ppm_neg} the red boxes correspond to the available annotations and the regions in white are the ones identified by PPM. Finally, we show detection results and failures of our approach in Figures \ref{fig:det_pos} and \ref{fig:det_neg} respectively.
{\small
\bibliographystyle{ieee_fullname}

\section{Proposed Approach}
\label{sec:approach}

In this section we give the intuition in Section \ref{sec:intuition} followed by specifics of proposed approach. Our approach works with the original and augmented views of the input as described in Section \ref{sec:method}. For both of these inputs backbone features are extracted and concatenated for the proposed Common RPN described in Section \ref{sec:crpn}. Using the generated proposals a Pseudo-Positive mining step (Section \ref{sec:ppm}) is performed which helps separate out possible missing object regions. For the last part (Section \ref{sec:dethead}), at the detector head, the original image features are utilized to generate pseudo labels for the strongly augmented branch using the non-mined boxes. For the mined boxes a consistency regularization based loss guides the strongly augmented features to be similar to original image features. The main components of the pipeline are highlighted in Figure \ref{fig:pipeline}.

\subsection{Method}
\label{sec:method}

For every image in the training set $I$ with annotations $\mathcal{B}$, an augmented view $\widetilde{I}$ is generated. The augmentations are selected such that the annotations $\mathcal{B}$ are not affected i.e., the augmentations do not change the relative position of pixels. For augmentation a cascade of random contrast, brightness, saturation, lighting and bounding box erase with randomly selected parameters are used. A detector backbone network is employed to extract two features, $f_{I}$ and $f_{\widetilde{I}}$, from $I$ and $\widetilde{I}$ respectively.

\subsubsection{Common Region Proposal Network}
\label{sec:crpn}
Traditional two stage object detectors~\cite{Lin2017FeaturePN, Ren2015FasterRT, He2020MaskR} use a region proposal network (RPN)~\cite{Ren2015FasterRT} to generate region of interests (ROI). Unlike existing SSOD and SAOD approaches which pass the two views of the images twice through the RPN to get two sets of proposals, we modify the architecture to work on the concatenated set of features $\mathbf{f} = \mathbf{f}_{I} \oplus \mathbf{f}_{\widetilde{I}}$.
By having a common RPN, there are a common set of proposals for both views of the image. In the absence of a common RPN, a heuristic matching like Kuhn-Munkres algorithm~\cite{Kuhn55thehungarian} based on Intersection over Union (IoU) scores. Additionally, such heuristic approaches lead to slower computation and can introduce noise. Thus a common RPN makes it possible to perform the next stages of our pipeline in quick and accurate manner.
The RPN is trained with $\mathcal{L}_{rpn}$ which is a combination of BCE loss and smooth $L_1$ \cite{fastrcnn} loss for objectness classification and bounding box regression respectively.

\subsubsection{Pseudo Positive Mining}
\label{sec:ppm}
The proposals obtained in the previous section using the common RPN consist of foreground, background and regions corresponding to missing annotations. To differentiate the region proposals corresponding to missing annotations from the rest we propose Pseudo Positive Mining (PPM). PPM selects the proposal boxes which have an objectness score above a score threshold $\tau$. This ensures that the boxes which the model is confident about but which do not overlap with any ground truth due to missing annotations do not lead to unnecessarily penalizing of the detector in the later stages. It should be noted that the removal of these boxes can be done from the RPN loss computation as well but given no major difference in RPN recall for a vanilla model trained on full as well as sparse data  \s{Refer to Supp Table XX}, we only use the mined boxes to improve the detector head.

\subsubsection{Detector Head}
\label{sec:dethead}
For the original view $I$, $\mathcal{L}_{det}$ which is a combination of classification and regression based losses is computed using the ground truth labels. On the other hand before the loss computation for the augmented view $\widetilde{I}$, the ground truth annotations and the predictions generated using $I$ are merged. This step provides one way to get region level labels for the unlabelled regions using the pseudo labels to perform region level semi-supervised learning. While merging predictions we perform a class-wise NMS to remove repeating/highly overlapping predictions which are already present in the ground truth making sure a ground truth annotation is never removed.

Another way region level semi-supervised learning is performed is by utilizing the boxes mined at RPN. To do this these boxes are passed through the detector head, \s{make clear that there are unlabeled regions as o/p of normal branch whihc were missed and then there are those which PPM mined} first using $\mathbf{f}_{I}$ and $\mathbf{f}_{\hat{I}}$ respectively to get corresponding detector head features. Following this, consistency regularization based loss is applied between these sets of features as they correspond to the same regions. On extracting corresponding box features $\mathbf{f}_{I_1}, \mathbf{f}_{I_2} ...$ and $\mathbf{f}_{\widetilde{I}_1}, \mathbf{f}_{\widetilde{I}_2} ...$ the following consistency regularization loss is computed:
\begin{equation}
    \mathcal{L}_{cr} = \sum\limits_i\|\mathbf{f}_{I_i} - \mathbf{f}_{\widetilde{I}_i}\|^2_2,
\end{equation}
This loss enforces that the features of an original and augmented region should be consistent with each other.

The network is trained end to end with the following loss equation:
\begin{equation}
    \mathcal{L} = \frac{1}{2}\left(\mathcal{L}_{det} + \mathcal{L}_{det}^+\right) + \mathcal{L}_{cr} + \mathcal{L}_{rpn}
\end{equation}
Where $\mathcal{L}_{det}$ is computed between the non-mined predictions from $I$ and ground truth annotations while $\mathcal{L}_{det}^+$ is the loss between the appended ground truth annotations and the non-mined $\hat{I}$ predictions. Both $\mathcal{L}_{det} \text{ and } \mathcal{L}_{det}^+$ comprise of cross entropy based classification and smooth $L_1$ \cite{fastrcnn} based regression losses.

\noindent\textbf{Discussion: }During inference, we compute the backbone features ($f_I$ and $f_{\widetilde{I}}$) for both the original  and augmented versions of the input to obtain the region proposals and only $f_I$ is used for the final detections. We use a two-stage object detector to describe the proposed approach. Note that this method is not specific to two-stage object detectors and can be easily extended to one-stage detectors.
\section{Conclusion}
\label{sec:conclusion}
\vspace{-0.5em}
We present a novel formulation that casts sparsely annotated object detection as semi supervised object detection at a region level. We propose a simple yet effective technique for identifying the unlabeled regions using pseudo-positive mining and apply unsupervised loss on them.  We show the effectiveness of our approach with an exhaustive set of experiments on both sparsely annotated and semi-supervised object detection.  We propose a new setup, unifying both the problem domains, that evaluates sparsely annotated and semi-supervised methods on a common benchmark. \\
\noindent\textbf{Limitations: }Despite its impressive performance, one major limitation of our work is the introduction of additional hyperparameters and the model's sensitivity to them. Reporting results on multiple runs with confidence thresholds needs to be carried out. Another limitation of the current approach is the need for longer training times as each image is duplicated for generating the augmented counterpart.\\
\noindent\textbf{Broader Impact: } We do not foresee an immediate negative impact of the proposed approach.

\subsection{Data and Metrics}\label{sec:data}
We conduct all our experiments on the COCO~\cite{Lin2014MicrosoftCC} (2017 split) and PASCAL-VOC~\cite{pascal-voc-2007, pascal-voc-2012} (both 2007 and 2012 versions) datasets. The 2017 version of COCO~\cite{Lin2014MicrosoftCC} dataset consists of $118000$ and $5000$ images for training and validation respectively. Experiments on the PASCAL-VOC~\cite{pascal-voc-2007} 2007 version are conducted on $5011$ trainval images and performance is computed on $4952$ images of the test set. The PASCAL-VOC~\cite{pascal-voc-2007} 2012 version consists of $11530$ (trainval) images for training and evaluation is done on the PASCAL-VOC 2007 test set. Following past literature~\cite{niitani2019sampling, wu2018soft, zhang2020solving, yang2020object, Wang2021CominingSL} we create five different splits (See Section~\ref{subsec:split}) and report results on them. For splits on the COCO~\cite{Lin2014MicrosoftCC} dataset, we use the standard COCO style Average Precision (AP) which is AP computed at $11$ different IoU thresholds. For splits on the PASCAL-VOC~\cite{pascal-voc-2007} dataset, we use both the COCO style AP and the standard PASCAL-VOC style AP$_{50}$, which is Average Precision computed at an IoU threshold of $0.5$.

\subsection{Sparsely annotated splits}\label{subsec:split}
An extensive review of contemporary sparsely annotated object detection methods reveals that there are a total of five different splits in use for creating sparsely annotated training data on the COCO~\cite{Lin2014MicrosoftCC} and PASCAL-VOC~\cite{pascal-voc-2007, pascal-voc-2012} datasets combined. Most methods evaluate their models on a subset of them which makes it harder to compare across methods. Additionally, these methods haven't released their splits, so we created them in house to evaluate our method and the baselines. We will release all the splits for future research. We will briefly describe the splits below while more details can be found in the supplementary.\\
\noindent\textbf{Split-1{\normalfont ~\cite{yang2020object, Wang2021CominingSL}}: }For this split, the COCO~\cite{Lin2014MicrosoftCC} 2017 train set is used for training and the results are reported on the validation set. For each object category, $p\%$ of annotations are randomly removed. We experiment with $p=\{30, 50, 70\}$. Note, this split might contain images with no annotations. \\
\noindent\textbf{Split-2{\normalfont~\cite{niitani2019sampling}}: }This split is constructed using the COCO 2017~\cite{Lin2014MicrosoftCC} train set for training and the COCO 2017 validation set for evaluation. For each category in the dataset, among the set of images containing the category, all annotations of the category among the selected images are removed with a probability $p=\{0.30, 0.50, 0.70\}$. This split also ensures that there exists one annotation per image.\\
\noindent\textbf{Split-3: }This split, which uses the COCO 2017~\cite{Lin2014MicrosoftCC} train set for training and the validation set for evaluation, deletes annotations with a specific probability $p$ in a class agnostic fashion for each image ensuring at least one annotation per image. For the experiments, we use $p=\{0.3, 0.5, 0.7\}$. \\
\noindent\textbf{Split-4{\normalfont~\cite{zhang2020solving, Wang2021CominingSL}}: }This split uses the PASCAL-VOC 2007+2012~\cite{pascal-voc-2007, pascal-voc-2012} trainval set for training and the PASCAL-VOC 2007~\cite{pascal-voc-2007} test set for evaluation. It requires evaluating models on three different sets namely \textit{easy}, \textit{hard} and \textit{extreme} ensuring atleast one annotation per image.

\noindent\textbf{Split-5{\normalfont~\cite{wu2018soft}}: }This split uses the PASCAL-VOC 2007~\cite{pascal-voc-2007} train set for training and the PASCAL-VOC 2007 test set for evaluation. This split is constructed similar to Split-2.

\begin{table*}[!t]
\begin{minipage}[t]{0.72\linewidth}
\renewcommand{\arraystretch}{1.2}
\centering
\footnotesize
\caption{\textbf{Sparsely annotated object detection} results on two splits of PASCAL-VOC dataset. ``Oracle" and ``Vanilla" corresponds to training models using all annotations and available ground truths respectively. A/B correspond to AP/AP$_{50}$. $\dagger$: Uses a RetinaNet backbone.}\label{tab:saod_voc}
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}lccccccc@{}}
\toprule
 \textbf{Method}& \textbf{Backbone} & \multicolumn{3}{c}{\textbf{Split-4}}& \multicolumn{3}{c}{\textbf{Split-5}}  \\
 \cmidrule[\cmidrulewidth](l){3-5}
 \cmidrule[\cmidrulewidth](l){6-8}

 & & Easy & Hard & Extreme & 30\% &40\% & 50\% \\
 \midrule
 Oracle & R101 & \multicolumn{3}{c}{- - - - - - - - - - - 51.61/81.45 - - - - - - - - - - -}&\multicolumn{3}{c}{- - - - - - - - - - - 46.21/76.59 - - - - - - - - - - -} \\
 Vanilla & R101 & 49.61/80.14 & 49.55/79.481 & 46.06/75.18 & 43.87/73.65 & 42.51/71.74 & 41.77/71.03 \\
 \midrule
 Co-mining$^\dagger$~\cite{Wang2021CominingSL} & R101-FPN & 51.89/79.59 & 51.23/78.38 & 44.84/69.60 & 46.53/74.42 & 45.98/73.30 & 43.21/69.89 \\
 Ours & R101 & 50.91/81.09 & 51.45/81.13&47.64/75.95 & \textbf{47.84/77.84} & \textbf{47.09/76.48} & \textbf{46.47/75.79} \\
 Ours & R101-FPN & \textbf{52.63/82.38} & \textbf{51.95/81.76} & \textbf{48.42/76.69 }& 47.80/77.25& 46.77/75.91 & 45.68/74.44\\
\bottomrule
\end{tabular}
}
\end{minipage}\hfill
\begin{minipage}[t]{0.25\linewidth}
 \setlength{\cmidrulewidth}{0.01em}
\renewcommand{\tabcolsep}{4pt}
\renewcommand{\arraystretch}{1.2}
\centering
\footnotesize
\caption{Analysis of various components of proposed approach}
\label{tab:cr}
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}ccccc@{}}
\toprule
 \textbf{CRPN} & \textbf{PPM}&$\mathbf{\mathcal{L}^{u}_{cr}}$&$\mathbf{\mathcal{L}^{s}_{det,a}}$&\textbf{mAP}   \\
 \midrule
 \xmark & \xmark & \xmark & \xmark & 15.60\\
 \xmark & \xmark & \xmark & \cmark & 18.14\\
 \cmark & \xmark & \xmark & \cmark & 17.86  \\
  \cmark & \cmark & \xmark &\cmark &18.26	\\
     \cmark & \cmark & \cmark &\cmark &\textbf{18.82}	  \\
  \xmark & \cmark & \cmark &\cmark &18.41\\
  \cmark & \cmark & \cmark & \xmark & 18.52\\

\bottomrule
\end{tabular}
 }
\end{minipage}

\end{table*}

\subsection{Implementation details}\label{sec:impl}
For all our experiments, we use a Faster RCNN~\cite{Ren2015FasterRT} object detector with ResNet-101~\cite{resnet} backbone. We train all our models with a batch size of $8$ for $270000$ and $18000$ iterations on the COCO and PASCAL-VOC splits respectively with a learning rate of $0.01$. The learning rate is decreased ten fold twice at $\{210000, 250000\}$ and $\{12000, 15000\}$ for the COCO and PASCAL-VOC trainings respectively. We adopt a warm-up strategy for $1000$ and $100$ iterations for the COCO and PASCAL-VOC respectively. We use a ROI head batch size of $256$ and a positive to negative ratio of $0.5$. Our approach has been built using Detectron2~\cite{wu2019detectron2} .

\subsection{Baselines}\label{sec:baseline}
We compare our method against two contemporary methods. The first method is the recently proposed sparsely annotated object detection method, i.e. Co-Mining~\cite{Wang2021CominingSL}. Since Co-Mining~\cite{Wang2021CominingSL} outperforms previous sparsely annotated methods, we choose to compare only against them. Co-mining~\cite{Wang2021CominingSL} uses a RetinaNet~\cite{Lin2017FocalLF} detector with a ResNet-101~\cite{resnet} FPN~\cite{Lin2017FeaturePN} backbone. Authors of Co-Mining~\cite{Wang2021CominingSL} only report results on Split-1, so we use the code provided by the authors to run on rest of the splits with the batch size of $8$, a learning rate of $0.005$ and the same number of iterations as mentioned above. The second method is a recently proposed semi supervised object detection method called UBTeacher~\cite{liu2021unbiased}. We use the publicly available code for all the experiments keeping the hyperparameters similar to what we use for fair comparisons.

\subsection{Comparison with state-of-the-art}\label{sec:sota}
In this section we compare our method with contemporary methods. We evaluate all the models in three different setups. We name the first setup \textit{Sparsely annotated} setup which evaluates models on sparsely annotated object detection. The second setup, called the \textit{Semi-Supervised} setup, evaluates models for semi supervised object detection. The final setup, combines the two setups above and in a sense is a ``true" semi supervised object detection setup where the labeled dataset is not assumed to be exhaustively labeled i.e. contains labeled and unlabeled images and regions.\\
\noindent\textbf{\textit{Sparsely annotated} setup: }
We show results of this setup in Table~\ref{tab:saod_coco} and Table~\ref{tab:saod_voc}. Note that it is unfair to evaluate Unbiased Teacher~\cite{liu2021unbiased} in this setup as semi-supervised approaches expect a large unlabeled dataset for training. Splits-2 and 3 ensure atleast one annotation per image is retained. So, we use the available $1000$ unlabeled images for training Unbiased Teacher~\cite{liu2021unbiased}. On the other hand, split-1 doesn't ensure this and has significantly more unlabeled data than the other splits. Splits-4 and 5 have no unlabeled images at all. Hence, we don't evaluate the Unbiased teacher on splits-4 and 5.
In both the tables, the rows named ``oracle" refers to the models trained using the standard techniques using all annotations.
Similarly, rows named ``Vanilla" are models trained using the sparsely annotated data with same hyperparameters.

From Table~\ref{tab:saod_coco}, we observe that our approach outperforms the other two baselines and is closer to the oracle performance on the $30\%$ splits. The other SAOD method, Co-mining~\cite{Wang2021CominingSL}, performs competitively on all the splits at $30\%$ but its performance drops drastically at higher rates of dropping annotations. We observe a reverse trend of performance improvement for Unbiased Teacher~\cite{liu2021unbiased} on split-1. This is due to the fact that, on split-1, higher rate of dropping annotations is proportional to the number of unlabeled images. On split-1, our method obtains a performance improvement of $(1.57,  10.06)$, $(3.34, 5.05)$, $(8.67, 1.59)$ at $30\%$, $40\%$ and 50\% over (Co-mining~\cite{Wang2021CominingSL} and Unbiased Teacher~\cite{liu2021unbiased}) respectively. On split-2, our method obtains an improvement of $(2.6, 2.95)$, $(4.00, 4.12)$ and $(10.55, 3.98)$ at $30\%$, $40\%$ and $50\%$ over the two baselines. On split-3, an improvement of $(2.21, 2.98)$, $(4.35, 4.81)$ and $(10.68, 5.54)$ at $30\%$, $40\%$ and $50\%$ over the two baselines was observed. On average, all the methods achieve lower performance on split-1 compared to the other splits as it employs an aggressive strategy to drop annotations.
\begin{table*}[!t]
\begin{minipage}[t]{0.48\linewidth}
\setlength{\cmidrulewidth}{0.01em}
\renewcommand{\tabcolsep}{4pt}
\renewcommand{\arraystretch}{1.2}
\caption{\textbf{Semi-supervised object detection} results on PASCAL-VOC (top) and COCO (bottom) settings. A/B corresponds to AP/AP$_{50}$. *-results reported in the paper. $\dagger$-results using our backbone and hyperparameters.}\label{tab:ssl}
\vspace{-0.5em}
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}lcccccccccccccccc@{}}
\toprule
 \textbf{Method}& \textbf{Backbone} &\textbf{Labeled Data}& \textbf{Unlabeled Data}& \textbf{AP}  \\

 \midrule
 Unbiased Teacher$^\dagger$~\cite{liu2021unbiased} &R101-FPN & VOC 2007 & VOC 2012 & 48.67/79.43 \\
 Unbiased Teacher$^{*}$~\cite{liu2021unbiased} & R50-FPN & VOC 2007 & VOC 2012 & 48.69/77.37 \\
Ours & Res101 & VOC 2007 & VOC 2012 & \textbf{50.73/80.13}\\
 \midrule
  Unbiased Teacher$^\dagger$~\cite{liu2021unbiased} & R101-FPN & COCO 10\% & COCO 90\% & 27.25\\%/45.32\\
  Unbiased Teacher$^{*}$~\cite{liu2021unbiased} & R50-FPN & COCO 10\% & COCO 90\% & \textbf{31.50} \\
Ours & Res101 & COCO 10\% & COCO 90\% & 28.66\\%/47.7\\
\bottomrule
\end{tabular}
}
\end{minipage}\hfill
\begin{minipage}[t]{0.48\linewidth}
\setlength{\cmidrulewidth}{0.01em}
\renewcommand{\tabcolsep}{4pt}
\renewcommand{\arraystretch}{1.2}
\caption{Results on the proposed \textbf{``true" semi-supervised object detection} setup. Results are shown with different values of removal probability $p$ for Split-5 as the labeled data.}\label{tab:tssl}
\vspace{-0.5em}
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}lcccccccccccccccc@{}}
\toprule
 \textbf{Method}& \textbf{Backbone} &\textbf{p}& \textbf{Unlabeled Data}& \textbf{AP/AP}$_{50}$  \\
 \midrule
 Unbiased Teacher~\cite{liu2021unbiased} & R101-FPN &  30\% & VOC 2012 & 47.01/78.11 \\
 Ours & R101 & 30\% & VOC 2012 & \textbf{48.43/ 78.26}\\
 \hdashline
 Unbiased Teacher~\cite{liu2021unbiased} & R101-FPN & 40\%& VOC 2012 & 46.64/78.01 \\
 Ours & R101 & 40\% & VOC 2012 & \textbf{48.23/77.51}\\
 \hdashline
 Unbiased Teacher~\cite{liu2021unbiased} & R101-FPN & 50\% & VOC 2012 & 46.23/77.92 \\
Ours & R101 & 50\% & VOC 2012 & \textbf{46.93/76.01}\\
\bottomrule
\end{tabular}
}
\end{minipage}\vspace{-1em}
\end{table*}

From Table~\ref{tab:saod_voc}, we see a similar trend observed above. Both the methods perform competitively on \textit{easy} setting on split-4 and $30\%$ of split-5. However, the performance gap between the proposed approach and Co-mining~\cite{Wang2021CominingSL} increases at higher rates. It is interesting to observe that on split-5, a model trained using the proposed approach using $30\%$ of the annotations $(47.84/77.84)$ outperforms the oracle model, which is trained using all the annotations $(46.21/76.59)$. This is because the PASCAL-VOC dataset is not exhaustively labeled. To verify this, we train a model using all the annotations on Split-4 and our method achieves $49.77/79.66$ outperforming the fully supervised baseline which achieves $(46.21/76.59)$. On split-4, we get an improvement of -0.98/1.5, $0.22/2.75$ and $2.8/6.35$ (AP/AP$_{50}$) percentage points on the \textit{easy}, \textit{hard} and \textit{extreme} settings respectively over Co-mining. On split-5, an improvement of $1.31/3.42$, $1.11/3.18$ and $3.26/5.9$ (AP/AP$_{50}$) percentage points was observed. To make a fair comparison, we evaluate our method with a Res101 FPN backbone and show that using a stronger backbone further improves the performance in most cases.
Since, this is an orthogonal direction for improvement, it is not used further.
\\
\noindent\textbf{\textit{Semi-Supervised} setup: }To evaluate models' capabilities to work with unlabeled data, we evaluate our model in a semi-supervised learning setup and report results in Table~\ref{tab:ssl}. Co-mining~\cite{Wang2021CominingSL} drops all images without a single annotation and hence cannot be evaluated in this setup. To the best of our knowledge, our method is the first sparsely annotated method which can be evaluated in this setup. We evaluate models in this setup on two standard semi-supervised settings. For the first setting, we use VOC 2007 trainset as the labeled data and VOC 2012 trainval set as the unlabeled data. In the second setting, performed on the COCO dataset, we treat $10\%$ of the training set as the labeled data and the rest of the images as the unlabeled data.  We formulate the problem of sparsely annotated object detection as semi-supervised object detection at a region level. Our general formulation, enables the proposed approach to perform traditional semi-supervised object detection. On both settings, our method performs competitively demonstrating that it can be used for semi-supervised object detection. In the first setting, we obtain an improvement of $2.06/0.7$ (AP/AP$_{50}$) percentage points. In the second setting, an improvement of 1.41 percentage points was observed.
For more details on training, refer to the supplementary section.
\\
\noindent\textbf{\textit{``true" Semi-Supervised} setup: }
Finally, we evaluate our model on a truly semi-supervised setup where the labeled data is not exhaustively labeled. For this setup, we use the different sets of Split-4 (Section~\ref{subsec:split}) as the labeled set and VOC 2012 trainval set as the unlabeled dataset. Note that this setup has both unlabeled regions and images and in a sense is a ``true" semi-supervised setup. Table~\ref{tab:tssl} shows that formulating SAOD as a more general region-wise SSOD, can provide improvements on both SAOD and SSOD. \begin{figure*}[!t]
    \centering
\begin{minipage}[t]{\linewidth}
 \centering
\includegraphics[width=\linewidth]{latex/figures/ppm_viz_3.pdf}
\caption{Qualitative results showing the unlabeled regions identified by the pseudo-positive mining step. The red boxes correspond to the available ground truth. A class agnostic NMS was performed on the regions and the result is shown in white. }\label{fig:ppm}\vspace{1em}
\end{minipage}

\begin{minipage}[t]{\linewidth}
\centering
\includegraphics[width=\linewidth]{latex/figures/qual1.pdf}
    \caption{Qualitative results comparing the output of a ``vanilla" model trained using available ground truths (top) to a model trained using our approach (bottom).  Predictions with a class confidence score greater than $0.9$ are shown. \textcolor{red}{Red}: Person, \textcolor{cyan}{Cyan}: Dog, \textcolor{purple}{Purple}: Horse, \textcolor{yellow}{Yellow}: Clock, \textcolor{YellowGreen}{Green}: Stop sign, \textcolor{Cerulean}{Blue}: Parking meter, \textcolor{violet}{Violet}: Giraffe, \textcolor{orange}{Orange}: Potted plant, Black: Surfboard, \textcolor{ForestGreen}{Dark green}: Boat}
    \label{fig:good_res}
\end{minipage}
\vspace{-1em}
\end{figure*}

\subsection{Ablation Experiments}\label{sec:ablation}
In this section we conduct experiments to understand the effect of the components in our proposed approach. For all the ablation experiments we create an in-house split of the COCO dataset with $6000$ images dropping annotations similar to Split-1 with $p = 0.5$. The COCO 2017 validation set is used to evaluate the performance of the models. For all these experiments, we train the networks with the same hyperparameters used for the PASCAL-VOC dataset as discussed in~\ref{sec:impl}.
We provide more details of this set and justify its use in the supplementary material.

From Table~\ref{tab:cr}, a model trained on the ablation dataset using the available ground truth obtains $15.31$ on the validation set.  Introducing $\mathcal{L}^s_{det,a}$ improves the performance by $2.54$ points. Adding a common RPN (C-RPN) results in a drop in performance. The improvement is realized when C-RPN is combined with pseudo-positive mining step. The combination of C-RPN and PPM improves the performance by $0.12$ percentage points. PPM identifies and removes the unlabeled regions  from the set of negatives to avoid penalizing the network. By applying the unsupervised consistency regularization loss, $\mathcal{L}^u_{cr}$, we achieve the best performance of $18.82$. As mentioned in Section~\ref{sec:approach}, without C-RPN, an IoU based heuristic matching has to be performed to apply $\mathcal{L}^u_{cr}$ which can often be noisy. By removing the C-RPN , we observe a drop in performance by $0.41$ percentage points.

Our method identifies potential positives which can be mistaken for negatives due to missing annotations using a pseudo-positive mining step. We rely extensively on the objectness score of the RPN to identify these regions. In Table~\ref{tab:thresh}, we vary the threshold for picking pseudo-positives. For a low threshold, a few hard negatives might also be identified as pseudo-positive, thereby reducing the number of negatives to the network leading to a drop in performance. With a high threshold, a few potential positives might not be selected. We observe that a threshold of $0.8$ provides a good trade-off and is therefore used for all subsequent experiments unless stated otherwise. In all our experiments, PPM is performed after an initial warmup of $9000$ and $30000$ iterations on the PASCAL-VOC and COCO datasets respectively.
\begin{table}[H]
 \setlength{\cmidrulewidth}{0.01em}
\renewcommand{\tabcolsep}{4pt}
\renewcommand{\arraystretch}{1.03}
\centering
\footnotesize
\caption{Analysis of the threshold used for pseudo-positive mining}
\label{tab:thresh}
\begin{tabular}{@{}lccccc@{}}
\toprule
 \textbf{Threshold} & 0.6 & 0.7 & 0.8& 0.9 & 0.95   \\
 \midrule
 \textbf{AP} & 18.41 & 18.34 & \textbf{18.50} & 18.35 & 18.38\\
 \textbf{AP}$_{50}$ & 34.83 & 34.79 & \textbf{35.07} & 35.20 & 35.00\\
\bottomrule
\end{tabular}
\end{table}
For analysis on different data augmentations and the warm-up refer to the supplementary section.

\subsection{Discovery RPN Recall experiments}
PPM identifies foreground regions mistakenly assigned as negatives during training to avoid penalizing the network. To study the affect of PPM on novel~\cite{9093355} classes, we train a network using our approach on randomly chosen $6000$ images of the COCO dataset, containing annotations for only $20$ classes of the PASCAL-VOC. We evaluate the recall@0.5 of the RPN over the remaining 60 classes to study the affect of PPM on novel classes. A model trained using the standard techniques on this dataset achieves a recall@0.5 of $77.46\%$ and $29.06\%$ on the known classes (20 categories) and unknown classes (60 categories) respectively. Our proposed approach, with PPM, achieves a recall@0.5 of $78.47\%$ and $35.20\%$ respectively. This ability to localize objects not seen during training can be beneficial for object discovery methods like~\cite{Vo20rOSD, VoBCHLPP19, Rambhatla2021ThePO} which use RPN proposals to learn/discover new categories.
\subsection{Qualitative Results}
In Figure~\ref{fig:ppm} we show the pseudo positives mined by the pseudo-positive mining step. In each figure, the red boxes correspond to the ground truth annotations and the white boxes correspond to the post NMS pseudo positive boxes identified by PPM. It can be clearly seen that the PPM step is able to correctly mine proposals which correspond to missing object annotations.
We show the detection results of a model trained on the 50\% Split-1 of our approach. The results at shown in Figure~\ref{fig:good_res}. The top image in each column corresponds to the model trained using sparse annotations and the bottom image shows the output of our approach. We can see that our approach detects objects with higher confidence compared to vanilla training.

redpajama_set_name: RedPajamaArXiv


It was a great weekend to exhale by the lake!
The Mother-Daughter Retreat was special for everyone. Creating a space of relaxation, quality time together, and reflection.
Families and friends came together over the long weekend at Family Winter Weekend. They spent the weekend playing, worshiping, and learning. Take a look to see some of the fun!
Take a look at all the fun we had at Winterblast 2! The Junior High campers were able to spend time together learning, playing, and worshiping.
It was a jam-packed weekend for our trailblazer retreat, Winterjam! Take a look at all the fun we had!
Take a look at all the fun we had at Winterblast I! The Junior High campers were able to spend time together learning, playing, and worshiping.
We are so glad you received a postcard from Camp Squanto! Find out how to transform the postcard into a paper airplane!
There was worship, games, magic, workshops, and so much more! Joseph Sojourner brought an energizing message for today's youth, speaking on Best, Better, Good Enough.
We have a lot to be thankful for this season! We hope you had a Happy Thanksgiving! Check out some of the fun we had gathering for Thanksgiving this year!
Explore what the Men did this weekend!


Next time I make this, I might add some salsa verde, though, to give it even more of a kick. I left it out tonight because I don't want to mess with teachers' digestive systems too much during parent-teacher conferences. I'm a teacher too, and I know I would NOT appreciate an upset tummy particularly in that situation.
So, for now, this is what I have. It's pretty easy, especially if you already have the ingredients for the Cheesecake Factory White Chicken Chili. Have I mentioned how much I love that chili...?
Place the first eleven ingredients (everything BUT the chicken) in a blender or food processor, or in a tall cup if you have an immersion blender. Blend until smooth and the texture of a loose natural peanut butter.
Place chicken in a plastic zip-top bag and put enough marinade on the chicken to thoroughly coat all over, but it should not be swimming in the marinade. (That might be wasteful.) Zip the bag and refrigerate 1-2 hours.
Preheat grill to about 450 degrees Fahrenheit. Place chicken on grill, turning every 4-5 minutes until chicken is done/juices run clear, or a meat thermometer reads 160-165 degrees Fahrenheit (there will be a little carryover cooking, and most placed I've looked recommend 165 degrees for chicken).
You can either serve pieces of grilled chicken, or you can cut the chicken into bite-size pieces for people to spoon into tacos or burritos.
This looks fabulous. What would you say the heat level is with the Tabasco?


When the first Canadian contingent arrived in France, they were equipped with the Canadian-made Ross rifle. It was a finely crafted weapon, superior to the British Lee-Enfield in both range and accuracy. However, the Ross rifle was not designed to withstand the rigours of trench warfare. It jammed easily in the muddy conditions at the front, and the rapid firing often caused the rifle to overheat and seize up.
At the Second Battle of Ypres on 22 April 1915, many Canadian soldiers learned first-hand about the deficiencies of the Ross rifle in combat conditions. They quickly discarded their jammed weapons and armed themselves with Lee-Enfield rifles taken from the dead British soldiers. As a result, a significant change was made the following month. Despite strong objections from Minister of Militia Sam Hughes, Lieutenant-General Alderson re-equipped the 1st Canadian Division with British Lee-Enfield rifles. Many snipers, however, preferred the Ross rifle. Indeed, the weapon was still in service exclusively as a sniper rifle as late as the Korean War (1950-1953).


2 .How do you make excellent inferences?
3. What are sources and in what forms do they appear?
4. How do you write a high level answer in the exam?
Inferences are educated guesses or conclusions you make about something or someone, based on the clues available in your sources of information.
For example, you don't know these two boys and you stumble upon a photo of them on the internet, on a website collecting memories of old Singapore. What can you conclude about the boys? Hint: Look at the clues below.
Yep, you might have concluded that these boys are Star Wars fans. The picture doesn't carry any big flashing text that says 'These boys are Star Wars fans.' You concluded this (guessed) from the clues you see in the photo—both are grinning and standing proudly in front of a collection of Star Wars figurines, and both wear Star Wars T-shirts.
Many students have no idea why they consistently score L1/1m for inference SBQs. They think they would never crack the code.
Actually, you could. You should in fact aim for (almost) full marks, since it usually requires less writing than the other SBQs!
(a) Know the topic well. This means you need to know who the heroes and villains are, as well as a timeline of events. You don't know what our man Hitler looks like and when he was Chancellor, you're in deep trouble!
(b) Take note of the issue in question, if any.
(c) Take note of clues you see/read in the source.
Remember: If in doubt, always go back to the question and the source!
Sources are simply information about a certain topic you want to explore. Think of them as clues to answering a question or solving a mystery.
For example, if you want to find out which mobile phone model you should buy, you may go onto the internet and search for the latest Iphone and Samsung phone reviews.
The Youtube videos and written review articles you read are therefore sources of information for you.
• Artefacts (everyday objects) made by people.
Before writing your answer, you MUST do these 3-steps.
STEP 1: Read the question and underline the issue.
STEP 2: Underline/highlight the source clues related to the issue.
Let's see the 3 steps in action!
Once you have done Step 3, go ahead and write out your answer.
To continue to part 2, visit HERE.


text:
Olympics Home
Sabine Spitz profile
From the section Olympics
http://www.bbc.com/sport/olympics/18508756
Cycling (Women's Mountain Bike)
Career highlight
Sabine Spitz won Germany's first ever mountain bike gold at the 2008 Olympics in Beijing to improve on the silver medal she won in the event at the World Championships in the same year.
Career other
She won bronze in the mountain bike at the 2004 Olympics in Athens, but is off the pace this season against younger rivals and is an outsider to defend her title.
Spitz only started cross-country cycling aged 22 and turned professional aged 29.

redpajama_set_name: RedPajamaCommonCrawl


text:
Regional IMFAR
Regional IMFAR 2017
Session REcordings - 2015
Repetitive Behavior and Restricted Interests of Offspring in Adults with ASD and Other Neuropsychiatric Disorders
Friday, May 12, 2017: 5:00 PM-6:30 PM
Golden Gate Ballroom (Marriott Marquis Hotel)
D. W. Evans1,2, D. B. Hanson2 and M. Uljarevic3, (1)Psychology, Bucknell University, Lewisburg, PA, (2)Bucknell University, Lewisburg, PA, (3)Olga Tennison Autism Research Centre, Olga Tennison Autism Research Centre, Melbourne, CA
Background: The broader autism phenoptype (BAP) is an emerging construct suggesting that first-degree relatives of those diagnosed with autism spectrum disorders (ASD) exhibit behaviors that encompass the ASD spectrum without necessarily reaching diagnostic thresholds for clinical disorder. Generally the BAP refers to behaviors of parents or "unaffected" siblings of children with ASD. Some aspects of the BAP are relevant to other disorders as well. Repetitive Behaviors and Restricted Interests (RBRI) for example not only define ASD (and the BAP) but they also appear in a wide range of other neurodevelopmental and neuropsychiatric disorders, including, but not limited to, obsessive-compulsive disorder (OCD), tic disorders, intellectual and developmental disabilities, and psychotic disorders. What is not well understood is the manifestation of RBRI in the offspring of adults diagnosed with these disorders, as well as (high functioning) ASD. We know virtually nothing, about the trajectory of RBRI in children who are at risk for various neurodevelopmetnal and neuropsychiatric disorders, by virtue of parental diagnostic status.
Objectives: To examine the frequency/intensity, as well as the cross-sectional trajectory, of RBRI in offspring of adults diagnosed with a range of neurodevelopmental and neuropsychiatric disorders who represent a nationally-representative cohort.
Methods: 2897 adults who were ascertained from a representative United States cohort of adults with at least one children ranging in age from 1-17;11 years of age. Families were sent an online battery of surveys indicating demographic status including clinical neurodevelopmental and neuropsychatric diagnostic family history. Parents also completed the Childhood Routines Inventory-Revised (CRI-R) to assess Total RBRI, Repetitive Motor Behaviors/Compulsions (RMBC), and Resistence/Insistence on Sameness (RIS). Parents were grouped into several diagnostic clusters basad on their psychiatric history: ASD (n=38); OCD/tic disorders (n=66); bi-polar/schizophrenia (n=157); no diagnosis (n=1914); "other" diagnosis (n=738). Children were placed into one of three age groups (1-6 years, 7-12 years, 13-17;11 years).
Results: A 4 (parent diagnosis) X 3 (age group) analysis of variance revealed a significant interaction for RMBC (F(2882,8)= 2.08, p= 0.03), main effects for parent diagnosis for all three CRI-R factors (All p < 0.0001), and main effects for age group for Total CRI-R (p = 0.03) and RMBC (p= 0.02).
Conclusions: The offspring of adults with an ASD demonstrated signficantly higher RBRI relative to all other diagnositic groups, followed by OCD/tic disorders, bi-polar/schizophrenia, any other diagnosis, and no diagnosis. Children of parents with ASD had unique developmental trajectories that indicated progressive age effects across all three age groups, whereas all offspring of all other parental diagnostic groups showed a characteristic increase from agegroup 1 to agegroup 2, with a decrease for age group 3. These data indicate that the frequency/intensity and developmental trajectories of RBRI in children are impacted by parental diagnostic status.
See more of: Sensory, Motor, and Repetitive Behaviors and Interests
Marriott Marquis Hotel
Internation Society for Autism Research
342 North Main Street | West Hartford, CT 06117-2507 USA


text:
Neither Samsung nor Apple will buy Arm from SoftBank
Samsung Electronics Vice Chairman Lee Jae-yong, left, and SoftBank CEO Masayoshi Son, right, arrive at the Korea Furniture Museum in Seoul, in this July 4, 2019, file photo, to attend a dinner with the leaders of Korea's conglomerates. / Yonhap
By Baek Byung-yeul
Speculation has arisen that either Samsung Electronics or Apple may consider acquiring British chip-designing firm Arm, but an influential industry analyst expressed his doubts saying neither of the two IT giants will be interested in purchasing the company, which was put up for sale by its mother company SoftBank in Japan.
Jim Handy, a U.S.-based renowned semiconductor analyst guru working for Objective Analysis, said Samsung and Apple won't try to acquire Arm, the world's leading company in mobile chip architecture design, given the merger and acquisition deal is expected to amount to over $40 billion, which is too costly.
"It's hard to tell with Samsung. The company rarely acquires other companies, and the ones that I recall were all very small. Arm would be bigger than usual," the analyst told The Korea Times in a recent email interview.
SoftBank is reportedly considering either selling its chip-designing affiliate Arm or going through with an initial public offering for the unit. This is because the Japanese tech giant is strapped for cash after its $100 billion Vision Fund recorded losses for two consecutive quarters, hit by the pandemic impact on the global economy. In an effort to offset the investment losses, SoftBank has said it plans to sell off up to $41 billion in assets.
Some industry analysts have put up Samsung or Apple as potential suitors for Arm, which licenses its mobile chip architecture design to other companies, but Handy said the two companies won't look at intellectual property (IP) licensing business because they have used their IPs to help mitigate risks and avoid rises in costs.
From that standpoint, the analyst presumed Arm will be acquired by a consortium led by fabless companies or intelligent system design companies such as Rambus, Cadence and Synopsys. The rationale for that is because these firms are generating revenue by licensing their IP to other companies, which is exactly what Arm is doing.
They may try to form an association to join the acquisition race, expecting to create a synergistic effect in their licensing businesses.
"It is also unlike Samsung to license IP to other companies. Samsung's IP strategy is mostly defensive: They get patents so that they can defend themselves against other companies who try to use their patent portfolios against Samsung. Samsung usually tries to develop all of its IP internally," he said.
"Apple is very similar. Usually Apple buys small companies for their IP but does not license that IP to others the way that Arm does. So I find it hard to understand why either Samsung or Apple would want to acquire Arm."
Mobile application processors chips are used in most smartphones, and Apple's A-series, Samsung's Exynos and Qualcomm's Snapdragon chips are based on Arm's architecture. Every chipmaker using the architecture pays a licensing fee upfront, which depends on the complexity of the design, and royalties, which Arm receives for each chip sold.
Some concerns are either Samsung or Apple would experience an increase in their licensing fees if one of their competitors succeeds in acquiring Arm, but the analyst said the two companies can afford it.
"Consider what happened in the cell phone business. Motorola and Nokia were very big brands before smartphones became popular. Microsoft acquired Nokia's mobile business and Google acquired Motorola's mobile business. Neither Samsung nor Apple wanted to acquire these companies, and you know that they could have afforded to," Handy said.
Regarding any potential suitors for Arm, the analyst said IP licensing firms will be strong candidates.
"Companies that have synergies with Arm would include IP licensors. Rambus, Cadence, and Synopsys come to mind. Qualcomm does too, but to a lesser extent. I don't think that a software firm like Microsoft would be a good complement," he said.
Given Samsung's ample cash-equivalent assets, it is seen as one of the potential suitors to purchase Arm independently or indirectly. As of the first quarter of this year, Samsung's internal cash and cash-equivalent assets stood at 97.53 trillion won ($81 billion).
Despite the analyst predicting the possibility of Samsung pursuing the chip designer is slim, chances are still there that Samsung may join with other chipmakers including Qualcomm, MediaTek and Huawei to acquire controlling stakes in it. Given the ongoing trade conflict between Beijing and Washington, if Huawei participates in a possible consortium for Arm, then Washington may sanction it.
baekby@koreatimes.co.kr More articles by this reporter


Despite the fact that she wears strapless Dior gowns to just about every red carpet event she attends, Jennifer Lawrence, a spokeswoman for the brand, has still not mastered the art of walking in them.
In a truly unbelievable moment, Lawrence completely fell -- to her knees -- in the red Dior number she chose for tonight's festivities, just minutes after arriving. It was slightly less elegant than her first Oscars fall, which occurred as she was walking up the steps to accept her award.
This time, as you can watch above, the incident occurred as she was waving to someone. Then, down she went, nearly bringing a nearby friend down with her as she tried to grasp onto her for leverage. Some police men who were weirdly surrounding her helped her up. And then she laughed hysterically and, allegedly, pulled a Slim Jim out of her purse. She was thinking about cake during last year's fall, so maybe she had that meaty snack on her mind this time.
We like JLaw too much to speculate that this was a publicity stunt, as incredible as it is that she fell down at two Oscars events in a row. So either she's really clumsy, or gets really drunk before these things. We're fine with either and can't wait to see what she does next!
Yep, that happened. Jennifer Lawrence, no stranger to awkward awards show stair moments, fell on her way to accept her Oscar for best actress. That's probably every woman's ultimate fear in that kind of situation, right? But Lawrence handled it with such charm and humility her fall made everyone fall that much more in love with her (not to mention she fell beautifully--I mean, look at that photo!). "You guys are just standing up 'cause you feel bad that I fell and that's really embarrassing but thank you," she said when she finally made it to the podium. So, what happened exactly?
Prepare to fall in love with Jennifer Lawrence all over again.
The actress has reportedly signed a new multimillion dollar deal with the French couture house.


text:
James Madison Cotten
Full Name: James Madison Cotten
Date of birth: March 11, 1920
H 62 Jan 8, 1963 - Jan 12, 1965 58th (1) (2) Democrat Weatherford / Parker Hood, Parker, Wise
H 72 Jan 10, 1961 - Jan 8, 1963 57th (3) Democrat Weatherford / Parker Jack, Parker, Wise
H 72 Jan 13, 1959 - Jan 10, 1961 56th Democrat Weatherford / Parker Jack, Parker, Wise
H 72 Jan 8, 1957 - Jan 13, 1959 55th Democrat Weatherford / Parker Jack, Parker, Wise
(1) Cotten took oath of office for the 58th Legislature early—on Thursday, 11/8/1962. Dallas Morning News.
(2) Sworn 11/8/1962. Oaths of Office.
(3) "Wilson Opinion Awaited on Exact End of Terms" reports "a legislator can hold office until his successor has qualified or until 'the first of the year. . .'" State canvassing board to certify results of November election on 11/23/1962. Dallas Morning News.
Oral history interview available in Oral History Collection, University of North Texas.
United States Army Air Forces, World War II. Obituary, James Madison Cotten, 3/8/2014. Weatherford Democrat, 1895.
Photograph. James Cotten, 55th Legislature, State Preservation Board
Photograph. James Cotten, 56th Legislature, Texas Yearbook, 56th Legislature
Congressional and Legislative Districts
Constitutional Amendments (Chair)
Interstate Cooperation
Municipal and Private Corporations
Appropriations (Chair)
Local and Uncontested Bills
Local and Uncontested Bills (Vice Chair)
State Hospitals and Special Schools
Insurance (I. C. T. Insurance Company), Investigation, Special


Curious to know what is the best Bitcoin Wallet in Malaysia? A Bitcoin wallet is essential for storing your earned (or bought) Bitcoin safely and securely. For starters, let's look at what are the different types of wallets available used to store Bitcoin. Digital Bitcoin Wallet Exchange Wallets Hardware Wallet for Bitcoin Digital Bitcoin Wallet… Continue reading What Is The Best Bitcoin Wallet In Malaysia?
Bitcoin Rate Malaysia – How Much Is A Bitcoin Now?
Want to trade Bitcoin? The Bitcoin Rate Malaysia for today is: In the graph above, you can see the latest Bitcoin Rate for Malaysia according to the time. The price of Bitcoin will change almost every minute. The Bitcoin Rate in Malaysia is affected by supply and demand. The more the demand, the higher the… Continue reading Bitcoin Rate Malaysia – How Much Is A Bitcoin Now?


Nothing too surprising here - the tasting room hours that are provided are the normal hours that the Winery Tasting room is open during non-event days. If the Winery is having an event, the hours may change of course. If the Winery has a Tasting Room but there is also a separate Tasting Room in a more populated area, Winery-Sage.com has provided the hours for both.


You can use off-the-shelf, open source tools to build a code generator that will port your application design to multiple technologies and languages.
My project is to model a small book collection. This is a valuable tutorial because it contains multiple tables and is realistic enough to serve as a foundation for larger data model and data access layer development. I will first generate the SQL to build this database and then generate C# and PHP database access layers for the database.
C# and PHP are proven application development technologies and they are representative of both a latently-typed scripting language and a strongly-typed system languages. With this example in hand you will be able to generate your database application layer in the language of your choice; you will not be bound to the technology that you begin with.
The model should include the name of the book, the author, and the publisher. Using a normalized relational model, I will need one table for each of those fields. These tables relate to each other as shown in Figure 1.
Figure 1. Tables: You'll need three tables to hold the information for the book database.
Listing 1 shows a model of Figure 1 in XML.
For simplicity I've combined the table definition, the connection information for the database, and the test data, into one file.
The database node includes attributes which describe the connection to the database.
A table node defines a table in the database. It must include a <field> node for each field in the table. This <field> node has the name of the field and it's type.
If a <select> node is included then this node defines a query that should be associated with the table. The <select> node must include a <query> node that defines the SQL for the query, and a set of <param> nodes for each run-time parameter.
A data node defines a row of test data. It includes the name of the table to insert the data into, as well as a set of <field> nodes which include the data for each field in the row.
Your application may require more information to properly model your database layer. You can easily add this to the code generation templates in this article.
With the data model and test data in hand I can start building the code generators that will turn the design into reality. I've chosen PHP and C# as my generation targets, but I'll also generate SQL for the physical model so that the database schema always remains in sync with the database access layer sitting above it. For the generator I have chosen to use XSLT.
XSLT is not just about converting one XML form to another or converting XML to HTML—it can also be used for code generation by building source code as text.
In the process above, the XSL processor searches the templates against each of the nodes it encounters in the input XML. In this case it matches against <table> nodes within the <schema> node. It finds the <xsl:template> node because of the match="schema/table" attribute. Within the template the text is output to standard output. The <xsl:value-of> node tells XSL to output the value of the name attribute into the standard output. The name attribute is specified by use of the select="@name" attribute. The '@' sign tells XSLT to look for the attribute as opposed to an interior node. The XSL template registers its intention to create text (instead of XML or HTML) by using the method="text" attribute on the <xsl:output> tag.
My guess is that the reason that people often find XSL difficult is because it uses an event-driven coding technique rather than an imperative one (e.g. C, Java, etc.). In Java you say, "Open the XML file and find this node then do x," in XSL you say "when you see this node do x."
The next step is to finish the implementation of the SQL generator.


text:
Biografia
Terzo baronetto della dinastia, grande amico di Oscar Wilde già dai tempi di "Rome Invisited", cercò in tutti i modi di convincerlo a diventare cattolico, grazie al suo aiuto dopo Wilde a Roma, riuscì a condurlo in presenza del Papa Pio IX.

Note

Collegamenti esterni

redpajama_set_name: RedPajamaWikipedia