We study {\em online} active learning of homogeneous s-sparse halfspaces in R^d with adversarial noise \mbox{%DIFAUXCMD kearns1992toward where the overall probability of a noisy label is constrained to be at most \nu and the marginal distribution over unlabeled data is unchanged . Our main contribution is a state-of-the-art online active learning algorithm that achieves near-optimal attribute efficiency, label and sample complexity under mild distributional assumptions. In particular, under the conditions that the marginal distribution is isotropic log-concave and \nu = \Omega(\epsilon), where \epsilon \in (0, 1) is the target error rate, we show that our algorithm PAC learns the underlying halfspace in polynomial time with near-optimal label complexity bound of O\big( s \cdot polylog( d, \frac{1}{\epsilon})\big) and sample complexity bound of \tilde{O}\big( \frac{s{\epsilon} \cdot polylog(d)} {\epsilon} } \big). Prior to this work, existing online algorithms designed for tolerating the adversarial noise are either subject to label complexity polynomial in d or \frac{1}{\epsilon}, or work under the restrictive uniform marginal distribution.\tilde{O}\big( {\epsilon}) }\big) \tilde{O}\big({\epsilon} \cdot polylog(d) }\big) As an immediate corollary of our main result , we show that under the more challenging agnostic model \mbox{%DIFAUXCMD \cite{kearns1992toward }\hspace{0pt}%DIFAUXCMD }  where no assumption is made on the noise rate  , our active learner achieves an error rate of O(OPT) + \epsilon with the same running time and label and sample complexity, where OPT is the best possible error rate achievable by any homogeneous s-sparse halfspace. Our algorithm builds upon the celebrated Perceptron while leveraging novel localized sampling and semi-random gradient update to tolerate the adversarial noise. We believe that our algorithmic design and analysis are of independent interest, and may shed light on learning halfspaces with broader noise models .
We study {\em online} active learning of homogeneous halfspaces in R^d with adversarial noise where the overall probability of a noisy label is constrained to be at most \nu . Our main contribution is a Perceptron-like online active learning algorithm that runs in polynomial time, and under the conditions that the marginal distribution is isotropic log-concave and \nu = \Omega(\epsilon), where \epsilon \in (0, 1) is the target error rate, our algorithm PAC learns the underlying halfspace with near-optimal label complexity of O\big( d \cdot polylog( \frac{1}{\epsilon})\big) and sample complexity of \tilde{O}\big( {\epsilon} \cdot polylog(d)} \frac{d{\epsilon} } \big). Prior to this work, existing online algorithms designed for tolerating the adversarial noise are subject to either label complexity polynomial in \frac{1}{\epsilon}, or suboptimal noise tolerance, or restrictive marginal distributions. With the additional prior knowledge that the underlying halfspace is s-sparse, we obtain attribute-efficient label complexity of\tilde{O}\big( s \cdot polylog(d, \frac{1{\epsilon}) }\big) and sample complexity of\tilde{O}\big(\frac{s{\epsilon} \cdot polylog(d) }\big). As an immediate corollary , we show that under the }\hspace{0pt}%DIFAUXCMD } agnostic model where no assumption is made on the noise rate \nu , our active learner achieves an error rate of O(OPT) + \epsilon with the same running time and label and sample complexity, where OPT is the best possible error rate achievable by any homogeneous halfspace .
D
s-sparse
D
\mbox{%DIFAUXCMD kearns1992toward
D
and the marginal distribution over unlabeled data is unchanged
R
state-of-the-art
Perceptron-like
R
achieves near-optimal attribute efficiency, label and sample complexity under mild distributional assumptions. In particular, under
runs in polynomial time, and under
D
we show that
D
in polynomial time
D
bound
R
s
d
D
d,
D
bound
D
\frac{s
A
\frac{d
R
either subject to
subject to either
D
d or
R
work under the restrictive uniform marginal distribution.
suboptimal noise tolerance, or restrictive marginal distributions. With the additional prior knowledge that the underlying halfspace is s-sparse, we obtain attribute-efficient label complexity of
A
s \cdot polylog(d, \frac{1
A
and sample complexity of
A
\frac{s
A
.
D
of our main result
D
more challenging agnostic model \mbox{%DIFAUXCMD \cite{kearns1992toward
A
agnostic model
A
\nu
R
s-sparse halfspace. Our algorithm builds upon the celebrated Perceptron while leveraging novel localized sampling and semi-random gradient update to tolerate the adversarial noise. We believe that our algorithmic design and analysis are of independent interest, and may shed light on learning halfspaces with broader noise models
halfspace