How does Adaboost regression calculate 'the amount of say' of each weak leaner in regression?
In the case of Adaboost classification tree, 'the amount of say' of each weak learner is calculated by the following equation,
1/2 * log((1-total error)/total error)
this is easy to understand since the performance of the weak learner can be evaluated but the classification accuracy. However, I am curious about how to calculate 'the amount of say' in the case of regression. When looking at the source code in sklearn, it seems a similar equation is used
def _boost(self, iboost, X, y, sample_weight, random_state):
"""Implement a single boost for regression
Perform a single boost according to the AdaBoost.R2 algorithm and
return the updated sample weights.
Parameters
----------
iboost : int
The index of the current boost iteration.
X : {array-like, sparse matrix} of shape (n_samples, n_features)
The training input samples.
y : array-like of shape (n_samples,)
The target values (class labels in classification, real numbers in
regression).
sample_weight : array-like of shape (n_samples,)
The current sample weights.
random_state : RandomState
The RandomState instance used if the base estimator accepts a
`random_state` attribute.
Controls also the bootstrap of the weights used to train the weak
learner.
replacement.
Returns
-------
sample_weight : array-like of shape (n_samples,) or None
The reweighted sample weights.
If None then boosting has terminated early.
estimator_weight : float
The weight for the current boost.
If None then boosting has terminated early.
estimator_error : float
The regression error for the current boost.
If None then boosting has terminated early.
"""
estimator = self._make_estimator(random_state=random_state)
# Weighted sampling of the training set with replacement
bootstrap_idx = random_state.choice(
np.arange(_num_samples(X)),
size=_num_samples(X),
replace=True,
p=sample_weight,
)
# Fit on the bootstrapped sample and obtain a prediction
# for all samples in the training set
X_ = _safe_indexing(X, bootstrap_idx)
y_ = _safe_indexing(y, bootstrap_idx)
estimator.fit(X_, y_)
y_predict = estimator.predict(X)
error_vect = np.abs(y_predict - y)
sample_mask = sample_weight > 0
masked_sample_weight = sample_weight[sample_mask]
masked_error_vector = error_vect[sample_mask]
error_max = masked_error_vector.max()
if error_max != 0:
masked_error_vector /= error_max
if self.loss == "square":
masked_error_vector **= 2
elif self.loss == "exponential":
masked_error_vector = 1.0 - np.exp(-masked_error_vector)
# Calculate the average loss
estimator_error = (masked_sample_weight * masked_error_vector).sum()
if estimator_error <= 0:
# Stop if fit is perfect
return sample_weight, 1.0, 0.0
elif estimator_error >= 0.5:
# Discard current estimator only if it isn't the only one
if len(self.estimators_) > 1:
self.estimators_.pop(-1)
return None, None, None
beta = estimator_error / (1.0 - estimator_error)
# Boost weight using AdaBoost.R2 alg
estimator_weight = self.learning_rate * np.log(1.0 / beta)
if not iboost == self.n_estimators - 1:
sample_weight[sample_mask] *= np.power(
beta, (1.0 - masked_error_vector) * self.learning_rate
)
return sample_weight, estimator_weight, estimator_error
I don't understand why the estimator_weight (or 'the amount of say') can be calculated using the error_vector normalized with error_max. 392