Gradient Boosting Mapping for Dimensionality Reduction and Feature Extraction
<|reference_start|>Gradient Boosting Mapping for Dimensionality Reduction and Feature Extraction: A fundamental problem in supervised learning is to find a good set of features or distance measures. If the new set of features is of lower dimensionality and can be obtained by a simple transformation of the original data, they can make the model understandable, reduce overfitting, and even help to detect distribution drift. We propose a supervised dimensionality reduction method Gradient Boosting Mapping (GBMAP), where the outputs of weak learners -- defined as one-layer perceptrons -- define the embedding. We show that the embedding coordinates provide better features for the supervised learning task, making simple linear models competitive with the state-of-the-art regressors and classifiers. We also use the embedding to find a principled distance measure between points. The features and distance measures automatically ignore directions irrelevant to the supervised learning task. We also show that we can reliably detect out-of-distribution data points with potentially large regression or classification errors. GBMAP is fast and works in seconds for dataset of million data points or hundreds of features. As a bonus, GBMAP provides a regression and classification performance comparable to the state-of-the-art supervised learning methods.<|reference_end|>
@article{patron2024gradient,
    title={Gradient Boosting Mapping for Dimensionality Reduction and Feature
  Extraction},
    author={Anri Patron, Ayush Prasad, Hoang Phuc Hau Luu and Kai Puolam"aki},
    journal={arXiv preprint arXiv:2405.08486},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.08486},
    primaryClass={cs.LG}
}