Tempered Calculus for ML: Application to Hyperbolic Model Embedding
<|reference_start|>Tempered Calculus for ML: Application to Hyperbolic Model Embedding: Most mathematical distortions used in ML are fundamentally integral in nature: $f$-divergences, Bregman divergences, (regularized) optimal transport distances, integral probability metrics, geodesic distances, etc. In this paper, we unveil a grounded theory and tools which can help improve these distortions to better cope with ML requirements. We start with a generalization of Riemann integration that also encapsulates functions that are not strictly additive but are, more generally, $t$-additive, as in nonextensive statistical mechanics. Notably, this recovers Volterra's product integral as a special case. We then generalize the Fundamental Theorem of calculus using an extension of the (Euclidean) derivative. This, along with a series of more specific Theorems, serves as a basis for results showing how one can specifically design, alter, or change fundamental properties of distortion measures in a simple way, with a special emphasis on geometric- and ML-related properties that are the metricity, hyperbolicity, and encoding. We show how to apply it to a problem that has recently gained traction in ML: hyperbolic embeddings with a "cheap" and accurate encoding along the hyperbolic vs Euclidean scale. We unveil a new application for which the Poincar\'e disk model has very appealing features, and our theory comes in handy: \textit{model} embeddings for boosted combinations of decision trees, trained using the log-loss (trees) and logistic loss (combinations).<|reference_end|>
@article{nock2024tempered,
    title={Tempered Calculus for ML: Application to Hyperbolic Model Embedding},
    author={Richard Nock and Ehsan Amid and Frank Nielsen and Alexander Soen and
  Manfred K. Warmuth},
    journal={arXiv preprint arXiv:2402.04163},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.04163},
    primaryClass={cs.LG}
}