  Meta-learners and ensembles aim to combine a set of relevant yet diverse base
models to improve predictive performance. However, determining an appropriate
set of base models is challenging, especially in online environments where the
underlying distribution of data can change over time. In this paper, we present
a novel approach for estimating the conceptual similarity of base models, which
is calculated using the Principal Angles (PAs) between their underlying
subspaces. We propose two methods that use conceptual similarity as a metric to
obtain a relevant yet diverse subset of base models: (i) parameterised
threshold culling and (ii) parameterless conceptual clustering. We evaluate
these methods against thresholding using common ensemble pruning metrics,
namely predictive performance and Mutual Information (MI), in the context of
online Transfer Learning (TL), using both synthetic and real-world data. Our
results show that conceptual similarity thresholding has a reduced
computational overhead, and yet yields comparable predictive performance to
thresholding using predictive performance and MI. Furthermore, conceptual
clustering achieves similar predictive performances without requiring
parameterisation, and achieves this with lower computational overhead than
thresholding using predictive performance and MI when the number of base models
becomes large.

Machine Learning