How do we decide the fair value of individual classifiers in an ensemble model? We introduce a new class of transferable utility cooperative games to answer this question. The players in ensemble games are pre-trained binary classifiers that collaborate in an ensemble to correctly label points from a dataset. We design Troupe a scalable algorithm that designates payoffs to individual models based on the Shapley value of those in the ensemble game. We show that the approximate  Shapley value of classifiers in these games is an adequate measure for selecting a subgroup of highly predictive models . In addition, we introduce the Shapley entropy a new metric to quantify the heterogeneity of machine learning ensembles when it comes to model quality. We analytically prove that our Shapley value approximation algorithm is accurate and scales to large ensembles and big data. Experimental results on graph classification tasks establish that Troupe gives precise estimates of the Shapley value in ensemble games. We demonstrate that the Shapley value can be used for pruning large ensembles, show that complex classifiers have a prime role in correct and incorrect classification decisions, and provide evidence that adversarial models receive a low valuation  .
What is the value of an individual model in an ensemble of binary classifiers? We answer this question by introducing a class of transferable utility cooperative games calledensemble games. In machine learning ensembles, pre-trained models cooperate to make classification decisions. To quantify the importance of models in these ensemble games, we defineTroupe -- an efficient algorithm which allocates payoffs based on approximate Shapley values of the classifiers. We argue that the Shapley value of models in these games is an effective decision metric for choosing a high performing subset of models from the ensemble. Our analytical findings prove that our Shapley value estimation scheme is precise and scalable; its performance increases with size of the dataset and ensemble. Empirical results on real world graph classification tasks demonstrate that our algorithm produces high quality estimates of the Shapley value . We find that Shapley values can be utilized for ensemble pruning, and that adversarial models receive a low valuation . Complex classifiers are frequently found to be responsible for both correct and incorrect classification decisions .
R
How do we decide the fair value of individual classifiers
What is the value of an individual model
R
model? We introduce a new
of binary classifiers? We answer this question by introducing a
R
to answer this question. The players in ensemble games are
called
A
ensemble games
A
. In machine learning ensembles,
R
binary classifiers that collaborate in an ensemble to correctly label points from a dataset. We design Troupe a scalable algorithm that designates payoffs to individual models based on the Shapley value of those in the ensemble game. We show that the approximate
models cooperate to make classification decisions. To quantify the importance of models in these ensemble games, we define
A
Troupe
A
-- an efficient algorithm which allocates payoffs based on approximate Shapley values of the classifiers. We argue that the
R
classifiers
models
R
adequate measure for selecting a subgroup of highly predictive models . In addition, we introduce the Shapley entropy a new metric to quantify the heterogeneity of machine learning ensembles when it comes to model quality. We analytically
effective decision metric for choosing a high performing subset of models from the ensemble. Our analytical findings
R
approximation algorithm is accurate and scales to large ensembles and big data. Experimental results on
estimation scheme is precise and scalable; its performance increases with size of the dataset and ensemble. Empirical results on real world
R
establish that Troupe gives precise
demonstrate that our algorithm produces high quality
R
in ensemble games. We demonstrate that the Shapley value can be used for pruning large ensembles, show that complex classifiers have a prime role in correct and incorrect classification decisions, and provide evidence
. We find that Shapley values can be utilized for ensemble pruning, and
A
. Complex classifiers are frequently found to be responsible for both correct and incorrect classification decisions