Hi!
So I tried this with MNIST + Cross Entropy Loss. What you probably guessed is that I’m trying to build neural network ensembles whose individual learners are as diverse as possible. I’m evaluating this by comparing their outputs. For example the “accuracy” for one learner, given the output of another learner.
As expected with this similarity cost function I couldn’t see any difference in my evaluation to just using plain cross entropy loss. I don’t have a working cuda installation at the moment though, so I had to do the testing on my laptop cpu. I couldn’t really use a lot of networks and couldn’t train them for long. I think I’ll try to fix the cuda installation errors first before I do any further testing.
I think as my next similarity measure I’ll use 1 / (cross entropy loss of the 2nd model, given the outputs of the first model).
So yeah, the original questions I had about autograd are pretty much solved. Thanks again!