That looks cool, i’ll look into it! I don’t really need to use ensembles for anything, I’m just doing this for fun atm.
I think I made the loss thing sound more clever than it actually is.
Usually one computes cross entropy between the network output and the actual labels. And the goal is to minimize this loss. Additionally now, I compute the cross entropy loss between the outputs of my already trained network and my new 2nd network and try to maximize this loss with my 2nd network.
So essentially I’m telling the network: “If there are 2 amost equally good predictions, choose the one that’s different from what the first network predicted.” Well, at least that’s what im trying to do.
So in order to archieve this I could either subtract this 2nd loss from my normal cross entropy loss or I do the (1 / 2nd loss) - thing and add it to my first loss. The goal is the same: Higher loss when the 2 networks’ predictions are equal.