Okay so I’ve tried all your suggestions: using different subsets of the training set for each nn, using different hyperparams for each nn. None of these produces networks that combined together give me more than 20-30% more accuracy.
Instead of relying on the committee I trained one more network, which also has 24 neurons in a layer, but has two (hidden) layers, not one. And that network managed to get 10x more accuracy than each of the 1-hidden-layer network. Nevertheless I would be happy to figure out how to get the committee working.
I wonder what the authors mean by “selection of the best result among neural-network
results in the committee machine”. Since they’re training 10 nns and say that only 6 are needed I would guess this “selection” refers to picking these 6 nns out of those 10 nns.