Screenshot 2019-02-03 at 11.33.10 AM.png1162×714 50.1 KB

Just a quick update.  This is how the scores in ensamble improve with each model added. The keras ensambles are ran on two different machines, with different keras and tf versions. The Pytorch models are slightly different: one is with BCEWithLogitsLoss, the other is with BCELoss after a sigmoid layer. Yet the difference is still there, and consistent. There are other tests that I’ve done, with the same result.
The interesting part is that the PyTorch models taken individually, on average, score better than the keras models.
Keras average score: 0.09892 and 0.09898
PyTorch average score: 0.09890 and 0.09884
My intuition says that keras is introducing some randomness that slightly decreases the score of each model, but increses the diversity in the ensamble and improves the final score.