Don’t average the weights of the networks! Then nothing makes sense anymore. For example you have two convolution kernels at the start of the network, they have been trained and fine-tuned to be specific features in the network. Averaging them will most likely not correspond to anything meaningful that the network has learned to deal with.
From the very little I’ve read from the iMet challenge, it is multi-class multi-label classification, right? Then I would suggest averaging the outputs of the model instead, if you really need to do it that way. If you have linear layers just take the sum and divide by the number of models you have, and try with that.
Another approach, which is better IMHO, is to train for 9h and save checkpoints just before the session stops (you need to save the model’s state dict and the optimizer’s also, maybe some other things that you need to train, for example on what step you last iterated in the dataset). Then you start from that point with the next session, instead of starting from scratch to train another suboptimal model.
Ensembles work, but I think they work better if the models have already reached a performance plateau.