Could you explain your approach of calculating the loss of one model given the outputs of another one?
I’m not sure, how you would implement it.
Also, unrelated to your question, but if you need to create an ensemble, you could try snapshot ensembles 22. Basically you apply a cyclic learning rate schedule and save some good snapshots on the way. You can find the learning rate schedulers here 8.