You could try to use the output of the penultimate layers, concatenate them, and train a final classifier. This would be similar to the vanilla fine tuning approach, but with multiple inputs.
If this doesn’t provide much benefit, I would go for e.g. XGBoost and try to create a model ensemble out of your trained models.