How do you measure the degree of influence of pre-training on a fine-tuning task?
Context
I have a randomly initialized (deep learning) model A, and a pre-trained model Ap. The two are trained (or fine-tuned, more specifically for Ap) on the same target task* and I call them A' and Ap', respectively.
p(.) is the performance of a model on the target task. Let's admit that, in general, p(Ap') > p(A').
*Assuming that either with everything identical (seeds, etc), or that I have enough seeds so I can rely on the average behavior.
What I am looking for
A measure/metric of the robustness/dependency of my method with respect to the pre-training; i.e. 458
something that tells me the degree of dependency of my method on the pre-training process.
In the end I want to repeat this process over many fine-tunning methods and compare them.
Edit 1
I would like have some conclusion that looks like: "the fine-tunning method X is more robust to pre-training than fine-tunning method Y based on the fact that even with a worse* pre-training it retains more performance.
A bit more formally : * say that the training (or, again, fine-tuning) methods X and Y are being compared; * their respective performances after they are applied to the (raw, pre-trained) models are (p(Ax), p(Apx)) and (p(Ay), p(Apy));
I'd like to make conclusions, based on comparisons between (p(Ax), p(Apx)) and (p(Ay), p(Apy)), that X is more/less robust to the pre-training than Y.
Ideally... given a method X, I'd like to generate a sequence (p(Ax), p(Ap1x), ..., p(ApNx)) such that the pre-trained models Ap1, ..., ApN are increasingly better in their original (surrogate) task -- and A (without pre-training) is a reference. Then the sequence of performance on the target task (p(Ax), p(Ap1x), ..., p(ApNx)) allows one to take conclusions about X [compared to another fine-tuning mehtod Y].
A practical (optional) constraint: for the sake of reproducibility, I'd like to use publicly availabe pre-trained models so in a perfect scenario all the model Ap1, ..., ApN should be available or "easily" reproducible (i.e. can be programatically generated). 458