Hi, basically I would like to see what BERT learns across the different layers (how much it already has figured out by the nth layer).
So I was thinking to take a fine-tuned model for QA, cut lets say the last 2 layers. I was assuming that the rest of the layers keep their old weights.
Now it has 10 layers + q_a head. Now I freeze all the layers and fine-tune it so that the final layer ‘connects’ to the qa_head so that I can probe how much that layer knows.
Does that make sense for such an experiment?