One paper I know of is this one, which looked at freezing various encoder blocks of BERT / RoBERTa. As Sylvain mentions, the performance tends to drop as you freeze more layers.

Screen Shot 2021-12-09 at 16.15.23793×236 15.4 KB

There are more recent proposals like AutoFreeze which circumvent this drop in performance, but at the expense of a fairly complex training procedure (i.e. you’d be better off distilling the model if training speed is a concern):

Screen Shot 2021-12-09 at 16.18.08890×249 21 KB