I would be very happy to help support this if anyone has results that demonstrate its effectiveness!
There is also a freeze_params helper function here: https://github.com/huggingface/transformers/blob/e92efcf7286c955e6901f894be39cf6154af48b7/examples/seq2seq/utils.py#L277 102