Freeze the token embeddings for finetuning
Hi ,

I have finetuned the T5 model using the community notebooks given . But when i looked into the code under examples/seq2seq finetuning code token embedding weights are frozen.

Can some one throw some light on how does this effect the finetuning process?


Thanks
freezing embedding weights will result in faster training and will also consume less GPU memory. It might result in slightly less performance but I guess it ultimately depends on the task, the dataset etc.<|||||>A tangent question: is it possible to freeze only a subset of the token embedding weights? Background: I am adding some new tokens and I want the model to only fine tune on those.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>It would be a very interesting feature proposed by @AzharSultan like for ***prompt learning*** purpose. 
PyTorch doesn't allow to only update a part of the embeddings lookup table. A solution would be either masking gradients or creating a new lookup table just for the added tokens. Any plan to add that ? <|||||>@montellasebastien did you experiment with these workarounds?<|||||>I am also interested in @AzharSultan's question. What is the recommended for doing this right now? 