Hi all, I recently raised an issue 9 asking why the lm_head layer is not a parameters. The response I got is that “It is tied to the input layer”. After reading the docs as suggested I found that the lm_head layer is a “linear layer with weights tied to the input embeddings”.
I still don’t understand what that means, as I thought the lm_head layer would output a tensor shaped (*, vocab_size) whereas the embedding is shaped (vocab_size, embedding_size)?
Does that mean if I want to fine-tune the lm_head layer, I would need to fine-tune the embedding layer (wte)?