content.raw: minimal single script implementation of knowledge distillation in LLMs. In this implementation, we use GPT-2 (124M) as student model and GPT-2 Medium (340M) as teacher via reverse Kullback-Leibler (KL) divergence, trained on a small chunk of openwebtext. 
content.type: text
content.value: minimal single script implementation of knowledge distillation in LLMs. In this implementation, we use GPT-2 (124M) as student model and GPT-2 Medium (340M) as teacher via reverse Kullback-Leibler (KL) divergence, trained on a small chunk of openwebtext. 
content.type: new_line
content.type: new_line
content.raw: Code: 
content.type: text
content.value: Code: 
content.href: https://github.com/Jaykef/ai-algorithms/blob/main/llm_knowledge_distillation.ipynb
content.raw: https://github.com/Jaykef/ai-algorithms/blob/main/llm_knowledge_distillation.ipynb
content.type: link
rawContent: minimal single script implementation of knowledge distillation in LLMs. In this implementation, we use GPT-2 (124M) as student model and GPT-2 Medium (340M) as teacher via reverse Kullback-Leibler (KL) divergence, trained on a small chunk of openwebtext. 
  
  Code: https://github.com/Jaykef/ai-algorithms/blob/main/llm_knowledge_distillation.ipynb
reactions.reaction: 👍
reactions.users: yoeldcd daniel-ltw vinu9999 sylvain471
reactions.reaction: 👀
reactions.users: John6666 TheFernandoM