Knowledge ****Distillation****
[knowledge-distillation-pytorch](https://github.com/haitongli/knowledge-distillation-pytorch) [kdtf](https://github.com/DushyantaDhyani/kdtf) [Soft Label](https://texonom.com/soft-label-a9e9a46dd208446cb9511764a0052c86) [AI Ensemble](https://texonom.com/ai-ensemble-03bf3a1926dd46f18400b5830d8fdf0b) [Transfer Learning](https://texonom.com/transfer-learning-442feb66465944eebf144d4e9dd1dbf8)


### Distillation from teacher network to student network (less parameter)
NIPS 2014 [Geoffrey Hinton](https://texonom.com/geoffrey-hinton-441d5ce2b78146d0935454042b4f06d9), 오리올 비니알스, [Jeff Dean](https://texonom.com/jeff-dean-dd38bba08cea419eb45e1029e7c3aa15) 
Pre-trained Teacher network → Student network
lighter than Ensemble
### Knowledge ****Distillation Notion****
|Title|
|:-:|
|[Teacher Network](https://texonom.com/teacher-network-bf45e9f87cd3474992c2ab38b3eaae9d)|
|[Student Network](https://texonom.com/student-network-894b29040685444e9d3d07919ffe9342)|
|[Hinton’s KD](https://texonom.com/hintons-kd-5dd4cda27117403d9154a069dc7e54f4)|
|[Dark Knowledge](https://texonom.com/dark-knowledge-180df27f31144a66b429d420841de821)|
|[Distillation loss](https://texonom.com/distillation-loss-cbc11ca184d24defbacc267d3bcfd641)|
|[Instruction Tuning](https://texonom.com/instruction-tuning-f1620fadb6694407b678276d09e077a3)|


### Knowledge ****Distillation Usages****
|Title|
|:-:|
|[LaMini LM](https://texonom.com/lamini-lm-7312cefcc508434aac19acbf61cd2968)|


### [Geoffrey Hinton](https://texonom.com/geoffrey-hinton-441d5ce2b78146d0935454042b4f06d9), [Jeff Dean](https://texonom.com/jeff-dean-dd38bba08cea419eb45e1029e7c3aa15) 
> [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531)

> [딥러닝 모델 지식의 증류기법, Knowledge Distillation](https://baeseongsu.github.io/posts/knowledge-distillation/)

> [딥러닝 용어 정리, Knowledge distillation 설명과 이해](https://light-tree.tistory.com/196)









