Transformer Model
[Attention Mechanism](https://texonom.com/attention-mechanism-762711860abb45f59904f1ac4e4af285) [Attention is all you need](https://texonom.com/attention-is-all-you-need-f52e664318eb47c0aa6cbd27b9a4c491)


## Self Attention is the core feature
the Transformer gains a wider perspective and can attend to multiple interaction levels within the input sentence.
어텐션을 Encoder 보정을 위한 용도로서 사용하는 것이 아니라 어텐션만으로 인코더와 디코더 생성
모든 토큰을 동시에 받아 연산하기 때문에 병렬연산이 가능
인코더 블럭과 디코더 블럭이 각각 6개씩 모여있는 구조
인코더 블록은 2개의 sub-layer(Multi-Head(self) Attention, Feed Forward)로 나눌 수 있고
디코더 블록은 3개의 sub-layer(Masked Multi-Head(self) Attention, Multi-Head (Encoder-Decoder) Attention, Feed Forward)로 나눌 수 있다
기존의 Attention과는 다르게 각 벡터가 모두 가중치 벡터라는 점
### Transformer Model Notion
|Title|
|:-:|
|[Attention is all you need](https://texonom.com/attention-is-all-you-need-f52e664318eb47c0aa6cbd27b9a4c491)|
|[NLP Transformer Encoder](https://texonom.com/nlp-transformer-encoder-d5bc8cebe43d4fc7896d8e0290683b18)|
|[NLP Transformer Decoder](https://texonom.com/nlp-transformer-decoder-1ee79f4754d24aa8b64312399782cbec)|
|[Transformer Attention](https://texonom.com/transformer-attention-84d340d69722409a96ec1d806970608a)|
|[Transformer Model Tool  ](https://texonom.com/transformer-model-tool-01c749c1e9254d8c85ec5ef2feb0566b)|


### Transformer Models
|Title|
|:-:|
|[RETRO Transformer](https://texonom.com/retro-transformer-e3efdb06419948ebb1be444f4e124867)|
|[Torchscale](https://texonom.com/torchscale-c4b82c1245d74c7092617e70b9ddab51)|
|[BART](https://texonom.com/bart-48bd2258eee74af3b2d929b03bb9553b)|
|[RMT](https://texonom.com/rmt-f076841449f2419a82b3ce09281b9bb9)|
|[Transformer-XL](https://texonom.com/transformer-xl-eb2128e01dc744539b6825f3880da761)|



### Architecture
> [Transformer’s Encoder-Decoder: Let’s Understand The Model Architecture - KiKaBeN](https://kikaben.com/transformers-encoder-decoder/)

### Pseudo Source code
> [Transformers for software engineers](https://blog.nelhage.com/post/transformers-for-software-engineers/)

### Korean
> [점프 투 파이썬](https://wikidocs.net/31379)

> [[딥러닝] 언어모델, RNN, GRU, LSTM, Attention, Transformer, GPT, BERT 개념 정리](https://velog.io/@rsj9987/딥러닝-용어정리)





