Parallel Training
[ML Compiler Optimization](https://texonom.com/ml-compiler-optimization-011e7bd0ba8f417bb111ec5ea2171c8e) 


### data parallelism or model parallelism
- In data parallelism, the data is split into multiple parts
- in model parallelism, different parts of the model are processed by separate processors
### Parallel Training Notion
|Title|
|:-:|
|[Model Parallelism](https://texonom.com/model-parallelism-76dd813ada7b4e50b645af2f05821d48)|
|[Data Parallelism](https://texonom.com/data-parallelism-8e90f1c595a84dad9f4e921e74f86ba6)|



### Parallel Training Usages
|Title|
|:-:|
|[Parallel Learning Tool](https://texonom.com/parallel-learning-tool-2a9741aa76c14c16a1240f3422f11421)|
|[Parallel Training Example](https://texonom.com/parallel-training-example-3c16ddf97fbe43359e7da3dbd3ce96ee)|



![https://xiandong79.github.io](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F74479c70-d665-4f9e-a0df-dda36042a7ee%2FUntitled.png?table=block&id=cf76f928-10e0-4fb6-bb03-ec8afb321aa7&cache=v2)



> [딥러닝 모델의 분산학습이란? (Data parallelism과 Model parallelism)](https://lifeisenjoyable.tistory.com/21)





