QLoRA
[Quantization Aware Training](https://texonom.com/quantization-aware-training-e0fe4518abdc43c2ad661911b87a597c)


### LoRA + [Model Quantization](https://texonom.com/model-quantization-88320068bdd94ddab6f44c0c7d66de31) 
4-bit Normalized FP + Double Quantized + Paged Optimizer = Memory Optimization



### Implementation
- [gptqlora](https://github.com/qwopqwop200/gptqlora)
- [qlora](https://github.com/artidoro/qlora)



> [Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA](https://huggingface.co/blog/4bit-transformers-bitsandbytes)

> [Untitled](https://towardsdatascience.com/qlora-fine-tune-a-large-language-model-on-your-gpu-27bed5a03e2b)

> [QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314)

> [QLoRA: 48GB GPU로 65B 모델의 미세조정(파인튜닝)이 가능하다고요?](https://discuss.pytorch.kr/t/qlora-48gb-gpu-65b/1682)





