Lit GPT


[lit-llama](https://github.com/Lightning-AI/lit-llama)
[lit-gpt](https://github.com/Lightning-AI/lit-gpt)
- [nanoGPT](https://texonom.com/nanogpt-b1755b382e244584915cc7e837dd0773) 
- [Flash Attention](https://texonom.com/flash-attention-ca8093deb6c648059aff1580b0ddbf68) 
- [GPTQ](https://texonom.com/gptq-87428aee2a774b93906ab2213a1b6dc6) for int4
- [bitsandbytes](https://texonom.com/bitsandbytes-1575b433faaf455ba0d86cf5f7e5190b) for int8, fp4, nf4
- [LoRA](https://texonom.com/lora-fda3706be3674496898ad2e5e00007c9) 
not available for 
```typegit clone https://github.com/Lightning-AI/lit-gptcd lit-gpt# pip install pytorchpip install --index-url https://download.pytorch.org/whl/nightly/cu118 --pre 'torch>=2.1.0dev'pip install -r requirements.txtpip uninstall -y lightning; pip install -r requirements.txtpip install sentencepiece```

### Download mode and so annoying~
```typepython scripts/download.py --repo_id openlm-research/open_llama_3bpython scripts/convert_hf_checkpoint.py --checkpoint_dir checkpoints/openlm-research/open_llama_3bpython quantize/gptq.py --checkpoint_dir checkpoints/openlm-research/open_llama_3bpython generate/base.py --prompt "Hello, my name is" --checkpoint_dir checkpoints/openlm-research/open_llama_3b --quantize gptq.int4```

### mac
```typeexport PYTORCH_ENABLE_MPS_FALLBACK=1--precision 16-true```





> [Why We're Building Lit-LLaMA - Lightning AI](https://lightning.ai/pages/community/why-were-building-lit-llama/)





