A 7B model originally takes 13GB of disk space and RAM to load. It only takes about 4 GB
after 4-bit quantization.

Due to its native Apple Silicon support, llama.cpp is an excellent choice for running LLaMA
models on Mac M1/M2.

However, it only supports usage in a text terminal. Technically, you can use text-generation-
webui as a GUI for llama.cpp. But, as of writing, it could be a lot slower.

See the installation guide on Mac.
text-generation-webui

Oobabooga text-generation-webui is a GUI for using LLaMA models. It can be run on
Windows, Linux and Mac.

https://agi-sphere.com/llama-models/

14/18

02/05/2023, 07:05

A brief history of LLaMA models - AGI Sphere

You should go with this GUI if you have a GPU card on Windows or Linux.

Like llama.cpp, it supports 4-bit quantization (but in a different file format) for model size
reduction.

See the installation guide on Windows and Mac.

Share this…

Tags:

BEGINNER

LLAMA.CPP

MODEL

TEXT-GENERATION-WEBUI

Leave a Reply
A brief history of LLaMA models - AGI Sphere