Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp
It's not a new technique at this point, but it still blows my mind that you can quantize a model trained at fp16 down to 4 bits, and still have coherent inference results.<p>> While running, the model uses about 4GB of RAM and Activity Monitor shows it using 748% CPU - which makes sense since I told it to use 8 CPU cores.<p>> I imagine it's possible to run a larger model such as 13B on this hardware, but I've not figured out how to do that yet.<p>Naively it seems like you could repeat the same process outlined in the article but with "13B" in place of "7B". What's the catch?