Llama.cpp 30B runs with only 6GB of RAM now
Author here. For additional context, please read <a href="https://github.com/ggerganov/llama.cpp/discussions/638#discussioncomment-5492916">https://github.com/ggerganov/llama.cpp/discussions/638#discu...</a> The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome, but like all things in science a small amount of healthy skepticism is warranted.