frameworks will do a few things:Quantization: Reduce the memory footprint of the raw model weightsEfficient implementation for inference: Support inference on consumer hardware (e.g., CPU or laptop GPU)In particular, see this excellent post on the importance of quantization.With less precision, we radically decrease the memory needed to store the LLM in memory.In addition, we can see the importance of GPU memory bandwidth sheet!A Mac M2 Max is 5-6x faster than a M1 for inference due to the larger GPU memory bandwidth.Quickstart‚ÄãOllama is one way to easily run inference on macOS.The instructions here provide details, which we summarize:Download and run the appFrom command line, fetch a model from this list of options: e.g., ollama pull llama2When the app is running, all models are automatically served on localhost:11434from langchain.llms import Ollamallm = Ollama(model="llama2")llm("The first man on the moon was ...")    ' The first man on the moon was Neil Armstrong, who landed on the moon on July 20, 1969 as part of the Apollo 11 mission. obviously.'Stream tokens as they are being generated.from langchain.callbacks.manager import CallbackManagerfrom langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler llm = Ollama(model="llama2",              callback_manager = CallbackManager([StreamingStdOutCallbackHandler()]))llm("The first man on the moon was ...")     The first man to walk on the moon was Neil Armstrong, an American astronaut who was part of the Apollo 11 mission in 1969. —Ñ–µ–≤—Ä—É–∞—Ä–∏ 20, 1969, Armstrong stepped out of the lunar module Eagle and onto the moon's surface, famously declaring "That's one small step for man, one giant leap for mankind" as he took his first steps. He was followed by fellow astronaut Edwin "Buzz" Aldrin, who also walked on the moon during the mission.    ' The first man to walk on the moon was Neil Armstrong, an American astronaut who was part of the Apollo 11 mission in 1969. —Ñ–µ–≤—Ä—É–∞—Ä–∏ 20, 1969,
Use case
Use case ->: frameworks will do a few things:Quantization: Reduce the memory footprint of the raw model weightsEfficient implementation for inference: Support inference on consumer hardware (e.g., CPU or laptop GPU)In particular, see this excellent post on the importance of quantization.With less precision, we radically decrease the memory needed to store the LLM in memory.In addition, we can see the importance of GPU memory bandwidth sheet!A Mac M2 Max is 5-6x faster than a M1 for inference due to the larger GPU memory bandwidth.Quickstart‚ÄãOllama is one way to easily run inference on macOS.The instructions here provide details, which we summarize:Download and run the appFrom command line, fetch a model from this list of options: e.g., ollama pull llama2When the app is running, all models are automatically served on localhost:11434from langchain.llms import Ollamallm = Ollama(model="llama2")llm("The first man on the moon was ...")    ' The first man on the moon was Neil Armstrong, who landed on the moon on July 20, 1969 as part of the Apollo 11 mission. obviously.'Stream tokens as they are being generated.from langchain.callbacks.manager import CallbackManagerfrom langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler llm = Ollama(model="llama2",              callback_manager = CallbackManager([StreamingStdOutCallbackHandler()]))llm("The first man on the moon was ...")     The first man to walk on the moon was Neil Armstrong, an American astronaut who was part of the Apollo 11 mission in 1969. —Ñ–µ–≤—Ä—É–∞—Ä–∏ 20, 1969, Armstrong stepped out of the lunar module Eagle and onto the moon's surface, famously declaring "That's one small step for man, one giant leap for mankind" as he took his first steps. He was followed by fellow astronaut Edwin "Buzz" Aldrin, who also walked on the moon during the mission.    ' The first man to walk on the moon was Neil Armstrong, an American astronaut who was part of the Apollo 11 mission in 1969. —Ñ–µ–≤—Ä—É–∞—Ä–∏ 20, 1969,