11 mission in 1969. —Ñ–µ–≤—Ä—É–∞—Ä–∏ 20, 1969, Armstrong stepped out of the lunar module Eagle and onto the moon\'s surface, famously declaring "That\'s one small step for man, one giant leap for mankind" as he took his first steps. He was followed by fellow astronaut Edwin "Buzz" Aldrin, who also walked on the moon during the mission.'Environment‚ÄãInference speed is a challenge when running models locally (see above).To minimize latency, it is desiable to run models locally on GPU, which ships with many consumer laptops e.g., Apple devices.And even with GPU, the available GPU memory bandwidth (as noted above) is important.Running Apple silicon GPU‚ÄãOllama will automatically utilize the GPU on Apple devices.Other frameworks require the user to set up the environment to utilize the Apple GPU.For example, llama.cpp python bindings can be configured to use the GPU via Metal.Metal is a graphics and compute API created by Apple providing near-direct access to the GPU. See the llama.cpp setup here to enable this.In particular, ensure that conda is using the correct virtual environment that you created (miniforge3).E.g., for me:conda activate /Users/rlm/miniforge3/envs/llamaWith the above confirmed, then:CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install -U llama-cpp-python --no-cache-dirLLMs‚ÄãThere are various ways to gain access to quantized model weights.HuggingFace - Many quantized model are available for download and can be run with framework such as llama.cppgpt4all - The model explorer offers a leaderboard of metrics and associated quantized models available for download Ollama - Several models can be accessed directly via pullOllama‚ÄãWith Ollama, fetch a model via ollama pull <model family>:<tag>:E.g., for Llama-7b: ollama pull llama2 will download the most basic version of the model (e.g., smallest # parameters and 4 bit quantization)We can also specify a particular version from the model list, e.g., ollama pull llama2:13bSee the full set of parameters
Use case
Use case ->: 11 mission in 1969. —Ñ–µ–≤—Ä—É–∞—Ä–∏ 20, 1969, Armstrong stepped out of the lunar module Eagle and onto the moon\'s surface, famously declaring "That\'s one small step for man, one giant leap for mankind" as he took his first steps. He was followed by fellow astronaut Edwin "Buzz" Aldrin, who also walked on the moon during the mission.'Environment‚ÄãInference speed is a challenge when running models locally (see above).To minimize latency, it is desiable to run models locally on GPU, which ships with many consumer laptops e.g., Apple devices.And even with GPU, the available GPU memory bandwidth (as noted above) is important.Running Apple silicon GPU‚ÄãOllama will automatically utilize the GPU on Apple devices.Other frameworks require the user to set up the environment to utilize the Apple GPU.For example, llama.cpp python bindings can be configured to use the GPU via Metal.Metal is a graphics and compute API created by Apple providing near-direct access to the GPU. See the llama.cpp setup here to enable this.In particular, ensure that conda is using the correct virtual environment that you created (miniforge3).E.g., for me:conda activate /Users/rlm/miniforge3/envs/llamaWith the above confirmed, then:CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install -U llama-cpp-python --no-cache-dirLLMs‚ÄãThere are various ways to gain access to quantized model weights.HuggingFace - Many quantized model are available for download and can be run with framework such as llama.cppgpt4all - The model explorer offers a leaderboard of metrics and associated quantized models available for download Ollama - Several models can be accessed directly via pullOllama‚ÄãWith Ollama, fetch a model via ollama pull <model family>:<tag>:E.g., for Llama-7b: ollama pull llama2 will download the most basic version of the model (e.g., smallest # parameters and 4 bit quantization)We can also specify a particular version from the model list, e.g., ollama pull llama2:13bSee the full set of parameters