Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKProvidersAnthropicAWSGoogleMicrosoftOpenAIMoreComponentsLLMsLLMsAI21Aleph AlphaAmazon API GatewayAnyscaleArceeAzure MLAzure OpenAIBaidu QianfanBananaBasetenBeamBedrockBittensorCerebriumAIChatGLMClarifaiCohereC TransformersCTranslate2DatabricksDeepInfraDeepSparseEden AIFireworksForefrontAIGCP Vertex AIGooseAIGPT4AllGradientHugging Face HubHugging Face Local PipelinesHuggingface TextGen InferenceJavelin AI Gateway TutorialJSONFormerKoboldAI APILlama.cppLLM Caching integrationsManifestMinimaxModalMosaicMLNLP CloudOctoAIOllamaOpaquePromptsOpenAIOpenLLMOpenLMPetalsPipelineAIPredibasePrediction GuardPromptLayer OpenAIRELLMReplicateRunhouseSageMakerEndpointStochasticAINebula (Symbl.ai)TextGenTitan TakeoffTogether AITongyi QwenvLLMWriterXorbits Inference (Xinference)YandexGPTChat modelsDocument loadersDocument transformersText embedding modelsVector storesRetrieversToolsAgents and toolkitsMemoryCallbacksChat loadersComponentsLLMsLlama.cppOn this pageLlama.cppllama-cpp-python is a Python binding for llama.cpp. It supports inference for many LLMs, which can be accessed on HuggingFace.This notebook goes over how to run llama-cpp-python within LangChain.Note: new versions of llama-cpp-python use GGUF model files (see here).This is a breaking change.To convert existing GGML models to GGUF you can run the following in llama.cpp:python ./convert-llama-ggmlv3-to-gguf.py --eps 1e-5 --input models/openorca-platypus2-13b.ggmlv3.q4_0.bin --output models/openorca-platypus2-13b.gguf.q4_0.binInstallation‚ÄãThere are different options on how to install the llama-cpp package: CPU usageCPU + GPU (using one of many BLAS backends)Metal GPU (MacOS with Apple Silicon Chip) CPU only installation‚Äãpip install llama-cpp-pythonInstallation with OpenBLAS / cuBLAS / CLBlast‚Äãlama.cpp supports multiple BLAS backends for faster processing. Use the FORCE_CMAKE=1 environment variable
llama-cpp-python is a Python binding for llama.cpp.
llama-cpp-python is a Python binding for llama.cpp. ->: Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKProvidersAnthropicAWSGoogleMicrosoftOpenAIMoreComponentsLLMsLLMsAI21Aleph AlphaAmazon API GatewayAnyscaleArceeAzure MLAzure OpenAIBaidu QianfanBananaBasetenBeamBedrockBittensorCerebriumAIChatGLMClarifaiCohereC TransformersCTranslate2DatabricksDeepInfraDeepSparseEden AIFireworksForefrontAIGCP Vertex AIGooseAIGPT4AllGradientHugging Face HubHugging Face Local PipelinesHuggingface TextGen InferenceJavelin AI Gateway TutorialJSONFormerKoboldAI APILlama.cppLLM Caching integrationsManifestMinimaxModalMosaicMLNLP CloudOctoAIOllamaOpaquePromptsOpenAIOpenLLMOpenLMPetalsPipelineAIPredibasePrediction GuardPromptLayer OpenAIRELLMReplicateRunhouseSageMakerEndpointStochasticAINebula (Symbl.ai)TextGenTitan TakeoffTogether AITongyi QwenvLLMWriterXorbits Inference (Xinference)YandexGPTChat modelsDocument loadersDocument transformersText embedding modelsVector storesRetrieversToolsAgents and toolkitsMemoryCallbacksChat loadersComponentsLLMsLlama.cppOn this pageLlama.cppllama-cpp-python is a Python binding for llama.cpp. It supports inference for many LLMs, which can be accessed on HuggingFace.This notebook goes over how to run llama-cpp-python within LangChain.Note: new versions of llama-cpp-python use GGUF model files (see here).This is a breaking change.To convert existing GGML models to GGUF you can run the following in llama.cpp:python ./convert-llama-ggmlv3-to-gguf.py --eps 1e-5 --input models/openorca-platypus2-13b.ggmlv3.q4_0.bin --output models/openorca-platypus2-13b.gguf.q4_0.binInstallation‚ÄãThere are different options on how to install the llama-cpp package: CPU usageCPU + GPU (using one of many BLAS backends)Metal GPU (MacOS with Apple Silicon Chip) CPU only installation‚Äãpip install llama-cpp-pythonInstallation with OpenBLAS / cuBLAS / CLBlast‚Äãlama.cpp supports multiple BLAS backends for faster processing. Use the FORCE_CMAKE=1 environment variable