Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKGet startedIntroductionInstallationQuickstartLangChain Expression LanguageInterfaceHow toCookbookLangChain Expression Language (LCEL)Why use LCEL?ModulesModel I/‚ÄãORetrievalChainsMemoryAgentsCallbacksModulesSecurityGuidesAdaptersDebuggingDeploymentEvaluationFallbacksLangSmithRun LLMs locallyModel comparisonPrivacyPydantic compatibilitySafetyMoreGuidesRun LLMs locallyOn this pageRun LLMs locallyUse case‚ÄãThe popularity of projects like PrivateGPT, llama.cpp, and GPT4All underscore the demand to run LLMs locally (on your own device).This has at least two important benefits:Privacy: Your data is not sent to a third party, and it is not subject to the terms of service of a commercial serviceCost: There is no inference fee, which is important for token-intensive applications (e.g., long-running simulations, summarization)Overview‚ÄãRunning an LLM locally requires a few things:Open-source LLM: An open-source LLM that can be freely modified and shared Inference: Ability to run this LLM on your device w/ acceptable latencyOpen-source LLMs‚ÄãUsers can now gain access to a rapidly growing set of open-source LLMs. These LLMs can be assessed across at least two dimensions (see figure):Base model: What is the base-model and how was it trained?Fine-tuning approach: Was the base-model fine-tuned and, if so, what set of instructions was used?The relative performance of these models can be assessed using several leaderboards, including:LmSysGPT4AllHuggingFaceInference‚ÄãA few frameworks for this have emerged to support inference of open-source LLMs on various devices:llama.cpp: C++ implementation of llama inference code with weight optimization / quantizationgpt4all: Optimized C backend for inferenceOllama: Bundles model weights and environment into an app that runs on device and serves the LLM In general, these frameworks will do a few things:Quantization:
Use case
Use case ->: Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKGet startedIntroductionInstallationQuickstartLangChain Expression LanguageInterfaceHow toCookbookLangChain Expression Language (LCEL)Why use LCEL?ModulesModel I/‚ÄãORetrievalChainsMemoryAgentsCallbacksModulesSecurityGuidesAdaptersDebuggingDeploymentEvaluationFallbacksLangSmithRun LLMs locallyModel comparisonPrivacyPydantic compatibilitySafetyMoreGuidesRun LLMs locallyOn this pageRun LLMs locallyUse case‚ÄãThe popularity of projects like PrivateGPT, llama.cpp, and GPT4All underscore the demand to run LLMs locally (on your own device).This has at least two important benefits:Privacy: Your data is not sent to a third party, and it is not subject to the terms of service of a commercial serviceCost: There is no inference fee, which is important for token-intensive applications (e.g., long-running simulations, summarization)Overview‚ÄãRunning an LLM locally requires a few things:Open-source LLM: An open-source LLM that can be freely modified and shared Inference: Ability to run this LLM on your device w/ acceptable latencyOpen-source LLMs‚ÄãUsers can now gain access to a rapidly growing set of open-source LLMs. These LLMs can be assessed across at least two dimensions (see figure):Base model: What is the base-model and how was it trained?Fine-tuning approach: Was the base-model fine-tuned and, if so, what set of instructions was used?The relative performance of these models can be assessed using several leaderboards, including:LmSysGPT4AllHuggingFaceInference‚ÄãA few frameworks for this have emerged to support inference of open-source LLMs on various devices:llama.cpp: C++ implementation of llama inference code with weight optimization / quantizationgpt4all: Optimized C backend for inferenceOllama: Bundles model weights and environment into an app that runs on device and serves the LLM In general, these frameworks will do a few things:Quantization: