Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKProvidersAnthropicAWSGoogleMicrosoftOpenAIMoreComponentsLLMsLLMsAI21Aleph AlphaAmazon API GatewayAnyscaleArceeAzure MLAzure OpenAIBaidu QianfanBananaBasetenBeamBedrockBittensorCerebriumAIChatGLMClarifaiCohereC TransformersCTranslate2DatabricksDeepInfraDeepSparseEden AIFireworksForefrontAIGCP Vertex AIGooseAIGPT4AllGradientHugging Face HubHugging Face Local PipelinesHuggingface TextGen InferenceJavelin AI Gateway TutorialJSONFormerKoboldAI APILlama.cppLLM Caching integrationsManifestMinimaxModalMosaicMLNLP CloudOctoAIOllamaOpaquePromptsOpenAIOpenLLMOpenLMPetalsPipelineAIPredibasePrediction GuardPromptLayer OpenAIRELLMReplicateRunhouseSageMakerEndpointStochasticAINebula (Symbl.ai)TextGenTitan TakeoffTogether AITongyi QwenvLLMWriterXorbits Inference (Xinference)YandexGPTChat modelsDocument loadersDocument transformersText embedding modelsVector storesRetrieversToolsAgents and toolkitsMemoryCallbacksChat loadersComponentsLLMsHuggingface TextGen InferenceOn this pageHuggingface TextGen InferenceText Generation Inference is a Rust, Python and gRPC server for text generation inference. Used in production at HuggingFace to power LLMs api-inference widgets.This notebooks goes over how to use a self hosted LLM using Text Generation Inference.To use, you should have the text_generation python package installed.# !pip3 install text_generationfrom langchain.llms import HuggingFaceTextGenInferencellm = HuggingFaceTextGenInference(    inference_server_url="http://localhost:8010/",    max_new_tokens=512,    top_k=10,    top_p=0.95,    typical_p=0.95,    temperature=0.01,    repetition_penalty=1.03,)llm("What did foo say about bar?")Streaming‚Äãfrom langchain.llms import HuggingFaceTextGenInferencefrom langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandlerllm = HuggingFaceTextGenInference(    inference_server_url="http://localhost:8010/",
Text Generation Inference is a Rust, Python and gRPC server for text generation inference. Used in production at HuggingFace to power LLMs api-inference widgets.
Text Generation Inference is a Rust, Python and gRPC server for text generation inference. Used in production at HuggingFace to power LLMs api-inference widgets. ->: Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKProvidersAnthropicAWSGoogleMicrosoftOpenAIMoreComponentsLLMsLLMsAI21Aleph AlphaAmazon API GatewayAnyscaleArceeAzure MLAzure OpenAIBaidu QianfanBananaBasetenBeamBedrockBittensorCerebriumAIChatGLMClarifaiCohereC TransformersCTranslate2DatabricksDeepInfraDeepSparseEden AIFireworksForefrontAIGCP Vertex AIGooseAIGPT4AllGradientHugging Face HubHugging Face Local PipelinesHuggingface TextGen InferenceJavelin AI Gateway TutorialJSONFormerKoboldAI APILlama.cppLLM Caching integrationsManifestMinimaxModalMosaicMLNLP CloudOctoAIOllamaOpaquePromptsOpenAIOpenLLMOpenLMPetalsPipelineAIPredibasePrediction GuardPromptLayer OpenAIRELLMReplicateRunhouseSageMakerEndpointStochasticAINebula (Symbl.ai)TextGenTitan TakeoffTogether AITongyi QwenvLLMWriterXorbits Inference (Xinference)YandexGPTChat modelsDocument loadersDocument transformersText embedding modelsVector storesRetrieversToolsAgents and toolkitsMemoryCallbacksChat loadersComponentsLLMsHuggingface TextGen InferenceOn this pageHuggingface TextGen InferenceText Generation Inference is a Rust, Python and gRPC server for text generation inference. Used in production at HuggingFace to power LLMs api-inference widgets.This notebooks goes over how to use a self hosted LLM using Text Generation Inference.To use, you should have the text_generation python package installed.# !pip3 install text_generationfrom langchain.llms import HuggingFaceTextGenInferencellm = HuggingFaceTextGenInference(    inference_server_url="http://localhost:8010/",    max_new_tokens=512,    top_k=10,    top_p=0.95,    typical_p=0.95,    temperature=0.01,    repetition_penalty=1.03,)llm("What did foo say about bar?")Streaming‚Äãfrom langchain.llms import HuggingFaceTextGenInferencefrom langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandlerllm = HuggingFaceTextGenInference(    inference_server_url="http://localhost:8010/",