Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKProvidersAnthropicAWSGoogleMicrosoftOpenAIMoreComponentsLLMsChat modelsChat modelsAnthropicAnthropic FunctionsAnyscaleAzureAzureML Chat Online EndpointBaichuan ChatBaidu QianfanBedrock ChatCohereERNIE-Bot ChatEverlyAIFireworksGCP Vertex AIJinaChatKonkoüöÖ LiteLLMLlama APIMiniMaxOllamaOpenAIPromptLayer ChatOpenAITongyi QwenvLLM ChatYandexGPTDocument loadersDocument transformersText embedding modelsVector storesRetrieversToolsAgents and toolkitsMemoryCallbacksChat loadersComponentsChat modelsvLLM ChatvLLM ChatvLLM can be deployed as a server that mimics the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API. This server can be queried in the same format as OpenAI API.This notebook covers how to get started with vLLM chat models using langchain's ChatOpenAI as it is.from langchain.chat_models import ChatOpenAIfrom langchain.prompts.chat import (    ChatPromptTemplate,    SystemMessagePromptTemplate,    AIMessagePromptTemplate,    HumanMessagePromptTemplate,)from langchain.schema import AIMessage, HumanMessage, SystemMessageinference_server_url = "http://localhost:8000/v1"chat = ChatOpenAI(    model="mosaicml/mpt-7b",    openai_api_key="EMPTY",    openai_api_base=inference_server_url,    max_tokens=5,    temperature=0,)messages = [    SystemMessage(        content="You are a helpful assistant that translates English to Italian."    ),    HumanMessage(        content="Translate the following sentence from English to Italian: I love programming."    ),]chat(messages)    AIMessage(content=' Io amo programmare', additional_kwargs={}, example=False)You can make use of templating by using a MessagePromptTemplate. You can build a ChatPromptTemplate from one or more MessagePromptTemplates. You can use ChatPromptTemplate's format_prompt -- this returns a PromptValue, which you can convert to a string
vLLM can be deployed as a server that mimics the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API. This server can be queried in the same format as OpenAI API.
vLLM can be deployed as a server that mimics the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API. This server can be queried in the same format as OpenAI API. ->: Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKProvidersAnthropicAWSGoogleMicrosoftOpenAIMoreComponentsLLMsChat modelsChat modelsAnthropicAnthropic FunctionsAnyscaleAzureAzureML Chat Online EndpointBaichuan ChatBaidu QianfanBedrock ChatCohereERNIE-Bot ChatEverlyAIFireworksGCP Vertex AIJinaChatKonkoüöÖ LiteLLMLlama APIMiniMaxOllamaOpenAIPromptLayer ChatOpenAITongyi QwenvLLM ChatYandexGPTDocument loadersDocument transformersText embedding modelsVector storesRetrieversToolsAgents and toolkitsMemoryCallbacksChat loadersComponentsChat modelsvLLM ChatvLLM ChatvLLM can be deployed as a server that mimics the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API. This server can be queried in the same format as OpenAI API.This notebook covers how to get started with vLLM chat models using langchain's ChatOpenAI as it is.from langchain.chat_models import ChatOpenAIfrom langchain.prompts.chat import (    ChatPromptTemplate,    SystemMessagePromptTemplate,    AIMessagePromptTemplate,    HumanMessagePromptTemplate,)from langchain.schema import AIMessage, HumanMessage, SystemMessageinference_server_url = "http://localhost:8000/v1"chat = ChatOpenAI(    model="mosaicml/mpt-7b",    openai_api_key="EMPTY",    openai_api_base=inference_server_url,    max_tokens=5,    temperature=0,)messages = [    SystemMessage(        content="You are a helpful assistant that translates English to Italian."    ),    HumanMessage(        content="Translate the following sentence from English to Italian: I love programming."    ),]chat(messages)    AIMessage(content=' Io amo programmare', additional_kwargs={}, example=False)You can make use of templating by using a MessagePromptTemplate. You can build a ChatPromptTemplate from one or more MessagePromptTemplates. You can use ChatPromptTemplate's format_prompt -- this returns a PromptValue, which you can convert to a string