and toolkitsMemoryCallbacksChat loadersProvidersMoreRay ServeOn this pageRay ServeRay Serve is a scalable model serving library for building online inference APIs. Serve is particularly well suited for system composition, enabling you to build a complex inference service consisting of multiple chains and business logic all in Python code. Goal of this notebook‚ÄãThis notebook shows a simple example of how to deploy an OpenAI chain into production. You can extend it to deploy your own self-hosted models where you can easily define amount of hardware resources (GPUs and CPUs) needed to run your model in production efficiently. Read more about available options including autoscaling in the Ray Serve documentation.Setup Ray Serve‚ÄãInstall ray with pip install ray[serve]. General Skeleton‚ÄãThe general skeleton for deploying a service is the following:# 0: Import ray serve and request from starlettefrom ray import servefrom starlette.requests import Request# 1: Define a Ray Serve deployment.@serve.deploymentclass LLMServe:    def __init__(self) -> None:        # All the initialization code goes here        pass    async def __call__(self, request: Request) -> str:        # You can parse the request here        # and return a response        return "Hello World"# 2: Bind the model to deploymentdeployment = LLMServe.bind()# 3: Run the deploymentserve.api.run(deployment)# Shutdown the deploymentserve.api.shutdown()Example of deploying and OpenAI chain with custom prompts‚ÄãGet an OpenAI API key from here. By running the following code, you will be asked to provide your API key.from langchain.llms import OpenAIfrom langchain.prompts import PromptTemplatefrom langchain.chains import LLMChainfrom getpass import getpassOPENAI_API_KEY = getpass()@serve.deploymentclass DeployLLM:    def __init__(self):        # We initialize the LLM, template and the chain here        llm = OpenAI(openai_api_key=OPENAI_API_KEY)        template = "Question: {question}\n\nAnswer: Let's think step
Ray Serve is a scalable model serving library for building online inference APIs. Serve is particularly well suited for system composition, enabling you to build a complex inference service consisting of multiple chains and business logic all in Python code.
Ray Serve is a scalable model serving library for building online inference APIs. Serve is particularly well suited for system composition, enabling you to build a complex inference service consisting of multiple chains and business logic all in Python code. ->: and toolkitsMemoryCallbacksChat loadersProvidersMoreRay ServeOn this pageRay ServeRay Serve is a scalable model serving library for building online inference APIs. Serve is particularly well suited for system composition, enabling you to build a complex inference service consisting of multiple chains and business logic all in Python code. Goal of this notebook‚ÄãThis notebook shows a simple example of how to deploy an OpenAI chain into production. You can extend it to deploy your own self-hosted models where you can easily define amount of hardware resources (GPUs and CPUs) needed to run your model in production efficiently. Read more about available options including autoscaling in the Ray Serve documentation.Setup Ray Serve‚ÄãInstall ray with pip install ray[serve]. General Skeleton‚ÄãThe general skeleton for deploying a service is the following:# 0: Import ray serve and request from starlettefrom ray import servefrom starlette.requests import Request# 1: Define a Ray Serve deployment.@serve.deploymentclass LLMServe:    def __init__(self) -> None:        # All the initialization code goes here        pass    async def __call__(self, request: Request) -> str:        # You can parse the request here        # and return a response        return "Hello World"# 2: Bind the model to deploymentdeployment = LLMServe.bind()# 3: Run the deploymentserve.api.run(deployment)# Shutdown the deploymentserve.api.shutdown()Example of deploying and OpenAI chain with custom prompts‚ÄãGet an OpenAI API key from here. By running the following code, you will be asked to provide your API key.from langchain.llms import OpenAIfrom langchain.prompts import PromptTemplatefrom langchain.chains import LLMChainfrom getpass import getpassOPENAI_API_KEY = getpass()@serve.deploymentclass DeployLLM:    def __init__(self):        # We initialize the LLM, template and the chain here        llm = OpenAI(openai_api_key=OPENAI_API_KEY)        template = "Question: {question}\n\nAnswer: Let's think step