It is broken into two parts: Modal installation and web endpoint deploymentUsing deployed web endpoint with LLM wrapper class.Installation and Setup‚ÄãInstall with pip install modalRun modal token newDefine your Modal Functions and Webhooks‚ÄãYou must include a prompt. There is a rigid response structure:class Item(BaseModel):    prompt: str@stub.function()@modal.web_endpoint(method="POST")def get_text(item: Item):    return {"prompt": run_gpt2.call(item.prompt)}The following is an example with the GPT2 model:from pydantic import BaseModelimport modalCACHE_PATH = "/root/model_cache"class Item(BaseModel):    prompt: strstub = modal.Stub(name="example-get-started-with-langchain")def download_model():    from transformers import GPT2Tokenizer, GPT2LMHeadModel    tokenizer = GPT2Tokenizer.from_pretrained('gpt2')    model = GPT2LMHeadModel.from_pretrained('gpt2')    tokenizer.save_pretrained(CACHE_PATH)    model.save_pretrained(CACHE_PATH)# Define a container image for the LLM function below, which# downloads and stores the GPT-2 model.image = modal.Image.debian_slim().pip_install(    "tokenizers", "transformers", "torch", "accelerate").run_function(download_model)@stub.function(    gpu="any",    image=image,    retries=3,)def run_gpt2(text: str):    from transformers import GPT2Tokenizer, GPT2LMHeadModel    tokenizer = GPT2Tokenizer.from_pretrained(CACHE_PATH)    model = GPT2LMHeadModel.from_pretrained(CACHE_PATH)    encoded_input = tokenizer(text, return_tensors='pt').input_ids    output = model.generate(encoded_input, max_length=50, do_sample=True)    return tokenizer.decode(output[0], skip_special_tokens=True)@stub.function()@modal.web_endpoint(method="POST")def get_text(item: Item):    return {"prompt": run_gpt2.call(item.prompt)}Deploy the web endpoint‚ÄãDeploy the web endpoint to Modal cloud with the modal deploy CLI command.
This page covers how to use the Modal ecosystem to run LangChain custom LLMs.
This page covers how to use the Modal ecosystem to run LangChain custom LLMs. ->: It is broken into two parts: Modal installation and web endpoint deploymentUsing deployed web endpoint with LLM wrapper class.Installation and Setup‚ÄãInstall with pip install modalRun modal token newDefine your Modal Functions and Webhooks‚ÄãYou must include a prompt. There is a rigid response structure:class Item(BaseModel):    prompt: str@stub.function()@modal.web_endpoint(method="POST")def get_text(item: Item):    return {"prompt": run_gpt2.call(item.prompt)}The following is an example with the GPT2 model:from pydantic import BaseModelimport modalCACHE_PATH = "/root/model_cache"class Item(BaseModel):    prompt: strstub = modal.Stub(name="example-get-started-with-langchain")def download_model():    from transformers import GPT2Tokenizer, GPT2LMHeadModel    tokenizer = GPT2Tokenizer.from_pretrained('gpt2')    model = GPT2LMHeadModel.from_pretrained('gpt2')    tokenizer.save_pretrained(CACHE_PATH)    model.save_pretrained(CACHE_PATH)# Define a container image for the LLM function below, which# downloads and stores the GPT-2 model.image = modal.Image.debian_slim().pip_install(    "tokenizers", "transformers", "torch", "accelerate").run_function(download_model)@stub.function(    gpu="any",    image=image,    retries=3,)def run_gpt2(text: str):    from transformers import GPT2Tokenizer, GPT2LMHeadModel    tokenizer = GPT2Tokenizer.from_pretrained(CACHE_PATH)    model = GPT2LMHeadModel.from_pretrained(CACHE_PATH)    encoded_input = tokenizer(text, return_tensors='pt').input_ids    output = model.generate(encoded_input, max_length=50, do_sample=True)    return tokenizer.decode(output[0], skip_special_tokens=True)@stub.function()@modal.web_endpoint(method="POST")def get_text(item: Item):    return {"prompt": run_gpt2.call(item.prompt)}Deploy the web endpoint‚ÄãDeploy the web endpoint to Modal cloud with the modal deploy CLI command.