in the returned json, and the value must be a string.# Return the results as a dictionaryresult = {'outputs': result}An example inference function would be:@app.handler("/")def handler(context: dict, request: Request) -> Response:    """Handle a request to generate code from a prompt."""    model = context.get("model")    tokenizer = context.get("tokenizer")    max_new_tokens = request.json.get("max_new_tokens", 512)    temperature = request.json.get("temperature", 0.7)    prompt = request.json.get("prompt")    prompt_template=f'''[INST] Write code to solve the following coding problem that obeys the constraints and passes the example test cases. Please wrap your code answer using ```:    {prompt}    [/INST]    '''    input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()    output = model.generate(inputs=input_ids, temperature=temperature, max_new_tokens=max_new_tokens)    result = tokenizer.decode(output[0])    return Response(json={"outputs": result}, status=200)This example is from the app.py file in CodeLlama-7B-Instruct-GPTQ.Wrappers‚ÄãLLM‚ÄãWithin Langchain, there exists a Banana LLM wrapper, which you can access withfrom langchain.llms import BananaYou need to provide a model key and model url slug, which you can get from the model's details page in the Banana.dev dashboard.llm = Banana(model_key="YOUR_MODEL_KEY", model_url_slug="YOUR_MODEL_URL_SLUG")PreviousBagelDBNextBasetenInstallation and SetupDefine your Banana TemplateBuild the Banana appWrappersLLMCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright ¬© 2023 LangChain, Inc.
Banana provided serverless GPU inference for AI models, including a CI/CD build pipeline and a simple Python framework (Potassium) to server your models.
Banana provided serverless GPU inference for AI models, including a CI/CD build pipeline and a simple Python framework (Potassium) to server your models. ->: in the returned json, and the value must be a string.# Return the results as a dictionaryresult = {'outputs': result}An example inference function would be:@app.handler("/")def handler(context: dict, request: Request) -> Response:    """Handle a request to generate code from a prompt."""    model = context.get("model")    tokenizer = context.get("tokenizer")    max_new_tokens = request.json.get("max_new_tokens", 512)    temperature = request.json.get("temperature", 0.7)    prompt = request.json.get("prompt")    prompt_template=f'''[INST] Write code to solve the following coding problem that obeys the constraints and passes the example test cases. Please wrap your code answer using ```:    {prompt}    [/INST]    '''    input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()    output = model.generate(inputs=input_ids, temperature=temperature, max_new_tokens=max_new_tokens)    result = tokenizer.decode(output[0])    return Response(json={"outputs": result}, status=200)This example is from the app.py file in CodeLlama-7B-Instruct-GPTQ.Wrappers‚ÄãLLM‚ÄãWithin Langchain, there exists a Banana LLM wrapper, which you can access withfrom langchain.llms import BananaYou need to provide a model key and model url slug, which you can get from the model's details page in the Banana.dev dashboard.llm = Banana(model_key="YOUR_MODEL_KEY", model_url_slug="YOUR_MODEL_URL_SLUG")PreviousBagelDBNextBasetenInstallation and SetupDefine your Banana TemplateBuild the Banana appWrappersLLMCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright ¬© 2023 LangChain, Inc.