readme: ---
  license: bsd-3-clause
  tags:
  - endpoints-template
  pipeline_tag: text-generation
  ---
  # Sharded fork of [Salesforce/codegen-6B-mono](https://huggingface.co/Salesforce/codegen-6B-mono) with a custom pipeline.py
  
  This repository implements a custom `pipeline` task for `text-generation` for 🤗 Inference Endpoints for LLM inference using bitsandbytes quantization. The code for the customized pipeline is in the [pipeline.py](https://huggingface.co/philschmid/codegen-6B-mono-sharded-bnb/blob/main/pipeline.py).
  
  There is also a [notebook](https://huggingface.co/philschmid/codegen-6B-mono-sharded-bnb/blob/main/create_handler.ipynb) included.
  
  ### expected Request payload
  ```json
  {
      "inputs": "# load distilbert model and initialize text-classification pipeline\nmodel_id = 'distil",
      "parameters": {
          "top_k": 100,
          "max_length": 64,
          "early_stopping": true,
          "do_sample": true,
          "eos_token_id": 50256,
      }
  }
  ```
  
  below is an example on how to run a request using Python and `requests`.
  
  ## Run Request 
  ```python
  import json
  from typing import List
  import requests as r
  import base64
  ENDPOINT_URL = ""
  HF_TOKEN = ""
  
  parameters={
          "top_k": 100,
          "max_length": 64,
          "early_stopping": True,
          "do_sample": True,
          "eos_token_id": 50256,
      }
  
  def predict(code_snippet:str=None):
      payload = {"inputs": code_snippet,"parameters": parameters}
      response = r.post(
          ENDPOINT_URL, headers={"Authorization": f"Bearer {HF_TOKEN}"}, json=payload
      )
      return response.json()
  prediction = predict(
      code_snippet="# load distilbert model and initialize text-classification pipeline\nmodel_id = 'distil"
  )
  ```
  expected output
  ```python
  {'generated_text': "# load distilbert model and initialize text-classification pipeline\nmodel_id = 'distilbert-base-uncased'\nmodel_url = 'https://tfhub.dev/tensorflow/small_bert/1'\n\nmodel_dir = './distilBERT'"}
  ```
  