readme: ---
  license: mit
  tags:
  - endpoints-template
  - optimum
  library_name: generic
  ---
  
  # Optimized and Quantized [deepset/roberta-base-squad2](https://huggingface.co/deepset/roberta-base-squad2) with a custom handler.py
  
  
  This repository implements a `custom` handler for `question-answering` for 🤗 Inference Endpoints for accelerated inference using [🤗 Optiumum](https://huggingface.co/docs/optimum/index). The code for the customized handler is in the [handler.py](https://huggingface.co/philschmid/roberta-base-squad2-optimized/blob/main/handler.py).
  
  Below is also describe how we converted & optimized the model, based on the [Accelerate Transformers with Hugging Face Optimum](https://huggingface.co/blog/optimum-inference) blog post. You can also check out the [notebook](https://huggingface.co/philschmid/roberta-base-squad2-optimized/blob/main/optimize_model.ipynb).
  
  ### expected Request payload
  
  ```json
  {
      "inputs": {
          "question": "As what is Philipp working?", 
          "context": "Hello, my name is Philipp and I live in Nuremberg, Germany. Currently I am working as a Technical Lead at Hugging Face to democratize artificial intelligence through open source and open science. In the past I designed and implemented cloud-native machine learning architectures for fin-tech and insurance companies. I found my passion for cloud concepts and machine learning 5 years ago. Since then I never stopped learning. Currently, I am focusing myself in the area NLP and how to leverage models like BERT, Roberta, T5, ViT, and GPT2 to generate business value."
      }
  }
  ```
  
  below is an example on how to run a request using Python and `requests`.
  
  ## Run Request 
  
  ```python
  import json
  from typing import List
  import requests as r
  import base64
  
  ENDPOINT_URL = ""
  HF_TOKEN = ""
  
  
  def predict(question:str=None,context:str=None):
      payload = {"inputs": {"question": question, "context": context}}
      response = r.post(
          ENDPOINT_URL, headers={"Authorization": f"Bearer {HF_TOKEN}"}, json=payload
      )
      return response.json()
  
  
  prediction = predict(
      question="As what is Philipp working?",
      context="Hello, my name is Philipp and I live in Nuremberg, Germany. Currently I am working as a Technical Lead at Hugging Face to democratize artificial intelligence through open source and open science."
  )
  ```
  
  expected output
  
  ```python
  {
      'score': 0.4749588668346405,
      'start': 88,
      'end': 102,
      'answer': 'Technical Lead'
  }
  ```
  
  
  
  # Convert & Optimize model with Optimum 
  
  Steps:
  1. [Convert model to ONNX](#1-convert-model-to-onnx)
  2. [Optimize & quantize model with Optimum](#2-optimize--quantize-model-with-optimum)
  3. [Create Custom Handler for Inference Endpoints](#3-create-custom-handler-for-inference-endpoints)
  4. [Test Custom Handler Locally](#4-test-custom-handler-locally)
  5. [Push to repository and create Inference Endpoint](#5-push-to-repository-and-create-inference-endpoint)
  
  Helpful links:
  * [Accelerate Transformers with Hugging Face Optimum](https://huggingface.co/blog/optimum-inference)
  * [Optimizing Transformers for GPUs with Optimum](https://www.philschmid.de/optimizing-transformers-with-optimum-gpu)
  * [Optimum Documentation](https://huggingface.co/docs/optimum/onnxruntime/modeling_ort)
  * [Create Custom Handler Endpoints](https://link-to-docs)
  
  ## Setup & Installation
  
  
  ```python
  %%writefile requirements.txt
  optimum[onnxruntime]==1.4.0
  mkl-include
  mkl
  ```
  
  
  ```python
  !pip install -r requirements.txt
  ```
  
  ## 0. Base line Performance
  
  
  ```python
  from transformers import pipeline
  
  qa = pipeline("question-answering",model="deepset/roberta-base-squad2")
  ```
  
  Okay, let's test the performance (latency) with sequence length of 128.
  
  
  ```python
  context="Hello, my name is Philipp and I live in Nuremberg, Germany. Currently I am working as a Technical Lead at Hugging Face to democratize artificial intelligence through open source and open science. In the past I designed and implemented cloud-native machine learning architectures for fin-tech and insurance companies. I found my passion for cloud concepts and machine learning 5 years ago. Since then I never stopped learning. Currently, I am focusing myself in the area NLP and how to leverage models like BERT, Roberta, T5, ViT, and GPT2 to generate business value." 
  question="As what is Philipp working?" 
  
  payload = {"inputs": {"question": question, "context": context}}
  ```
  
  
  ```python
  from time import perf_counter
  import numpy as np 
  
  def measure_latency(pipe,payload):
      latencies = []
      # warm up
      for _ in range(10):
          _ = pipe(question=payload["inputs"]["question"], context=payload["inputs"]["context"])
      # Timed run
      for _ in range(50):
          start_time = perf_counter()
          _ =  pipe(question=payload["inputs"]["question"], context=payload["inputs"]["context"])
          latency = perf_counter() - start_time
          latencies.append(latency)
      # Compute run statistics
      time_avg_ms = 1000 * np.mean(latencies)
      time_std_ms = 1000 * np.st