readme: ---
  pipeline_tag: question-answering
  datasets:
  - squad
  metrics:
  - exact_match
  - f1
  tags:
  - distilbert
  ---
  
  **task**: `question-answering`
  
  Fixed parameters:
  * **model_name_or_path**: `distilbert-base-uncased-distilled-squad`
  * **dataset**:
      * **path**: `squad`
      * **name**: `None`
      * **calibration_split**: `train`
      * **eval_split**: `validation`
      * **data_keys**: `{'question': 'question', 'context': 'context'}`
      * **ref_keys**: `['answers']`
      * **max_seq_length**: `128`
  * **node_exclusion**: `[]`
  * **per_channel**: `False`
  * **calibration**:
      * **method**: `minmax`
      * **num_calibration_samples**: `20`
      * **calibration_histogram_percentile**: `None`
      * **calibration_moving_average**: `None`
      * **calibration_moving_average_constant**: `None`
  * **framework**: `onnxruntime`
  * **framework_args**:
      * **opset**: `15`
      * **optimization_level**: `1`
  * **aware_training**: `False`
  
  Benchmarked parameters:
  * **quantization_approach**: `dynamic`,  `static`
  * **operators_to_quantize**: `['Add', 'MatMul']`,  `['Add']`
  
  ## Evaluation
  Below, time metrics for
  * Batch size: 8
  * Input length: 128
  | quantization_approach | operators_to_quantize |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |     | exact_match (original) | exact_match (optimized) |     | f1 (original) | f1 (optimized) |
  | :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: | :-: | :--------------------: | :---------------------: | :-: | :-----------: | :------------: |
  |       `dynamic`       |  `['Add', 'MatMul']`  |  \|  |           247.65            |            75.80             |  \|  |           4.50            |           13.50            |  \|  |         90.000         |         85.000          |  \|  |    90.000     |     86.667     |
  |       `dynamic`       |       `['Add']`       |  \|  |           234.61            |            191.78            |  \|  |           4.50            |            5.50            |  \|  |         90.000         |         90.000          |  \|  |    90.000     |     90.000     |
  |       `static`        |  `['Add', 'MatMul']`  |  \|  |           238.04            |            131.81            |  \|  |           4.50            |            8.00            |  \|  |         90.000         |         50.000          |  \|  |    90.000     |     59.362     |
  |       `static`        |       `['Add']`       |  \|  |           241.84            |            257.85            |  \|  |           4.50            |            4.00            |  \|  |         90.000         |         70.000          |  \|  |    90.000     |     76.333     |
  