readme: ---
  pipeline_tag: question-answering
  datasets:
  - squad
  metrics:
  - exact_match
  - f1
  - total_time_in_seconds
  - samples_per_second
  - latency_in_seconds
  tags:
  - distilbert
  ---
  
  **task**: `question-answering`  
  **Backend:** `sagemaker-training`  
  **Backend args:** `{'instance_type': 'ml.m5.2xlarge', 'supported_instructions': 'avx512'}`  
  **Number of evaluation samples:** `All dataset`  
  
  Fixed parameters:
  * **dataset**: [{'path': 'squad', 'eval_split': 'validation', 'data_keys': {'question': 'question', 'context': 'context'}, 'ref_keys': ['answers'], 'name': None, 'calibration_split': None}]
  * **name_or_path**: `distilbert-base-uncased-distilled-squad`
  * **from_transformers**: `True`
  * **quantization_approach**: `dynamic`
  
  Benchmarked parameters:
  * **framework**: `onnxruntime`,  `pytorch`
  * **operators_to_quantize**: `['Add', 'MatMul']`,  `['Add']`
  * **node_exclusion**: `[]`,  `['layernorm', 'gelu', 'residual', 'gather', 'softmax']`
  * **per_channel**: `False`,  `True`
  * **framework_args**: `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}`,  `{}`
  * **reduce_range**: `True`,  `False`
  * **apply_quantization**: `True`,  `False`
  
  # Evaluation
  ## Non-time metrics
  |   framework   | operators_to_quantize |                      node_exclusion                      | per_channel |                           framework_args                            | reduce_range | apply_quantization |     | exact_match |     |   f1   |
  | :-----------: | :-------------------: | :------------------------------------------------------: | :---------: | :-----------------------------------------------------------------: | :----------: | :----------------: | :-: | :---------: | :-: | :----: |
  | `onnxruntime` |        `None`         |                          `None`                          |   `None`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `None`    |      `False`       |  \|  |   78.884    |  \|  | 86.690 |
  | `onnxruntime` |  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |   76.764    |  \|  | 85.053 |
  | `onnxruntime` |  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |   69.622    |  \|  | 79.914 |
  | `onnxruntime` |  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |    0.435    |  \|  | 5.887  |
  | `onnxruntime` |  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |   78.165    |  \|  | 85.973 |
  | `onnxruntime` |  `['Add', 'MatMul']`  |                           `[]`                           |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |   76.764    |  \|  | 85.053 |
  | `onnxruntime` |  `['Add', 'MatMul']`  |                           `[]`                           |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |   69.622    |  \|  | 79.914 |
  | `onnxruntime` |  `['Add', 'MatMul']`  |                           `[]`                           |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |    0.435    |  \|  | 5.887  |
  | `onnxruntime` |  `['Add', 'MatMul']`  |                           `[]`                           |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |   78.165    |  \|  | 85.973 |
  | `onnxruntime` |       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |   78.884    |  \|  | 86.690 |
  | `onnxruntime` |       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |   78.884    |  \|  | 86.690 |
  | `onnxruntime` |       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |   78.884    |  \|  | 86.690 |
  | `onnxruntime` |       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |    