readme: ---
  pipeline_tag: question-answering
  datasets:
  - squad
  metrics:
  - exact_match
  - f1
  tags:
  - distilbert
  ---
  
  **task**: `question-answering`  
  **Backend:** `sagemaker-training`  
  **Backend args:** `{'instance_type': 'ml.g4dn.2xlarge', 'supported_instructions': None}`  
  **Number of evaluation samples:** `1000`  
  
  Fixed parameters:
  * **model_name_or_path**: `distilbert-base-uncased-distilled-squad`
  * **dataset**:
      * **path**: `squad`
      * **eval_split**: `validation`
      * **data_keys**: `{'question': 'question', 'context': 'context'}`
      * **ref_keys**: `['answers']`
      * **calibration_split**: `train`
  * **per_channel**: `False`
  * **calibration**:
      * **method**: `minmax`
      * **num_calibration_samples**: `100`
  * **framework**: `onnxruntime`
  * **framework_args**:
      * **opset**: `11`
      * **optimization_level**: `1`
  * **aware_training**: `False`
  
  Benchmarked parameters:
  * **quantization_approach**: `dynamic`,  `static`
  * **operators_to_quantize**: `['Add']`,  `['Add', 'MatMul']`
  * **node_exclusion**: `[]`,  `['layernorm', 'gelu', 'residual', 'gather', 'softmax']`
  
  # Evaluation
  ## Non-time metrics
  | quantization_approach | operators_to_quantize |                      node_exclusion                      |     | exact_match (original) | exact_match (optimized) |     | f1 (original) | f1 (optimized) |
  | :-------------------: | :-------------------: | :------------------------------------------------------: | :-: | :--------------------: | :---------------------: | :-: | :-----------: | :------------: |
  |       `dynamic`       |  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |  \|  |         82.300         |         80.600          |  \|  |    87.232     |     86.097     |
  |       `dynamic`       |  `['Add', 'MatMul']`  |                           `[]`                           |  \|  |         82.300         |         80.600          |  \|  |    87.232     |     86.097     |
  |       `dynamic`       |       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |  \|  |         82.300         |         82.300          |  \|  |    87.232     |     87.232     |
  |       `dynamic`       |       `['Add']`       |                           `[]`                           |  \|  |         82.300         |         82.300          |  \|  |    87.232     |     87.232     |
  |       `static`        |  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |  \|  |         82.300         |         72.900          |  \|  |    87.232     |     79.964     |
  |       `static`        |  `['Add', 'MatMul']`  |                           `[]`                           |  \|  |         82.300         |         54.500          |  \|  |    87.232     |     64.292     |
  |       `static`        |       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |  \|  |         82.300         |         76.900          |  \|  |    87.232     |     83.014     |
  |       `static`        |       `['Add']`       |                           `[]`                           |  \|  |         82.300         |         59.800          |  \|  |    87.232     |     69.217     |
  
  ## Time metrics
  Time benchmarks were run for 15 seconds per config.
  
  
  Below, time metrics for batch size = 1, input length = 32.
  
  | quantization_approach | operators_to_quantize |                      node_exclusion                      |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
  | :-------------------: | :-------------------: | :------------------------------------------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
  |       `dynamic`       |  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |  \|  |            47.87            |             7.23             |  \|  |           20.93           |           138.40           |
  |       `dynamic`       |  `['Add', 'MatMul']`  |                           `[]`                           |  \|  |            48.10            |             7.14             |  \|  |           20.80           |           140.13           |
  |       `dynamic`       |       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |  \|  |            43.83            |            17.16             |  \|  |           22.87           |           58.33            |
  |       `dynamic`       |       `['Add']`       |                           `[]`                           |  \|  |            34.13            |            17.02             |  \|  |           29.33           |           58.80            |
  |       `static`        |  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |  \|  |            35.07            |             9.21             |  \|  |           28.53           |           108.53           |