readme: ---
  pipeline_tag: image-classification
  datasets:
  - beans
  metrics:
  - accuracy
  tags:
  - vit
  ---
  
  **task**: `image-classification`  
  **Backend:** `sagemaker-training`  
  **Backend args:** `{'instance_type': 'ml.g4dn.2xlarge', 'supported_instructions': None}`  
  **Number of evaluation samples:** `All dataset`  
  
  Fixed parameters:
  * **model_name_or_path**: `nateraw/vit-base-beans`
  * **dataset**:
      * **path**: `beans`
      * **eval_split**: `validation`
      * **data_keys**: `{'primary': 'image'}`
      * **ref_keys**: `['labels']`
  * **quantization_approach**: `dynamic`
  * **node_exclusion**: `[]`
  * **framework**: `onnxruntime`
  * **framework_args**:
      * **opset**: `11`
      * **optimization_level**: `1`
  * **aware_training**: `False`
  
  Benchmarked parameters:
  * **operators_to_quantize**: `['Add', 'MatMul']`,  `['Add']`,  `[]`
  * **per_channel**: `False`,  `True`
  
  # Evaluation
  ## Non-time metrics
  | operators_to_quantize | per_channel |     | accuracy (original) | accuracy (optimized) |
  | :-------------------: | :---------: | :-: | :-----------------: | :------------------: |
  |  `['Add', 'MatMul']`  |   `False`   |  \|  |        0.980        |        0.980         |
  |  `['Add', 'MatMul']`  |   `True`    |  \|  |        0.980        |        0.980         |
  |       `['Add']`       |   `False`   |  \|  |        0.980        |        0.980         |
  |       `['Add']`       |   `True`    |  \|  |        0.980        |        0.980         |
  |         `[]`          |   `False`   |  \|  |        0.980        |        0.980         |
  |         `[]`          |   `True`    |  \|  |        0.980        |        0.980         |
  
  ## Time metrics
  Time benchmarks were run for 15 seconds per config.
  
  
  Below, time metrics for batch size = 1, input length = 32.
  
  | operators_to_quantize | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
  | :-------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
  |  `['Add', 'MatMul']`  |   `False`   |  \|  |           201.25            |            70.30             |  \|  |           5.00            |           14.27            |
  |  `['Add', 'MatMul']`  |   `True`    |  \|  |           203.52            |            72.48             |  \|  |           4.93            |           13.80            |
  |       `['Add']`       |   `False`   |  \|  |           166.03            |            150.93            |  \|  |           6.07            |            6.67            |
  |       `['Add']`       |   `True`    |  \|  |           200.82            |            163.17            |  \|  |           5.00            |            6.13            |
  |         `[]`          |   `False`   |  \|  |           190.99            |            162.06            |  \|  |           5.27            |            6.20            |
  |         `[]`          |   `True`    |  \|  |           155.15            |            162.52            |  \|  |           6.47            |            6.20            |
  
  
  Below, time metrics for batch size = 1, input length = 64.
  
  | operators_to_quantize | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
  | :-------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
  |  `['Add', 'MatMul']`  |   `False`   |  \|  |           165.85            |            70.60             |  \|  |           6.07            |           14.20            |
  |  `['Add', 'MatMul']`  |   `True`    |  \|  |           161.41            |            72.71             |  \|  |           6.20            |           13.80            |
  |       `['Add']`       |   `False`   |  \|  |           200.45            |            129.40            |  \|  |           5.00            |            7.73            |
  |       `['Add']`       |   `True`    |  \|  |           154.68            |            136.42            |  \|  |           6.47            |            7.40            |
  |         `[]`          |   `False`   |  \|  |           166.97            |            162.15            |  \|  |           6.00            |            6.20            |
  |         `[]`          |   `True`    |  \|  |           166.32            |            162.81            |  \|  |           6.07            |            6.20            |
  
  
  Below, time metrics for batch size = 1, input length = 128.
  
  | operators_to_quantize | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
  | :-------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
  |  `['Add', 'MatMul']`  |   `Fa