readme: ---
  pipeline_tag: image-classification
  datasets:
  - beans
  metrics:
  - accuracy
  tags:
  - vit
  ---
  
  **task**: `image-classification`  
  **Backend:** `sagemaker-training`  
  **Backend args:** `{'instance_type': 'ml.g4dn.2xlarge', 'supported_instructions': None}`  
  **Number of evaluation samples:** `All dataset`  
  
  Fixed parameters:
  * **model_name_or_path**: `nateraw/vit-base-beans`
  * **dataset**:
      * **path**: `beans`
      * **eval_split**: `validation`
      * **data_keys**: `{'primary': 'image'}`
      * **ref_keys**: `['labels']`
      * **calibration_split**: `train`
  * **quantization_approach**: `dynamic`
  * **calibration**:
      * **method**: `minmax`
      * **num_calibration_samples**: `100`
  * **framework**: `onnxruntime`
  * **framework_args**:
      * **opset**: `11`
      * **optimization_level**: `1`
  * **aware_training**: `False`
  
  Benchmarked parameters:
  * **operators_to_quantize**: `['Add']`,  `['Add', 'MatMul']`
  * **node_exclusion**: `[]`,  `['layernorm', 'gelu', 'residual', 'gather', 'softmax']`
  * **per_channel**: `False`,  `True`
  
  # Evaluation
  ## Non-time metrics
  | operators_to_quantize |                      node_exclusion                      | per_channel |     | accuracy (original) | accuracy (optimized) |
  | :-------------------: | :------------------------------------------------------: | :---------: | :-: | :-----------------: | :------------------: |
  |  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |        0.980        |        0.980         |
  |  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |        0.980        |        0.980         |
  |  `['Add', 'MatMul']`  |                           `[]`                           |   `False`   |  \|  |        0.980        |        0.980         |
  |  `['Add', 'MatMul']`  |                           `[]`                           |   `True`    |  \|  |        0.980        |        0.980         |
  |       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |        0.980        |        0.980         |
  |       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |        0.980        |        0.980         |
  |       `['Add']`       |                           `[]`                           |   `False`   |  \|  |        0.980        |        0.980         |
  |       `['Add']`       |                           `[]`                           |   `True`    |  \|  |        0.980        |        0.980         |
  
  ## Time metrics
  Time benchmarks were run for 15 seconds per config.
  
  
  Below, time metrics for batch size = 1, input length = 32.
  
  | operators_to_quantize |                      node_exclusion                      | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
  | :-------------------: | :------------------------------------------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
  |  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           200.50            |            63.00             |  \|  |           5.00            |           15.93            |
  |  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           198.19            |            72.65             |  \|  |           5.07            |           13.80            |
  |  `['Add', 'MatMul']`  |                           `[]`                           |   `False`   |  \|  |           191.44            |            63.27             |  \|  |           5.27            |           15.87            |
  |  `['Add', 'MatMul']`  |                           `[]`                           |   `True`    |  \|  |           154.84            |            72.51             |  \|  |           6.47            |           13.80            |
  |       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           155.84            |            130.95            |  \|  |           6.47            |            7.67            |
  |       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           201.76            |            131.25            |  \|  |           5.00            |            7.67            |
  |       `['Add']`       |                           `[]`                           |   `False`   |  \|  |           198.96            |            128.82            |  \|  |           5.07            |            7.80            |
  |       `['Add']`       |                           `[]`                           |   `True`    |  \|  |           163.76            |            129.62            |  \|  |           6.13            |            7.