README.md

metadata

title: WhisperKit Android Benchmarks
emoji: 🏆
colorFrom: green
colorTo: indigo
sdk: gradio
app_file: main.py
license: mit

Prerequisites

Ensure you have the following software installed:

• Python 3.10 or higher
• pip (Python package installer)

Installation

1. Clone the repository:

   git clone https://github.com/argmaxinc/model-performance-dashboard.git
   cd model-performance-dashboard
   

2. Create a virtual environment:

   python -m venv venv
   source venv/bin/activate
   

3. Install required packages:

   pip install -r requirements.txt
   

Usage

1. Run the application:

   gradio main.py
   

2. Access the application: After running main.py, a local server will start, and you will see an interface URL in the terminal. Open the URL in your web browser to interact with Argmax Android Benchmark dashboard.

Data Generation

1. Performance Data Update (performance_generate.py):
   • Downloads benchmark data from WhisperKit Evals Dataset.
   • Processes the data to extract performance metrics for various models, devices, and operating systems.
   • Calculates metrics such as speed, tokens per second for long and short-form data.
   • Saves the results in performance_data.json and support_data.csv.

Data Update

To update the dashboard with latest data from our HuggingFace datasets, run:

    make use-huggingface-data

Alternatively, you can use our on-device testing code [TODO:INSERT_LINK_TO_OS_TEST_CODE] on your device to update the dashboard with your own data. After generating the Xcode data, place the resulting .json files in the whisperkit-evals/xcresults/benchmark_data directory, then run:

    make use-local-data