Commit
Β·
d6f889d
1
Parent(s):
4f66b85
Update README.md
Browse files
README.md
CHANGED
@@ -1,14 +1,177 @@
|
|
1 |
---
|
2 |
-
title:
|
3 |
-
emoji:
|
4 |
colorFrom: indigo
|
5 |
colorTo: green
|
6 |
sdk: streamlit
|
7 |
sdk_version: 1.44.1
|
8 |
app_file: app.py
|
9 |
-
pinned:
|
10 |
license: apache-2.0
|
11 |
-
short_description: fbaldassarri
|
12 |
---
|
13 |
|
14 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
+
title: fbaldassarri Repository Eval
|
3 |
+
emoji: π»
|
4 |
colorFrom: indigo
|
5 |
colorTo: green
|
6 |
sdk: streamlit
|
7 |
sdk_version: 1.44.1
|
8 |
app_file: app.py
|
9 |
+
pinned: true
|
10 |
license: apache-2.0
|
11 |
+
short_description: Evaluation space for fbaldassarri Repository
|
12 |
---
|
13 |
|
14 |
+
# fbaldassarri Model Comparison Tool
|
15 |
+
|
16 |
+
## π Features
|
17 |
+
|
18 |
+
- **Model Discovery**: Automatically finds all quantized models in your HuggingFace account
|
19 |
+
- **Smart Filtering**: Filter models by quantization method, base model, or custom text
|
20 |
+
- **Detailed Metadata Analysis**: Compare model architectures, sizes, and quantization details
|
21 |
+
- **Repository Statistics**: Track downloads, likes, and update frequency
|
22 |
+
- **Interactive Visualizations**: Multiple charts and graphs for performance analysis
|
23 |
+
- **Selective Comparison**: Use `.modelsignore` to exclude irrelevant models
|
24 |
+
- **CPU-Optimized**: Works perfectly on free tier HuggingFace Spaces without GPU
|
25 |
+
- **Export Capabilities**: Download comparison results as CSV for further analysis
|
26 |
+
|
27 |
+
## π Usage Instructions
|
28 |
+
|
29 |
+
### Basic Usage
|
30 |
+
|
31 |
+
1. Enter your HuggingFace username
|
32 |
+
2. The tool will automatically discover all your quantized models
|
33 |
+
3. Use the filters to select models by quantization method or base model
|
34 |
+
4. Click "Run Comparison" to analyze the selected models
|
35 |
+
5. View the results in the "Model Comparison", "Model Details", and "Visualizations" tabs
|
36 |
+
|
37 |
+
### Advanced Features
|
38 |
+
|
39 |
+
#### Filtering Models
|
40 |
+
|
41 |
+
- **Quantization Method**: Select specific quantization methods (Intel AutoRound, AutoGPTQ, AutoAWQ)
|
42 |
+
- **Base Model**: Compare models derived from the same base architecture
|
43 |
+
- **Text Filter**: Find models containing specific text in their names
|
44 |
+
|
45 |
+
#### Using .modelsignore
|
46 |
+
|
47 |
+
Create a `.modelsignore` file in the same directory as `app.py` to exclude specific models from discovery:
|
48 |
+
|
49 |
+
```
|
50 |
+
# Comments start with #
|
51 |
+
fbaldassarri/llama-2-7b-* # Ignores all llama-2-7b models
|
52 |
+
*mistral* # Ignores anything with "mistral" in the name
|
53 |
+
fbaldassarri/exact-model-name # Ignores a specific model
|
54 |
+
```
|
55 |
+
|
56 |
+
Supports wildcards and exact matching patterns.
|
57 |
+
|
58 |
+
#### Comparison Methods
|
59 |
+
|
60 |
+
Two comparison methods are available:
|
61 |
+
|
62 |
+
1. **Metadata Comparison Only**: Fast analysis of model metadata and architecture
|
63 |
+
2. **Metadata + Estimated Size**: Additionally calculates the disk size of each model
|
64 |
+
|
65 |
+
## π Visualization Types
|
66 |
+
|
67 |
+
The tool provides multiple visualization options:
|
68 |
+
|
69 |
+
### 1. Quantization Methods
|
70 |
+
|
71 |
+
- Distribution of quantization methods (pie chart)
|
72 |
+
- Distribution of precision formats (bar chart)
|
73 |
+
- Distribution of group sizes (bar chart)
|
74 |
+
|
75 |
+
### 2. Model Architecture
|
76 |
+
|
77 |
+
- Model size comparison (bar chart)
|
78 |
+
- Layer count comparison (bar chart)
|
79 |
+
- Hidden size comparison (bar chart)
|
80 |
+
- Attention heads comparison (bar chart)
|
81 |
+
|
82 |
+
### 3. Repository Statistics
|
83 |
+
|
84 |
+
- Downloads comparison (bar chart)
|
85 |
+
- Likes comparison (bar chart)
|
86 |
+
- Days since update comparison (bar chart)
|
87 |
+
|
88 |
+
## π§ Technical Details
|
89 |
+
|
90 |
+
### Supported Quantization Methods
|
91 |
+
|
92 |
+
The tool is designed to detect and analyze:
|
93 |
+
|
94 |
+
- **Intel AutoRound**: Intel's quantization solution
|
95 |
+
- **AutoGPTQ**: Automatic GPTQ quantization
|
96 |
+
- **AutoAWQ**: Activation-aware weight quantization
|
97 |
+
|
98 |
+
### Metadata Extraction
|
99 |
+
|
100 |
+
For each model, the tool extracts:
|
101 |
+
|
102 |
+
- Quantization method (based on name and tags)
|
103 |
+
- Precision (INT4, INT8, FP16, FP32)
|
104 |
+
- Group size (e.g., gs128)
|
105 |
+
- Base model architecture
|
106 |
+
- Hidden size and layer count
|
107 |
+
- Attention head count
|
108 |
+
- Maximum sequence length
|
109 |
+
|
110 |
+
### Size Estimation
|
111 |
+
|
112 |
+
When the "Metadata + Estimated Size" option is selected, the tool:
|
113 |
+
|
114 |
+
1. Fetches the list of model files from the repository
|
115 |
+
2. Identifies `.bin` and `.safetensors` files
|
116 |
+
3. Calculates their total size in GB
|
117 |
+
|
118 |
+
This avoids downloading the actual model weights, making it CPU-friendly.
|
119 |
+
|
120 |
+
## π Example Use Cases
|
121 |
+
|
122 |
+
1. **Comparing Quantization Methods**: Determine which method (AutoGPTQ vs AutoAWQ) produces smaller models
|
123 |
+
2. **Quantization Parameter Analysis**: See how different group sizes affect model size
|
124 |
+
3. **Popularity Tracking**: Identify which of your quantized models has the most downloads
|
125 |
+
4. **Architecture Analysis**: Compare hidden sizes and layer counts across different quantized variants
|
126 |
+
|
127 |
+
## β οΈ Troubleshooting
|
128 |
+
|
129 |
+
### Common Issues
|
130 |
+
|
131 |
+
1. **Slow Model Discovery**
|
132 |
+
- The tool may take time to discover models if you have many repositories
|
133 |
+
- Use the `.modelsignore` file to exclude irrelevant models and speed up discovery
|
134 |
+
2. **Memory Issues**
|
135 |
+
- The application is optimized for CPU usage, but comparing many models simultaneously may cause memory pressure
|
136 |
+
- Use the "Maximum models to compare" slider to limit the number of models
|
137 |
+
3. **Error: "Cannot subtract tz-naive and tz-aware datetime-like objects"**
|
138 |
+
- This error has been fixed in the latest version
|
139 |
+
- Update to the latest version if you encounter it
|
140 |
+
4. **Missing Model Information**
|
141 |
+
- Some information may be unavailable if models don't have standard config files
|
142 |
+
- The tool will show "Unknown" for fields it couldn't extract
|
143 |
+
|
144 |
+
## π Requirements
|
145 |
+
|
146 |
+
- Python 3.9+
|
147 |
+
- huggingface_hub
|
148 |
+
- streamlit
|
149 |
+
- pandas
|
150 |
+
- plotly
|
151 |
+
- numpy
|
152 |
+
|
153 |
+
See `requirements.txt` for a complete list of dependencies.
|
154 |
+
|
155 |
+
## π License
|
156 |
+
|
157 |
+
This project is licensed under the MIT License - see the LICENSE file for details.
|
158 |
+
|
159 |
+
## π Acknowledgements
|
160 |
+
|
161 |
+
- HuggingFace for the Hub API and Spaces platform
|
162 |
+
- The developers of AutoGPTQ, AutoAWQ, and Intel AutoRound
|
163 |
+
- Streamlit for the interactive web framework
|
164 |
+
|
165 |
+
## π€ Author
|
166 |
+
|
167 |
+
Created by Francesco Baldassarri
|
168 |
+
|
169 |
+
## π€ Contributing
|
170 |
+
|
171 |
+
Contributions, issues, and feature requests are welcome!
|
172 |
+
|
173 |
+
1. Fork the Project
|
174 |
+
2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
|
175 |
+
3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)
|
176 |
+
4. Push to the Branch (`git push origin feature/AmazingFeature`)
|
177 |
+
5. Open a Pull Request
|