fbaldassarri commited on
Commit
d6f889d
Β·
1 Parent(s): 4f66b85

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +168 -5
README.md CHANGED
@@ -1,14 +1,177 @@
1
  ---
2
- title: Fbaldassarri Repository Eval
3
- emoji: πŸ¦€
4
  colorFrom: indigo
5
  colorTo: green
6
  sdk: streamlit
7
  sdk_version: 1.44.1
8
  app_file: app.py
9
- pinned: false
10
  license: apache-2.0
11
- short_description: fbaldassarri-repository-eval
12
  ---
13
 
14
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: fbaldassarri Repository Eval
3
+ emoji: πŸ’»
4
  colorFrom: indigo
5
  colorTo: green
6
  sdk: streamlit
7
  sdk_version: 1.44.1
8
  app_file: app.py
9
+ pinned: true
10
  license: apache-2.0
11
+ short_description: Evaluation space for fbaldassarri Repository
12
  ---
13
 
14
+ # fbaldassarri Model Comparison Tool
15
+
16
+ ## 🌟 Features
17
+
18
+ - **Model Discovery**: Automatically finds all quantized models in your HuggingFace account
19
+ - **Smart Filtering**: Filter models by quantization method, base model, or custom text
20
+ - **Detailed Metadata Analysis**: Compare model architectures, sizes, and quantization details
21
+ - **Repository Statistics**: Track downloads, likes, and update frequency
22
+ - **Interactive Visualizations**: Multiple charts and graphs for performance analysis
23
+ - **Selective Comparison**: Use `.modelsignore` to exclude irrelevant models
24
+ - **CPU-Optimized**: Works perfectly on free tier HuggingFace Spaces without GPU
25
+ - **Export Capabilities**: Download comparison results as CSV for further analysis
26
+
27
+ ## πŸ“‹ Usage Instructions
28
+
29
+ ### Basic Usage
30
+
31
+ 1. Enter your HuggingFace username
32
+ 2. The tool will automatically discover all your quantized models
33
+ 3. Use the filters to select models by quantization method or base model
34
+ 4. Click "Run Comparison" to analyze the selected models
35
+ 5. View the results in the "Model Comparison", "Model Details", and "Visualizations" tabs
36
+
37
+ ### Advanced Features
38
+
39
+ #### Filtering Models
40
+
41
+ - **Quantization Method**: Select specific quantization methods (Intel AutoRound, AutoGPTQ, AutoAWQ)
42
+ - **Base Model**: Compare models derived from the same base architecture
43
+ - **Text Filter**: Find models containing specific text in their names
44
+
45
+ #### Using .modelsignore
46
+
47
+ Create a `.modelsignore` file in the same directory as `app.py` to exclude specific models from discovery:
48
+
49
+ ```
50
+ # Comments start with #
51
+ fbaldassarri/llama-2-7b-* # Ignores all llama-2-7b models
52
+ *mistral* # Ignores anything with "mistral" in the name
53
+ fbaldassarri/exact-model-name # Ignores a specific model
54
+ ```
55
+
56
+ Supports wildcards and exact matching patterns.
57
+
58
+ #### Comparison Methods
59
+
60
+ Two comparison methods are available:
61
+
62
+ 1. **Metadata Comparison Only**: Fast analysis of model metadata and architecture
63
+ 2. **Metadata + Estimated Size**: Additionally calculates the disk size of each model
64
+
65
+ ## πŸ“Š Visualization Types
66
+
67
+ The tool provides multiple visualization options:
68
+
69
+ ### 1. Quantization Methods
70
+
71
+ - Distribution of quantization methods (pie chart)
72
+ - Distribution of precision formats (bar chart)
73
+ - Distribution of group sizes (bar chart)
74
+
75
+ ### 2. Model Architecture
76
+
77
+ - Model size comparison (bar chart)
78
+ - Layer count comparison (bar chart)
79
+ - Hidden size comparison (bar chart)
80
+ - Attention heads comparison (bar chart)
81
+
82
+ ### 3. Repository Statistics
83
+
84
+ - Downloads comparison (bar chart)
85
+ - Likes comparison (bar chart)
86
+ - Days since update comparison (bar chart)
87
+
88
+ ## πŸ”§ Technical Details
89
+
90
+ ### Supported Quantization Methods
91
+
92
+ The tool is designed to detect and analyze:
93
+
94
+ - **Intel AutoRound**: Intel's quantization solution
95
+ - **AutoGPTQ**: Automatic GPTQ quantization
96
+ - **AutoAWQ**: Activation-aware weight quantization
97
+
98
+ ### Metadata Extraction
99
+
100
+ For each model, the tool extracts:
101
+
102
+ - Quantization method (based on name and tags)
103
+ - Precision (INT4, INT8, FP16, FP32)
104
+ - Group size (e.g., gs128)
105
+ - Base model architecture
106
+ - Hidden size and layer count
107
+ - Attention head count
108
+ - Maximum sequence length
109
+
110
+ ### Size Estimation
111
+
112
+ When the "Metadata + Estimated Size" option is selected, the tool:
113
+
114
+ 1. Fetches the list of model files from the repository
115
+ 2. Identifies `.bin` and `.safetensors` files
116
+ 3. Calculates their total size in GB
117
+
118
+ This avoids downloading the actual model weights, making it CPU-friendly.
119
+
120
+ ## πŸ” Example Use Cases
121
+
122
+ 1. **Comparing Quantization Methods**: Determine which method (AutoGPTQ vs AutoAWQ) produces smaller models
123
+ 2. **Quantization Parameter Analysis**: See how different group sizes affect model size
124
+ 3. **Popularity Tracking**: Identify which of your quantized models has the most downloads
125
+ 4. **Architecture Analysis**: Compare hidden sizes and layer counts across different quantized variants
126
+
127
+ ## ⚠️ Troubleshooting
128
+
129
+ ### Common Issues
130
+
131
+ 1. **Slow Model Discovery**
132
+ - The tool may take time to discover models if you have many repositories
133
+ - Use the `.modelsignore` file to exclude irrelevant models and speed up discovery
134
+ 2. **Memory Issues**
135
+ - The application is optimized for CPU usage, but comparing many models simultaneously may cause memory pressure
136
+ - Use the "Maximum models to compare" slider to limit the number of models
137
+ 3. **Error: "Cannot subtract tz-naive and tz-aware datetime-like objects"**
138
+ - This error has been fixed in the latest version
139
+ - Update to the latest version if you encounter it
140
+ 4. **Missing Model Information**
141
+ - Some information may be unavailable if models don't have standard config files
142
+ - The tool will show "Unknown" for fields it couldn't extract
143
+
144
+ ## πŸ“ Requirements
145
+
146
+ - Python 3.9+
147
+ - huggingface_hub
148
+ - streamlit
149
+ - pandas
150
+ - plotly
151
+ - numpy
152
+
153
+ See `requirements.txt` for a complete list of dependencies.
154
+
155
+ ## πŸ“œ License
156
+
157
+ This project is licensed under the MIT License - see the LICENSE file for details.
158
+
159
+ ## πŸ™ Acknowledgements
160
+
161
+ - HuggingFace for the Hub API and Spaces platform
162
+ - The developers of AutoGPTQ, AutoAWQ, and Intel AutoRound
163
+ - Streamlit for the interactive web framework
164
+
165
+ ## πŸ‘€ Author
166
+
167
+ Created by Francesco Baldassarri
168
+
169
+ ## 🀝 Contributing
170
+
171
+ Contributions, issues, and feature requests are welcome!
172
+
173
+ 1. Fork the Project
174
+ 2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
175
+ 3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)
176
+ 4. Push to the Branch (`git push origin feature/AmazingFeature`)
177
+ 5. Open a Pull Request