devjas1 commited on
Commit
e7aa207
Β·
1 Parent(s): 933e0be

REMOVE: delete requirements.txt as it is no longer needed

Browse files

(DOCS)[New + Improved `README.md`: Revise README.md for clarity and updated features, including installation instructions and model setup

(DOCS)[Download models from HF Hub]: update README.md to include instructions for downloading the phi-2.Q4_0.gguf model from HuggingFace Hub

(DOCS): update README.md to clarify model download instructions and provide two options for users

FEAT: update README.md to include detailed instructions and requirements for EmbeddingGemma-300m model

FEAT: add sample data for query and document corpus in sample_data1.py

.gitignore CHANGED
@@ -44,3 +44,4 @@ ENV/
44
 
45
  # Vector cache
46
  vector_cache/
 
 
44
 
45
  # Vector cache
46
  vector_cache/
47
+ models/embeddinggemma-300m/README.md
README.md CHANGED
@@ -1,226 +1,328 @@
1
- # CodeMind
2
 
3
- A CLI tool for intelligent document analysis and commit message generation using EmbeddingGemma-300m for embeddings, FAISS for vector storage, and Phi-2 for text generation.
4
 
5
  ## Features
6
 
7
- - **Document Indexing**: Embed and index documents for semantic search
8
- - **Semantic Search**: Find relevant documents using natural language queries
9
- - **Smart Commit Messages**: Generate meaningful commit messages from staged git changes
10
- - **RAG (Retrieval-Augmented Generation)**: Answer questions using indexed document context
 
 
11
 
12
- ## Setup
13
 
14
- ### Prerequisites
 
 
 
15
 
16
- - Windows 11
17
- - Conda environment
18
- - Git
19
 
20
- ### Installation
 
 
21
 
22
- 1. **Create a Conda environment:**
23
 
24
- ```bash
25
- conda create -n codemind python=3.9
26
- conda activate codemind
27
- ```
28
 
29
- 2. **Clone the repository:**
 
 
 
30
 
31
- ```bash
32
- git clone https://github.com/devjas1/codemind.git
33
- cd codemind
34
- ```
35
 
36
- 3. **Install dependencies:**
37
 
38
- ```bash
39
- pip install -r requirements.txt
40
- ```
41
 
42
- 4. **Download models:**
 
43
 
44
- **Embedding Model (EmbeddingGemma-300m):**
 
45
 
46
- - Download from Hugging Face: `google/embeddinggemma-300m`
47
- - Place in `./models/embeddinggemma-300m/` directory
 
48
 
49
- **Generation Model (Phi-2 GGUF):**
50
 
51
- - Download the quantized Phi-2 model: `phi-2.Q4_0.gguf`
52
- - Place in `./models/` directory
53
- - Download from: [Microsoft Phi-2 GGUF](https://huggingface.co/microsoft/phi-2-gguf) or similar quantized versions
 
 
 
 
 
 
 
 
54
 
55
  ### Directory Structure
56
 
 
 
 
 
 
 
 
57
  ```
58
- CodeMind/
59
- β”œβ”€β”€ cli.py # Main CLI entry point
60
- β”œβ”€β”€ config.yaml # Configuration file
61
- β”œβ”€β”€ requirements.txt # Python dependencies
62
- β”œβ”€β”€ models/ # Model storage
63
- β”‚ β”œβ”€β”€ embeddinggemma-300m/ # Embedding model directory
64
- β”‚ └── phi-2.Q4_0.gguf # Phi-2 quantized model file
65
- β”œβ”€β”€ src/ # Core modules
66
- β”‚ β”œβ”€β”€ config_loader.py # Configuration management
67
- β”‚ β”œβ”€β”€ embedder.py # Document embedding
68
- β”‚ β”œβ”€β”€ retriever.py # Semantic search
69
- β”‚ β”œβ”€β”€ generator.py # Text generation
70
- β”‚ └── diff_analyzer.py # Git diff analysis
71
- β”œβ”€β”€ docs/ # Documentation
72
- └── vector_cache/ # FAISS index storage (auto-created)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
73
  ```
74
 
75
- ## Usage
 
 
 
 
 
76
 
77
- ### Initialize Document Index
78
 
79
- Index documents from a directory for semantic search:
80
 
81
  ```bash
82
- python cli.py init ./docs/
 
 
 
 
 
 
 
83
  ```
84
 
85
- This will:
 
 
 
 
 
86
 
87
- - Embed all documents in the specified directory
88
- - Create a FAISS index in `vector_cache/`
89
- - Save metadata for retrieval
90
 
91
- ### Semantic Search
 
 
92
 
93
- Search for relevant documents using natural language:
94
 
95
  ```bash
96
- python cli.py search "how to configure the model"
97
  ```
98
 
99
- Returns ranked results with similarity scores.
100
-
101
- ### Ask Questions (RAG)
102
 
103
- Get answers based on your indexed documents:
104
 
105
  ```bash
106
- python cli.py ask "What are the configuration options?"
107
  ```
108
 
109
- Uses retrieval-augmented generation to provide contextual answers.
110
 
111
- ### Git Commit Message Generation
 
 
 
 
112
 
113
- Generate intelligent commit messages from staged changes:
114
 
115
  ```bash
116
- # Preview commit message without applying
117
  python cli.py commit --preview
118
 
119
- # Show staged files and analysis without generating message
120
- python cli.py commit --dry-run
121
-
122
- # Generate and apply commit message
123
- python cli.py commit --apply
124
  ```
125
 
126
- ### Start API Server (Future Feature)
127
 
128
  ```bash
129
  python cli.py serve --port 8000
130
  ```
131
 
132
- _Note: API server functionality is planned for future releases._
133
 
134
- ## Configuration
135
 
136
- Edit `config.yaml` to customize behavior:
137
 
138
- ```yaml
139
- embedding:
140
- model_path: "./models/embeddinggemma-300m"
141
- dim: 768
142
- truncate_to: 128
143
 
144
- generator:
145
- model_path: "./models/phi-2.Q4_0.gguf"
146
- quantization: "Q4_0"
147
- max_tokens: 512
148
- n_ctx: 2048
149
 
150
- retrieval:
151
- vector_store: "faiss"
152
- top_k: 5
153
- similarity_threshold: 0.75
 
 
154
 
155
- commit:
156
- tone: "imperative"
157
- style: "conventional"
158
- max_length: 72
159
 
160
- logging:
161
- verbose: true
162
- telemetry: false
 
 
 
 
 
 
 
163
  ```
164
 
165
- ### Configuration Options
166
 
167
- - **embedding.model_path**: Path to the EmbeddingGemma-300m model
168
- - **generator.model_path**: Path to the Phi-2 GGUF model file
169
- - **retrieval.top_k**: Number of documents to retrieve for context
170
- - **retrieval.similarity_threshold**: Minimum similarity score for results
171
- - **generator.max_tokens**: Maximum tokens for generation
172
- - **generator.n_ctx**: Context window size for Phi-2
173
 
174
- ## Dependencies
 
 
175
 
176
- - `sentence-transformers>=2.2.2` - Document embedding
177
- - `faiss-cpu>=1.7.4` - Vector similarity search
178
- - `llama-cpp-python>=0.2.23` - Phi-2 model inference (Windows compatible)
179
- - `typer>=0.9.0` - CLI framework
180
- - `PyYAML>=6.0` - Configuration file parsing
181
 
182
- ## Troubleshooting
 
183
 
184
- ### Model Loading Issues
 
 
185
 
186
- If you encounter model loading errors:
187
 
188
- 1. **Embedding Model**: Ensure `embeddinggemma-300m` is a directory containing all model files
189
- 2. **Phi-2 Model**: Ensure `phi-2.Q4_0.gguf` is a single GGUF file
190
- 3. **Paths**: All paths in `config.yaml` should be relative to the project root
191
 
192
- ### Memory Issues
 
193
 
194
- For systems with limited RAM:
195
 
196
- - Use Q4_0 quantization for Phi-2 (already configured)
197
- - Reduce `n_ctx` in config.yaml if needed
198
- - Process documents in smaller batches
199
 
200
- ### Windows-Specific Issues
201
 
202
- - Ensure `llama-cpp-python` version supports Windows
203
- - Use PowerShell or Command Prompt for CLI commands
204
- - Check file path separators in configuration
 
 
 
 
205
 
206
- ## Development
207
 
208
- To test the modules:
209
 
210
  ```bash
211
- python -c "from src import *; print('All modules imported successfully')"
 
212
  ```
213
 
214
- To run in development mode:
215
-
216
- ```bash
217
- python cli.py --help
 
 
 
 
 
 
 
 
 
 
 
218
  ```
219
 
220
- ## License
 
 
 
 
 
 
 
 
 
 
 
 
221
 
222
- [Insert your license information here]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
223
 
224
  ## Contributing
225
 
226
- [Insert contribution guidelines here]
 
 
 
 
 
1
+ # CodeMind: Local AI Development Assistant
2
 
3
+ CodeMind is an AI-powered development assistant that runs entirely on your local machine. It helps you understand your codebase through semantic search and generates meaningful commit messages using locally hosted language models, ensuring complete privacy and no cloud dependencies.
4
 
5
  ## Features
6
 
7
+ - **Semantic Code Search**: Find relevant code and documentation using AI-powered semantic search
8
+ - **Commit Message Generation**: Automatically generate descriptive commit messages based on your changes
9
+ - **Local Processing**: All AI processing happens on your machine with no data sent to cloud services
10
+ - **Flexible Configuration**: Customize models and parameters to suit your specific needs
11
+ - **FAISS Integration**: Efficient vector similarity search for fast retrieval
12
+ - **Multiple Model Support**: Compatible with GGUF and SentenceTransformers models
13
 
14
+ ## Prerequisites
15
 
16
+ - **Python 3.8 or higher**
17
+ - **8GB+ RAM** recommended (for running language models)
18
+ - **4GB+ disk space** for model files
19
+ - **Git** for repository cloning
20
 
21
+ ### Platform Recommendations
 
 
22
 
23
+ - **Linux** (Recommended for best compatibility)
24
+ - **macOS** (Good compatibility)
25
+ - **Windows** (May require additional setup for some dependencies)
26
 
27
+ ## Installation
28
 
29
+ ### 1. Clone the Repository
 
 
 
30
 
31
+ ```bash
32
+ git clone https://github.com/devjas1/codemind.git
33
+ cd codemind
34
+ ```
35
 
36
+ ### 2. Set Up Python Environment
 
 
 
37
 
38
+ Create and activate a virtual environment:
39
 
40
+ ```bash
 
 
41
 
42
+ # Create virtual environment
43
+ python -m venv venv
44
 
45
+ # Activate on macOS/Linux
46
+ source venv/bin/activate
47
 
48
+ # Activate on Windows
49
+ venv\Scripts\activate
50
+ ```
51
 
52
+ ### 3. Install Dependencies
53
 
54
+ ```bash
55
+ pip install -r requirements.txt
56
+ ```
57
+
58
+ **Note**: If you encounter installation errors related to C++/PyTorch/FAISS:
59
+
60
+ - Ensure you have Python development tools installed
61
+ - Linux/macOS are preferred for FAISS compatibility
62
+ - On Windows, you may need to install Visual Studio Build Tools
63
+
64
+ ## Model Setup
65
 
66
  ### Directory Structure
67
 
68
+ Create the following directory structure for model files:
69
+
70
+ ```text
71
+ models/
72
+ β”œβ”€β”€ phi-2.Q4_0.gguf # For commit message generation (Phi-2 model)
73
+ └── embeddinggemma-300m/ # For document embedding (EmbeddingGemma model)
74
+ └── [model files here]
75
  ```
76
+
77
+ ### Downloading Models
78
+
79
+ 1. **Phi-2 Model** (for commit message generation):
80
+
81
+ - Download `phi-2.Q4_0.gguf` from a trusted source
82
+ - Place it in the `models/` directory
83
+
84
+ 2. **EmbeddingGemma Model** (for document embedding):
85
+
86
+ - Download the EmbeddingGemma-300m model files
87
+ - Place all files in the `models/embeddinggemma-300m/` directory
88
+
89
+ > **Note**: The specific process for obtaining these models may vary. Check the documentation in each model folder for detailed instructions.
90
+
91
+ ## Configuration
92
+
93
+ Edit the `config.yaml` file to match your local setup:
94
+
95
+ ```yaml
96
+ # Model configuration for commit message generation
97
+ generator:
98
+ model_path: "./models/phi-2.Q4_0.gguf"
99
+ quantization: "Q4_0"
100
+ max_tokens: 512
101
+ n_ctx: 2048
102
+
103
+ # Model configuration for document embedding
104
+ embedding:
105
+ model_path: "./models/embeddinggemma-300m"
106
+
107
+ # Retrieval configuration for semantic search
108
+ retrieval:
109
+ vector_store: "faiss"
110
+ top_k: 5 # Number of results to return
111
+ similarity_threshold: 0.7 # Minimum similarity score (0.0 to 1.0)
112
  ```
113
 
114
+ ### Configuration Tips
115
+
116
+ - Adjust `top_k` to control how many results are returned for each query
117
+ - Modify `similarity_threshold` to filter results by relevance
118
+ - Ensure all file paths are correct for your system
119
+ - For larger codebases, you may need to increase `max_tokens`
120
 
121
+ ## Indexing Documents
122
 
123
+ To enable semantic search over your documentation or codebase, you need to create a FAISS index:
124
 
125
  ```bash
126
+ # Basic usage
127
+ python src/embedder.py path/to/your/documents config.yaml
128
+
129
+ # Example with docs directory
130
+ python src/embedder.py ./docs config.yaml
131
+
132
+ # Example with specific code directory
133
+ python src/embedder.py ./src config.yaml
134
  ```
135
 
136
+ This process:
137
+
138
+ 1. Reads all documents from the specified directory
139
+ 2. Generates embeddings using the configured model
140
+ 3. Creates a FAISS index in the `vector_cache/` directory
141
+ 4. Enables fast semantic search capabilities
142
 
143
+ > **Note**: The indexing process may take several minutes depending on the size of your codebase and your hardware capabilities.
 
 
144
 
145
+ ## Usage
146
+
147
+ ### Command Line Interface
148
 
149
+ Run the main CLI interface:
150
 
151
  ```bash
152
+ python cli.py
153
  ```
154
 
155
+ ### Available Commands
 
 
156
 
157
+ #### Get Help
158
 
159
  ```bash
160
+ python cli.py --help
161
  ```
162
 
163
+ #### Ask Questions About Your Codebase
164
 
165
+ ```bash
166
+ python cli.py ask "How does this repository work?"
167
+ python cli.py ask "Where is the main configuration handled?"
168
+ python cli.py ask "Show me examples of API usage"
169
+ ```
170
 
171
+ #### Generate Commit Messages
172
 
173
  ```bash
174
+ # Preview a generated commit message
175
  python cli.py commit --preview
176
 
177
+ # Generate commit message without preview
178
+ python cli.py commit
 
 
 
179
  ```
180
 
181
+ #### API Server (Placeholder)
182
 
183
  ```bash
184
  python cli.py serve --port 8000
185
  ```
186
 
187
+ > **Note**: The API server functionality is not yet implemented. This command will display: "API server functionality not implemented yet."
188
 
189
+ ### Advanced Usage
190
 
191
+ For more advanced usage, you can modify the configuration to:
192
 
193
+ - Use different models for specific tasks
194
+ - Adjust the context window size for larger documents
195
+ - Customize the similarity threshold for retrieval
196
+ - Use different vector stores (though FAISS is currently the only supported option)
 
197
 
198
+ ## Troubleshooting
 
 
 
 
199
 
200
+ ### Common Issues
201
+
202
+ #### Model Errors
203
+
204
+ **Problem**: Model files not found or inaccessible
205
+ **Solution**:
206
 
207
+ - Verify model files are in the correct locations
208
+ - Check file permissions
209
+ - Ensure the paths in `config.yaml` are correct
 
210
 
211
+ #### FAISS Errors
212
+
213
+ **Problem**: "No FAISS index found" error
214
+ **Solution**:
215
+
216
+ - Run the embedder script to create the index
217
+ - Ensure the `vector_cache/` directory has write permissions
218
+
219
+ ```bash
220
+ python src/embedder.py path/to/documents config.yaml
221
  ```
222
 
223
+ #### SentenceTransformers Issues
224
 
225
+ **Problem**: Compatibility errors with SentenceTransformers
226
+ **Solution**:
 
 
 
 
227
 
228
+ - Check that the model format is compatible with SentenceTransformers
229
+ - Verify the version in requirements.txt
230
+ - Ensure all model files are present in the model directory
231
 
232
+ #### Performance Issues
 
 
 
 
233
 
234
+ **Problem**: Slow response times
235
+ **Solution**:
236
 
237
+ - Ensure you have adequate RAM
238
+ - Consider using smaller quantized models
239
+ - Close other memory-intensive applications
240
 
241
+ #### Platform-Specific Issues
242
 
243
+ **Windows-specific issues**:
 
 
244
 
245
+ - FAISS may require additional compilation
246
+ - Path separators may need adjustment in configuration
247
 
248
+ **macOS/Linux**:
249
 
250
+ - Generally fewer compatibility issues
251
+ - Ensure you have write permissions for all directories
 
252
 
253
+ ### Validation Checklist
254
 
255
+ - All model files present in correct directories
256
+ - FAISS index built in `vector_cache/`
257
+ - `config.yaml` paths match your local setup
258
+ - Python environment activated
259
+ - All dependencies installed
260
+ - Adequate disk space available
261
+ - Sufficient RAM available
262
 
263
+ ### Getting Detailed Error Information
264
 
265
+ For specific errors, run commands with verbose output:
266
 
267
  ```bash
268
+ # Add debug flags if available
269
+ python cli.py --verbose ask "Your question"
270
  ```
271
 
272
+ ## Project Structure
273
+
274
+ ```text
275
+ codemind/
276
+ β”œβ”€β”€ models/ # AI model files
277
+ β”‚ β”œβ”€β”€ phi-2.Q4_0.gguf # Phi-2 model for generation
278
+ β”‚ └── embeddinggemma-300m/ # Embedding model
279
+ β”‚ └── [model files]
280
+ β”œβ”€β”€ src/ # Source code
281
+ β”‚ └── embedder.py # Document embedding script
282
+ β”œβ”€β”€ vector_cache/ # FAISS vector store (auto-generated)
283
+ β”œβ”€β”€ config.yaml # Configuration file
284
+ β”œβ”€β”€ requirements.txt # Python dependencies
285
+ β”œβ”€β”€ cli.py # Command-line interface
286
+ └── README.md # This file
287
  ```
288
 
289
+ ## FAQ
290
+
291
+ ### Q: Can I use different models?
292
+
293
+ > **A**: Yes, you can use any GGUF-compatible model for generation and any SentenceTransformers-compatible model for embeddings. Update the paths in `config.yaml` accordingly.
294
+
295
+ ### Q: How much RAM do I need?
296
+
297
+ > **A**: For the Phi-2 Q4_0 model, 8GB RAM is recommended. Larger models will require more memory.
298
+
299
+ ### Q: Can I index multiple directories?
300
+
301
+ > **A**: Yes, you can run the embedder script multiple times with different directories, or combine your documents into one directory before indexing.
302
 
303
+ ### Q: Is my data sent to the cloud?
304
+
305
+ > **A**: No, all processing happens locally on your machine. No code or data is sent to external services.
306
+
307
+ ### Q: How often should I re-index my documents?
308
+
309
+ > **A**: Re-index whenever your documentation or codebase changes significantly to keep search results relevant.
310
+
311
+ ## Support
312
+
313
+ If you encounter issues:
314
+
315
+ 1. Check the troubleshooting section above
316
+ 2. Verify all model files are in correct locations
317
+ 3. Confirm Python and library versions match requirements
318
+ 4. Ensure proper directory permissions
319
+
320
+ For specific errors, please include the full traceback when seeking assistance.
321
 
322
  ## Contributing
323
 
324
+ Contributions to CodeMind are welcome! Please feel free to submit pull requests, create issues, or suggest new features.
325
+
326
+ ## License
327
+
328
+ This project is licensed under the terms of the LICENSE file included in the repository.
data/sample_data1.py ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # This file holds our sample data (query and documents/corpus).
2
+ # We define it here to keep our main script clean.
3
+
4
+ # A single query string we want to find an answer for.
5
+ QUERY = "Which planet is known as the Red Planet?"
6
+
7
+ # A list of document strings that form our knowledge corpus.
8
+ # We will convert these to embeddings and compare them to the query.
9
+ DOCUMENTS = [
10
+ "Venus is the second planet from the Sun and is often called Earth's twin because of its similar size and proximity.",
11
+ "Mars is the fourth planet from the Sun and is frequently referred to as the Red Planet due to its reddish appearance caused by iron oxide on its surface.",
12
+ "Jupiter is the fifth and largest planet in our solar system, a gas giant known for its prominent Great Red Spot, a giant storm.",
13
+ "Saturn is the sixth planet from the Sun, famous for its extensive and visible ring system made of ice particles and dust.",
14
+ ]
15
+
16
+ # Optional: A more complex example using a list of dictionaries.
17
+ # This is useful if you have metadata like titles, which work well with EmbeddingGemma's prompts.
18
+ DOCUMENTS_WITH_META = [
19
+ {
20
+ "title": "Venus",
21
+ "text": "Venus is often called Earth's twin because of its similar size and proximity.",
22
+ },
23
+ {
24
+ "title": "Mars",
25
+ "text": "Mars, known for its reddish appearance, is often referred to as the Red Planet.",
26
+ },
27
+ {
28
+ "title": "Jupiter",
29
+ "text": "Jupiter, the largest planet in our solar system, has a prominent red spot.",
30
+ },
31
+ {
32
+ "title": "Saturn",
33
+ "text": "Saturn, famous for its rings, is sometimes mistaken for the Red Planet.",
34
+ },
35
+ ]
models/README.md ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Download Models for CodeMind
2
+
3
+ You need to download compatible model files (such as `phi-2.Q4_0.gguf`) to use CodeMind locally. There are two main ways to obtain these models:
4
+
5
+ ---
6
+
7
+ ## Option 1: Download via Hugging Face Web Interface
8
+
9
+ 1. Visit the [Hugging Face model page](https://huggingface.co/TheBloke/phi-2-GGUF).
10
+ 2. Locate the desired file (e.g., `phi-2.Q4_0.gguf`) in the "Files and versions" section.
11
+ 3. Click the file name to download it directly.
12
+ 4. Move the downloaded file into your `./models` directory within your CodeMind repository.
13
+
14
+ [![HuggingFace](https://img.shields.io/badge/-HuggingFace-FDEE21?style=for-the-badge&logo=HuggingFace&logoColor=black)](https://huggingface.co/TheBloke/phi-2-GGUF)
15
+
16
+ ---
17
+
18
+ ## Option 2: Download Using Python and `huggingface_hub`
19
+
20
+ If you prefer automation, you can use the `huggingface_hub` Python library:
21
+
22
+ ### 1. Install the library
23
+
24
+ ```bash
25
+ pip install huggingface_hub
26
+ ```
27
+
28
+ ### 2. Download the model file
29
+
30
+ Create a Python script (e.g., `download_model.py`) with the following code:
31
+
32
+ ```python
33
+ from huggingface_hub import hf_hub_download
34
+
35
+ repo_id = "TheBloke/phi-2-GGUF" # Change if using a different repo
36
+ filename = "phi-2.Q4_0.gguf"
37
+ local_dir = "./models"
38
+
39
+ hf_hub_download(repo_id=repo_id, filename=filename, local_dir=local_dir)
40
+ print(f"Downloaded {filename} to {local_dir}")
41
+ ```
42
+
43
+ - Replace `repo_id` and `filename` if you need a different model or file.
44
+ - Make sure `local_dir` matches your desired storage location.
45
+
46
+ ### 3. Run the script
47
+
48
+ ```bash
49
+ python download_model.py
50
+ ```
51
+
52
+ This will download the specified file into your `models` directory.
53
+
54
+ ---
55
+
56
+ ## Notes
57
+
58
+ - Always verify the model license and compatibility before use.
59
+ - For other models, repeat the steps above with the appropriate repository and filename.
60
+ - For more details, see the [Hugging Face documentation](https://huggingface.co/docs/huggingface_hub).
models/embeddinggemma-300m/README.md ADDED
@@ -0,0 +1,71 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # EmbeddingGemma-300m Model Files
2
+
3
+ This directory should contain all files required for the EmbeddingGemma-300m model in a format compatible with [SentenceTransformers](https://www.sbert.net/).
4
+
5
+ ---
6
+
7
+ ## 1. Required Files
8
+
9
+ Typically, you need the following files:
10
+
11
+ - `config.json`
12
+ - `pytorch_model.bin` or `model.safetensors`
13
+ - `tokenizer.json`
14
+ - `tokenizer_config.json`
15
+ - `vocab.txt` (if applicable)
16
+
17
+ ---
18
+
19
+ ## 2. How to Obtain the Files
20
+
21
+ ### Option 1: Download via Hugging Face Web Interface
22
+
23
+ 1. Visit the [EmbeddingGemma-300m model page](https://huggingface.co/google/embeddinggemma-300m).
24
+ 2. Download each file listed above manually.
25
+ 3. Place all files in the `models/embeddinggemma-300m/` directory.
26
+
27
+ [![HuggingFace](https://img.shields.io/badge/-HuggingFace-FDEE21?style=for-the-badge&logo=HuggingFace&logoColor=black)](https://huggingface.co/google/embeddinggemma-300m)
28
+
29
+ ### Option 2: Download Using Git
30
+
31
+ If you have [Git LFS](https://git-lfs.com/) installed, you can clone the entire repository:
32
+
33
+ ```bash
34
+ git lfs install
35
+ git clone https://huggingface.co/google/embeddinggemma-300m models/embeddinggemma-300m
36
+ ```
37
+
38
+ This will download all necessary files into the correct directory.
39
+
40
+ ---
41
+
42
+ ## 3. Directory Structure Example
43
+
44
+ ```
45
+ models/
46
+ └── embeddinggemma-300m/
47
+ β”œβ”€β”€ config.json
48
+ β”œβ”€β”€ pytorch_model.bin
49
+ β”œβ”€β”€ tokenizer.json
50
+ β”œβ”€β”€ tokenizer_config.json
51
+ └── vocab.txt
52
+ ```
53
+
54
+ ---
55
+
56
+ ## 4. Validation
57
+
58
+ After placing the files, you can load the model in Python:
59
+
60
+ ```python
61
+ from sentence_transformers import SentenceTransformer
62
+ model = SentenceTransformer("./models/embeddinggemma-300m")
63
+ ```
64
+
65
+ ---
66
+
67
+ ## Notes
68
+
69
+ - Ensure all files are present for proper embedding functionality.
70
+ - For updates or troubleshooting, refer to the [Hugging Face documentation](https://huggingface.co/docs).
71
+ - Always verify model license and compatibility before use.
models/models/README.md DELETED
@@ -1 +0,0 @@
1
- # Place phi-2.Q4_0.gguf model file here
 
 
models/models/embeddinggemma-300m/README.md DELETED
@@ -1 +0,0 @@
1
- # Place EmbeddingGemma-300m model files here