docs: update the transformers and API codes
Browse files
README.md
CHANGED
@@ -26,13 +26,15 @@ Embeddings produced by `jina-embeddings-v4` serve as the backbone for neural inf
|
|
26 |
|
27 |
|
28 |
Built based on [Qwen/Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct), `jina-embeddings-v4` has the following features:
|
29 |
-
|
|
|
30 |
- **Multilingual support** (20+ languages) and compatibility with a wide range of domains, including technical and visually complex documents.
|
31 |
- **Task-specific adapters** for retrieval, text matching, and code-related tasks, which can be selected at inference time.
|
32 |
- **Flexible embedding size**: dense embeddings are 2048 dimensions by default but can be truncated to as low as 128 with minimal performance loss.
|
33 |
|
34 |
|
35 |
Summary of features:
|
|
|
36 |
| Feature | Jina Embeddings V4 |
|
37 |
|------------|------------|
|
38 |
| Base Model | Qwen2.5-VL-3B-Instruct |
|
@@ -42,8 +44,8 @@ Summary of features:
|
|
42 |
| Single-Vector Dimension | 2048 |
|
43 |
| Multi-Vector Dimension | 128 |
|
44 |
| Matryoshka dimensions | 128, 256, 512, 1024, 2048 |
|
45 |
-
| Attention Mechanism | FlashAttention2 |
|
46 |
| Pooling Strategy | Mean pooling |
|
|
|
47 |
|
48 |
|
49 |
|
@@ -58,6 +60,7 @@ Please refer to our [technical report of jina-embeddings-v4](https://puginarug.c
|
|
58 |
<summary>Requirements</a></summary>
|
59 |
|
60 |
The following Python packages are required:
|
|
|
61 |
- `transformers>=4.52.0`
|
62 |
- `torch>=2.6.0`
|
63 |
- `peft>=0.15.2`
|
@@ -68,25 +71,21 @@ The following Python packages are required:
|
|
68 |
- **flash-attention**: Installing [flash-attention](https://github.com/Dao-AILab/flash-attention) is recommended for improved inference speed and efficiency, but not mandatory.
|
69 |
- **sentence-transformers**: If you want to use the model via the `sentence-transformers` interface, install this package as well.
|
70 |
|
71 |
-
|
72 |
</details>
|
73 |
|
74 |
|
75 |
<details>
|
76 |
-
<summary>via
|
|
|
77 |
|
78 |
-
Needs to be adjusted for V4
|
79 |
```bash
|
80 |
curl https://api.jina.ai/v1/embeddings \
|
81 |
-H "Content-Type: application/json" \
|
82 |
-
-H "Authorization: Bearer
|
83 |
-d @- <<EOFEOF
|
84 |
{
|
85 |
"model": "jina-embeddings-v4",
|
86 |
-
"
|
87 |
-
"task": "retrieval.query",
|
88 |
-
"normalized": true,
|
89 |
-
"embedding_type": "float",
|
90 |
"input": [
|
91 |
{
|
92 |
"text": "غروب جميل على الشاطئ"
|
@@ -136,37 +135,41 @@ EOFEOF
|
|
136 |
|
137 |
```python
|
138 |
# !pip install transformers>=4.52.0 torch>=2.6.0 peft>=0.15.2 torchvision pillow
|
139 |
-
# !pip install
|
140 |
from transformers import AutoModel
|
|
|
141 |
|
142 |
# Initialize the model
|
143 |
model = AutoModel.from_pretrained("jinaai/jina-embeddings-v4", trust_remote_code=True)
|
|
|
|
|
|
|
144 |
# ========================
|
145 |
# 1. Retrieval Task
|
146 |
# ========================
|
147 |
# Configure truncate_dim, max_length (for texts), max_pixels (for images), vector_type, batch_size in the encode function if needed
|
148 |
|
149 |
# Encode query
|
150 |
-
|
151 |
texts=["Overview of climate change impacts on coastal cities"],
|
152 |
task="retrieval",
|
153 |
prompt_name="query",
|
154 |
-
)
|
155 |
|
156 |
# Encode passage (text)
|
157 |
-
|
158 |
texts=[
|
159 |
"Climate change has led to rising sea levels, increased frequency of extreme weather events..."
|
160 |
],
|
161 |
task="retrieval",
|
162 |
prompt_name="passage",
|
163 |
-
)
|
164 |
|
165 |
# Encode image/document
|
166 |
-
|
167 |
images=["https://i.ibb.co/nQNGqL0/beach1.jpg"],
|
168 |
task="retrieval",
|
169 |
-
)
|
170 |
|
171 |
# ========================
|
172 |
# 2. Text Matching Task
|
@@ -183,25 +186,43 @@ texts = [
|
|
183 |
"해변 위로 아름다운 일몰", # Korean
|
184 |
]
|
185 |
|
186 |
-
text_embeddings = model.
|
187 |
|
188 |
# ========================
|
189 |
# 3. Code Understanding Task
|
190 |
# ========================
|
191 |
|
192 |
# Encode query
|
193 |
-
query_embedding = model.
|
194 |
texts=["Find a function that prints a greeting message to the console"],
|
195 |
task="code",
|
196 |
prompt_name="query",
|
197 |
)
|
198 |
|
199 |
# Encode code
|
200 |
-
code_embeddings = model.
|
201 |
texts=["def hello_world():\n print('Hello, World!')"],
|
202 |
task="code",
|
203 |
prompt_name="passage",
|
204 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
205 |
```
|
206 |
</details>
|
207 |
|
|
|
26 |
|
27 |
|
28 |
Built based on [Qwen/Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct), `jina-embeddings-v4` has the following features:
|
29 |
+
|
30 |
+
- **Unified embeddings** for text, images, and visual documents, supporting both dense (single-vector) and late-interaction (multi-vector) retrieval.
|
31 |
- **Multilingual support** (20+ languages) and compatibility with a wide range of domains, including technical and visually complex documents.
|
32 |
- **Task-specific adapters** for retrieval, text matching, and code-related tasks, which can be selected at inference time.
|
33 |
- **Flexible embedding size**: dense embeddings are 2048 dimensions by default but can be truncated to as low as 128 with minimal performance loss.
|
34 |
|
35 |
|
36 |
Summary of features:
|
37 |
+
|
38 |
| Feature | Jina Embeddings V4 |
|
39 |
|------------|------------|
|
40 |
| Base Model | Qwen2.5-VL-3B-Instruct |
|
|
|
44 |
| Single-Vector Dimension | 2048 |
|
45 |
| Multi-Vector Dimension | 128 |
|
46 |
| Matryoshka dimensions | 128, 256, 512, 1024, 2048 |
|
|
|
47 |
| Pooling Strategy | Mean pooling |
|
48 |
+
| Attention Mechanism | FlashAttention2 |
|
49 |
|
50 |
|
51 |
|
|
|
60 |
<summary>Requirements</a></summary>
|
61 |
|
62 |
The following Python packages are required:
|
63 |
+
|
64 |
- `transformers>=4.52.0`
|
65 |
- `torch>=2.6.0`
|
66 |
- `peft>=0.15.2`
|
|
|
71 |
- **flash-attention**: Installing [flash-attention](https://github.com/Dao-AILab/flash-attention) is recommended for improved inference speed and efficiency, but not mandatory.
|
72 |
- **sentence-transformers**: If you want to use the model via the `sentence-transformers` interface, install this package as well.
|
73 |
|
|
|
74 |
</details>
|
75 |
|
76 |
|
77 |
<details>
|
78 |
+
<summary>via <a href="https://jina.ai/embeddings/">Jina AI Embeddings API</a></summary>
|
79 |
+
|
80 |
|
|
|
81 |
```bash
|
82 |
curl https://api.jina.ai/v1/embeddings \
|
83 |
-H "Content-Type: application/json" \
|
84 |
+
-H "Authorization: Bearer $JINA_AI_API_TOKEN" \
|
85 |
-d @- <<EOFEOF
|
86 |
{
|
87 |
"model": "jina-embeddings-v4",
|
88 |
+
"task": "text-matching",
|
|
|
|
|
|
|
89 |
"input": [
|
90 |
{
|
91 |
"text": "غروب جميل على الشاطئ"
|
|
|
135 |
|
136 |
```python
|
137 |
# !pip install transformers>=4.52.0 torch>=2.6.0 peft>=0.15.2 torchvision pillow
|
138 |
+
# !pip install
|
139 |
from transformers import AutoModel
|
140 |
+
import torch
|
141 |
|
142 |
# Initialize the model
|
143 |
model = AutoModel.from_pretrained("jinaai/jina-embeddings-v4", trust_remote_code=True)
|
144 |
+
|
145 |
+
model.to("cuda")
|
146 |
+
|
147 |
# ========================
|
148 |
# 1. Retrieval Task
|
149 |
# ========================
|
150 |
# Configure truncate_dim, max_length (for texts), max_pixels (for images), vector_type, batch_size in the encode function if needed
|
151 |
|
152 |
# Encode query
|
153 |
+
query_embeddings = model.encode_text(
|
154 |
texts=["Overview of climate change impacts on coastal cities"],
|
155 |
task="retrieval",
|
156 |
prompt_name="query",
|
157 |
+
)
|
158 |
|
159 |
# Encode passage (text)
|
160 |
+
passage_embeddings = model.encode_text(
|
161 |
texts=[
|
162 |
"Climate change has led to rising sea levels, increased frequency of extreme weather events..."
|
163 |
],
|
164 |
task="retrieval",
|
165 |
prompt_name="passage",
|
166 |
+
)
|
167 |
|
168 |
# Encode image/document
|
169 |
+
image_embeddings = model.encode_image(
|
170 |
images=["https://i.ibb.co/nQNGqL0/beach1.jpg"],
|
171 |
task="retrieval",
|
172 |
+
)
|
173 |
|
174 |
# ========================
|
175 |
# 2. Text Matching Task
|
|
|
186 |
"해변 위로 아름다운 일몰", # Korean
|
187 |
]
|
188 |
|
189 |
+
text_embeddings = model.encode_text(texts=texts, task="text-matching")
|
190 |
|
191 |
# ========================
|
192 |
# 3. Code Understanding Task
|
193 |
# ========================
|
194 |
|
195 |
# Encode query
|
196 |
+
query_embedding = model.encode_text(
|
197 |
texts=["Find a function that prints a greeting message to the console"],
|
198 |
task="code",
|
199 |
prompt_name="query",
|
200 |
)
|
201 |
|
202 |
# Encode code
|
203 |
+
code_embeddings = model.encode_text(
|
204 |
texts=["def hello_world():\n print('Hello, World!')"],
|
205 |
task="code",
|
206 |
prompt_name="passage",
|
207 |
)
|
208 |
+
|
209 |
+
# ========================
|
210 |
+
# 4. Use multivectors
|
211 |
+
# ========================
|
212 |
+
|
213 |
+
multivector_embeddings = model.encode_text(
|
214 |
+
texts=texts,
|
215 |
+
task="retrieval",
|
216 |
+
prompt_name="query",
|
217 |
+
return_multivector=True,
|
218 |
+
)
|
219 |
+
|
220 |
+
images = ["https://i.ibb.co/nQNGqL0/beach1.jpg", "https://i.ibb.co/r5w8hG8/beach2.jpg"]
|
221 |
+
multivector_image_embeddings = model.encode_image(
|
222 |
+
images=images,
|
223 |
+
task="retrieval",
|
224 |
+
return_multivector=True,
|
225 |
+
)
|
226 |
```
|
227 |
</details>
|
228 |
|