add posts

src/posts/2025-02-13-qwen2_5-vl-mlx-vlm/index.qmd

@@ -1,5 +1,6 @@
 ---
 title: "Qwen2.5-vl with MLX-VLM"
+author: "Joana Levtcheva"
 date: "2025-02-13"
 categories: [Machine Learning, mlx, vlm]
 draft: false

src/posts/2025-04-06-fine-tuning-function-calling/index.qmd

@@ -0,0 +1,368 @@
+---
+title: "Fine-Tuning a Model for Function-Calling with MLX-LM"
+author: "Joana Levtcheva"
+date: "2025-04-06"
+categories: [Machine Learning, mlx, llm]
+draft: false
+---
+
+In this post, we explore the process of fine-tuning a language model for function-calling using [MLX-LM](https://github.com/ml-explore/mlx-lm). Following the Hugging Face Agents course [notebook](https://huggingface.co/agents-course/notebooks/blob/main/bonus-unit1/bonus-unit1.ipynb), we’ll walk through the steps from setting up the environment to training the model with LoRA adapters. The goal is to empower the model with the ability to intelligently plan and generate function calls, making it a versatile tool for interactive applications. Medium post can be found [here](https://medium.com/@levchevajoana/fine-tuning-a-model-for-function-calling-with-mlx-lm-d00d587e2559)
+
+## Introduction
+
+Modern AI models can do much more than generate plain text — they can now integrate with external tools by “calling” functions based on user intent. In this tutorial, we demonstrate how to adapt a pre-trained model (in our case, the [gemma-2-2b-it-4bit](https://huggingface.co/mlx-community/gemma-2-2b-it-4bit) model from the [MLX Community](https://huggingface.co/mlx-community)) to handle function-calling by using the `mlx-lm` package. This involves creating a specialized chat template, preprocessing a dataset of function call interactions, and applying LoRA for efficient fine-tuning.
+
+## Setting Up the Model and Tokenizer
+
+We start by importing the necessary libraries and modules, including the MLX-LM package, dataset utilities, and LoRA functions.
+
+```python
+import json
+import os
+from enum import Enum
+from typing import Dict, List, Tuple, Union
+
+import mlx.optimizers as optim
+from datasets import load_dataset
+from mlx.utils import tree_flatten
+from mlx_lm import generate, load
+from mlx_lm.tuner import TrainingArgs, datasets, linear_to_lora_layers, train
+```
+
+After loading our model and tokenizer,
+
+```python
+model_path = "mlx-community/gemma-2-2b-it-4bit"
+model, tokenizer = load(model_path)
+```
+
+we customize the tokenizer’s chat template to define the structure of our conversational interactions.
+
+```python
+tokenizer.chat_template = (
+    "{{ bos_token }}"
+    "{% if messages[0]['role'] == 'system' %}{{ raise_exception('System role not supported') }}{% endif %}"
+    "{% for message in messages %}"
+    "{{ '<start_of_turn>' + message['role'] + '\n' + message['content'] | trim + '<end_of_turn><eos>\n' }}"
+    "{% endfor %}"
+    "{% if add_generation_prompt %}{{'<start_of_turn>model\n'}}{% endif %}"
+)
+```
+
+This template embeds special tokens (like `<bos>`, `<start_of_turn>`,` <think>`, and `<tool_call>`) that mark the different stages of the conversation - from the user’s prompt to the model’s internal reasoning and eventual function call.
+
+## Dataset Preparation and Preprocessing
+
+We use the dataset [Jofthomas/hermes-function-calling-thinking-V1](https://huggingface.co/datasets/Jofthomas/hermes-function-calling-thinking-V1) which contains conversations involving function calls.
+
+```python
+dataset_path = "Jofthomas/hermes-function-calling-thinking-V1"
+```
+
+Let’s load the dataset.
+
+```python
+dataset = load_dataset(dataset_path)
+dataset
+```
+
+This outputs
+
+```
+DatasetDict({
+    train: Dataset({
+        features: ['conversations'],
+        num_rows: 3570
+    })
+})
+```
+
+showing that the dataset originally includes a “conversations” column, and has 3570 rows. We rename this column to “messages” for consistency
+
+```python
+dataset = dataset.rename_column("conversations", "messages")
+dataset
+```
+
+and then apply the following preprocessing function
+
+```python
+def preprocess(sample):
+    messages = sample["messages"]
+    first_message = messages[0]
+
+    # Instead of adding a system message, we merge the content into the first user message
+    if first_message["role"] == "system":
+        system_message_content = first_message["content"]
+        # Merge system content with the first user message
+        messages[1]["content"] = (
+            system_message_content
+            + "Also, before making a call to a function take the time to plan the function to take. Make that thinking process between <think>{your thoughts}</think>\n\n"
+            + messages[1]["content"]
+        )
+        # Remove the system message from the conversation
+        messages.pop(0)
+
+    return {"text": tokenizer.apply_chat_template(messages, tokenize=False)}
+```
+
+to the dataset
+
+```python
+dataset = dataset.map(preprocess, remove_columns="messages")
+dataset = dataset["train"].train_test_split(0.1)
+dataset
+```
+
+This function merges any system messages into the first user message, ensuring the context is maintained without extra role annotations. This outputs
+
+```text
+DatasetDict({
+    train: Dataset({
+        features: ['text'],
+        num_rows: 3213
+    })
+    test: Dataset({
+        features: ['text'],
+        num_rows: 357
+    })
+})
+```
+
+showing that we have successfully separated our original dataset into a train set with 3213 records, and a test set with 357 records. Each sample is now a formatted text string ready for fine-tuning. Let’s see one train example
+
+```text
+<bos><start_of_turn>human
+You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags.You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions.Here are the available tools:<tools> [{'type': 'function', 'function': {'name': 'create_todo', 'description': 'Create a new todo item', 'parameters': {'type': 'object', 'properties': {'task': {'type': 'string', 'description': 'The task description'}, 'due_date': {'type': 'string', 'format': 'date', 'description': 'The due date of the task'}, 'priority': {'type': 'integer', 'description': 'The priority of the task (1-5)'}}, 'required': ['task', 'due_date']}}}, {'type': 'function', 'function': {'name': 'calculate_bmi', 'description': 'Calculate the Body Mass Index (BMI)', 'parameters': {'type': 'object', 'properties': {'weight': {'type': 'number', 'description': 'The weight in kilograms'}, 'height': {'type': 'number', 'description': 'The height in meters'}}, 'required': ['weight', 'height']}}}] </tools>Use the following pydantic model json schema for each tool call you will make: {'title': 'FunctionCall', 'type': 'object', 'properties': {'arguments': {'title': 'Arguments', 'type': 'object'}, 'name': {'title': 'Name', 'type': 'string'}}, 'required': ['arguments', 'name']}For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
+<tool_call>
+{tool_call}
+</tool_call>Also, before making a call to a function take the time to plan the function to take. Make that thinking process between <think>{your thoughts}</think>
+
+I need to create a new task.<end_of_turn><eos>
+<start_of_turn>model
+Sure, I can help with that. Could you please provide me with the task description, the due date, and the priority level?<end_of_turn><eos>
+<start_of_turn>human
+The task is to prepare a presentation for the annual meeting. The due date is 2022-09-15 and the priority level is 3.<end_of_turn><eos>
+<start_of_turn>model
+<think>Okay, so I need to figure out the reasoning that goes between the conversation and the next function call. Let me break this down step by step.
+
+First, looking at the conversation, the user starts by saying, "I need to create a new task." The model responds by asking for the task description, due date, and priority level. The user then provides all the necessary details: the task is to prepare a presentation for the annual meeting, the due date is 2022-09-15, and the priority is 3.
+
+Now, the model's next move should be to call the appropriate function. The available functions are 'create_todo' and 'calculate_bmi'. Since the user is talking about creating a new task, 'create_todo' is the relevant function here.
+
+Examining the function's parameters, it requires 'task', 'due_date', and takes 'priority' as optional. The user provided all three, so we can include them in the arguments.
+
+Therefore, the model will execute the 'create_todo' function with the provided task details. This makes sense because the conversation is about setting up a new task, and the function is designed for that exact purpose.
+</think><tool_call>
+{'name': 'create_todo', 'arguments': {'task': 'Prepare a presentation for the annual meeting', 'due_date': '2022-09-15', 'priority': 3}}
+</tool_call><end_of_turn><eos>
+<start_of_turn>tool
+<tool_response>
+{'status': 'success', 'message': 'Todo item successfully created', 'todo_id': '12345'}
+</tool_response><end_of_turn><eos>
+<start_of_turn>model
+Your task has been successfully created. The ID for your new task is 12345.<end_of_turn><eos>
+```
+
+## Training Setup with LoRA Adapters
+
+To efficiently fine-tune the model without retraining all of its parameters, we leverage LoRA. First, we create a directory to store adapter configurations and weights.
+
+```python
+adapter_path = "adapters_fc"
+os.makedirs(adapter_path, exist_ok=True)
+adapter_config_path = os.path.join(adapter_path, "adapter_config.json")
+adapter_file_path = os.path.join(adapter_path, "adapters.safetensors")
+```
+
+Then we define our LoRA configuration, with parameters like number of layers 8, a rank of 16, scale of 64, a dropout of 0.05,
+
+```python
+lora_config = {
+    "num_layers": 8,
+    "lora_parameters": {
+        "rank": 16,
+        "scale": 64,
+        "dropout": 0.05,
+    },
+}
+```
+
+and save it as a JSON file.
+
+```python
+with open(adapter_config_path, "w") as f:
+    json.dump(lora_config, f, indent=4)
+```
+
+Next, we define the training arguments, specifically setting a single iteration,
+
+```python
+training_args = TrainingArgs(
+    adapter_file=adapter_file_path,
+    iters=1,
+    steps_per_eval=50,
+)
+```
+
+and freeze the original model parameters.
+
+```python
+_ = model.freeze()
+```
+
+Then, we convert selected linear layers to LoRA layers to make only a small subset of parameters trainable.
+
+```python
+linear_to_lora_layers(model, lora_config["num_layers"], lora_config["lora_parameters"])
+```
+
+In our example, this results in
+
+```python
+num_train_params = sum(v.size for _, v in tree_flatten(model.trainable_parameters()))
+print(f"Number of trainable parameters: {num_train_params}")
+```
+
+983,040 trainable parameters. Finally, we should not forget to activate training mode while still preserving the frozen state of the main model parameters.
+
+```python
+_ = model.train()
+```
+
+## Fine-Tuning Process and Metrics
+
+With our model and dataset ready, we configure a metrics tracker to log both training and validation losses,
+
+```python
+class Metrics:
+    def __init__(self) -> None:
+        self.train_losses: List[Tuple[int, float]] = []
+        self.val_losses: List[Tuple[int, float]] = []
+
+    def on_train_loss_report(self, info: Dict[str, Union[float, int]]) -> None:
+        self.train_losses.append((info["iteration"], info["train_loss"]))
+
+    def on_val_loss_report(self, info: Dict[str, Union[float, int]]) -> None:
+        self.val_losses.append((info["iteration"], info["val_loss"]))
+```
+
+and create an instance of this class.
+
+```python
+metrics = Metrics()
+```
+
+We also create mlx-lm–suitable datasets by first defining the following configuration about our datasets,
+
+```python
+configs = {
+    "mask_prompt": False,
+    "prompt_feature": "prompt",
+    "text_feature": "text",
+    "completion_feature": "completion",
+    "chat_feature": "messages",
+}
+```
+
+and then create a train set with the help of the mlx-lm function `datasets.create_dataset` and passing the configuration from above.
+
+```python
+train_set = datasets.create_dataset(
+    dataset["train"],
+    tokenizer,
+    configs
+)
+```
+
+Similarly, we create our validation set.
+
+```python
+val_set = datasets.create_dataset(
+    dataset["test"],
+    tokenizer,
+    configs
+)
+```
+
+Finally, we start the fine-tuning process by calling the `train()` function.
+
+```python
+train(
+    model=model,
+    tokenizer=tokenizer,
+    args=training_args,
+    optimizer=optim.Adam(learning_rate=1e-5),
+    train_dataset=train_set,
+    val_dataset=val_set,
+    training_callback=metrics,
+)
+```
+
+The training logs report both training and validation losses, along with performance metrics like tokens processed per second and memory usage. After training, the adapter weights are saved and can later be reloaded to quickly deploy the fine-tuned model.
+
+```
+Starting training..., iters: 1
+Iter 1: Val loss 1.821, Val took 128.584s
+Iter 1: Train loss 1.861, Learning Rate 1.000e-05, It/sec 0.430, Tokens/sec 160.427, Trained Tokens 3735, Peak mem 20.665 GB
+Saved final weights to adapters_fc/adapters.safetensors.
+```
+
+## Evaluating the Fine-Tuned Model
+
+After training, we reload the model with the newly learned LoRA weights,
+
+```python
+model_lora, _ = load(model_path, adapter_path=adapter_path)
+```
+
+set our prompt to
+
+```python
+prompt="""<bos><start_of_turn>human
+You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags.You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions.Here are the available tools:<tools> [{'type': 'function', 'function': {'name': 'convert_currency', 'description': 'Convert from one currency to another', 'parameters': {'type': 'object', 'properties': {'amount': {'type': 'number', 'description': 'The amount to convert'}, 'from_currency': {'type': 'string', 'description': 'The currency to convert from'}, 'to_currency': {'type': 'string', 'description': 'The currency to convert to'}}, 'required': ['amount', 'from_currency', 'to_currency']}}}, {'type': 'function', 'function': {'name': 'calculate_distance', 'description': 'Calculate the distance between two locations', 'parameters': {'type': 'object', 'properties': {'start_location': {'type': 'string', 'description': 'The starting location'}, 'end_location': {'type': 'string', 'description': 'The ending location'}}, 'required': ['start_location', 'end_location']}}}] </tools>Use the following pydantic model json schema for each tool call you will make: {'title': 'FunctionCall', 'type': 'object', 'properties': {'arguments': {'title': 'Arguments', 'type': 'object'}, 'name': {'title': 'Name', 'type': 'string'}}, 'required': ['arguments', 'name']}For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
+<tool_call>
+{tool_call}
+</tool_call>Also, before making a call to a function take the time to plan the function to take. Make that thinking process between <think>{your thoughts}</think>
+
+Hi, I need to convert 500 USD to Euros. Can you help me with that?<end_of_turn><eos>
+<start_of_turn>model
+<think>"""
+```
+
+and generate a response
+
+```python
+generate(model_lora, tokenizer, prompt=prompt, verbose=True, max_tokens=1000)
+```
+
+which returns
+
+```text
+==========
+
+To convert USD to Euros, I need to use the 'convert_currency' function from the provided tools.  I need to provide the amount to convert, the currency to convert from (USD), and the currency to convert to (Euros).  I should also make sure the amount is a number.
+</think>
+
+<tool_call>
+{
+  'name': 'convert_currency',
+  'arguments': {
+    'amount': 500,
+    'from_currency': 'USD',
+    'to_currency': 'EUR'
+  }
+}
+</tool_call> 
+
+==========
+Prompt: 460 tokens, 862.170 tokens-per-sec
+Generation: 135 tokens, 68.472 tokens-per-sec
+Peak memory: 20.665 GB
+```
+
+The model first walks through a thought process before generating a function call to convert USD to Euros. This demonstrates the model’s improved ability to generate precise JSON function calls within `<tool_call>` XML tags.
+
+## Conclusion
+
+Fine-tuning a model for function-calling can significantly enhance its interactivity and real-world utility. By adapting the chat template, preprocessing the dataset, and applying LoRA adapters, we’ve demonstrated a streamlined approach to training a model that can generate executable function calls with clear reasoning. It is impressive that we achieved this by using only the `mlx-lm` package. Happy fine-tuning!

src/posts/2025-04-15-paligemma-2-mix/index.qmd

@@ -0,0 +1,643 @@
+---
+title: "Image Segmentation with PaliGemma 2 Mix and MLX"
+author: "Joana Levtcheva"
+date: "2025-04-15"
+categories: [Machine Learning, mlx, vlm]
+draft: false
+---
+
+In this post, we are going to explore Google’s [**PaliGemma 2 mix**](https://developers.googleblog.com/en/introducing-paligemma-2-mix/) vision-language model (VLM), and its capabilities to perform image segmentation. What’s interesting is that we are going to perform this task by only using Apple’s MLX framework, and MLX-VLM. This would eliminate the dependency of using JAX/Flax as in the original Google’s segmentation [script](https://github.com/google-research/big_vision/blob/main/big_vision/evaluators/proj/paligemma/transfers/segmentation.py), and would allow us to fully and seamlessly utilise Apple’s unified memory. Medium post can be found [here](https://medium.com/@levchevajoana/image-segmentation-with-paligemma-2-mix-and-mlx-7e69e077968b).
+
+# Introduction
+
+## PaliGemma 2
+
+In December 2024 Google introduced the [PaliGemma 2](https://developers.googleblog.com/en/introducing-paligemma-2-powerful-vision-language-models-simple-fine-tuning/) vision-language models (VLMs). These are pre-trained (**pt**) models coming in three different sizes: `3B`, `10B`, and `28B`, as well as three different input resolutions for images: `224x224`, `448x448`, and `896x896` pixels. These models represent the latest evolution of vision-language models developed by Google, building upon the foundation laid by its predecessor, PaliGemma. Below, we can see the architecture of the PaliGemma 2 model.
+
+<figure>
+  <img src="images/paligemma2-architecture.png" alt="PaliGemma 2 architecture" style="display: block; margin: 0 auto">
+  <figcaption style="text-align: center">Figure 1. PaliGemma 2 Architecture Overview <span style="font-size: 0.8em;">[<a href="https://arxiv.org/pdf/2412.03555">Source</a>]</span></figcaption>
+</figure>
+
+PaliGemma 2 processes images at resolutions of `224×224`, `448×448`, or `896×896` pixels using a **[SigLIP-400m](https://arxiv.org/abs/2303.15343) vision encoder** with a patch size of 14×14 pixels. This design yields 256, 1024, or 4096 tokens, respectively. After a linear projection, the resulting image tokens are concatenated with the input text tokens, and [**Gemma 2**](https://blog.google/technology/developers/google-gemma-2/) is used as a **text decoder** to autoregressively complete the combined prefix to generate an answer.
+
+## PaliGemma 2 Mix
+
+As already mentioned, PaliGemma 2 models are pre-trained models, but they are also designed to be easy to fine-tune and adapt to various specific vision-language tasks and domains. Google wanted to demonstrate the performance of a fine-tuned version of the pt PaliGemma 2 models on downstream tasks, and thus a few months later, in February 2025, they introduced [**PaliGemma 2 mix**](https://developers.googleblog.com/en/introducing-paligemma-2-mix/). These models are fine-tuned to a mixture of vision language tasks that can be used out-of-the-box for common use cases. They are available in three sizes: `3B`, `10B`, and `28B`, and support resolutions of `224×224` and `448×448` pixels.
+
+### Tasks
+
+PaliGemma 2 mix can perform the following types of tasks:
+
+- Short and long captioning
+- Optical character recognition (OCR)
+- Image question answering
+- (Multiple) object detection
+- (Multiple) image segmentation
+
+### Prompting
+
+In general, the PaliGemma models are very sensitive to the prompt’s syntax and patterns. But based on the following Hugging Face [article](https://huggingface.co/blog/paligemma2mix) when using PaliGemma 2 mix models, open-ended prompts yield better performance than the previously required task-prefixed prompts. Earlier, task-specific prefixes were essential, like
+
+- `"caption {lang}\n"`: Short captions
+- `"describe {lang}\n"`: More descriptive captions
+- `"ocr"`: Optical character recognition
+- `"answer {lang} {question}\n"`: Question answering about the image contents
+- `"question {lang} {answer}\n"`: Question generation for a given answer
+
+However, two specific tasks - **object detection** and **image segmentation** - still exclusively require task prefixes:
+
+- `"detect {object description} ; {object description} ; ...\n"`: Locate multiple objects in an image and return the bounding boxes for those objects
+- `"segment {object description} ; {object description} ; ...\n"`: Locate the area occupied by multiple objects in an image to create an image segmentation for that object
+
+# Image Segmentation
+
+## What is Image Segmentation?
+
+Image segmentation is a key computer vision technique that divides an image into pixel groups, or segments, enabling tasks like object detection, scene understanding, and advanced image processing. Traditional methods use pixel features such as color, brightness, contrast, and intensity to separate objects from the background, often relying on simple heuristics or basic machine learning. Recently, deep learning models with complex neural networks have dramatically improved segmentation accuracy.
+
+Unlike image classification, which labels an entire image, or object detection, which locates objects with bounding boxes, image segmentation provides detailed pixel-level annotations. This approach assigns every pixel to a specific category, with variants including semantic segmentation (classifying pixels), instance segmentation (distinguishing between instances of the same object), and panoptic segmentation (combining both methods).
+
+## Image Segmentation with VLMs
+
+VLMs enhance traditional image segmentation by enabling open-vocabulary segmentation through textual instructions, moving away from closed-set methods that rely on predefined categories. By merging text and image data into a common feature space, these models reduce adaptation costs and excel at tasks like referring expression segmentation. For example, a user might prompt the model to *“segment the cat sitting on the chair”*, and the VLM would identify and segment the pixels corresponding to that specific cat.
+
+To achieve this, VLMs harness visual features from encoders like CNNs or Vision Transformers, using cross-attention to focus on image regions relevant to the text. Some models are fine-tuned to produce bounding boxes or segmentation masks directly, and careful prompting guides them to accurately segment based on the integrated understanding of visual content and language.
+
+## Image Segmentation the PaliGemma 2 Way
+
+Earlier, in **Figure 1** we saw that PaliGemma 2’s architecture combines a Transformer decoder based on the Gemma 2 language model with a Vision Transformer image encoder initialised from SigLIP-So400m/14. The SigLIP encoder divides input images into `14x14` pixel patches to generate “soft tokens” that capture spatial relationships. Then, a linear projection layer is used to map the visual tokens into the same dimensional space as the input embeddings of the Gemma 2 language model. This projection ensures that the visual information can be seamlessly combined with textual information for processing by the language model.
+
+The Gemma 2 language model functions as the decoder, processing concatenated image tokens and text tokens to produce autoregressive text output, predicting one token at a time based on the preceding context. To enhance its capabilities for vision-language tasks, PaliGemma extends the vocabulary of the standard Gemma tokenizer (having 256,000 tokens) with additional special tokens. These include 1024 tokens representing coordinates in a normalised image space, denoted as `<loc0000>` through `<loc1023>`, and another 128 tokens, `<seg000>` through `<seg127>`, which are codewords used for a lightweight referring-expression segmentation vector-quantized approach.
+
+### Segmentation Output
+
+When processing a segmentation prompt, PaliGemma 2 mix produces a sequence that begins with four location tokens defining the bounding box for the segmented object. These four tokens specify the bounding box coordinates in the normalized image space. This is followed by 16 segmentation tokens, which can be decoded via a learned codebook into a binary segmentation mask confined within the identified region. Below is an example output:
+
+```text
+<loc0336><loc0049><loc0791><loc0941><seg106><seg074><seg114><seg081><seg082><seg028><seg018><seg037><seg120><seg073><seg061><seg125><seg045><seg059><seg052><seg084>
+```
+
+### Segmentation Mask
+
+If we want to further process the 16 segmentation tokens to generate a binary segmentation mask within the identified bounding box, we have to decode the segmentation tokens by using the Decoder from Google’s big vision repository related to the PaliGemma models. It is available in the following [script](https://github.com/google-research/big_vision/blob/main/big_vision/evaluators/proj/paligemma/transfers/segmentation.py). As we can see, the script uses JAX and Flax, and it is known that the Metal plug-in for JAX is still not fully supported as stated in the [Accelerated JAX on Mac](https://developer.apple.com/metal/jax/) article. In the next part of this post, we are going to show not only how to reconstruct the binary mask with the help of the above script, but we are also going to show how to translate JAX/Flax to [mlx](https://github.com/ml-explore/mlx) so that we can fully utilise the unified memory in Apple’s chips.
+
+# Tutorial
+
+In this section, we are going to generate a segmentation mask with the PaliGemma 2 mix model, specifically [mlx-community/paligemma2–10b-mix-448–8bit](https://huggingface.co/mlx-community/paligemma2-10b-mix-448-8bit), by using only the packages `mlx-vlm` and `mlx`. We are also going to overlay the mask on top of the image we are segmenting.
+
+## Overview of the Process
+
+Let’s first begin by outlining the steps of the process for generating a segmentation mask. An illustrative diagram can be seen in **Figure 2**.
+
+- We start with passing a **prompt** to the model of the form *"segment cat\n"*, and the image we want to segment. This is our **original image** with dimensions $x_{\text{orig}}$ by $y_{\text{orig}}$.
+- Then, the model’s image processor (SiglipImageProcessor) yields to an **input image** with dimensions $x_{\text{input}}$ by $y_{\text{input}}$. In the PaliGemma 2 mix case this would be either `224x224` or `448x448`, depending on the model we have chosen to use. In our case, it would be `448x448`.
+- The model generates an output with 4 location coordinates and 16 segmentation tokens. The `<locXXXX><locXXXX><locXXXX><locXXXX>` sequence corresponds to the $y_{\text{min}}$, $x_{\text{min}}$, $y_{\text{max}}$, $x_{\text{max}}$ coordinates defining the **bounding box**. These coordinates should be normalised to an image size of `1024x1024` to obtain the bounding box coordinates of the object we want to segment with respect to the input image dimensions.
+
+<figure>
+  <img src="images/input_bb.png" alt="Model input and bounding box" style="display: block; margin: 0 auto">
+  <figcaption style="text-align: center">Figure 2. Model input and bounding box coordinates</figcaption>
+</figure>
+
+Now that we’ve defined the bounding box by its coordinates, let’s zoom in on its details as shown in **Figure 3**, and dicuss how we would overlay the segmentation mask on top of the image we are segmenting.
+
+- The model has returned the 16 segmentation tokens of the form `<segXXX>`. After decoding them via the codebook we end up reconstructing the **segmentation mask**. This mask has a size of `64x64` pixels.
+- Next, we need to map the segmentation mask onto the bounding box that was previously defined. This is accomplished using classical interpolation techniques to scale the mask to the bounding box’s dimensions.
+
+<figure>
+  <img src="images/map_mask.png" alt="Mapping mask to bounding box" style="display: block; margin: 0 auto">
+  <figcaption style="text-align: center">Figure 3. Mapping the 64x64 mask to the bounding box</figcaption>
+</figure>
+
+- Once resized, the mask is aligned to fit within the bounding box. To overlay this mask on the original image, we create an empty array matching the dimensions of the input image and then replace the array values corresponding to the bounding box coordinates with those from the interpolated segmentation mask.
+
+# MLX
+
+Finally, it’s time to dive into the coding section of this blog and focus specifically on the `mlx` components. The code can be found in [GitHub](https://github.com/JoeJoe1313/LLMs-Journey/blob/main/VLMs/paligemma_segmentation_mlx.py).
+
+We begin by importing the necessary libraries and modules,
+
+```python
+import argparse
+import functools
+import logging
+import re
+from typing import Callable, List, Tuple
+
+import cv2
+import matplotlib.pyplot as plt
+import mlx.core as mx
+import mlx.nn as nn
+import numpy as np
+from mlx_vlm import apply_chat_template, generate, load
+from mlx_vlm.utils import load_image
+from tensorflow.io import gfile
+```
+
+then, we establish the paths for the models and image resources. The `MODEL_PATH` points to the specific PaliGemma model that we are going to use for segmentation tasks. The `IMAGE_PATH` is the location of the image that we will process, and the `_KNOWN_MODELS` dictionary provides a reference to the VAE checkpoint needed for mask reconstruction.
+
+```python
+MODEL_PATH = "mlx-community/paligemma2-10b-mix-448-8bit"
+IMAGE_PATH = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg"
+_KNOWN_MODELS = {"oi": "gs://big_vision/paligemma/vae-oid.npz"}
+```
+
+Before diving into the core functionality, we set up logging to keep track of the execution flow and for debugging purposes. The following snippet initializes Python’s built-in logging system:
+
+```python
+logging.basicConfig()
+log = logging.getLogger(__name__)
+log.setLevel(logging.INFO)
+```
+
+The `ResBlock` class implements a basic residual block typical for convolutional architectures. It comprises three convolution layers:
+
+- Two `3x3` convolutions with ReLU activations, which process the input.
+- One `1x1` convolution to adjust dimensions if needed.
+
+The output of the block is computed by summing the result of the convolutions with the original input. This residual connection helps maintain gradient flow during training and preserves information across layers.
+
+```python
+class ResBlock(nn.Module):
+    def __init__(self, features: int):
+        super().__init__()
+        self.conv1 = nn.Conv2d(
+            in_channels=features, out_channels=features, kernel_size=3, padding=1
+        )
+        self.conv2 = nn.Conv2d(
+            in_channels=features, out_channels=features, kernel_size=3, padding=1
+        )
+        self.conv3 = nn.Conv2d(
+            in_channels=features, out_channels=features, kernel_size=1, padding=0
+        )
+
+    def __call__(self, x: mx.array) -> mx.array:
+        original_x = x
+        x = nn.relu(self.conv1(x))
+        x = nn.relu(self.conv2(x))
+        x = self.conv3(x)
+        return x + original_x
+```
+
+The `Decoder` class takes quantized vectors (obtained from segmentation tokens) and upscales them to produce a mask:
+
+- An initial convolution reduces the channel dimension.
+- A series of configurable residual blocks further process the features.
+Multiple transpose convolution layers (upsample layers) scale the feature maps until the desired resolution is reached.
+- A final convolution produces the output mask.
+
+```python
+class Decoder(nn.Module):
+    """Decoder that upscales quantized vectors to produce a mask.
+    The architecture is parameterized to avoid hardcoded layer definitions.
+    Takes channels-last input data (B, H, W, C).
+    """
+
+    def __init__(
+        self,
+        in_channels: int = 512,
+        res_channels: int = 128,
+        out_channels: int = 1,
+        num_res_blocks: int = 2,
+        upsample_channels: Tuple[int, ...] = (128, 64, 32, 16),
+    ):
+        super().__init__()
+        self.conv_in = nn.Conv2d(
+            in_channels=in_channels, out_channels=res_channels, kernel_size=1, padding=0
+        )
+        self.res_blocks = [
+            ResBlock(features=res_channels) for _ in range(num_res_blocks)
+        ]
+        self.upsample_layers = []
+        out_up_ch = res_channels
+        for ch in upsample_channels:
+            self.upsample_layers.append(
+                nn.ConvTranspose2d(
+                    in_channels=out_up_ch,
+                    out_channels=ch,
+                    kernel_size=4,
+                    stride=2,
+                    padding=1,
+                )
+            )
+            out_up_ch = ch
+        self.conv_out = nn.Conv2d(
+            in_channels=upsample_channels[-1],
+            out_channels=out_channels,
+            kernel_size=1,
+            padding=0,
+        )
+
+    def __call__(self, x: mx.array) -> mx.array:
+        x = nn.relu(self.conv_in(x))
+        for block in self.res_blocks:
+            x = block(x)
+        for layer in self.upsample_layers:
+            x = nn.relu(layer(x))
+
+        return self.conv_out(x)
+```
+
+The helper function `_get_params` is designed to convert a PyTorch checkpoint into a format that MLX can work with. It does so by
+
+- Transposing kernel weights to match the expected output format: from PyTorch’s format to MLX’s (Out, H, W, In) format.
+- Organizing the parameters into a structured dictionary that reflects the architecture of the decoder, including the convolutional layers, residual blocks, and upsample layers.
+
+This organized set of parameters is then used to initialize the decoder network.
+
+```python
+def _get_params(checkpoint: dict) -> dict:
+    """Converts PyTorch checkpoint to MLX params (nested dict).
+    Uses transpositions yielding (Out, H, W, In) format weights."""
+
+    def transp(kernel: np.ndarray) -> mx.array:
+        return mx.transpose(mx.array(kernel), (0, 2, 3, 1))
+
+    def transp_transpose(kernel: np.ndarray) -> mx.array:
+        intermediate = mx.transpose(mx.array(kernel), (1, 0, 2, 3))
+
+        return mx.transpose(intermediate, (0, 2, 3, 1))
+
+    def conv(name: str) -> dict:
+        return {
+            "bias": mx.array(checkpoint[f"{name}.bias"]),
+            "weight": transp(checkpoint[f"{name}.weight"]),
+        }
+
+    def conv_transpose(name: str) -> dict:
+        return {
+            "bias": mx.array(checkpoint[f"{name}.bias"]),
+            "weight": transp_transpose(checkpoint[f"{name}.weight"]),
+        }
+
+    def resblock(name: str) -> dict:
+        return {
+            "conv1": conv(f"{name}.0"),
+            "conv2": conv(f"{name}.2"),
+            "conv3": conv(f"{name}.4"),
+        }
+
+    params = {
+        "_embeddings": mx.array(checkpoint["_vq_vae._embedding"]),
+        "conv_in": conv("decoder.0"),
+        "res_blocks": [
+            resblock("decoder.2.net"),
+            resblock("decoder.3.net"),
+        ],
+        "upsample_layers": [
+            conv_transpose("decoder.4"),
+            conv_transpose("decoder.6"),
+            conv_transpose("decoder.8"),
+            conv_transpose("decoder.10"),
+        ],
+        "conv_out": conv("decoder.12"),
+    }
+
+    return params
+```
+
+The function `_quantized_values_from_codebook_indices` takes the segmentation tokens (represented as codebook indices) and uses the embeddings from the codebook to retrieve the corresponding encoded representations. These values are reshaped to fit the expected dimensions (batch, height, width, channels) so that they are ready for processing by the decoder.
+
+```python
+def _quantized_values_from_codebook_indices(
+    codebook_indices: mx.array, embeddings: mx.array
+) -> mx.array:
+    batch_size, num_tokens = codebook_indices.shape
+    expected_tokens = 16
+    if num_tokens != expected_tokens:
+        log.error(f"Expected {expected_tokens} tokens, got {codebook_indices.shape}")
+
+    encodings = mx.take(embeddings, codebook_indices.reshape((-1,)), axis=0)
+
+    return encodings.reshape((batch_size, 4, 4, embeddings.shape[1]))
+```
+
+The `get_reconstruct_masks` function loads the VAE checkpoint and initializes the decoder with the appropriate parameters. By extracting and setting up the necessary embeddings and decoder weights, this function returns another function (`reconstruct_masks`) that, when given segmentation tokens, decodes them into a binary segmentation mask.
+
+```python
+@functools.cache
+def get_reconstruct_masks(model: str) -> Callable[[mx.array], mx.array]:
+    """Loads the checkpoint and returns a function that reconstructs masks
+    from codebook indices using a preloaded MLX decoder.
+    """
+    checkpoint_path = _KNOWN_MODELS.get(model, model)
+    with gfile.GFile(checkpoint_path, "rb") as f:
+        checkpoint_data = dict(np.load(f))
+
+    params = _get_params(checkpoint_data)
+    embeddings = params.pop("_embeddings")
+    log.info(f"VAE embedding dimension: {embeddings.shape[1]}")
+
+    decoder = Decoder()
+    decoder.update(params)
+
+    def reconstruct_masks(codebook_indices: mx.array) -> mx.array:
+        quantized = _quantized_values_from_codebook_indices(
+            codebook_indices, embeddings
+        )
+        return decoder(quantized)
+
+    return reconstruct_masks
+```
+
+The function `extract_and_create_arrays` parses a given string pattern for segmentation tokens. It extracts these token numbers, converts them into integers, and then wraps them in MLX arrays for further mask reconstruction processing.
+
+```python
+def extract_and_create_arrays(pattern: str) -> List[mx.array]:
+    """Extracts segmentation tokens from each object in the pattern and returns a list of MLX arrays."""
+    object_strings = [obj.strip() for obj in pattern.split(";") if obj.strip()]
+
+    seg_tokens_arrays = []
+    for obj in object_strings:
+        seg_tokens = re.findall(r"<seg(\d{3})>", obj)
+        if seg_tokens:
+            seg_numbers = [int(token) for token in seg_tokens]
+            seg_tokens_arrays.append(mx.array(seg_numbers))
+
+    return seg_tokens_arrays
+```
+
+The `parse_bbox` function interprets the model's output string to extract bounding box coordinates. Each detected object's location is denoted by a string format (`<loc1234>`). This function finds four numbers per object, corresponding to the box boundaries, and aggregates them into a list of bounding boxes.
+
+```python
+def parse_bbox(model_output: str):
+    entries = model_output.split(";")
+
+    results = []
+    for entry in entries:
+        entry = entry.strip()
+        numbers = re.findall(r"<loc(\d+)>", entry)
+        if len(numbers) == 4:
+            bbox = [int(num) for num in numbers]
+            results.append(bbox)
+
+    return results
+```
+
+The `gather_masks` function combines the reconstruction and bounding box parsing steps. For each object:
+
+- It reconstructs the mask from its codebook indices.
+- It obtains the corresponding bounding box coordinates.
+- It normalizes these coordinates relative to a target image resolution (448×448 in this example).
+
+Each mask is then paired with its coordinates and stored in a list, making it straightforward to later overlay these onto the original image.
+
+```python
+def gather_masks(output, codes_list, reconstruct_fn):
+    masks_list = []
+
+    target_width, target_height = 448, 448
+    for i, codes in enumerate(codes_list):
+        codes_batch = codes[None, :]
+        masks = reconstruct_fn(codes_batch)
+        mask_np = np.array(masks[0, :, :, 0], copy=False)
+
+        y_min, x_min, y_max, x_max = parse_bbox(output)[i]
+        x_min_norm = int(x_min / 1024 * target_width)
+        y_min_norm = int(y_min / 1024 * target_height)
+        x_max_norm = int(x_max / 1024 * target_width)
+        y_max_norm = int(y_max / 1024 * target_height)
+
+        masks_list.append(
+            {
+                "mask": mask_np,
+                "coordinates": (x_min_norm, y_min_norm, x_max_norm, y_max_norm),
+            }
+        )
+
+    return masks_list
+```
+
+The function `plot_masks` handles the visualization of the segmentation outcomes. It loads the original image and processes it for display. Two types of visualizations are provided:
+
+- **Composite Overlay**: All masks are combined and overlaid on the original image.
+- **Reconstructed Mask**: Each reconstructed mask is plotted next to the composite overlay.
+
+Using OpenCV for mask resizing and Matplotlib for plotting, the function creates a series of subplots to clearly display both composite and individual mask overlays.
+
+```python
+def plot_masks(args, processor, masks_list):
+
+    image = load_image(args.image_path)
+    img_array = processor.image_processor(image)["pixel_values"][0].transpose(1, 2, 0)
+    img_array = (img_array * 0.5 + 0.5).clip(0, 1)
+
+    full = np.ones((448, 448, 1)) * (-1)
+    for mask_info in masks_list:
+        mask_np = mask_info["mask"]
+        x_min_norm, y_min_norm, x_max_norm, y_max_norm = mask_info["coordinates"]
+
+        width = x_max_norm - x_min_norm
+        height = y_max_norm - y_min_norm
+
+        resized_mask = cv2.resize(
+            mask_np, (width, height), interpolation=cv2.INTER_NEAREST
+        )
+        resized_mask = resized_mask.reshape((height, width, 1))
+
+        full[y_min_norm:y_max_norm, x_min_norm:x_max_norm] = resized_mask
+
+    n_masks = len(masks_list)
+    _, axs = plt.subplots(1, n_masks + 1, figsize=(5 * (n_masks + 1), 6))
+
+    axs[0].imshow(img_array)
+    axs[0].imshow(full, alpha=0.5)
+    axs[0].set_title("Mask Overlay")
+    axs[0].axis("on")
+
+    for i, mask_info in enumerate(masks_list, start=1):
+        mask_np = mask_info["mask"]
+        axs[i].imshow(mask_np)
+        axs[i].set_title(f"Reconstructed Mask {i}")
+        axs[i].axis("on")
+
+    plt.tight_layout()
+    plt.show()
+```
+
+The `main` function ties all the pieces together. It performs the following steps:
+
+- **Loading**: Reads the specified PaliGemma model and image.
+- **Setup**: Initializes the VAE checkpoint and extracts the reconstruction function.
+- **Prompting**: Formats the prompt using the processor and generates a segmentation output via the model.
+- **Processing**: Extracts segmentation tokens, reconstructs masks, and parses bounding box coordinates.
+- **Visualization**: Finally, it calls the plotting function to display the results.
+
+This function serves as the central point where data processing, model inference, mask reconstruction, and visualization are integrated into one complete pipeline.
+
+```python
+def main(args) -> None:
+    log.info(f"Loading PaliGemma model: {args.model_path}")
+    model, processor = load(args.model_path)
+    config = model.config
+
+    image = load_image(args.image_path)
+    log.info(f"Image size: {image.size}")
+
+    vae_path = _KNOWN_MODELS.get(args.vae_checkpoint_path, args.vae_checkpoint_path)
+    reconstruct_fn = get_reconstruct_masks(vae_path)
+
+    prompt = args.prompt.strip() + "\n"
+    log.info(f"Using prompt: '{prompt.strip()}'")
+    formatted_prompt = apply_chat_template(processor, config, prompt, num_images=1)
+
+    log.info("Generating segmentation output...")
+    output = generate(model, processor, formatted_prompt, image, verbose=False)
+    log.info(f"Model output: {output}")
+
+    codes_list = extract_and_create_arrays(output)
+    log.info(f"Extracted codes: {codes_list}")
+
+    log.info("Reconstructing mask from codes...")
+    masks_list = gather_masks(output, codes_list, reconstruct_fn)
+
+    log.info("Plotting masks...")
+    plot_masks(processor, masks_list)
+```
+
+Finally, the script includes an entry point that parses command-line arguments. Users can specify the image path, the prompt for the segmentation task, the model path, and the VAE checkpoint path. Once these are provided via `argparse`, the `main` function is invoked to start the processing pipeline.
+
+```python
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Vision tasks using PaliGemma 2 mix.")
+    parser.add_argument(
+        "--image_path", type=str, default=IMAGE_PATH, help="Path to the input image."
+    )
+    parser.add_argument(
+        "--prompt", type=str, required=True, help="Prompt for the model."
+    )
+    parser.add_argument(
+        "--model_path", type=str, default=MODEL_PATH, help="Path to the mlx model."
+    )
+    parser.add_argument(
+        "--vae_checkpoint_path", type=str, default="oi", help="Path to the .npz file."
+    )
+
+    cli_args = parser.parse_args()
+    main(cli_args)
+```
+
+# Results
+
+Let’s take a look at some examples and the segmentations we obtained from the model.
+
+## Single Object Segmentation
+
+In this section, we are going to show to examples of single object segmentation.
+
+---
+
+**Prompt:**
+
+```text
+"segment cow"
+```
+
+**Image:**
+
+<figure>
+  <img src="images/cow_in.png" alt="Cow input" style="display: block; margin: 0 auto">
+  <figcaption style="text-align: center">Figure 4. Original image of size 400x400</figcaption>
+</figure>
+
+**Model output:**
+
+```text
+<loc0410><loc0528><loc0884><loc1023><seg072><seg055><seg062><seg079><seg104><seg009><seg104><seg096><seg068><seg041><seg103><seg019><seg100><seg004><seg091><seg067>
+```
+
+**Mask overlay and reconstructed mask:**
+
+<figure>
+  <img src="images/cow_out.png" alt="Cow output" style="display: block; margin: 0 auto">
+  <figcaption style="text-align: center">Figure 5. Left: mask overlay onto the input image of size 448x448 | Right: reconstructed mask of size 64x64</figcaption>
+</figure>
+
+**Observation:**
+
+Based on the overlay image, the model manages to detect the precise location of the cow but struggles a bit with the detailed outlines of the cow. Looking only at the reconstructed mask would not persuade me that this is a cow.
+
+---
+
+**Prompt:**
+
+```text
+"segment cat"
+```
+
+**Image:**
+
+<figure>
+  <img src="images/cat_in.png" alt="Cat input" style="display: block; margin: 0 auto">
+  <figcaption style="text-align: center">Figure 6. Original image of size 400x400</figcaption>
+</figure>
+
+**Model output:**
+
+```text
+<loc0060><loc0000><loc0920><loc0879><seg039><seg107><seg018><seg006><seg056><seg120><seg058><seg042><seg079><seg094><seg009><seg099><seg074><seg010><seg078><seg012>
+```
+
+**Mask overlay and reconstructed mask:**
+
+<figure>
+  <img src="images/cat_out.png" alt="Cat output" style="display: block; margin: 0 auto">
+  <figcaption style="text-align: center">Figure 7. Left: mask overlay onto the input image of size 448x448 | Right: reconstructed mask of size 64x64</figcaption>
+</figure>
+
+**Observation:**
+
+Based on the overlay image, the model manages to detect the precise location of the cat, and is generally doing a good job with the cat’s outlines.
+
+## Multiple Object Segmentation
+
+It was tricky to find a working example for segmenting multiple objects, so there is only one example in this section. My observation is that the PaliGemma models are indeed very sensitive to the prompt formatting, and the 448–10B-8bit model might just not be powerful enough for the task of segmenting multiple objects.
+
+---
+
+**Prompt:**
+
+```text
+"segment left wheel ; right wheel"
+```
+
+**Image:**
+
+<figure>
+  <img src="images/car_in.png" alt="Car input" style="display: block; margin: 0 auto">
+  <figcaption style="text-align: center">Figure 8. Original image of size 640x480</figcaption>
+</figure>
+
+**Model output:**
+
+```text
+<loc0591><loc0157><loc0794><loc0311> <seg092><seg004><seg044><seg092><seg120><seg061><seg029><seg120><seg090><seg023><seg021><seg090><seg015><seg041><seg044><seg073> right wheel ; <loc0586><loc0728><loc0794><loc0882> <seg092><seg004><seg089><seg092><seg120><seg048><seg054><seg038><seg119><seg029><seg021><seg090><seg095><seg041><seg044><seg073> right wheel
+```
+
+**Mask overlay and reconstructed mask:**
+
+<figure>
+  <img src="images/car_out.png" alt="Car output" style="display: block; margin: 0 auto">
+  <figcaption style="text-align: center">Figure 9. Left: masks overlay onto the input image of size 448x448 | Right: reconstructed masks of size 64x64</figcaption>
+</figure>
+
+**Observation:**
+
+Looking at the model output we can see that both segmentations are labeled as right wheel. Despite this, based on the overlay image, the model manages to detect the precise location of the wheels, and their outlines.
+
+# Conclusion
+
+In summary, we implemented a unified segmentation pipeline by combining Google’s PaliGemma 2 Mix model with Apple’s MLX framework. Our workflow involved formatting segmentation prompts, preprocessing images, extracting segmentation tokens and bounding box coordinates, decoding these tokens into segmentation masks, and finally overlaying the masks on the original images.
+
+For single object segmentation, the model generally performed well: it accurately localised the object areas, as evidenced by both the “cat” and the “cow” examples. However, the segmentation for the “cow” revealed some challenges with capturing fine details, indicating areas for potential refinement.
+
+The multiple object segmentation proved challenging, as we struggled to find more than one working example that produced multiple segmentations. In our single example, the model demonstrated the ability to detect the general locations of objects — successfully identifying both wheels — but it also suffered from issues such as prompt sensitivity and duplicate labelling. This difficulty may be attributed to the inherent prompt sensitivity of the model or potential limitations of the specific model variant, particularly the 448–10B-8bit configuration. These observations suggest that either refining prompt structures or exploring more powerful models may be essential for reliably handling segmentation tasks involving multiple objects.
+
+# References
+
+- [Introducing PaliGemma 2 mix: A vision-language model for multiple tasks](https://developers.googleblog.com/en/introducing-paligemma-2-mix/)
+- [PaliGemma prompt and system instructions](https://ai.google.dev/gemma/docs/paligemma/prompt-system-instructions)
+- [PaliGemma 2 Mix — New Instruction Vision Language Models by Google](https://huggingface.co/blog/paligemma2mix)
+- [Welcome PaliGemma 2 — New vision language models by Google](https://huggingface.co/blog/paligemma2)
+- [Introducing PaliGemma 2: Powerful Vision-Language Models, Simple Fine-Tuning](https://developers.googleblog.com/en/introducing-paligemma-2-powerful-vision-language-models-simple-fine-tuning/)
+- [PaliGemma 2: A Family of Versatile VLMs for Transfer](https://arxiv.org/abs/2412.03555)

src/posts/2025-05-06-chat-qwen3-ios/index.qmd

@@ -0,0 +1,162 @@
+---
+title: "Chat with Qwen3 on your iPhone: A Step-by-Step Guide"
+author: "Joana Levtcheva"
+date: "2025-05-06"
+categories: [Machine Learning, mlx, swift, llm, ios]
+draft: false
+---
+
+Have you ever wanted to run a powerful large language model directly on your iPhone without sending your data to the cloud? Thanks to Apple’s [MLX Swift](https://github.com/ml-explore/mlx-swift) framework, you can now run the remarakably capable [Qwen3](https://qwenlm.github.io/blog/qwen3/) models right on your iPhone.
+
+What makes this exciting is how capable these new models are even with as little as 4B parameters. Based on Qwen’s benchmark reports, Qwen3–4B can rival the performance of Qwen2.5–72B-Instruct, delivering impressive results with just a fraction of the parameters. This development means truly powerful AI can now fit in your pocket.
+
+This blog post will guide you through the entire process from setting up your development environment to testing the model on your device. You can embrace true privacy, and take ownership of your AI experience. Your data stays on your device, where it belongs, and most importantly - you have access to this knowledge powerhouse even with no internet connection.
+
+Medium post can be found [here](https://medium.com/@levchevajoana/chat-with-qwen3-on-your-iphone-a-step-by-step-guide-515bb957cd02).
+
+# Main Steps
+
+The whole process consists of three main steps:
+
+- Enable Developer Mode on your iPhone
+- Clone Apple’s MLX Swift Examples [repository](https://github.com/ml-explore/mlx-swift-examples)
+- Configure and deploy the Xcode project to your device
+
+# Enable Developer Mode on iPhone
+
+In order to deploy custom apps to your iPhone, you need to enable **Developer Mode**. On your iPhone:
+
+- Go to **Settings → Privacy & Security → Developer Mode**  
+- Toggle Developer Mode to **On**
+Restart your iPhone when prompted
+
+> **NOTE:** It is important to note that your device’s security is reduced in Developer Mode.
+
+# Apple’s MLX Swift Examples Repository
+
+Now that your iPhone is ready for development, let’s get the necessary code. We are going to run the **MLXChatExample**, an example chat app supporting LLMs and VLMs for iOS and macOS, which can be found in the Applications folder in the [mlx-swift-examples](https://github.com/ml-explore/mlx-swift-examples) repository. So, the first step is to clone the repository:
+
+```bash
+git clone [email protected]:ml-explore/mlx-swift-examples.git
+```
+
+Then, navigate to the cloned directory:
+
+```bash
+cd mlx-swift-examples
+```
+
+And finally, open the Xcode project:
+
+```bash
+open mlx-swift-examples.xcodeproj
+```
+
+The above command should open the project directly in Xcode.
+
+# Configure and Deploy the Xcode Project
+
+With the project open in Xcode, you should connect your iPhone to the Mac, and then make a few adjustments.
+
+## Configure the Project
+
+We are interested in deploying the **MLXChatExample** app on your connected iPhone, so you should:
+
+- Change your scheme to **MLXChatExample** by going to **Product → Scheme → Choose Scheme** (this opens a dropdown menu from the Xcode’s top bar, which can be selected directly from the bar as well)
+- Set the destination to your connected iPhone by going to **Product → Destination → Choose Destination**
+
+<figure>
+  <img src="images/config_scheme_dest.png" alt="Configure scheme and destination" style="display: block; margin: 0 auto">
+  <figcaption style="text-align: center">Figure 1. Configure scheme and destination</figcaption>
+</figure>
+
+## Add Developer Team
+
+In order to build and run the app we have to assign a development team:
+
+- Select the **mlx-swift-examples** project in the Project Navigator (on the left)
+- Go to the **Signing & Capabilities** tab
+- Choose your **Team** from the dropdown menu
+
+<figure>
+  <img src="images/developer_team.png" alt="Developer team" style="display: block; margin: 0 auto">
+  <figcaption style="text-align: center">Figure 2. Add developer team</figcaption>
+</figure>
+
+# Deploy to Your iPhone
+
+Now we are ready to deploy **MLXChatExample** to our actual device:
+
+- Click the Run button (▶) on the left to run the project, or alternatively choose **Product &#8594; Run**
+- You might be asked to authorize Xcode development on your device
+- If successful, the app will be downloaded to your iPhone
+
+## Trust Developer Certificate
+
+If this is your first app from this developer team, you would get a message on your iPhone that the app is from an **Untrusted Developer** and you are not allowed to use it. We should allow applications from this developer, so on your iPhone:
+
+- Go to **Settings &#8594; General &#8594; VPN & Device Management**
+- Under **Developer App**, tap on your app
+Select **Trust “[Developer Name]”** and confirm
+
+## Using Qwen3 on Your iPhone
+
+With that, we are ready to test the MLXChatExample app. Let’s open it:
+
+<figure style="display: flex; justify-content: center; gap: 1rem; margin: 0;">
+  <div>
+    <img src="images/3_1.png" alt="App launch 1" />
+  </div>
+  <div>
+    <img src="images/3_2.png" alt="App launch 2" />
+  </div>
+</figure>
+<figcaption style="text-align: center; width: 100%; margin-top: 0.5rem;">
+    Figure 3. Left: MLXChatExample app default screen | Right: models dropdown menu
+</figcaption>
+
+By default, the app loads with the **llama3.2:1b** model. You can select different models from the dropdown menu in the upper right corner, see **Figure 3**. Let’s choose **qwen3:4b**, which runs successfully on an iPhone 14 Pro, and type our first query.
+
+If this is the first run for a given model, we have to wait for the model to download from Hugging Face to our cache. Below, in **Figure 4**, we can see our first query and how it triggers the model download (the arrow on top), and when clicking on the arrow we can see the download progress.
+
+<figure style="display: flex; justify-content: center; gap: 1rem; margin: 0;">
+  <div>
+    <img src="images/4_1.png" alt="Init query" />
+  </div>
+  <div>
+    <img src="images/4_2.png" alt="Download model" />
+  </div>
+</figure>
+<figcaption style="text-align: center; width: 100%; margin-top: 0.5rem;">
+    Figure 4. Left: first query for a model triggering model download | Right: model downloading progress
+</figcaption>
+
+The Qwen3 models can have thinking enabled and disabled. At the moment the app doesn’t support a toggle to manually switch between both modes. The default behaviour is to use thinking (the thought process output is encapsulated between the tags `<think></think>`), and in order to disable the thinking we have to add **/no_think** at the end of our prompt. In this case the think tags appear with nothing between them. Refer to **Figure 5** below.
+
+<figure style="display: flex; justify-content: center; gap: 1rem; margin: 0;">
+  <div>
+    <img src="images/5_1.png" alt="Gen 1" />
+  </div>
+  <div>
+    <img src="images/5_2.png" alt="Gen 2" />
+  </div>
+</figure>
+<figcaption style="text-align: center; width: 100%; margin-top: 0.5rem;">
+    Figure 5. Left: query with default thinking behaviour | Right: query with disabled thinking
+</figcaption>
+
+# The Benefits of On-Device AI
+
+Running models like Qwen3 directly on your device offers several significant advantages:
+
+- **Complete Privacy:** Your conversations never leave your device. No server logs, no data collection, just you and your AI.
+- **Works Offline:** You have all the world’s knowledge in your pocket wherever you are, without depending on internet connection availability. Need AI assistance while traveling, hiking, or in areas with poor connectivity? Local models work anywhere, anytime.
+- **No Subscription Fees:** Once you’ve set up the model, there are no ongoing costs.
+- **No Rate Limits:** Chat as much as you want without hitting quotas.
+- **Reduced Environmental Impact:** On-device inference eliminates the carbon footprint associated with massive data centers processing your requests.
+
+# Conclusion
+
+While cloud-based solutions may offer larger models with excellent performance, the gap is closing rapidly as efficiency improvements make smaller models surprisingly capable. The Qwen3 models are an excellent example for this, and are a “living” proof of the rapid advancements in the field of AI. The MLX Swift framework from Apple makes powerful machine learning accessible to everyday users on their personal devices. This shift from cloud-dependent to local-first AI is just beginning.
+
+***Own your AI. Own your data. Go local.***

src/posts/2025-05-23-app-docker-fastapi/index.qmd

@@ -0,0 +1,546 @@
+---
+title: "Image Segmentation with PaliGemma 2 mix, Transformers, Docker, FastAPI, and GitHub Actions"
+author: "Joana Levtcheva"
+date: "2025-05-23"
+categories: [Machine Learning, FastAPI, Docker, LLM, GitHub-Actions]
+draft: false
+---
+
+In today’s fast-paced machine learning landscape, deploying AI models is just as important as developing them. In this blog post, we are going to walk through an image segmentation application using Google’s **PaliGemma 2 Mix** model and **transformers**, containerized with **Docker**, and served through a **FastAPI** backend. We are also going to discuss the CI/CD pipeline using **GitHub Actions** to automate building the Docker image and pushing it to Docker Hub. Let’s explore this service, why we chose these technologies, and how you can get started and use the service yourself!
+
+The complete code is available on [GitHub](https://github.com/JoeJoe1313/PaliGemma-Image-Segmentation). 
+
+# What is This Project All About?
+
+At its core, this project provides a **FastAPI service** that allows you to perform image segmentation using natural language. You simply provide as input to the REST API:
+
+- A **text prompt** describing what to segment
+- An **image** via URL or file upload
+- The specific **PaliGemma 2 model** to perform image segmentation
+
+The service then returns:
+
+- The base64 encoded **model input image**
+- The base64 encoded segmentation **masks** clearly outlining the desired objects
+- The **bounding box coordinates** for each segmented object
+
+The **FastAPI** application is also containerized with **Docker** for consistent deployment across environments. A CI/CD pipeline with **GitHub Actions** is created for automated container builds and registry publishing to Docker Hub.
+
+# Architectural Blueprint: How It All Works Together
+
+Understanding the flow of data and the interaction of components is key. Let’s first take a look at our project structure:
+
+```plaintext
+project_folder/
+├── app/
+│   ├── __init__.py
+│   ├── main.py            # FastAPI application and endpoints
+│   └── segmentation.py    # Image segmentation logic
+├── models/
+│   ├── huggingface/       # Cache directory for Hugging Face models
+│   └── vae-oid.npz        # VAE model for mask generation
+├── .dockerignore
+├── .github/
+│   └── workflows/         # GitHub Actions for Docker build and push
+│     └── docker-build.yml # Workflow to build and push Docker images
+├── .gitignore
+├── docker-compose.yml
+├── Dockerfile
+├── README.md
+└── requirements.txt
+```
+
+## User & Developer Workflow
+
+Our system is designed with both the end-user and the developer in mind.
+
+```{mermaid}
+graph LR
+    User([User]) -->|Provides Image & Prompt| ClientApp[Client Application]
+    ClientApp -->|POST Request| FastAPI_Service[FastAPI Service]
+    FastAPI_Service -->|Process Input| PaliGemma_Model[PaliGemma Model]
+    PaliGemma_Model -->|Generate Segmentation Tokens| VAE_Model[VAE Model]
+    VAE_Model -->|Decode Masks| FastAPI_Service
+    FastAPI_Service -->|JSON Response | ClientApp
+    ClientApp -->|Display Results| User
+
+    Developer([Developer]) -->|Push Code| GitHubRepo[GitHub Repository]
+    GitHubRepo -->|Trigger| GitHubActions[GitHub Actions]
+    GitHubActions -->|Build & Push Image| DockerRegistry[Docker Hub]
+    DockerRegistry -->|Pull Image| DeploymentEnv[Deployment Environment]
+    DeploymentEnv -.->|Runs| FastAPI_Service
+```
+
+<figcaption style="text-align: center">Figure 1. User & Developer Workflow</figcaption>
+
+**Figure 1** shows the workflow:
+
+- A **User** interacts with a client application, providing an image and a text prompt, and optionally the specific PaliGemma 2 model.
+- The **Client Application** sends an HTTP POST request to our **FastAPI Service**.
+- The **FastAPI Service** preprocesses the input and feeds it to the **PaliGemma Model**.
+- PaliGemma generates segmentation tokens, which are then passed to the **VAE Model**.
+- The VAE Model decodes these into pixel-level masks and, along with bounding boxes, sends them back to the API.
+The API returns a JSON response to the client.
+
+To visualize the precise sequence of operations when a user requests an image segmentation, the following diagram details the interactions between the core components:
+
+```{mermaid}
+sequenceDiagram
+    participant User
+    participant Client
+    participant FastAPI
+    participant SegmentationPy as segmentation.py
+    participant PaliGemma as PaliGemma Model
+    participant VAE as VAE Model
+
+    User->>+Client: Upload image & prompt
+    Client->>+FastAPI: POST /segment
+    FastAPI->>+SegmentationPy: call segment_image()
+    SegmentationPy->>+PaliGemma: infer with PaliGemma
+    PaliGemma-->>-SegmentationPy: (tokens/features)
+    SegmentationPy->>+VAE: generate masks
+    VAE-->>-SegmentationPy: (pixel masks)
+    SegmentationPy-->>-FastAPI: return mask & coords
+    FastAPI-->>-Client: JSON response
+    Client-->>-User: display results
+```
+<figcaption style="text-align: center">Figure 2. Segmentation process</figcaption>
+
+This sequence highlights how FastAPI acts as the entry point, delegating the complex segmentation logic to the `segmentation.py` module, which in turn leverages the PaliGemma and VAE models to produce the desired output.
+
+For **developers**, pushing code to GitHub triggers **GitHub Actions**, which automatically builds a Docker image and pushes it to a **Container Registry** (Docker Hub), ready for deployment.
+
+## Inside the Application
+
+Within the Docker container, the application is neatly structured:
+
+```{mermaid}
+graph TD
+    subgraph "Docker Container"
+        subgraph "app/"
+            main[main.py
+            FastAPI Application]
+            segmentation[segmentation.py
+            Image Segmentation Logic]
+            main -->|imports| segmentation
+        end
+        
+        subgraph "External Dependencies"
+            NP[numpy]
+            TR[transformers]
+            PT[PyTorch]
+            JF[JAX/Flax]
+        end
+        
+        subgraph "Models"
+            PaliGemma[PaliGemma 2 mix]
+            VAE[VAE Checkpoint]
+        end
+        
+        segmentation -->|uses| TR
+        segmentation -->|uses| PT
+        segmentation -->|uses| JF
+        segmentation -->|uses| NP
+        
+        main -->|loads| PaliGemma
+        segmentation -->|loads| VAE
+    end
+    
+    Client[Client Application] -->|HTTP Requests| main
+    
+    subgraph "API Endpoints"
+        segment[POST /segment/]
+        root[GET /]
+    end
+    
+    main -->|defines| segment
+    main -->|defines| root
+    Client -->|calls| segment
+    Client -->|calls| root
+
+    style main fill:#c2e0ff,stroke:#0078d7
+    style segmentation fill:#c2e0ff,stroke:#0078d7
+    style Client fill:#ffd7b5,stroke:#ff8c00
+    style segment fill:#d5e8d4,stroke:#82b366
+    style root fill:#d5e8d4,stroke:#82b366
+```
+
+<figcaption style="text-align: center">Figure 3. Application Architecture</figcaption>
+
+- `app/main.py`: This is the heart of our API, built using FastAPI. It defines the API endpoints like `/` (for a welcome message) and `/segment` (for the actual segmentation).
+- `app/segmentation.py`: This module contains all the core logic for image processing, interacting with the PaliGemma and VAE models, and generating the final masks and coordinates. It uses the libraries **JAX**, **Flax** and **Transformers** for efficient model execution and inference.
+
+# Technology Stack Overview
+
+Let’s first understand the key technologies used in the project.
+
+## PaliGemma 2 mix
+
+[PaliGemma 2 mix](https://developers.googleblog.com/en/introducing-paligemma-2-mix/) is a state-of-the-art vision-language model from Google that can comprehend both images and text. It represents a significant advancement in multimodal AI, allowing for natural language-guided image understanding. Once PaliGemma identifies the segments (as tokens), a Variational Autoencoder (VAE) model steps in. It decodes these segmentation tokens into the detailed, pixel-level masks that you see as the output. This particular service uses a VAE checkpoint specifically `vae-oid.npz` for this task. A comprehensive overview of the image segmentation process with Paligemma 2 mix can be found in one of my previous posts [here](https://joejoe1313.github.io/2025-04-15-paligemma-2-mix.html).
+
+## FastAPI
+
+[FastAPI](https://fastapi.tiangolo.com) is a web framework for building APIs with Python. We chose this framework for several compelling reasons:
+
+- Automatic API documentation with **Swagger UI** and ReDoc
+Data validation and serialization through **Pydantic** models
+- Asynchronous request handling with support for `async`/`await` syntax
+- Excellent performance comparable to Node.js and Go
+- Built-in dependency injection system
+- Security and authentication features out of the box
+- WebSocket support and background tasks
+
+In our project, `main.py` uses FastAPI to define the `/segment` endpoint which accepts form data including the prompt, and optionally an image URL or an uploaded image file, and the desired `model_id`.
+
+## Transformers
+
+**Transformers** is a library by **Hugging Face** that provides state-of-the-art pre-trained models for natural language processing and computer vision. For our project, it’s essential because:
+
+- It provides easy access to the PaliGemma models
+- Offers a unified API for loading and using different model architectures
+- Handles model preprocessing and tokenization
+- Supports efficient model inference
+- Enables fine-tuning of models if needed
+
+## JAX/Flax
+
+JAX is a high-performance numerical computing library developed by Google. Flax is a neural network library built on top of JAX. Together, they provide:
+
+- Accelerated computation on GPUs and TPUs
+- Just-in-time compilation for optimized performance
+- Automatic differentiation capabilities
+- Functional programming approach to machine learning
+
+In our app **JAX/Flax & Transformers** are used for scalable model execution and inference, JAX/Flax is used for the VAE model which decodes segmentation tokens into pixel-level masks.
+
+## Docker & Docker Compose
+
+### What is Docker?
+
+Docker is a platform that uses OS-level virtualization to deliver software in packages called **containers**. A container is a standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries, and settings.
+
+### Why Docker for This Project?
+
+- **Consistency**: _“It works on my machine”_ is a phrase Docker aims to eliminate. By packaging the application, its dependencies (like specific versions of PyTorch, JAX, Transformers), we ensure it runs identically everywhere - from a developer’s laptop to a production server.
+- **Isolation**: Containers run in isolation, preventing conflicts between different applications or dependencies on the same host system.
+- **Simplified Deployment**: Docker abstracts away the underlying infrastructure. With a simple `docker-compose up` command, anyone can get the service running without manually installing Python, various libraries, or configuring complex environments.
+- **Scalability**: Dockerized applications are inherently easier to scale. Orchestration tools like Kubernetes can manage multiple instances of our container to handle increased load.
+- **Multi-Architecture Support**: Our Docker setup supports both `amd64` (common for desktops and servers) and `arm64` (increasingly used in cloud instances and devices like Raspberry Pi) architectures, broadening its usability.
+
+### Key Docker Components Used:
+
+- **`Dockerfile`**
+
+This is the **recipe for building our Docker image**. It specifies the base image (e.g., a Python image), copies our application code (`app/` directory, `requirements.txt`, etc.) into the image, installs all necessary Python packages, and defines the command to run when the container starts (e.g., `uvicorn app.main:app`).
+
+```Dockerfile
+FROM python:3.11-slim
+
+WORKDIR /app
+
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY . .
+
+ENV MODEL_ID=google/paligemma2-3b-mix-448
+ENV MODELS_DIR=/app/models
+
+EXPOSE 8000
+
+CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
+```
+
+- **`docker-compose.yml`**
+
+While a `Dockerfile` builds a single image, Docker Compose is used to define and run multi-container Docker applications. In our case, it simplifies running our FastAPI service. It can also manage networks, volumes, and other aspects of the application stack. For this project, it handles setting up the service and importantly, the volume mounting for model persistence.
+
+```yaml
+services:
+  paligemma-api:
+    image: joejoe1313/paligemma-image-segmentation:latest
+    ports:
+      - "8000:8000"
+    environment:
+      - MODEL_ID=google/paligemma2-3b-mix-448
+      - MODELS_DIR=/app/models
+    secrets:
+      - hf_token
+    volumes:
+      - $HOME/.cache/huggingface/hub:/app/models/huggingface
+    restart: unless-stopped
+
+secrets:
+  hf_token:
+    file: $HOME/.cache/huggingface/token
+```
+
+- **`.dockerignore`:**
+
+Similar to `.gitignore`, this file lists files and directories that should not be copied into the Docker image during the build process (e.g., local virtual environments, `.git` directory). This keeps the image lean and build times faster.
+
+### Smart Model Management with Volume Mounting
+
+Models, especially large ones like PaliGemma, take time to download. We use Docker’s **volume mounting** feature to map a directory on the host machine to a directory inside the container. This means:
+
+- Models are downloaded once and **persisted** on your host machine.
+- They are **reused** across container restarts.
+- They can be **shared** if you run multiple instances.
+- Updating models becomes easier as you can manage them directly on your host.
+
+The default mount point in our `docker-compose.yaml` file is `$HOME/.cache/huggingface/hub:/app/models/huggingface` which maps the local `$HOME/.cache/huggingface/hub` directory to `/app/models/huggingface` in the container.
+
+## CI/CD with GitHub Actions
+
+GitHub Actions provides automated workflows for continuous integration and continuous delivery (CI/CD). This is crucial for modern software development, thus we have implemented a CI/CD pipeline using **GitHub Actions**. We can see our workflow `.github/workflows/docker-build.yml` below:
+
+```yaml
+name: Docker Build and Push
+
+on:
+  push:
+    branches: main
+  pull_request:
+    branches: main
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Login to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: ${{ vars.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build and push
+        uses: docker/build-push-action@v6
+        with:
+          push: ${{ github.event_name != 'pull_request' }}
+          platforms: linux/amd64,linux/arm64
+          tags: ${{ vars.DOCKERHUB_USERNAME }}/paligemma-image-segmentation:latest
+```
+
+**What does it do?**
+
+- **Trigger**: Every time code is pushed to the `main` branch (or a pull request is made to `main`), the GitHub Actions workflow is automatically triggered.
+- **Build**: The workflow checks out the code and builds a multi-architecture (`amd64`/`arm64`) Docker image using the `Dockerfile`.
+- **Push**: The newly built image is then pushed to a container registry (like Docker Hub).
+- **Tag**: The image is tagged with the unique commit SHA (for traceability) and also with `latest` (for convenience).
+
+**Figure 4** below shows a high level overview of the workflow:
+
+```{mermaid}
+sequenceDiagram
+  participant G as GitHub Repo
+  participant A as GitHub Actions
+  participant D as Docker Registry
+
+  G->>A: on: push / pull_request to main
+  activate A
+  A-->>A: Login to Docker Registry
+  A-->>A: Set up QEMU (for multi-arch)
+  A-->>A: Set up Docker Buildx
+  A-->>A: Build Docker image (multi-arch)
+  A-->>D: Push image (tagged with SHA & latest)
+  deactivate A
+```
+
+<figcaption style="text-align: center">Figure 4. GitHub Actions workflow</figcaption>
+
+**How to use it in your fork**
+
+If you fork this repository, you can set up your own CI/CD pipeline by adding the following secrets to your GitHub repository settings:
+
+- `DOCKERHUB_USERNAME`: Your Docker Hub username.
+- `DOCKERHUB_TOKEN`: Your Docker Hub access token (not your password!).
+- You should also update the image in docker-compose.yaml with your username: `{DOCKERHUB_USERNAME}/paligemma-image-segmentation:latest`
+
+This automated pipeline ensures that a fresh, deployable image is always available.
+
+# Getting Started: Installation and Setup
+
+Ready to try it yourself? Here’s how:
+
+## Prerequisites
+
+- **Docker**: Ensure Docker Desktop or Docker Engine is installed and running.
+- **Hugging Face Token**: To access gated models like PaliGemma, you’ll need a Hugging Face token. Make sure it’s stored at `$HOME/.cache/huggingface/token` on your host machine, or adjust the path accordingly. The application will use this token when downloading models.
+
+## Setup with Docker Compose:
+
+- Clone the repository:
+
+```bash
+git clone https://github.com/JoeJoe1313/PaliGemma-Image-Segmentation.git
+cd PaliGemma-Image-Segmentation
+```
+
+- Ensure your **Hugging Face token** is in place as mentioned above. The `docker-compose.yml` is set up to mount your local Hugging Face cache, including the token.
+- Run the application:
+
+```bash
+docker-compose up -d
+```
+
+> **Warning**: It’s generally good practice to avoid running Docker commands as a root user unless necessary. Docker Compose should typically be run by a `user` in the `docker group`.
+
+This command will pull the pre-built Docker image and start the FastAPI service on `http://localhost:8000`. The `-dflag` runs it in detached mode.
+
+## Choosing Your PaliGemma Model
+
+You have two ways to specify which PaliGemma model variant to use:
+
+- At **runtime via API**: Pass the `model_id` parameter in your POST request to the `/segment` endpoint.
+- Via **Docker Environment Variable**: Set the `MODEL_ID` environment variable when starting the container.
+
+> **Note**: If both are set, the `model_id` parameter in the API request takes precedence.
+
+If a specified model isn’t found in the local cache (`/app/models/huggingface` inside the container, which maps to your `$HOME/.cache/huggingface/hub`), the application will attempt to download it from Hugging Face. **Figure 5** below shows a comprehensive overview of the possible steps.
+
+```{mermaid}
+flowchart TD
+    A[API Request] --> B{Check Local Cache}
+    B -->|Found| H[Load from Local Cache]
+    B -->|Not Found| C{Has HF Token?}
+    
+    %% Style definitions
+    classDef process fill:#e0e0ff,stroke:#9999ff,color:black
+    classDef decision1 fill:#ffe0b0,stroke:#ffbb66,color:black
+    classDef decision2 fill:#d0f0d0,stroke:#aaddaa,color:black
+    classDef cache fill:#d0e0ff,stroke:#aabbee,color:black
+    
+    %% Apply styles
+    class A,D,F,I,E,Z process
+    class B decision1
+    class C decision2
+    class G,H cache
+
+    C -->|Yes| D[Authenticate with HF]
+    C -->|No| E[Try Loading Public Model]
+    D --> F[Download Model]
+    F --> G[Save to Cache]
+    E -->|Success| G
+    E -->|Failure| Z[Auth Error]
+    G --> H
+    H --> I[Use Model]
+```
+
+<figcaption style="text-align: center">Figure 5. Model download scheme</figcaption>
+
+# Examples: Putting the API to the Test
+
+Once the service is running (default: `http://localhost:8000`):
+
+- **API Docs**: Visit `http://localhost:8000/docs` for interactive API documentation.
+
+## Health Check (GET /)
+
+Verify the API is up and running.
+
+**Request:**
+
+```python
+import requests
+
+response = requests.get("http://localhost:8000/")
+print(response.json())
+```
+
+**Response:**
+
+```json
+{
+    "message": "Welcome to the PaliGemma Segmentation API!"
+}
+```
+
+## Segmenting an Image (POST /segment/)
+
+Form Parameters:
+
+- `prompt` (str, required): Text description of objects to segment (e.g., "segment the red car").
+- `image_url` (str, optional): URL of the image to segment.
+- `image_file` (UploadFile, optional): Uploaded image file to segment.
+- `model_id` (str, optional): Specific PaliGemma model ID to use.
+
+### Using an Image URL
+
+```{mermaid}
+        sequenceDiagram
+            participant C as Client
+            participant S as /segment Endpoint
+            C->>S: POST Request with JSON body: { "image_url": "your_image_url.jpg", "prompt": "object to segment" }
+            S-->>S: Download Image
+            S-->>S: Process with PaliGemma & VAE
+            S-->>C: JSON Response: { "image": "base64_input_image", "masks": [ { "mask": "base64_mask_data", "coordinates": [x_min,y_min,x_max,y_max] } ] }
+```
+<figcaption style="text-align: center">Figure 6. Segment request</figcaption>
+
+**Request:**
+
+```python
+import requests
+
+data = {
+    "prompt": "segment left wheel",
+    "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg"
+}
+
+response = requests.post("http://localhost:8000/segment", data=data)
+print(response.json())
+```
+
+### Uploading an Image File
+
+**Request:**
+
+```python
+import os
+import requests
+
+segm_url = "http://localhost:8000/segment"
+image_path = "your_image.png"
+
+with open(image_path, "rb") as image_file:
+    data = {
+        "prompt": "segment the main object" # Adjust prompt as needed
+    }
+    files = {
+        "image_file": (os.path.basename(image_path), image_file, "image/png") # Or image/jpeg
+    }
+
+    response = requests.post(segm_url, files=files, data=data)
+    print(response.json())
+```
+
+### Expected JSON Response Structure:
+
+**Response:**
+
+```python
+{
+  "image": "base64_encoded_model_input_image_data",
+  "masks": [
+    {
+      "mask": "base64_encoded_mask_data_for_object_1",
+      "coordinates": [x_min, y_min, x_max, y_max]
+    }
+    # ...more masks if multiple instances of the prompt are found
+  ]
+}
+```
+
+See [example.ipynb](https://github.com/JoeJoe1313/PaliGemma-Image-Segmentation/blob/main/example.ipynb) for a demonstration of the segmentation pipeline.
+
+# Conclusion
+
+In this guide, we explored an image segmentation API using PaliGemma 2 mix, FastAPI, Transformers, and Docker. This service enables users to segment objects in images using natural language prompts, opening up a wide range of applications in computer vision, image editing, and data analysis. The containerized application provides a flexible, scalable solution that can be easily deployed across various environments. The combination of FastAPI’s performance and Docker’s portability makes this architecture ideal for production ML applications.
+
+_Happy coding!_