Documentation/Technical_Report.md

# Documentation for Gemma Chat Demo

## Model Choice and Research

Basing off of the Geema 3 technical report, I will analyze and compare different model sizes to determine the most suitable one for deployment in this project. For this portion, I will take the model's size, performance benchmarks, inference efficiency and memory requirements into consideration. The goal of this analysis is to strike a balance between computational cost and model quality.

### Model Overview

The Gemma 3 family consists of four model sizes, each with increasing capabilities and resource requirements:

| Model       | Parameters                                         | Vision Encoder | Total Size | Context Length | Key Capabilities                                                    |
| ----------- | -------------------------------------------------- | -------------- | ---------- | -------------- | ------------------------------------------------------------------- |
| Gemma 3-1B  | 698M non-embedding + 302M embedding                | None           | 1B         | 32K tokens     | Basic text generation; no vision capabilities, extrmely lightweight |
| Gemma 3-4B  | 3.2B non-embedding + 675M embedding + 417M vision  | SigLIP         | 4.3B       | 128K tokens    | Multimodal with good balance of performance and efficiency          |
| Gemma 3-12B | 10.8B non-embedding + 1B embedding + 417M vision   | SigLIP         | 12.2B      | 128K tokens    | Strong performance across all tasks with reasonable resource needs  |
| Gemma 3-27B | 25.6B non-embedding + 1.4B embedding + 417M vision | SigLIP         | 27.4B      | 128K tokens    | Best performance; comparable to Gemini 1.5 Pro on benchmarks        |

From this chart, the 1B model does not support vision encoding, thus it is limited to pure text-based tasks. As such, in order to fulloy demonstrate the capability demonstrated by the Gemma models, I will be moving forward with the other three models.

### Performance Analysis

#### Coding Performance

| Model | HumanEval | MBPP  | LiveCodeBench |
| ----- | --------- | ----- | ------------- |
| 1B    | 41.5%     | 35.2% | 5.0%          |
| 4B    | 71.3%     | 63.2% | 23.0%         |
| 12B   | 85.4%     | 73.0% | 32.0%         |
| 27B   | 87.8%     | 74.4% | 39.0%         |

The 12B and 27B models show strong coding capabilities, with 27B achieving the highest accuracy across all code-focused benchmarks. These results indicate that both are well-suited for code generation, debugging assistance, and live programming support. The 4B model, while not at the top, still demonstrates reliable code performance and may serve well in resource-constrained environments.

#### Research Capabilities

| Model | MMLU  | MATH  | GSM8K | GPQA Diamond |
| ----- | ----- | ----- | ----- | ------------ |
| 1B    | 38.8% | 48.0% | 62.8% | 19.2%        |
| 4B    | 58.1% | 75.6% | 89.2% | 30.8%        |
| 12B   | 71.9% | 83.8% | 94.4% | 40.9%        |
| 27B   | 76.9% | 89.0% | 95.9% | 42.4%        |

In tasks requiring factual recall, mathematical reasoning, and complex QA, performance improves significantly with scale. The 27B model again leads across all metrics.

### Hardware Requirements

| Model   | bf16 | Int4 | Int4 (Blocks=32) | SFP8 |
| ------- | ---- | ---- | ---------------- | ---- |
| **1B**  | 2.0  | 0.5  | 0.7              | 1.0  |
| +KV     | 2.9  | 1.4  | 1.6              | 1.9  |
| **4B**  | 8.0  | 2.6  | 2.9              | 4.4  |
| +KV     | 12.7 | 7.3  | 7.6              | 9.1  |
| **12B** | 24.0 | 6.6  | 7.1              | 12.4 |
| +KV     | 38.9 | 21.5 | 22.0             | 27.3 |
| **27B** | 54.0 | 14.1 | 15.3             | 27.4 |
| +KV     | 72.7 | 32.8 | 34.0             | 46.1 |

These are the required VRAM sizes (in GB) for running the Gemma 3 models under different precision formats and with or without Key-Value (KV) caching. For this application, I will be considering Key-Value pairing as essential for optimizing inference latency and enabling efficient long-context performance; since those are important factors in providing a smooth and responsive user experience in real-time chat scenarios.

### HuggingFace Spaces Resources

Since I am deploying on HuggingFace Spaces, I will be utilizing their **ZeroGPU**, which offers access to high-performance virtual GPUs, particularly the Nvidia H200 with 70GB of VRAM. This configuration provides ample memory and compute power to run large-scale language models with Key-Value (KV) caching, long context windows, and multimodal inference, all with low latency and high throughput.

### Final Choice

For my final choice, in order to fully demonstrate the capabilities of the Gemma 3 family, I have selected the Gemma 3-27B model with Key-Value caching enabled. This setup leverages the full compute and memory bandwidth of the NVIDIA H200 (70GB VRAM) provided by HuggingFace's ZeroGPU environment. Overall, this configuration strikes a strong balance between maximum model capability and inference efficiency, ensuring that the demo remains smooth, accurate, and production-ready—even when scaling to complex or multimodal inputs.

## User Research

To ensure this demo is intuitive and useful, I will identify key user types, their goals, and pain points. This informs both the interface design and prompt engineering strategy.

### User Profiles

Here, I will be listing some common user types and typical use cases for the demo:

1. Student Researchers

   - Demographics: University students
   - Needs: Literature research assistance, data analysis help, visualization of complex concepts

2. Software Developers

   - Demographics: Professional developers across experience levels
   - Needs: Code generation, debugging assistance, API documentation exploration

3. Content Creators

   - Demographics: Writers, marketers, social media managers
   - Needs: Creative ideation, content improvement, image analysis and description

4. General Knowledge Seekers
   - Demographics: Diverse, general public
   - Needs: Factual information, explanations of complex topics, assistance with everyday tasks

### User Stories

- As a student researcher, I want to input a research question and receive a concise, well-referenced summary so that I can quickly assess relevant sources.

- As a software developer, I want to submit code and receive debugging suggestions and performance tips so I can improve my code faster without switching tabs.

- As a content creator, I want to upload an image and get a tone-aligned caption with hashtags so I can streamline my content workflow for social media.

- As a general knowledge seeker, I want to ask layered questions in a natural language format so I can learn about complex topics step by step.

### Engineering Tasks

#### Core Infrastructure

- Multi-modal Input Processing: Implement handlers for text, code, images, and documents to support diverse user content types

- Context Memory Management: Design session-based context retention to enable follow-up questions and iterative refinement