[node] estimation

README.md

@@ -1,6 +1,6 @@
 ---
 title: Tp 1 Dgx Node Estimator
-emoji: 🐠
+emoji: ⚙️
 colorFrom: purple
 colorTo: yellow
 sdk: gradio
@@ -12,3 +12,133 @@ short_description: for NVIDIA TRDC estimation
 ---
 
 Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+
+# 🚀 H100 Node & CUDA Version Estimator
+
+An interactive Gradio application for estimating H100 GPU node requirements and CUDA version recommendations based on your machine learning workload specifications.
+
+## Features
+
+- **Model Support**: Supports popular models including LLaMA-2/3/3.1, Nemotron-4, and Qwen2/2.5 variants
+- **Smart Estimation**: Calculates memory requirements including model weights, KV cache, and operational overhead
+- **Use Case Optimization**: Provides different estimates for inference, training, and fine-tuning scenarios
+- **Precision Support**: Handles different precision formats (FP32, FP16, BF16, INT8, INT4)
+- **Interactive Visualizations**: Memory breakdown charts and node utilization graphs
+- **CUDA Recommendations**: Suggests optimal CUDA versions and driver requirements
+
+## Installation
+
+1. Clone the repository:
+```bash
+git clone <repository-url>
+cd tp-1-dgx-node-estimator
+```
+
+2. Install dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+## Usage
+
+1. Run the application:
+```bash
+python app.py
+```
+
+2. Open your browser and navigate to `http://localhost:7860`
+
+3. Configure your parameters:
+   - **Model**: Select from supported models (LLaMA, Nemotron, Qwen2)
+   - **Input Tokens**: Number of input tokens per request
+   - **Output Tokens**: Number of output tokens per request
+   - **Batch Size**: Number of concurrent requests
+   - **Use Case**: Choose between inference, training, or fine-tuning
+   - **Precision**: Select model precision/quantization level
+
+4. Click "💡 Estimate Requirements" to get your recommendations
+
+## Key Calculations
+
+### Memory Estimation
+- **Model Memory**: Base model weights adjusted for precision
+- **KV Cache**: Calculated based on sequence length and model architecture
+- **Overhead**: Use-case specific multipliers:
+  - Inference: 1.2x (20% overhead)
+  - Training: 3.0x (gradients + optimizer states)
+  - Fine-tuning: 2.5x (moderate overhead)
+
+### Node Calculation
+- **H100 Memory**: 80GB HBM3 per GPU (90% usable)
+- **Model Parallelism**: Automatic consideration for large models
+- **Memory Efficiency**: Optimal distribution across nodes
+
+## Example Scenarios
+
+| Model | Tokens (In/Out) | Batch Size | Use Case | Precision | Estimated Nodes |
+|-------|----------------|------------|----------|-----------|----------------|
+| LLaMA-3-8B | 2048/512 | 1 | Inference | FP16 | 1 |
+| LLaMA-3-70B | 4096/1024 | 4 | Inference | FP16 | 3-4 |
+| Qwen2.5-72B | 8192/2048 | 2 | Fine-tuning | BF16 | 4-5 |
+| Nemotron-4-340B | 2048/1024 | 1 | Inference | INT8 | 6-8 |
+
+## CUDA Recommendations
+
+The application provides tailored CUDA version recommendations:
+
+- **Optimal**: CUDA 12.4 + cuDNN 8.9+
+- **Recommended**: CUDA 12.1+ + cuDNN 8.7+
+- **Minimum**: CUDA 11.8 + cuDNN 8.5+
+
+## Output Features
+
+### 📊 Detailed Analysis
+- Complete memory breakdown
+- Parameter counts and model specifications
+- Step-by-step calculation explanation
+
+### 🔧 CUDA Recommendations
+- Version compatibility matrix
+- Driver requirements
+- Compute capability information
+
+### 📈 Memory Utilization
+- Visual memory breakdown (pie chart)
+- Node utilization distribution (bar chart)
+- Efficiency metrics
+
+## Technical Details
+
+### Supported Models
+- **LLaMA**: 2-7B, 2-13B, 2-70B, 3-8B, 3-70B, 3.1-8B, 3.1-70B, 3.1-405B
+- **Nemotron**: 4-15B, 4-340B
+- **Qwen2**: 0.5B, 1.5B, 7B, 72B
+- **Qwen2.5**: 0.5B, 1.5B, 7B, 14B, 32B, 72B
+
+### Precision Impact
+- **FP32**: Full precision (4 bytes per parameter)
+- **FP16/BF16**: Half precision (2 bytes per parameter)
+- **INT8**: 8-bit quantization (1 byte per parameter)
+- **INT4**: 4-bit quantization (0.5 bytes per parameter)
+
+## Limitations
+
+- Estimates are approximate and may vary based on:
+  - Specific model implementation details
+  - Framework overhead (PyTorch, TensorFlow, etc.)
+  - Hardware configuration
+  - Network topology for multi-node setups
+
+## Contributing
+
+Feel free to submit issues and enhancement requests!
+
+## License
+
+This project is licensed under the MIT License - see the LICENSE file for details.
+
+## Notes
+
+- For production deployments, consider adding a 10-20% buffer to estimates
+- Network bandwidth and storage requirements are not included in calculations
+- Estimates assume optimal memory layout and efficient implementations

app.py

@@ -0,0 +1,361 @@
+import gradio as gr
+import pandas as pd
+import matplotlib.pyplot as plt
+import numpy as np
+import json
+from typing import Dict, Tuple, List
+
+# Model specifications (approximate parameter counts and memory requirements)
+MODEL_SPECS = {
+    "LLaMA-2-7B": {"params": 7e9, "base_memory_gb": 14},
+    "LLaMA-2-13B": {"params": 13e9, "base_memory_gb": 26},
+    "LLaMA-2-70B": {"params": 70e9, "base_memory_gb": 140},
+    "LLaMA-3-8B": {"params": 8e9, "base_memory_gb": 16},
+    "LLaMA-3-70B": {"params": 70e9, "base_memory_gb": 140},
+    "LLaMA-3.1-8B": {"params": 8e9, "base_memory_gb": 16},
+    "LLaMA-3.1-70B": {"params": 70e9, "base_memory_gb": 140},
+    "LLaMA-3.1-405B": {"params": 405e9, "base_memory_gb": 810},
+    "Nemotron-4-340B": {"params": 340e9, "base_memory_gb": 680},
+    "Nemotron-4-15B": {"params": 15e9, "base_memory_gb": 30},
+    "Qwen2-0.5B": {"params": 0.5e9, "base_memory_gb": 1},
+    "Qwen2-1.5B": {"params": 1.5e9, "base_memory_gb": 3},
+    "Qwen2-7B": {"params": 7e9, "base_memory_gb": 14},
+    "Qwen2-72B": {"params": 72e9, "base_memory_gb": 144},
+    "Qwen2.5-0.5B": {"params": 0.5e9, "base_memory_gb": 1},
+    "Qwen2.5-1.5B": {"params": 1.5e9, "base_memory_gb": 3},
+    "Qwen2.5-7B": {"params": 7e9, "base_memory_gb": 14},
+    "Qwen2.5-14B": {"params": 14e9, "base_memory_gb": 28},
+    "Qwen2.5-32B": {"params": 32e9, "base_memory_gb": 64},
+    "Qwen2.5-72B": {"params": 72e9, "base_memory_gb": 144},
+}
+
+# H100 specifications
+H100_MEMORY_GB = 80
+H100_COMPUTE_CAPABILITY = "9.0"
+
+# CUDA version recommendations based on model and use case
+CUDA_RECOMMENDATIONS = {
+    "inference": {
+        "recommended": "12.1+",
+        "minimum": "11.8",
+        "optimal": "12.4"
+    },
+    "training": {
+        "recommended": "12.1+", 
+        "minimum": "11.8",
+        "optimal": "12.4"
+    },
+    "fine_tuning": {
+        "recommended": "12.1+",
+        "minimum": "11.8", 
+        "optimal": "12.4"
+    }
+}
+
+def calculate_kv_cache_memory(num_tokens: int, model_params: float, num_layers: int = None) -> float:
+    """Calculate KV cache memory requirements in GB"""
+    if num_layers is None:
+        # Estimate layers based on model size
+        if model_params < 1e9:
+            num_layers = 24
+        elif model_params < 10e9:
+            num_layers = 32
+        elif model_params < 100e9:
+            num_layers = 80
+        else:
+            num_layers = 96
+    
+    # KV cache memory per token (approximate)
+    # 2 (K + V) * 2 (fp16) * hidden_dim * num_layers
+    hidden_dim = int((model_params / (num_layers * 4)) ** 0.5) * 64  # Rough estimate
+    kv_memory_per_token = 2 * 2 * hidden_dim * num_layers / (1024**3)  # GB
+    
+    return num_tokens * kv_memory_per_token
+
+def estimate_h100_nodes(
+    model_name: str,
+    input_tokens: int,
+    output_tokens: int,
+    batch_size: int,
+    use_case: str,
+    precision: str
+) -> Tuple[int, str, Dict]:
+    """
+    Estimate the number of H100 nodes required
+    
+    Returns:
+        - Number of nodes required
+        - Detailed explanation
+        - Dictionary with breakdown
+    """
+    
+    if model_name not in MODEL_SPECS:
+        return 1, f"Model {model_name} not found in specifications", {}
+    
+    model_spec = MODEL_SPECS[model_name]
+    base_memory = model_spec["base_memory_gb"]
+    
+    # Adjust memory based on precision
+    precision_multiplier = {
+        "FP32": 1.0,
+        "FP16": 0.5,
+        "BF16": 0.5,
+        "INT8": 0.25,
+        "INT4": 0.125
+    }
+    
+    model_memory = base_memory * precision_multiplier.get(precision, 0.5)
+    
+    # Calculate KV cache memory
+    total_tokens = input_tokens + output_tokens
+    kv_cache_memory = calculate_kv_cache_memory(total_tokens, model_spec["params"]) * batch_size
+    
+    # Use case specific memory overhead
+    overhead_multiplier = {
+        "inference": 1.2,  # 20% overhead
+        "training": 3.0,   # 3x for gradients, optimizer states
+        "fine_tuning": 2.5 # 2.5x for fine-tuning
+    }
+    
+    total_memory_per_instance = (model_memory + kv_cache_memory) * overhead_multiplier.get(use_case, 1.2)
+    
+    # Calculate nodes needed
+    memory_per_node = H100_MEMORY_GB * 0.9  # Reserve 10% for system
+    nodes_needed = max(1, int(np.ceil(total_memory_per_instance / memory_per_node)))
+    
+    # For very large models, consider model parallelism
+    if model_memory > memory_per_node:
+        min_nodes_for_model = int(np.ceil(model_memory / memory_per_node))
+        nodes_needed = max(nodes_needed, min_nodes_for_model)
+    
+    # Generate explanation
+    explanation = f"""
+    **Estimation Breakdown:**
+    
+    • **Model**: {model_name} ({model_spec['params']/1e9:.1f}B parameters)
+    • **Precision**: {precision}
+    • **Model Memory**: {model_memory:.1f} GB
+    • **KV Cache Memory**: {kv_cache_memory:.1f} GB (for {total_tokens:,} tokens × {batch_size} batch size)
+    • **Use Case Overhead**: {overhead_multiplier.get(use_case, 1.2):.1f}x ({use_case})
+    • **Total Memory Required**: {total_memory_per_instance:.1f} GB
+    • **H100 Usable Memory**: {memory_per_node:.1f} GB per node
+    
+    **Recommendation**: {nodes_needed} H100 node(s)
+    """
+    
+    breakdown = {
+        "model_memory_gb": model_memory,
+        "kv_cache_memory_gb": kv_cache_memory,
+        "total_memory_gb": total_memory_per_instance,
+        "h100_memory_per_node_gb": memory_per_node,
+        "nodes_required": nodes_needed
+    }
+    
+    return nodes_needed, explanation, breakdown
+
+def get_cuda_recommendation(use_case: str) -> str:
+    """Get CUDA version recommendation based on use case"""
+    cuda_info = CUDA_RECOMMENDATIONS.get(use_case, CUDA_RECOMMENDATIONS["inference"])
+    
+    recommendation = f"""
+    **CUDA Version Recommendations for {use_case.title()}:**
+    
+    • **Optimal**: CUDA {cuda_info['optimal']} + cuDNN 8.9+
+    • **Recommended**: CUDA {cuda_info['recommended']} + cuDNN 8.7+
+    • **Minimum**: CUDA {cuda_info['minimum']} + cuDNN 8.5+
+    
+    **Additional Requirements:**
+    • **Driver Version**: 525.60.13+ (Linux) / 527.41+ (Windows)
+    • **Compute Capability**: {H100_COMPUTE_CAPABILITY} (H100 native)
+    • **Memory**: ECC enabled recommended for production
+    """
+    
+    return recommendation
+
+def create_performance_chart(breakdown: Dict) -> plt.Figure:
+    """Create a memory utilization chart"""
+    if not breakdown:
+        fig, ax = plt.subplots(figsize=(8, 6))
+        ax.text(0.5, 0.5, 'No data to display', ha='center', va='center')
+        ax.set_xlim(0, 1)
+        ax.set_ylim(0, 1)
+        return fig
+    
+    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))
+    
+    # Memory breakdown pie chart
+    labels = ['Model Memory', 'KV Cache', 'Overhead']
+    model_mem = breakdown['model_memory_gb']
+    kv_mem = breakdown['kv_cache_memory_gb']
+    overhead_mem = breakdown['total_memory_gb'] - model_mem - kv_mem
+    sizes = [model_mem, kv_mem, overhead_mem]
+    
+    colors = ['#ff9999', '#66b3ff', '#99ff99']
+    ax1.pie(sizes, labels=labels, colors=colors, autopct='%1.1f%%', startangle=90)
+    ax1.set_title('Memory Breakdown')
+    
+    # Node utilization bar chart
+    nodes = breakdown['nodes_required']
+    total_memory = breakdown['total_memory_gb']
+    memory_per_node = breakdown['h100_memory_per_node_gb']
+    
+    node_labels = [f'Node {i+1}' for i in range(nodes)]
+    utilization = []
+    
+    for i in range(nodes):
+        if i < nodes - 1:
+            utilization.append(memory_per_node)
+        else:
+            remaining_memory = total_memory - (nodes - 1) * memory_per_node
+            utilization.append(remaining_memory)
+    
+    utilization_pct = [u / memory_per_node * 100 for u in utilization]
+    
+    bars = ax2.bar(node_labels, utilization_pct, color='skyblue', alpha=0.7)
+    ax2.axhline(y=100, color='red', linestyle='--', alpha=0.7, label='Max Capacity')
+    ax2.set_ylabel('Memory Utilization (%)')
+    ax2.set_title('H100 Node Memory Utilization')
+    ax2.set_ylim(0, 110)
+    ax2.legend()
+    
+    # Add value labels on bars
+    for bar, pct in zip(bars, utilization_pct):
+        ax2.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 1,
+                f'{pct:.1f}%', ha='center', va='bottom')
+    
+    plt.tight_layout()
+    return fig
+
+def estimate_nodes_interface(
+    model_name: str,
+    input_tokens: int,
+    output_tokens: int,
+    batch_size: int,
+    use_case: str,
+    precision: str
+):
+    """Main interface function"""
+    
+    # Validate inputs
+    if input_tokens <= 0 or output_tokens <= 0:
+        return "Please enter valid token counts (> 0)", "", None, ""
+    
+    if batch_size <= 0:
+        return "Please enter a valid batch size (> 0)", "", None, ""
+    
+    # Calculate node requirements
+    nodes_needed, explanation, breakdown = estimate_h100_nodes(
+        model_name, input_tokens, output_tokens, batch_size, use_case, precision
+    )
+    
+    # Get CUDA recommendations
+    cuda_rec = get_cuda_recommendation(use_case)
+    
+    # Create performance chart
+    fig = create_performance_chart(breakdown)
+    
+    return explanation, cuda_rec, fig, f"**Estimated H100 Nodes Required: {nodes_needed}**"
+
+# Create Gradio interface
+def create_interface():
+    with gr.Blocks(title="H100 Node Estimator", theme=gr.themes.Soft()) as demo:
+        gr.Markdown("# 🚀 H100 Node & CUDA Version Estimator")
+        gr.Markdown("Get recommendations for H100 node count and CUDA version based on your model and workload requirements.")
+        
+        with gr.Row():
+            with gr.Column(scale=1):
+                gr.Markdown("## Input Parameters")
+                
+                model_dropdown = gr.Dropdown(
+                    choices=list(MODEL_SPECS.keys()),
+                    value="LLaMA-3-8B",
+                    label="Model",
+                    info="Select the model you want to run"
+                )
+                
+                input_tokens = gr.Number(
+                    value=2048,
+                    label="Input Tokens",
+                    info="Number of input tokens per request"
+                )
+                
+                output_tokens = gr.Number(
+                    value=512,
+                    label="Output Tokens", 
+                    info="Number of output tokens per request"
+                )
+                
+                batch_size = gr.Number(
+                    value=1,
+                    label="Batch Size",
+                    info="Number of concurrent requests"
+                )
+                
+                use_case = gr.Dropdown(
+                    choices=["inference", "training", "fine_tuning"],
+                    value="inference",
+                    label="Use Case",
+                    info="What will you use the model for?"
+                )
+                
+                precision = gr.Dropdown(
+                    choices=["FP32", "FP16", "BF16", "INT8", "INT4"],
+                    value="FP16",
+                    label="Precision",
+                    info="Model precision/quantization"
+                )
+                
+                estimate_btn = gr.Button("💡 Estimate Requirements", variant="primary")
+            
+            with gr.Column(scale=2):
+                gr.Markdown("## Results")
+                
+                node_count = gr.Markdown("**Ready to estimate...**")
+                
+                with gr.Tab("📊 Detailed Analysis"):
+                    detailed_output = gr.Markdown()
+                
+                with gr.Tab("🔧 CUDA Recommendations"):
+                    cuda_output = gr.Markdown()
+                
+                with gr.Tab("📈 Memory Utilization"):
+                    chart_output = gr.Plot()
+        
+        # Connect the interface
+        estimate_btn.click(
+            fn=estimate_nodes_interface,
+            inputs=[model_dropdown, input_tokens, output_tokens, batch_size, use_case, precision],
+            outputs=[detailed_output, cuda_output, chart_output, node_count]
+        )
+        
+        # Add examples
+        gr.Markdown("## 💡 Example Scenarios")
+        
+        examples = [
+            ["LLaMA-3-8B", 2048, 512, 1, "inference", "FP16"],
+            ["LLaMA-3-70B", 4096, 1024, 4, "inference", "FP16"], 
+            ["Qwen2.5-72B", 8192, 2048, 2, "fine_tuning", "BF16"],
+            ["Nemotron-4-340B", 2048, 1024, 1, "inference", "INT8"],
+        ]
+        
+        gr.Examples(
+            examples=examples,
+            inputs=[model_dropdown, input_tokens, output_tokens, batch_size, use_case, precision],
+            outputs=[detailed_output, cuda_output, chart_output, node_count],
+            fn=estimate_nodes_interface,
+            cache_examples=False
+        )
+        
+        gr.Markdown("""
+        ## ℹ️ Notes
+        - Estimates are approximate and may vary based on actual implementation details
+        - Memory calculations include model weights, KV cache, and operational overhead
+        - Consider network bandwidth and storage requirements for multi-node setups
+        - For production deployments, add 10-20% buffer for optimal performance
+        """)
+    
+    return demo
+
+if __name__ == "__main__":
+    demo = create_interface()
+    demo.launch(share=True, server_name="0.0.0.0", server_port=7860)

requirements.txt

@@ -0,0 +1,13 @@
+APScheduler
+black
+datasets
+gradio>=4.0.0
+gradio[oauth]
+gradio_leaderboard==0.0.13
+gradio_client
+huggingface-hub>=0.18.0
+matplotlib>=3.6.0
+numpy>=1.24.0
+pandas>=1.5.0
+python-dateutil
+tqdm