lamm-mit/Cephalo-Gemma-3-4b-it-04-15-2025

Cephalo-Gemma-3-4b

Load model

import torch
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from transformers.image_utils import load_image
from PIL import Image as PILImage

ckpt = "lamm-mit/Cephalo-Gemma-3-4b-it-04-15-2025"
model = Gemma3ForConditionalGeneration.from_pretrained(
    ckpt, device_map="auto", torch_dtype=torch.bfloat16,
)
processor = AutoProcessor.from_pretrained(ckpt)

Inference example #1

image=PILImage.open(f'./spiderweb.png').convert("RGB")
messages = [
    {
        "role": "system",
        "content": [
             {"type": "text", "text": "You are a materials scientist."}
        ],
        "role": "user",
        "content": [
             {"type": "image", "image": image},
             {"type": "text", "text": "What does this image show? Provide a detailed analysis."}
        ]
    }
]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device)

input_len = inputs["input_ids"].shape[-1]

generation = model.generate(**inputs, max_new_tokens=512, do_sample=False)
generation = generation[0][input_len:]

decoded = processor.decode(generation, skip_special_tokens=True)
print(decoded)

Output:

The image shows a comparison between a 3D model of a structure and its physical 3D printed counterpart. 

The top part of the image displays a 3D model of a structure, which is a complex geometric design with multiple interconnected lines and angles. The model is likely created using computer-aided design (CAD) software, which allows for precise and detailed representation of the structure.

The bottom part of the image shows the physical 3D printed version of the same structure. The printed object is a tangible representation of the CAD model, with the same geometric design and intricate details. The printed object is placed on a surface, which could be a table or a platform, and is illuminated to highlight its three-dimensional form.

The comparison between the 3D model and the physical 3D printed object demonstrates the accuracy and fidelity of the 3D printing process. The printed object closely resembles the CAD model, indicating that the 3D printing technology can accurately reproduce complex geometric designs.

The results shown in the image highlight the potential of 3D printing for creating complex and intricate structures with high precision and accuracy. This technology has various applications in fields such as manufacturing, engineering, and design, where the ability to create precise and detailed objects is crucial.

Inference example #2

Another example:

image=PILImage.open(f'./scan.jpg').convert("RGB")
messages = [
    {
        "role": "system",
        "content": [
             {"type": "text", "text": "You are a biologist."}
        ],
        "role": "user",
        "content": [
             {"type": "image", "image": image},
             {"type": "text", "text": "Extract details of the material, type, design features, and use cases. Respond in JSON."}
        ]
    }
]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device)

input_len = inputs["input_ids"].shape[-1]

generation = model.generate(**inputs, max_new_tokens=512, do_sample=False)
generation = generation[0][input_len:]

decoded = processor.decode(generation, skip_special_tokens=True)
print(decoded)

Output:

Here's a detailed description of the material, type, design features, and use cases, presented in JSON format:

{
  "material": "Spider silk",
  "type": "Natural protein fiber",
  "design_features": [
    "High tensile strength and elasticity",
    "Lightweight and flexible",
    "Self-healing properties",
    "Biocompatible and biodegradable"
  ],
  "use_cases": [
    "Medical applications: sutures, scaffolds for tissue engineering, drug delivery systems",
    "Aerospace: lightweight composites for aircraft and spacecraft",
    "Protective gear: bulletproof vests, helmets, and body armor",
    "Industrial applications: high-performance fibers for ropes, cables, and nets",
    "Environmental applications: biodegradable packaging and textiles"
  ],
  "properties": {
    "tensile_strength": "Up to 1.5 GPa",
    "elastic_modulus": "1.5 GPa",
    "elongation_at_break": "Up to 1500%"
  },
  "source": "Spider silk is produced by spiders and is composed of proteins such as spidroin, which are arranged in a hierarchical structure to form the silk fibers."
}

**Explanation of the JSON fields:**

- **material**: The primary material used, which is spider silk.
- **type**: The type of material, which is a natural protein fiber.
- **design_features**: Key characteristics of the material, including its strength, elasticity, and self-healing properties.
- **use_cases**: Various applications where spider silk is used, ranging from medical to industrial.
- **properties**: Physical properties of spider silk, such as tensile strength, elastic modulus, and elongation at break.
- **source**: A brief description of where spider silk comes from and its composition.

This JSON provides a comprehensive overview of spider silk, highlighting its unique properties and potential applications.

Reference

@article{Buehler_Cephalo_2024_journal,
  title={Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Design},
  author={Markus J. Buehler},
  journal={Advanced Functional Materials},
  year={2024},
  volume={34},
  issue={49},
  doi={2409531},
  url={https://advanced.onlinelibrary.wiley.com/doi/full/10.1002/adfm.202409531}
}