Intel/GLM-4.5V-int4-AutoRound

Model Details

This model is an int4 model with group_size 128 and symmetric quantization of zai-org/GLM-4.5V generated by intel/auto-round algorithm. Please follow the license of the original model.

How To Use

INT4 Inference

requirement

pip install transformers-v4.55.0-GLM-4.5V-preview

from transformers import AutoProcessor, Glm4vMoeForConditionalGeneration
import torch

MODEL_PATH = "Intel/GLM-4.5V-int4-AutoRound"
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "url": "https://upload.wikimedia.org/wikipedia/commons/f/fa/Grayscale_8bits_palette_sample_image.png"
            },
            {
                "type": "text",
                "text": "describe this image"
            }
        ],
    }
]
processor = AutoProcessor.from_pretrained(MODEL_PATH)
model = Glm4vMoeForConditionalGeneration.from_pretrained(
    pretrained_model_name_or_path=MODEL_PATH,
    torch_dtype="auto",
    device_map="auto",
)
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt"
).to(model.device)
inputs.pop("token_type_ids", None)
generated_ids = model.generate(**inputs, max_new_tokens=8192)
output_text = processor.decode(generated_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False)
print(output_text)
"""
<think>Got it, let's see. The user wants a description of the image. First, I need to look at the details. The image is black and white, so that's a key point. The main subject is a parrot. Let's check its features: it has a large beak, maybe a crest on its head, feathers that look soft. It's perched on a circular perch, maybe a metal ring. The background has some foliage, but it's blurred, so the focus is on the parrot. I should mention the color (black and white), the parrot's position, its physical features like beak, feathers, the perch, and the background. Make sure to describe it clearly so the user gets a good picture.</think>
This is a black - and - white photograph featuring a parrot as the central subject. The parrot is perched on a circular metal perch. It has a prominent, curved beak and its feathers appear soft and textured. The bird’s head is turned slightly, and its body is positioned in a way that shows off its wings and tail. In the background, there is some blurred foliage, which helps to keep the focus on the parrot. The overall composition highlights the parrot’s details and posture.<|user|>
"""

Generate the model

auto_round --mllm --model zai-org/GLM-4.5V --output_dir tmp_autoround --group_size 128 --seqlen 2048

Ethical Considerations and Limitations

The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.

Therefore, before deploying any applications of the model, developers should perform safety testing.

Caveats and Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

Here are a couple of useful links to learn more about Intel's AI software:

• Intel Neural Compressor link

Disclaimer

The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.

Cite

@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }

arxiv github