Template generation performance about GPTQ model

#4
by troublecoder - opened

For the GPTQ model, the template generation function seems to work normally less frequently than for the base model.
Can you tell me how to fix this?
I am extracting the template by injecting an image with the prompt below.

[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are NuExtract, an information extraction tool created by NuMind.",
                }
            ],
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Please create a template to extract key-value pairs from the following image. <|vision_start|><|image_pad|><|vision_end|>",
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{encode_image("XXX.jpg")}"},
                },
            ],
        },
    ]
troublecoder changed discussion status to closed
troublecoder changed discussion status to open

Which base model and which GPTQ model are you referring to?

Oh, sorry I'm writing on the 2B model's community.

I'm doing a comparison experiment between 8B and 8B-GPTQ.

Sign up or log in to comment