Cannot apply chat template from tokenizer
The tokenizer loaded from AutoTokenizer cannot be used to apply the chat template, which is quite unexpected.
>>> from transformers import AutoProcessor, AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("mistral-community/pixtral-12b")
>>> tokenizer.chat_template # The result is None
>>> processor = AutoProcessor.from_pretrained("mistral-community/pixtral-12b")
>>> processor.chat_template # The result is correct
I have tested various versions of transformers and this happens from v4.44+ (possibly earlier), so I don't think it's a problem with Transformers library.
Hey! This is because the tokenizer config doesn't have a "chat_template" field in https://huggingface.co/mistral-community/pixtral-12b/blob/main/tokenizer_config.json. We usually do not add tokenizer templates if the model is a multimodal model, i.e. we don't expect anyone to use tokenizer and image processor separately
But in Vision LLM, like gemma3, the chat template is duplicated as the model has a Gemma3ForCausaLM class which allows users to do simple text-only inferece
You should definitely expect either text or vision queries separately. The original Pixtral model from mistralai worked fine, but no longer works in new vllm. But this community one has the problem that it has no tokenizer for vllm to use for normal text queries.