Hugging Face's logo

microsoft
/
Phi-4-multimodal-instruct

Automatic Speech Recognition
Transformers
Safetensors
phi4mm
text-generation
nlp
code
audio
speech-summarization
speech-translation
visual-question-answering
phi-4-multimodal
phi
phi-4-mini
custom_code
Model card Files
xet
Community
79

Error when loading audio processor saved with processor.save_pretrained(path_to_dir), and num_logits_to_keep wrong init value

#73
by YaelAiola - opened
Discussion
YaelAiola
  1. After fine-tuning the model on a speech task, I attempted to save both the model and the processor using processor.save_pretrained(path_to_dir). However, when loading the model, I encountered an error due to mismatches between the processor's saved fields and the expected input arguments:

audio_compression_rate → compression_rate

qformer_compression_rate → audio_downsample_rate

audio_feat_stride → feat_stride

Additionally, some arguments such as feature_size, sampling_rate, and padding_value are hardcoded in the processor, causing the loading process to fail when they are encountered again
preprocessor_config.png

  1. If the user does not provide the num_logits_to_keep argument, the code fails because in modeling_phi4mm.py, within the prepare_inputs_for_generation function, the argument is initialized with None instead of a default value- 0.
KevinKibe

I'm experiencing the same issue:

TypeError: Phi4MMAudioFeatureExtractor.__init__() missing 3 required positional arguments: 'audio_compression_rate', 'audio_downsample_rate', and 'audio_feat_stride'

To work around this, I used the base model's processor. However, my fine-tuned model is now hallucinating, and I'm unsure if this is the cause.

Does anyone have any solutions or suggestions for this issue?

Borrison

Running into the same issue.

morenolq

Same issue here, what can solve the problem?

morenolq
•
edited 6 days ago

What I exactly did to "temporary" solve:

  1. in preprocessor_config.json
    1. compression_rate renamed into audio_compression_rate (as suggested by @YaelAiola )
    2. feat_stride renamed into audio_feat_stride (as suggested by @YaelAiola )
    3. qformer_compression_rate renamed into audio_downsample_rate(as suggested by @YaelAiola )
    4. deleted padding_value field tout-court
    5. deleted feature_size field tout-court
    6. deleted sampling_rate field tout-court
  2. still got the error about chat_template having no setter -> I deleted the chat_template.json file (actually renamed it to do not loose it, but have the same effect), as it is on tokenizer_config.json

Hopefully this will save a few hours to some of you.

Sign up or log in to comment