VSLLaVA: a pipeline of large multimodal foundation model for industrial vibration signal analysis
<|reference_start|>VSLLaVA: a pipeline of large multimodal foundation model for industrial vibration signal analysis: Large multimodal foundation models have been extensively utilized for image recognition tasks guided by instructions, yet there remains a scarcity of domain expertise in industrial vibration signal analysis. This paper presents a pipeline named VSLLaVA that leverages a large language model to integrate expert knowledge for identification of signal parameters and diagnosis of faults. Within this pipeline, we first introduce an expert rule-assisted signal generator. The generator merges signal provided by vibration analysis experts with domain-specific parameter identification and fault diagnosis question-answer pairs to build signal-question-answer triplets. Then we use these triplets to apply low-rank adaptation methods for fine-tuning the linear layers of the Contrastive Language-Image Pretraining (CLIP) and large language model, injecting multimodal signal processing knowledge. Finally, the fine-tuned model is assessed through the combined efforts of large language model and expert rules to evaluate answer accuracy and relevance, which showcases enhanced performance in identifying, analyzing various signal parameters, and diagnosing faults. These enhancements indicate the potential of this pipeline to build a foundational model for future industrial signal analysis and monitoring.<|reference_end|>
@article{li2024vsllava:,
    title={VSLLaVA: a pipeline of large multimodal foundation model for industrial
  vibration signal analysis},
    author={Qi Li, Jinfeng Huang, Hongliang He, Xinran Zhang, Feibin Zhang, Zhaoye
  Qin and Fulei Chu},
    journal={arXiv preprint arXiv:2409.07482},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.07482},
    primaryClass={eess.SP cs.AI}
}