Multi-Modal Instruction-Tuning Small-Scale Language-and-Vision Assistant for Semiconductor Electron Micrograph Analysis
<|reference_start|>Multi-Modal Instruction-Tuning Small-Scale Language-and-Vision Assistant for Semiconductor Electron Micrograph Analysis: We present a novel framework for analyzing and interpreting electron microscopy images in semiconductor manufacturing using vision-language instruction tuning. The framework employs a unique teacher-student approach, leveraging pre-trained multimodal large language models such as GPT-4 to generate instruction-following data for zero-shot visual question answering (VQA) and classification tasks, customizing smaller multimodal models (SMMs) for microscopy image analysis, resulting in an instruction-tuned language-and-vision assistant. Our framework merges knowledge engineering with machine learning to integrate domain-specific expertise from larger to smaller multimodal models within this specialized field, greatly reducing the need for extensive human labeling. Our study presents a secure, cost-effective, and customizable approach for analyzing microscopy images, addressing the challenges of adopting proprietary models in semiconductor manufacturing.<|reference_end|>
@article{srinivas2024multi-modal,
    title={Multi-Modal Instruction-Tuning Small-Scale Language-and-Vision Assistant
  for Semiconductor Electron Micrograph Analysis},
    author={Sakhinana Sagar Srinivas, Geethan Sannidhi, Venkataramana Runkana},
    journal={arXiv preprint arXiv:2409.07463},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.07463},
    primaryClass={cs.CV cs.LG}
}