  Instruction finetuning is a popular paradigm to align large language models
(LLM) with human intent. Despite its popularity, this idea is less explored in
improving the LLMs to align existing foundation models with scientific
disciplines, concepts and goals. In this work, we present SciTune as a tuning
framework to improve the ability of LLMs to follow scientific multimodal
instructions. To test our methodology, we use a human-generated scientific
instruction tuning dataset and train a large multimodal model LLaMA-SciTune
that connects a vision encoder and LLM for science-focused visual and language
understanding. In comparison to the models that are finetuned with machine
generated data only, LLaMA-SciTune surpasses human performance on average and
in many sub-categories on the ScienceQA benchmark.

Computer Vision and Pattern Recognition;Artificial Intelligence;Computation and Language;Machine Learning