MVLLaVA: An Intelligent Agent for Unified and Flexible Novel View
  Synthesis
This paper introduces MVLLaVA, an intelligent agent designed for novel view
synthesis tasks. MVLLaVA integrates multiple multi-view diffusion models with a
large multimodal model, LLaVA, enabling it to handle a wide range of tasks
efficiently. MVLLaVA represents a versatile and unified platform that adapts to
diverse input types, including a single image, a descriptive caption, or a
specific change in viewing azimuth, guided by language instructions for
viewpoint generation. We carefully craft task-specific instruction templates,
which are subsequently used to fine-tune LLaVA. As a result, MVLLaVA acquires
the capability to generate novel view images based on user instructions,
demonstrating its flexibility across diverse tasks. Experiments are conducted
to validate the effectiveness of MVLLaVA, demonstrating its robust performance
and versatility in tackling diverse novel view synthesis challenges.
66e256dc239f8302156d9417
Hanyu Jiang
66e256dc239f8302156d9418
Jian Xue
66e256dc239f8302156d9419
Xing Lan
66e256dc239f8302156d941a
Guohong Hu
66e256dc239f8302156d941b
Ke Lu
MVLLaVA: An Intelligent Agent for Unified and Flexible Novel View
  Synthesis
This paper introduces MVLLaVA, an intelligent agent designed for novel view
synthesis tasks. MVLLaVA integrates multiple multi-view diffusion models with a
large multimodal model, LLaVA, enabling it to handle a wide range of tasks
efficiently. MVLLaVA represents a versatile and unified platform that adapts to
diverse input types, including a single image, a descriptive caption, or a
specific change in viewing azimuth, guided by language instructions for
viewpoint generation. We carefully craft task-specific instruction templates,
which are subsequently used to fine-tune LLaVA. As a result, MVLLaVA acquires
the capability to generate novel view images based on user instructions,
demonstrating its flexibility across diverse tasks. Experiments are conducted
to validate the effectiveness of MVLLaVA, demonstrating its robust performance
and versatility in tackling diverse novel view synthesis challenges.
MVLLaVA combines multi-view diffusion models with LLaVA to efficiently generate novel views based on user instructions for diverse tasks.
multi-view diffusion models LLaVA novel view synthesis viewpoint generation task-specific instruction templates