MAP-Elites with Transverse Assessment for Multimodal Problems in Creative Domains
<|reference_start|>MAP-Elites with Transverse Assessment for Multimodal Problems in Creative Domains: The recent advances in language-based generative models have paved the way for the orchestration of multiple generators of different artefact types (text, image, audio, etc.) into one system. Presently, many open-source pre-trained models combine text with other modalities, thus enabling shared vector embeddings to be compared across different generators. Within this context we propose a novel approach to handle multimodal creative tasks using Quality Diversity evolution. Our contribution is a variation of the MAP-Elites algorithm, MAP-Elites with Transverse Assessment (MEliTA), which is tailored for multimodal creative tasks and leverages deep learned models that assess coherence across modalities. MEliTA decouples the artefacts' modalities and promotes cross-pollination between elites. As a test bed for this algorithm, we generate text descriptions and cover images for a hypothetical video game and assign each artefact a unique modality-specific behavioural characteristic. Results indicate that MEliTA can improve text-to-image mappings within the solution space, compared to a baseline MAP-Elites algorithm that strictly treats each image-text pair as one solution. Our approach represents a significant step forward in multimodal bottom-up orchestration and lays the groundwork for more complex systems coordinating multimodal creative agents in the future.<|reference_end|>
@article{zammit2024map-elites,
    title={MAP-Elites with Transverse Assessment for Multimodal Problems in
  Creative Domains},
    author={Marvin Zammit, Antonios Liapis, Georgios N. Yannakakis},
    journal={arXiv preprint arXiv:2403.07182},
    year={2024},
    doi={10.1007/978-3-031-56992-0_26},
    archivePrefix={arXiv},
    eprint={2403.07182},
    primaryClass={cs.NE}
}