Multimodal generative semantic communication based on latent diffusion model
<|reference_start|>Multimodal generative semantic communication based on latent diffusion model: In emergencies, the ability to quickly and accurately gather environmental data and command information, and to make timely decisions, is particularly critical. Traditional semantic communication frameworks, primarily based on a single modality, are susceptible to complex environments and lighting conditions, thereby limiting decision accuracy. To this end, this paper introduces a multimodal generative semantic communication framework named mm-GESCO. The framework ingests streams of visible and infrared modal image data, generates fused semantic segmentation maps, and transmits them using a combination of one-hot encoding and zlib compression techniques to enhance data transmission efficiency. At the receiving end, the framework can reconstruct the original multimodal images based on the semantic maps. Additionally, a latent diffusion model based on contrastive learning is designed to align different modal data within the latent space, allowing mm-GESCO to reconstruct latent features of any modality presented at the input. Experimental results demonstrate that mm-GESCO achieves a compression ratio of up to 200 times, surpassing the performance of existing semantic communication frameworks and exhibiting excellent performance in downstream tasks such as object classification and detection.<|reference_end|>
@article{fu2024multimodal,
    title={Multimodal generative semantic communication based on latent diffusion
  model},
    author={Weiqi Fu, Lianming Xu, Xin Wu, Haoyang Wei, Li Wang},
    journal={arXiv preprint arXiv:2408.05455},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.05455},
    primaryClass={cs.CV cs.NI}
}