  With the advancement of telemedicine, both researchers and medical
practitioners are working hand-in-hand to develop various techniques to
automate various medical operations, such as diagnosis report generation. In
this paper, we first present a multi-modal clinical conversation summary
generation task that takes a clinician-patient interaction (both textual and
visual information) and generates a succinct synopsis of the conversation. We
propose a knowledge-infused, multi-modal, multi-tasking medical domain
identification and clinical conversation summary generation
(MM-CliConSummation) framework. It leverages an adapter to infuse knowledge and
visual features and unify the fused feature vector using a gated mechanism.
Furthermore, we developed a multi-modal, multi-intent clinical conversation
summarization corpus annotated with intent, symptom, and summary. The extensive
set of experiments, both quantitatively and qualitatively, led to the following
findings: (a) critical significance of visuals, (b) more precise and medical
entity preserving summary with additional knowledge infusion, and (c) a
correlation between medical department identification and clinical synopsis
generation. Furthermore, the dataset and source code are available at
https://github.com/NLP-RL/MM-CliConSummation.

Computation and Language;Artificial Intelligence