The present invention provides a cross-modal processing method, a device thereof, an electronic device, and a computer readable storage medium, and relates to the technical field of natural language processing. According to an embodiment, the cross-modal processing method comprises the steps of: acquiring sample sets including a plurality of corpora and a plurality of images; generating a plurality of training samples according to the sample set, wherein each of the plurality of the training samples is a combination of at least one of the corpus and at least one of the images corresponding thereto; using the plurality of the training samples to train a semantic model, so that the semantic model acquires semantic vectors containing the combinations of the corpus and the images through learning; and using the trained semantic model to perform a cross-modality process between the corpus and the images. By training the combination of the corpus and the corresponding image, the semantic model learns the semantic relation between the corpus and the corresponding image, so that the training effect of the semantic model is increased. In the case of multi-modal processing existing in the existing technology, by separately training each modal, semantic associations between different modals are separated to solve the problem of deterioration of the effectiveness of the trained model.
CROSS-MODALITY PROCESSING METHOD AND APPARATUS AND COMPUTER STORAGE MEDIUM