Disclosed are a language representation model system, a pre-training method and apparatus for a language representation model, a natural language processing method and apparatus, and a device and a medium. The language representation model system comprises: a word granularity language representation sub-model with a word as a word segmentation unit, and a phrase granularity language representation sub-model with a phrase as a word segmentation unit, wherein the word granularity language representation sub-model is used for outputting, on the basis of a sentence with a word as a word segmentation unit, a first semantic vector corresponding to semantics expressed by each segmented word in the sentence; and the phrase granularity language representation sub-model is used for outputting, on the basis of the sentence with a phrase as a word segmentation unit, a second semantic vector corresponding to semantics expressed by each segmented phrase in the sentence. According to the technical solution of the embodiments of the present disclosure, provided is a mixed granularity language representation model, which provides a model basis for a downstream natural language processing task, thereby facilitating improvement of the processing precision of the downstream natural language processing task, and improving the migration effect of a language representation model.
LANGUAGE REPRESENTATION MODEL SYSTEM, PRE-TRAINING METHOD AND APPARATUS, DEVICE AND MEDIUM