The present invention relates to a technique for recognizing a sleeping state of a driver. According to an embodiment of the present invention, a method for recognizing a state of a driver based on multimodal learning comprises the steps of: (a) receiving bio-information of a driver and face image information of the driver; (b) extracting characteristics on the received face image information; (c) forming a multi-modal network based on the extracted characteristics on the face image information and the bio-information; (d) arranging an output value of the formed multi-modal network in accordance with an order of a predetermined mode and an order of a predetermined time and mixing the output value to generate representative information; and (e) classifying a state of the driver based on the generated representative information.
THE SYSTEM AND THE METHOD FOR RECOGNIZING DRIVER'S CONDITION OF MULTIMODAL LEARNING