From Multimedia Logs to Personal Chronicles
Multimodal data streams are essential for analyzing personal life, environmental conditions, and social situations. Since these data streams have different granularities and semantics, the semantic gap becomes even more formidable. To make sense of all the multimodal correlated streams we must first synchronize them in the context of the application, and then analyze them to extract meaningful information. In this paper, we consider the problem of modeling an individual by using daily activity in order to understand their health and behavior. The first step is to correlate diverse data streams with atomic interval, and segment a person's day into her daily activities. We collect the diverse data streams from the person's smartphone to classify every atomic interval into a daily activity. Next, we use an interval growing technique for determining daily activity intervals and their attributes. Then, these daily activity intervals are labeled as the daily activities by using Bagging Formal Concept Analysis (BFCA) Finally, we build a personal chronicle, which is a person's time ordered list of daily activities. This personal chronicle can then be used to model the person using learning techniques applied to daily activities in the chronicle and relating them to biomedical or behavioral signals. We present the results for this daily activity segmentation and recognition by using lifelogs of 23 participants.
ACM Multimedia