Mobile devices can perceive greater details of user states with the increasing integration of mobile sensors into a pervasive computing framework, yet they consume large amounts of batteries and computational resources. This paper proposes a semantic management method which efficiently integrates multiple contexts into the mobile system by analyzing the semantic hierarchy and temporal relations. The proposed method semantically decides the recognition order of the contexts and identifies each context using a corresponding dynamic Bayesian network (DBN). To sort out the contexts, we designed a semantic network using a knowledge-driven approach, whereas DBNs are constructed with a data-driven approach. The proposed method was validated on a pervasive computing framework, which included multiple mobile sensors (such as motion sensors, data-gloves, and bio-signal sensors). Experimental results showed that the semantic management of multiple contexts dramatically reduced the recognition cost.