His mother died when he was only nine, and Kao-Tsung grew up dominated by his authoritarian father, T'ai-Tsung, who frequently worried that his son was too soft.
高宗素稱「仁孝」，這與九歲喪母，後來又長期伴隨極其權威的父親﹘﹘太宗有關。太宗將立太子，還屢屢擔心他是否「仁弱」。