His father was one of the few Aboriginal people of his generation to receive an advanced education, and during the Japanese occupation era, he was a very successful insurance salesman, allowing the family to live very comfortably. During WWII, when Lee was just five, they moved to Taipei.
李泰祥的父親是少數受過高等教育的原住民，母親是埔里人。父親在日據時代是位優秀的保險業務員，家境相當好，二次大戰期間為了躲避戰亂，舉家遷到台北，那年李泰祥才五歲。