col. james madison sr. was the son of planter and politician ambrose madison and his wife frances taylor, and was born in 1723 in orange county, virginia. when he was nine, his family moved to their new plantation of mount pleasant in 1732. his father had hired slaves and an overseer to clear it, work that had been going on for five years to establish cultivation. that summer (1732) his father died at age 36 in august after a short illness.