Of the U.S. ' first twelve presidents , the only two never to own slaves were James Madison , and his son John Quincy Adams ; the first of which famously said that the American Revolution would not be complete until all slaves were freed .