baker's wife, matilda died in 1855, and baker remarried in 1858 to charlotte frances chute. the couple had two daughters and one son. he was in evansville when the american civil war began, and was part of a large crowd that gathered to discuss the event. he took the podium and delivered a speech calling on the crowd to take an oath of allegiance to the union, which was administered by his brother, the mayor. he then called on all the able-bodied men to follow him to war. baker and his brother began actively recruiting a full regiment of men to serve in the war, and he was promoted to serve as colonel of the 1st regiment indiana cavalry.