after baker retired from the nfl, he worked as an aide for then-u.s. senator robert morgan. he was also appointed to the state parole commission in 1970, the first black to do so. baker was a member of the "oval table gang", an informal group of community leaders that met in ralph campbell jr.'s home to discuss strategies to desegregate raleigh schools, plan demonstrations, and assist black candidates for public office. he began speaking to youth in community centers and in prison, trying to help them change their lives for good. in 1976 baker worked on the state presidential campaign of jimmy carter.