Peabody's early company Riggs, Peabody and Co. was based in Maryland, a slave state. It bought and sold slaves as well as related goods like clothing for slaves. When he lived in London, he received visits from his relative Herbert C. Peabody, a slave owner and cotton planter in Mobile, Alabama.