 He entered a cloth house early in life and subsequently became a merchant tailor with large establishments in New York City, Boston, and Philadelphia.He retired from active business pursuits in 1875.Castor built an eighteen bedroom mansion, "Stoneyhurst", on Solly Avenue overlooking the Pennypack Creek in Holmesburg.