born in london, ponsonby was the eldest son of john ponsonby, 4th earl of bessborough, and his wife lady maria fane, third daughter of john fane, 10th earl of westmorland. he was a cricketer in his youth and played five first-class matches for marylebone cricket club (mcc) in the 1830s.