He was reared and educated in Pollock in Grant Parish. Holt's older brother, Jack Holt (1924-2013), was a B-24 pilot who flew in the last bombing missions over Germany in World War II. A lawyer in practice in Alexandria, Jack Holt was the first municipal judge in Pineville, a position which he held for two decades. A graduate of Louisiana College in Pineville and the Louisiana State University Law Center in Baton Rouge, Jack Holt was an active deacon and leader in the First Baptist Church in Pineville. Jack's son and Ben Holt's nephew, Robert Earl "Bobby" Holt, Sr. (1948-2017), was a cattle rancher and then real estate developer in Rapides Parish who also lived for a time in Alaska and Colorado. Ben Holt's widow was the former Joyce Ivoyne Lemmons (September 30, 1926 – January 24, 2016), a Pollock native who resided in Tioga in Rapides Parish. The Holts are interred at Forest Lawn Memorial Park in Ball, north of Pineville; his brother, at Greenwood Memorial Park in Pineville.