 Butler's father, George Butler, was teaching practical agriculture to Gerald Gallagher on the farm at Maiden Hall when Gallagher applied for a position in the British colonial service, where he became the first officer-in-charge of the Phoenix Islands Settlement Scheme, the last colonial expansion of the British Empire.Upon the death of George Butler in 1941, Hubert Butler inherited Maiden Hall and returned to live with his family in the house on the banks of the River Nore until his death in 1991.His wife, Susan Margaret — usually referred to as Peggy — was sister of the theatre director Tyrone Guthrie and the moving force behind foundation of the Kilkenny Art Gallery Society.