 James Butler was the son of Edmund Butler, Earl of Carrick, (126813 September 1321), Justiciar of Ireland and Joan FitzGerald, Countess of Carrick.His paternal grandparents were Theobald le Botiller (1242–1285), (the son of Theobald le Botiller and Margery de Burgh) and Joan FitzJohn (FitzGeffrey) (d. 4 April 1303), who was the daughter of John FitzGeoffrey, Lord of Shere, Justiciar of Ireland and Isabel Bigod.His maternal grandfather was John FitzThomas FitzGerald, 1st Earl of Kildare.