Sir Philip Le Despencer, Knt., of Goxhill, Lincolnshire was the son of Hugh Le Despencer, 1st Earl of Winchester and his wife, Lady Isabella Beauchamp, daughter of William de Beauchamp, 9th Earl of Warwick and Maud FitzJohn. He was born ca. 1290 in Stoke, Gloucester, England. He married Margaret de Goushill, daughter of Ralph De Gousille and his wife Hawise Fitzwarine. Philip was brother to Hugh Despenser, the Younger, a favorite of King Edward II.