Over the preceding years Braose had promised Gower to a number of persons, including Humphrey de Bohun, the Earl of Hereford, Hugh Despenser the Younger, and Roger Mortimer of Wigmore. Mowbray then in late 1319 took custody of Gower to protect his rights. Despenser persuaded the king in 1320 to take Gower into royal hands in October, and was appointed keeper of the honour in November. The other lords in the Welsh Marches resented this seizure, feeling that the king's excuse for it was not applicable. The seizure was one of the precipitating causes of the baronial rebellion that led to the exile of the Despensers in 1321. In 1322 Gower was given to the younger Despenser again, who then traded it for the honours of Usk and Caerleon. Braose was then induced to sue the new holder of Gower for the return of the barony in April 1324, which action succeeded in June 1324. Braose then promptly gave Gower to the elder Despenser, returning the property to the Despenser family once more. The lordship of Gower eventually ended up in the hands of the Beauchamp family, but it was not until the 1350s that the issue was decided. The name of Braose's first wife was Agnes, but her family is not known. His second wife was Elizabeth, the daughter and heiress of Raymund de Sully. He had two daughters with his first wife, but no children with his second wife, who outlived him. It appears that there was a son named William, who was the subject of a military summons from King Edward in 1311, but nothing further is mentioned of him after 1315. In 1316 a settlement of William the father's estates made no mention of this son making it likely that the son died before this date. Braose died not long before 1 May 1326 and his heirs were his daughter Aline and his grandson John de Bohun. Aline, the elder daughter, married John de Mowbray and Richard de Peschale. The second daughter, Joan, married James de Bohun and Richard Foliot, son of Jordan Foliot. Mowbray received the lands of Gower and Bramber before Braose's death. Braose was known as a man often in debt and as being unable to manage his cash flow well. Thomas Walsingham stated in his chronicle that Braose was "very rich by descent but a dissipater of the property left to him".