There were no children of the marriage, and the baronetcy became extinct in 1764, when he died at Hill Street, London. Per the third baronet's will, if Jacob Downing had no son, George Downing's estate was supposed to be used by the University of Cambridge to establish a college in his name. However, Jacob instead left the estates to his widow, which led to 40 years of litigation between Lady Margaret and the university. During this time, the estates were neglected and diminished considerably in value. Lady Margaret demolished Gamlingay Park house in 1776, "selling it brick by brick at auction for a fraction of its value." After Downing died in 1764, in 1768, she married Admiral Sir George Bowyer, 5th Baronet, who in 1794 was created a baronet in his own right, several years after her death (18 September 1778). The legal fight for the third baronet's estates continued after her death, as she left them to her widower and a nephew. The legal battle continued until March 1800, when the university and Lady Margaret's heirs reached an agreement. However, during this time, the value of the estate was massively reduced to a fraction of what it had been. Only 10 per cent was recovered from the £100,000 () owed to the estate in rents. Even though the charter for the new college was finally granted in September 1800, it was not until 1806 that the first building went up, and a further 16 years passed before the first Downing College students began their studies—56 years after the death of Sir Jacob Downing.