They shortly after moved into a row house, 172 South Street in Newburgh, with more accessible conveniences. In 1877, the family moved to 277 Grand Street, where Headley died of paralysis  at age 84 in 1897. Harriet Pardee Musgrave took ownership of the house in 1870, purchased by her father. Harriet and her husband, Stephen B. Musgrave, supervised subsequent renovations and had a child that same year. George E. Harney, architect of the renovations, did business in Newburgh, replicating the original tastes of Downing in his work. In 1873, he contributed revisions to the new edition of Downing's "Cottage Residences". One of Harney's most substantial additions, the porte-cochère, created a small solarium above the front entrance, jutting from the second floor. He made many interior changes on this floor, removing a dressing room and attic stairs to extend the central hallway. It joined the main house with the principal "kitchen wing", allowing for more servant bed chambers and general space on the second floor. Ownership shifted to John A. Corcoran in 1904, who commissioned New York architecture firm Rogers & Haniman to construct the current back wing. The firm's work included demolishing the wall between the original dining room and library to create a larger space. Following the Corcorans, a carpet distributor named O'Connor came to own the house, furnishing it according to Newport tastes. For an unknown period, the house sat semi-abandoned, similar to the condition of other nearby mansions during this time. In 1942, Andrew Lucas purchased the house for his family, priced inexpensively. They remained there until 1985, but initially found living conditions difficult due to its small number of bedrooms and bathrooms. Artist Kenny Scharf owned the house during the late 1980s and early 1990s, using it as a studio and weekend residence. It was not until the late 1990s that Scott Henry, a subsequent owner, discovered the house had been a Downing & Vaux design. The latter 20th century brought a period of historic home renovations to Newburgh, sparking a renewed interest in the abundance of local work of Downing and Vaux. When first tasked with designing the house, Downing & Vaux used a similar approach echoed by their partner Frederick Clarke Withers in his 1858 design for the Calvary Presbyterian Church in Newburgh. The building would harmonize with its natural landscape, incorporating steep roof-lines and colors akin to the mountain scenery.