cramond owned the mansion for a short period, as he was forced to sell it in 1806 due to bankruptcy. another merchant, samuel mifflin, bought the sedgeley mansion, but he then sold the house in 1812 to james cowles fisher, a shipping merchant, who used it as a summer residence. for some period of time, fisher left the house vacant or leased it to a tenant. fisher sold it in 1836 to isaac s. loyd, a real estate speculator, who decided to subdivide the house and did not take good care of it, letting it fall into disrepair.