 In 1796, Blackwell's great-grandson Jacob Blackwell constructed the Blackwell House, which is the island's oldest landmark, New York City's sixth oldest house, and one of the city's few remaining examples of 18th-century architecture.Through the 19th century, the island housed several hospitals and a prison.In 1828, the City of New York purchased the island for $32,000 (), and four years later, the city erected a penitentiary on the island; the Penitentiary Hospital was built to serve the needs of the prison inmates.