They had a son, Isaac Osgood Jr., whose daughter Charlotte married Moses T. Stevens, and a daughter, Sally Pickman Osgood, who married Bailey Loring and had a son, George B. Loring. Thirdly, Isaac married Mary Pickman in 1802, the cousin of his first two wives. His first two wives were cousins of, and his third wife a sister of, Benjamin Pickman. He died in New York City in 1813. His birthplace in North Andover, Massachusetts is located on a street named for his family, and is listed on the National Register of Historic Places, as is his New York residence. Osgood's portrait has been housed in the President's Room of the U.S. Capitol since Lincoln's presidency.