They had two sons and three daughters together. In the 1860s Bartley built a house on the highest point in Hamilton which had unrivalled views of Brisbane and the Brisbane River. However, as the house was surrounded by dense bushland, his wife refused to live in it fearing attacks by Aboriginals leading to its nickname of Bartley's Folly, a name which persisted long after Bartley had sold the property. In 1879, Bartley objected to the continued use of the nickname, as he felt the "folly" implied a financial misjudgement on his part, whereas, taking all costs into account, he had actually made a profit of on the house. In 1892 in Brisbane, he published the book "Opals and Agates" which was based on his diaries and provided chatty sketches of people and events. On 10 March 1894 Bartley, who had previously resided with his son for two-and-a-half years, left Brisbane for Sydney to secure subscribers to a second book to be entitled "Pioneers of Queensland," which he intended publishing shortly. However in early July 1894 in Sydney, he suffered from an internal pain which he blamed on bad baking powder in a scone that he had eaten and it was suggested this may have contributed to his death from heart failure on 10 July 1894 at his residence Richmond House, 1 Richmond Terrace, Domain, Sydney. He was survived by his wife, one of his two sons, and by three daughters. He was buried on 11 July 1894 in Waverley Cemetery. His book "Pioneers of Queensland" was published posthumously in 1896 under the title "Australian Pioneers and Reminiscences" after being edited by John James Knight. Bartley's Hill () in Ascot, Brisbane is named after him. Bartley had owned the hilltop. Bartley's Folly was demolished in the 1920s to create a water reservoir on the top of the hill. Richmond Terrace was demolished in 1935 to allow the State Library of New South Wales to expand.