Their son died young of small-pox, having caught the disease from his father, who had been volunteering on a ship that had that disease on board "and had carried home the infection in his clothes".[2] One of their daughters, Margaret, (1766 - Newtown, Ireland, April 1839) married Robert Smith (1754 - Chatham, Kent, 2 July 1813), a Major in the Royal Marines. Robert and Margaret Smith's daughter, Catherine Rebecca Smith (1796 - 10 October 1856), married Sir Thomas Osborne, 9th Baronet 6 April 1816 at St. Margaret's Church, Rochester, Kent. James and Rebecca Ramsay's youngest daughter, Jane, married her cousin Aretas Akers.