Ramsay left St Kitts in 1777, exhausted by the continuing conflict with influential planters and businessmen. He returned to Britain and briefly lived with Sir Charles Middleton at Teston, Kent where Lady Middleton joined the cause of the campaign against the slave trade.