Baker returned to England with the squadron of which so many of the ships were lost in the Scilly naval disaster on 22 October 1707, resulting in the deaths of nearly 2,000 men and Admiral Sir Cloudesley Shovell, and, having arrived at the Nore, was ordered to refit and keep the men on board with a view to their being sent to other ships. Baker remonstrated; he thought their case was hard, and that they ought to be allowed to go home. 'Most of them,' he wrote, on 3 November, 'have been with me in this ship for almost six years, and many have followed me from ship to ship for several years before.'