It was alleged that a backdoor deal was made between the British government and Norman-Walker that he would take up the post of Colonial Secretary only when the British government promised to make him the successor to Sir David Trench, the incumbent Governor of Hong Kong. However, this information was never verified by Norman-Walker. The rumour died out in 1971 when Sir Murray MacLehose, a career diplomat, was announced by the British government as the Governor-designate. Nevertheless, another rumour followed when the new Governor MacLehose, who brought many new policies to Hong Kong, was said to be 'diplomatic' in style and did not get on well with some of his colleagues. Some of the more old-fashioned colonial officials, such as (later Sir) Donald Luddington were alleged to have found it difficult to get on with the new Governor and were soon posted out of Hong Kong. Norman-Walker was one of these "old-fashioned colonial officials" and news emerged in the beginning of 1972 that he was going to retire from the government at the turn of 1973. This was not confirmed until July 1972 when the UK government announced that the term of Norman-Walker as Colonial Secretary would be extended to the autumn of 1973. In May 1973, it was further announced that Norman-Walker would become the Lieutenant Governor of the Isle of Man. As a kind of benefit, overseas civil servants in Hong Kong were entitled to a free exclusive sea passage when they retired. Norman-Walker was eligible and chose to enjoy the benefit when he left Hong Kong on 30 September 1973. He was supposed to get on board an ocean liner like his counterparts, but it turned out that there was no ocean liner calling at Hong Kong at the time of his departure. As a result, he was forced to enjoy the entitlement on a cargo ship. On his departure, in the presence of the members of the Executive and Legislative Councils and senior government officials, Norman-Walker boarded "Lady Maurine" from and travelled across the Victoria Harbour to the Kowloon side. From there he travelled to Kwai Chung Container Port and boarded the cargo ship, which sailed for Britain. Norman-Walker was supposed to take up the new post as the Lieutenant-Governor of the Isle of Man after he had returned to England. But a problem arose when it became known that his wife would not accompany him. The reaction in the Isle of Man was negative, and the Tynwald insisted that Lady Norman-Walker should accompany her husband to assist the Lieutenant Governor in carrying out his social duties.