After a period of training as a midshipman he was promoted to Lieutenant in 1936 and given command of a minesweeper. He was a friend of Reinhard Heydrich, who also served in the Navy in the 1920s. Heydrich was hired by SS chief Heinrich Himmler in 1931 to serve as the head of the SS counter-intelligence system, and when Rauff resigned from the Navy in 1937, Heydrich took him under his wing.