However, when Ben-Gurion led a breakaway from Mapai to form Rafi shortly before the 1965 elections, Almogi followed, losing his cabinet position in the process. Elected back to the Knesset on Rafi's list, Almogi became Minister of Labour in July 1968 when together with Mapam, Rafi merged into Eshkol's Alignment (a merger of Mapai and Ahdut HaAvoda). He retained his position after the 1969 elections, but was not included in Golda Meir's cabinet after the 1973 elections.