he was promoted colonel (unattached) in 1851. he received the cb on 9 september 1853 and retired from the army the following year. in 1858, he received the kcb. phipps was much in the confidence of victoria and albert, and aided charles grey in his duties as private secretary to the sovereign after the death of albert in 1861. he was appointed receiver-general of the duchy of cornwall on 26 may 1862, and was appointed to the council of the prince of wales on 27 january 1863. on 8 february 1864, he received the additional offices of secretary, chamberlain and receiver-general and keeper of the signet to the prince of wales as prince and steward of scotland. phipps died of bronchitis at st james's palace on 24 february 1866, and was buried in st george's chapel at windsor castle on 2 march 1866.