on 1 august 1846, phipps was appointed an equerry in ordinary to queen victoria, succeeding charles george james arbuthnot, and hereafter was much in the royal household. he was appointed private secretary to prince albert on 1 january 1847, and went on half-pay from the army later that month as a lieutenant-colonel. on 10 october 1849 phipps replaced george edward anson as keeper of the privy purse, treasurer and cofferer to the prince of wales, and treasurer to prince albert, resigning his post as private secretary.