During the reign of Sultan Jaafar Mua'azzam Shah (died in 1865), and also during the reign of Sultan Ali Al-Mukammal Inayat Shah, he was not elected to be the Raja Bendahara although he was eligible and entitled. This is because many rulers and officials of Perak do not like him. When Raja Muda Raja Abdullah was elected to be the 26th Sultan of Perak, he was appointed as the Raja Muda to strengthen his position, and at that time he was residing in Senggang.