Xiahou Xuan subsequently grew anxious and advised Cao Shuang to lead his troops back, which he eventually resorted to by June or July of the same year, further incurring losses during his retreat. In September 245, Cao Shuang wanted to make changes to the structure of the military so that he could put his brothers Cao Xi () and Cao Xun () in command of troops. Sima Yi opposed these changes, but Cao Shuang ignored him and went ahead. In January 246, the Wei emperor, Cao Fang, granted Sima Yi the privilege of riding to the imperial court in a type of horse-drawn carriage traditionally reserved for emperors. In February 246, when Eastern Wu forces attacked Zhazhong (柤中; west of present-day Nanzhang County, Hubei), over 10,000 households living in the area fled to the north across the Mian River (). When news of the Wu invasion reached the Wei imperial court, Sima Yi argued that they should let the civilians remain on the north side of the Mian River since the south side was near enemy territory and hence too dangerous for them. However, Cao Shuang said: "It isn't in our long-term interests to allow the civilians to remain here and give up trying to secure the south of the Mian River." Sima Yi disagreed: "If the enemy sends 20,000 troops to cut off passage across the Mian River, sends another 30,000 troops to fight our forces at the south of the Mian River, and sends another 10,000 troops to occupy Zhazhong, what can we do to save those civilians?" Cao Shuang refused and ordered the refugees to return to the south of the Mian River. As Sima Yi foresaw, the Wu forces occupied Zhazhong, captured the civilians, and relocated them to Wu territory. Around late May or early June 247, Cao Shuang wanted to further dominate the Wei government, so he used a series of political manoeuvres to consolidate and concentrate power in the hands of himself and his clique. He heeded the advice of his close aides He Yan, Deng Yang, and Ding Mi (), and relocated Empress Dowager Guo (Cao Rui's widow) to Yongning Palace () so that she could not interfere in politics. Sima Yi was unable to stop this, among other contrivances, pushing the relationship between him and Cao Shuang to a breaking point. Cao Shuang himself became increasingly distrustful and wary of Sima Yi.