Layered Ni-rich transition metal oxides exert great potential as high-capacity cathode materials for lithium-ion batteries. However, structural degradation during lithiation/delithiation hinders the cathode materials for commercial utilization. Herein, PO4 3− polyanion and Mn4+ cation are co-doped into Ni-rich LiNi0.80Co0.15Al0.05O2 cathode to improve the structural stability and electrochemical performance. The effects of PO4 3− and Mn4+ co-existence on phase, crystal structure, element valence state, electrochemical performance and phase transition during lithiation/delithihation are systematically investigated. The results show that moderate content of PO4 3− and Mn4+ co-doping can enlarge the channel for Li+ lithiation/delithiation, lower the cationic mixing, and suppress the structural degradation during cycling. With the stabilization role of Mn4+ and PO4 3−, the material with moderate amount of dopants shows remarkable enhanced electrochemical performance, especially at harsh condition. In the cell potential of 2.7–4.3 V, the 3% PO4 3− and Mn4+ co-doped cathode shows a reversible discharge capacity of 204 mAh g−1 at 0.1C, outstanding cycling stability with a capacity of 174 mAh g−1 and capacity retention of 85.5% at 1C after 100 cycles, especially, a superior discharge capacity of 157.8 mAh g−1 at 5C. Even at elevated temperature of 55 °C, the cathode retains 80.9% of initial capacity (195 mAh g−1) at 1C after 100 cycles.