Electrochemical capacitors (ECs) are slow devices with charging and discharging rates limited below 1Hz. They run at direct current and function as power source, but cannot afford the role of a conventional capacitor for current ripple filtering or pulse energy harvesting. Recently, developing ultrafast ECs that work at hundreds to kilohertz (kHz) frequency scope have attracted great interests, with the aim to replace the traditional aluminum electrolytic capacitors (AECs) that have bulky size and large equivalent series resistance. Compact kHz ECs would produce huge impacts on power design, power electronics and environmental pulse energy harvesting. Towards such a goal, the electrode material and its nanostructure are the keys to boost the response frequency of an EC from below 1Hz to above 1kHz. In this Review, we summarize guidelines on the electrode nanostructure design for kHz response, discuss the various carbonaceous materials and other highly conductive materials based electrode structures for kHz ECs. The configurations of higher voltage kHz ECs, and their dimension advantage over AECs are critically evaluated, followed with the outlook on the further study and development in this promising area.