A series of hard carbon nanofiber-based electrodes derived from electrospun polyacrylonitrile (PAN) nanofibers (PAN-CNFs) have been fabricated by stabilization in air at about 280°C and then carbonization in N2 at heat treatment temperatures (HTT) between 800 and 1500°C. The electrochemical performances of the binder-free, current collector-free carbon nanofiber-based anodes in lithium-ion batteries and sodium-ion batteries are systematically investigated and compared. We demonstrate the presence of similar alkali metal insertion mechanisms in both cases, but just the differences of the layer spacing and pore size available for lithium and sodium ion lead the discharge capacity delivered at sloping region and plateau region to vary from the kinds of alkali elements. Although the anodes in sodium-ion batteries show poorer rate capability than that in lithium-ion batteries, they still achieve a reversible sodium intercalation capacity of 275 mAh g−1 and similar cycling stability due to the conductive 3-D network, weakly ordered turbostratic structure and a large interlayer spacing between graphene sheets. The feature of high capacity and stable cycling performance makes PAN-CNFs to be promising candidates as electrodes in rechargeable sodium-ion batteries and lithium-ion batteries.