Hard carbon is a promising anode candidate for lithium/sodium ion batteries due to the key features of low operation potential and low cost, but its practical utilization is hindered by a challenging issue of poor initial Coulombic efficiency (ICE), which has not been understood well and resolved properly. Herein, we report a new in-situ engineering approach to deliberately tune the residual oxygen atoms/defects of hard carbon by controlling the atmosphere of pyrolysis synthesis process and reveal important correlations between the ICE and residual oxygen atoms/defects. When used as an anode in sodium ion battery, the hard carbon electrode with reduced residual oxygen atoms and defects can achieve a high average ICE above 85%, which is considerably higher than the commonly observed ~70% and ~30% ICE values for pristine and acid treated hard carbon electrodes. Encouragingly, a high reversible capacity of 310 mAh g−1 with good cycling stability (93% after 100 cycles) is demonstrated at a current density of 20 mA g−1. The density functional theory (DFT) calculation and experimental results reveal that the trap effects of residual oxygen atoms and defects on Na+ are the key factors that impact the ICE of the hard carbon electrode. When the hard carbon is coupled with Na3V2(PO4)2F3 cathode to form a sodium ion full cell, the battery delivers an impressively high energy density of 239 Wh/kg (based on the active mass of anode and cathode without additives), which is among the best performing sodium ion full cells. This work not only provides an effective approach to engineer the heteroatoms and defects in carbon-based materials but also sheds light on the design principle of practical hard carbon electrodes.