A new strategy is proposed for the preparation of Fe3O4 encapsulated in N-doped carbon (denoted as Fe3O4@NC) through a self-assembly of the colloidal FeOOH with polyaniline (PANI) and then a subsequent pyrolysis. Benefiting from the well-designed heterostructure, the resultant Fe3O4@NC shows an ultra-high gravimetric capacitance of 313 F g−1 at 1 A g−1 and still delivers 193 F g−1 at a high current density of 15 A g−1, demonstrating an excellent rate performance. In addition, it has an outstanding volumetric capacitance of 20.25 F cm−3 at 1 A g−1. The asymmetric supercapacitors (ASCs) are constructed by using Fe3O4@NC as anode and layered double hydroxides (LDHs) as cathode, respectively. Due to the enlarged voltage window and matched capacity and kinetics, the obtained CoMn-LDH//Fe3O4@NC ASC device shows a remarkable electrochemical performance with a large energy density up to 35 Wh·kg−1 at a power density of 900 W kg−1 and a high energy density of 24 Wh·kg−1 at a power density of 13.5 kW kg−1 as well as an excellent rate capability and a good cycling stability. The present work may provide an insight into the design of novel anode materials for high-performance ASCs.