The secondary lithium battery using lithium metal as a negative electrode has attracted more attention due to its extremely high theoretical specific energy. During the charge and discharge cycle, lithium ions are reduced and nonuniformly deposit on the surface of the lithium electrode, which leads to the formation and growth of lithium dendrites. The growth of lithium dendrites can pierce through the separator and causes internal short circuit of the battery as well as battery catastrophes like thermal runaway. The lithium dendrite growth process is complicated and can be affected by various factors. In this paper, a phase field model is developed to simulate the growth process of lithium dendrites, and the effects of anisotropic strength, applied voltage and microstructure of solid electrolyte interphase (SEI) on the growth of lithium dendrites are considered and investigated. The geometric model of solid electrolyte interphase (SEI) is established by numerical reconstruction method. Then, the growth process of lithium dendrites in solid electrolyte interphase (SEI) is investigated, which provides insight on revealing the growth mechanism of lithium dendrites.