The mechanical instability of the Solid Electrolyte Interphase (SEI) layer in lithium ion (Li-ion) batteries causes significant side reactions resulting in Li-ion consumption and cell impedance rise by forming further SEI layers, which eventually leads to battery capacity fade and power fade. In this paper, the composition-/structure-dependent elasticity of the SEI layer is investigated via Atomic Force Microscopy (AFM) measurements coupled with X-ray Photoelectron Spectroscopy (XPS) analysis, and atomistic calculations. It is observed that the inner layer is stiffer than the outer layer. The measured Young's moduli are mostly in the range of 0.2–4.5 GPa, while some values above 80 GPa are also observed. This wide variation of the observed elastic modulus is elucidated by atomistic calculations with a focus on chemical and structural analysis. The numerical analysis shows the Young's moduli range from 2.4 GPa to 58.1 GPa in the order of the polymeric, organic, and amorphous inorganic components. The crystalline inorganic component (LiF) shows the highest value (135.3 GPa) among the SEI species. This quantitative observation on the elasticity of individual components of the SEI layer must be essential to analyzing the mechanical behavior of the SEI layer and to optimizing and controlling it.