The enhancement of the electrochemical properties of poly(ethylene) oxide (PEO)–lithium salt composite electrolytes has been reported for many years, but the mechanism is still not fully understood. Therefore, this study focuses on the reason why the ionic conductivity is increased when inorganic fillers are added to the composite electrolyte. A maximum ionic conductivity is obtained by adding a nano-sized inorganic filler and by post-heat treatment. The conductivity reaches 2.3×10−5 Scm−1 at room temperature. The nano-sized SiO2 particles appear to play two roles, namely, the simultaneous arrangement of dipoles in the composite electrolytes and interruption of the crystallization of PEO. The PEO and the inorganic filler generally tend to orient their dipoles to minimize their total energy, so that PEO chains are forced to rotate and orient in a direction parallel to the applied electric field. Thermal treatment increases the amorphous phase of PEO such that subsequent segmental motion of the PEO chain follows, while nano-sized SiO2 prevents the recrystallization of PEO after heat treatment.