Triethylene glycol diacetate-2-propenoic acid butyl ester (TEGDA-BA) based composite polymer electrolytes (CPE) are fabricated by incorporating alumina (Al2O3) nanoparticles (average particle size 10–20 nm) as inorganic filler via in situ polymerization. Effects of Al2O3 concentration on ionic conductivities, Li+ transfer numbers and charge/discharge properties are studied in details. Due to the uniformly dispersed Al2O3 nanoparticles, significant improvements in the mechanical flexibility and bendability are presented in the resulting polymer electrolytes. The CPE with 5 wt% Al2O3 nanoparticles exhibits the highest ionic conductivity up to 6.02 × 10−3 S cm−1 at 25 °C and the highest Li+ transference number (0.675), coupled with the most stable electrochemical window (>4.5 V vs. Li/Li+). With the presence of Al2O3, the growth of interface resistance is retarded, which increases the interface stability. The Li|CPE|Li4Ti5O12 and Li|CPE|LiFePO4 cells demonstrate remarkably stable charge/discharge performance and excellent capacity retention during cycling test. The results suggest that the CPE holds great application potential in flexible lithium ion batteries.