In this work, the possibility of employing aluminium terephthalic acid metal organic framework (Al-TPA-MOF)-laden composite polymer membranes as electrolyte for all-solid-state lithium-sulfur (Li-S) and lithium-metal (Li-metal) polymer batteries is explored. The prepared composite polymer electrolytes (CPEs) based on a poly(ethylene oxide) (PEO) network with lithium bis(trifluoromethane)sulfonimide (LiTFSI) and Al-TPA-MOF are mechanically robust and thermally stable up to 270 °C, and provide appreciable ionic conductivity in the order of 0.1 mS cm−1 at 60 °C. The enhanced compatibility of CPEs with the lithium metal anode is attributed to the scavenging effect of Al-TPA-MOF. Laboratory scale all-solid-state Li-S and Li-metal polymer cells are assembled, which deliver specific capacities exceeding 800 and 130 mAh g−1, respectively, and a stable performance upon prolonged cycling even at 60 °C, which is superior to earlier reports on similar systems.