A new water-free proton conducting membrane based on an organic/inorganic hybrid is presented. Siloxane oligomer functionalized with imidazole-terminated side chains (CImSS) is employed as an organic phase and proton transporter. As a mechanical framework, organically modified silicate (ORMORSIL) is prepared by a sol–gel synthesis of tetraethoxysilane (TEOS) and methyltriethoxysilane (MTEOS). Polarity difference between the CImSS and the sol affects the phase separation of the resulting hybrids. Based on the phase separation, morphology of the hybrids and specific interactions between the CImSS and the silicates are investigated and discussed with the temperature- and the pressure-dependent proton conductivity. Under the absence of water, high proton conductivity of around 10−4 Scm−1 at 160°C has been obtained for the CImSS/(MTEOS-based silicate) hybrid. This high proton conductivity is believed mainly due to well-developed percolation of the proton-conducting phase (CImSS) and weaker interaction between the CImSS and the MTEOS-based silicate.