A selenium/carbon (Se/C) composite has been suggested as an excellent substitute cathode material due to its high theoretical volumetric capacity. However, it exhibits severe capacity decay, which is caused by the dissolution of active materials as well as shuttle effect. One effective strategy to alleviate this dissolution is to confine the Se within porous carbon frameworks. Here, we report the effects of carbon structure and dopant on the electrochemical performances of Se infiltrated carbon composites with three types carbons; (i) nitrogen-doped ordered mesoporous carbon (NOMC), (ii) sulfur-doped ordered mesoporous carbon (SOMC), and (iii) macroporous spherical carbon, which have different porosities and dopants. The Se/C cathodes are prepared by meting selenium-sulfur compound and then evaporating sulfur from the compound, providing nanoscale space for efficient electrolyte access to the reaction site. Among the three composite cathodes, the Se/NOMC exhibits the highest rate-capability and cyclic stability due to an abundance of cylindrical mesopores (4–5 nm pore diameter) that can provide enough electrolyte diffusion path and effectively impede the dissolution of selenium. As a result, the Se/NOMC has a high volumetric capacity of 1605 mAh cm−3 and excellent capacity retention of 100% during 500 cycles at 0.648 A h cm−3 (C-rate of C/2.5).