The idea of Hybrid Energy Storage System (HESS) lies on the fact that heterogeneous Energy Storage System (ESS) technologies have complementary characteristics in terms of power and energy density, life cycle, response rate, and so on. In other words, high power ESS devices possess fast response rate while in the contrary, high energy ESS devices possess slow response rate. Therefore, it may be beneficial to hybridize ESS technologies in the way that synergize functional advantages of two heterogeneous existing ESS technologies As a consequence, this hybridization provides excellent characteristics not offered by a single ESS unit. This new technology has been proposed and investigated by several researchers in the literature particularly in the fields of renewable energy and electrified transport sector. In this context and according to an extensive literature survey, this paper is to review the concept of the HESS, hybridization principles and proposed topologies, power electronics interface architectures, control and energy management strategies, and application arenas.