This paper deals with an optimal battery energy storage capacity for the smart grid operation. Distributed renewable generator and conventional thermal generator are considered as the power generation sources for the smart grid. Usually, a battery energy storage system (BESS) is used to satisfy the transmission constraints but installation cost of battery energy storage is very high. Sometimes, it is not possible to install a large capacity of the BESS. On the other hand, the competition of the electricity market has been increased due to the deregulation and liberalization of the power market. Therefore, the power companies are required to reduce the generation cost in order to maximize the profit. In this paper, a thermal units commitment program considers the demand response system to satisfy the transmission constraints. The BESS capacity can be reduced by the demand response system. The electric vehicle (EV) and heat pump (HP) in the smart house are considered as the controllable loads of the demand side. The effectiveness of the proposed method is validated by extensive simulation results which ensure the reduction of BESS capacity and power generation cost, and satisfy the transmission constraints.