The aim of this work is to present an optimization methodology for the installation capacity of a stand-alone hybrid generation system, taking into consideration the cost and reliability. Firstly, on the basis of derived steady state models of a wind generator (WG), a photovoltaic array (PV), a battery and an inverter, the hybrid generation system is modeled for the purpose of capacity optimization. Secondly, the power system is analyzed for determining both the system structure and the operation control strategy. Thirdly, according to hourly weather database of wind speed, temperature and solar irradiation, annual power generation capacity is estimated for the system match design in order that an annual power load demand can be met. The capacity determination of a hybrid generation system becomes complicated as a result of the uncertainty in the renewable energy together with load demand and the nonlinearity of system components. Aimed at the power system reliability and the cost minimization, the capacity of a hybrid generation system is optimized by application of an adaptive genetic algorithm (AGA) to individual power generation units. A total cost investigation is made under various conditions, such as wind generator power curves, battery discharge depth and the loss of load probability (LOLP). At the end of this work, the capacity of a hybrid generation system is optimized at two installation sites, namely the offshore Orchid Island and Wuchi in Taiwan. The optimization scheme is validated to optimize power capacities of a photovoltaic array, a battery and a wind turbine generator with a relative computational simplicity.