Why doesn't the vapor pressure of water depend on the humidity? Correct me if I'm wrong, I'd like to think of vapor pressure of water as the number of water molecules escaping from the water surface per unit area. Greater the leaving number, greater the vapor pressure. If this is right, can I conclude that the rate of evaporation is directly proportional to the vapor pressure?

My textbook says that the vapor pressure of water depends only on the temperature of the water. It doesn't depend on external pressure or humidity. But we know that the rate of evaporation decreases with increase in humidity.(Wet clothes dry faster in less humid air.) Doesn't this imply a decrease in vapor pressure with increase in humidity?
The partial pressure of water vapor is equal to the equilibrium vapor pressure at the temperature of the liquid only at the surface of the water.  Above the surface of the water, the partial pressure of the water vapor decreases with distance, and far from the surface, approaches the partial pressure consistent with the relative humidity of the room air.  So there is a concentration gradient of water vapor in the air above the water surface.  Based on Fick's law of diffusion, this provides a driving force for diffusion of water away from the surface.  We call this diffusive flux of water vapor evaporation.  It is driven by the difference between the equilibrium vapor pressure of water at the surface and the partial pressure of water vapor in the bulk of the room air.