Some of the swarm is ignited by the gaze, some are quickly extinguished again by the wind being forced past them by the wings of their cohorts. In all the gaze doe have an effect, but  not a large one. They do, however, change targets. Cassia nw being the closest and most viable option.