Leo is able to recall that the goblins that threatened Sunglade were never really a problem; the villagers could more or less defend themselves from that and, worst case, hide behind the palisades of the town.  It was when a clutch of lizardfolk came from the bogs that things started to go downhill.  The residents decided that it was easier to farm elsewhere than wait for reinforcements to help with that threat.