it did nothing , however , to dampen his fury with lavinia and her lot , and as gareth stole a glance at his wife , hurrying along beside him with her head bent over charlotte to protect her from the rain , he fought the impulse to go back to the brothel and take out his wrath on not just the one who 'd happened to be nearest , but each and every one of the vulgar louts who 'd been spying on them .