You have been purposefully avoiding a confrontation for quite a while. You don't want to get caught in a battle with someone, and so you have playing the role of "Mr./Ms. Nice Guy/Gal". Don't be afraid of being aggressive. It will behoove you to do so.