I encourage you to block people to make your experience better, regardless of whether someone has attacked you directly or not. Your experience being better will make your followers experience better.