Social justice is a good thing. Social justice at the expense of everything else is not. Violence or extremism to promote equality is bad, just as violence is bad to advance racism (which itself is abhorrent). Reasonable debate is good.
You are right that passionate defenders of social justice are unlikely to agree that their goal to achieve  a fairer world is not worth working towards. Are you against fairness and equality? Do you want social justice advocates to stop advocating?