Nothing about free speech says anyone has to listen to your speech. No one has to grant you a platform. If your opinion gets drowned out by other people using their voices, tough.

This isn't even a free speech first amendment issue. Freedom of speech only prevents the government from limiting your speech. Your fellow citizens however are free to limit and impeded your speech how ever they want, within legal bounds. So if a thousand people want to assemble to drown out the hate speech of some random nazi on the speech, the Nazi isn't having his freedom of speech taken away because the government was never limiting his ability to speak.

It's his fellow citizens taking away his platform he was using for his hate speech. You're trying to apply constitutional protections to things that they don't relate to.

Is the government preventing his speech? No? Then his 1A rights aren't being violated.

https://xkcd.com/1357/

His fellow citizens showed him the door.