The neo-Nazis rallied under the banner of "Unite the Right". They think they are part of your movement. Maybe if more conservatives would condemn the nazis in your midst as stridently and as often as you proffer deflections and false equivalencies, the connection between mainstream conservatism and white nationalism wouldn't be so easy to make. 

Remember, the mainstream Republican Party in the state of North Carolina made it illegal for local municipalities in their state to remove white supremacist traitorous Confederate statues. Maybe that's why these white nationalists who rallied around Confederate statues in VA feel at home in your ranks.