Last Friday, in New York, I discovered a strip club near the site of the planned Islamic centre, described by its opponents as "the mosque at Ground Zero". As pole dancers gyrated with all the sizzling eroticism of a weary Wal-Mart checkout assistant at the end of a long shift, I asked the burly front-of-house man – Scott, from Brooklyn – whether they had faced any protests about this profanation of hallowed ground. Had any Fox News commentators, for example, been beating an angry path to their door? Well, he replied, one or two passers-by had raised objections since the controversy erupted about the Islamic centre. "People are entitled to their opinions," said Scott, but the "New York Dolls" Gentlemen's Club had been here for 30 years and the folks working in it had to make a living.

Now a strip club at the memorial site of the worst terrorist atrocity on American soil would truly be a profanation. Though obviously not comparable to a strip club, planting a large new mosque directly on that site would nonetheless show an acute lack of sensitivity. Nine years on, the place where the twin towers stood is still a building site, but in a nearby exhibition you can see the plans for a commemorative ensemble of pools, trees and a museum, as well as a soaring new "freedom tower". As at the sites of Auschwitz, Katyn, Hiroshima or Ypres, so in the footprint of the World Trade Center, historical tact and commemorative mission should override all other considerations.

But here's the point: the strip club on Murray Street is not "at Ground Zero" any more than the site of the planned Islamic centre, a former Burlington coat factory in Park Place, is "at Ground Zero". They are, respectively, three and two blocks away. Neither would be visible from the World Trade Center memorial site, which may in some important if secular sense be considered hallowed ground. In New York, two blocks is a country mile. By the time you get to Park Place, there is no doubt that you are already somewhere else, amid the city's habitual huggermugger craziness, with the Amish Market on the corner selling Amish BBQ chicken, Amish fettucine and Amish sushi – all of them as authentically Amish as I am Chinese.

Then the critics of the proposed centre in Park Place – sorry, "Ground Zero Mega Mosque" – go on about dubious sources of funding and suspect statements by its principal protagonist, Imam Feisal Abdul Rauf. And so, they say, it should be built further away. The leap of illogic is as big as any leap of faith. Were the centre to have terrorist sources of finance, or radical, bloodthirsty Islamist leadership, it should be stopped anyway, whether it is two blocks away from Ground Zero or 200.

In the event, these claims too turn out to be twisted, or absurdly thin. The anti-Islam blogger Pamela Geller, for example, has a characteristic rant on her website, arguing that Rauf was associated with a Malaysian peace group which funded the Gaza aid flotilla. Her headline: "Ground Zero Imam Rauf's 'Charity' Funded Genocide Mission". The Daily Show's Jon Stewart did a fine riff on this kind of guilt by association, pointing out that the second-largest shareholder in Rupert Murdoch's News Corporation, which owns Fox News, is Saudi Prince Al-Waleed bin Talal – who is associated with the Carlyle Group, which has done business with the Bin Laden family, "one of whose sons – obviously I'm not going to say which one – may be anti-American".

In a clumsy, provocative comment during a television discussion soon after 9/11, Rauf said that US policies had been "an accessory to the crime that happened" and that Osama bin Laden was "made in the USA". That was wrong, and offensive. But it has to be put against the rest of his words and deeds, which have been devoted to promoting a gentle Sufi version of Islam compatible with a free society. I'm not a huge fan of his kind of interfaith waffle, but if the Muslim world were comprised entirely of Raufs, we would not have the problems we face today – and there would have been no 9/11 attacks. That is why the state department has been funding him to travel round the Middle East explaining American Islam.

There is therefore no reasonable objection to this Islamic centre, with its mission to promote peace, love, interfaith dialogue and swimming, being built in Park Place. Yet in the runup to the US mid-term elections on 2 November, senior politicians, pundits and even supposed opponents of religious discrimination are either condemning it or ducking out with weasel words. Newt Gingrich, the Republican former speaker of the House of Representatives, denounced the scheme, saying: "Nazis don't have the right to put up a sign next to the Holocaust Museum in Washington." Fox star Bill O'Reilly says it should not be built because "Muslims killed us on 9/11". Sarah Palin famously tweeted "Peaceful Muslims, please refudiate" (sic). Facing a tough re-election race even Harry Reid, the Democrat majority leader in the Senate, distanced himself from President Obama's cautious endorsement of Muslims' constitutional right to build the centre.

Most grotesquely, Abraham Foxman of the Anti-Defamation League insists it should be moved. Talking of the relatives of 9/11 victims who oppose it (though some other relatives support it), Foxman says "their anguish entitles them to positions that others would categorise as irrational or bigoted". An organisation established to combat bigotry thus comes out in defence of ... bigotry. And the upshot of all this is that, in a Pew poll this August, 51% of Americans asked said they opposed the building of the centre near the World Trade Center site.

There is now no good way forward. If it goes ahead, it will be a constant bone of contention. If it is moved, more Muslims will believe radical Islamists when they say: "You see, we told you so – America is Islamophobic." Either way, America is doing something extremely stupid. As if it did not have enough problems of its own, it is conspiring to give itself a problem which, up to now, it has not had – or at least, has had much less than most European countries.

Yes, there have been a few home-grown American jihadists, but there is a lot of evidence that American Muslims are generally better integrated, and more supportive of the state in which they live than most of their European counterparts. There are several reasons for this, but one of the biggest is the First Amendment tradition of free speech and freedom of religion, which is now at issue in those blocks just up the road from, but not at, Ground Zero.

That great tradition, which Scott, the doorman at "New York Dolls", seems to have understood better than Foxman, Gingrich or Reid, says: this is America, where Geller can rant, strippers can grind, Christians, Jews and Muslims can pray – and Stewart can make fun of them all. This is America, where no one has the right not to be offended. For God's sake, America, don't catch our European disease.